Approval Tests: Assert With Human Intelligence

In the previous article I described how the Approval Tests library can help reduce the amount of assert code that needs to be written. The second benefit of using Approval Tests is the ability to use innate human intelligence to decide if the result of the test is correct.

Imagine a scenario where you need to assert that a text-to-speech generator has generated the correct output. In this example the output could be a byte array representing a .WAV or .MP3 sound file. How would you write traditional asserts to test this output?

As another example, suppose you had to test code that applied a creative filter to an input photograph, this could be some sort of “make skin tones look nice” filter, the output in this case would be a modified image file. How would you assert that the output photo looked “nice”?

In cases like these using traditional asserts may be impossible or very time consuming to implement, there is no Assert.Speech(…) or Assert.LooksNice(…).

This is where the Approval Tests library offers great benefits. You could simply write Approvals.Verify(speechWavBytes); or Approvals.Verify(processedImageBytes); In the case of the sound file you could listen to it and decide if it sounds correct. In the case of the processed photo, you could look at it on screen and use human intelligence to decide if it “looks nice”.

Once you are happy you can approve the results and then in future tests runs if the output accidentally changes due to a bug the tests will fail.

If you want to see Approval Tests in action and learn more about how they can make your testing life easier check out my Approval Tests for .NET Pluralsight course which you can currently start watching for free today with a Pluralsight free trial.

SHARE:

Approval Tests: Write Tests More Quickly

Sometimes assert code in tests gets big and messy and complicated when the output we’re testing is complex.

Approval Tests is a library that can help simplify assert code. The library has other benefits/use cases which I’ll cover in future posts such as using human intelligence to judge if the output is correct; providing a safety net when refactoring legacy code that has no tests; and even testing view rendering.

In the following test code, notice the assert phase:

[Fact]
public void TraditionalAsserts()
{
    var lines = new List<string>
    {
        "Widget sales: 42",
        "Losses: 99",
        "Coffee overheads: 9,999,999"
    };

    var sut = new ReportGenerator(title: "Annual Report",
                                    footer: "Copyright 2020",
                                    lines);

    string report = sut.Generate();

    // Notice the complexity of the asserts below
    Assert.Equal("Annual Report", report.Split(Environment.NewLine)[0]);
    Assert.Equal("Widget sales: 42", report.Split(Environment.NewLine)[2]);
    Assert.Equal("Losses: 99", report.Split(Environment.NewLine)[3]);
    Assert.Equal("Coffee overheads: 9,999,999", report.Split(Environment.NewLine)[4]);
    Assert.Equal("Total lines: 3", report.Split(Environment.NewLine)[6]);
    Assert.Equal("Copyright 2020", report.Split(Environment.NewLine)[8]);
            

    // We could also have just asserted using a long expected string rather than individual line asserts
}

And for reference the ReportGenerator class looks like the following:

public class ReportGenerator
{
    public string Title { get; }
    public List<string> Lines { get; }
    public string Footer { get; }

    public ReportGenerator(string title, string footer, List<string> lines)
    {
        Title = title;
        Footer = footer;
        Lines = lines;
    }

    public string Generate()
    {
        var report = new StringBuilder();

        report.AppendLine(Title);
        report.AppendLine();

        foreach (var line in Lines)
        {
            report.AppendLine(line);
        }


        report.AppendLine();
        report.AppendLine($"Total lines: {Lines.Count}");

        report.AppendLine();
        report.AppendLine(Footer);

        return report.ToString();
    }
}

So in the test there are 6 lines of assert code:

Assert.Equal("Annual Report", report.Split(Environment.NewLine)[0]);
Assert.Equal("Widget sales: 42", report.Split(Environment.NewLine)[2]);
Assert.Equal("Losses: 99", report.Split(Environment.NewLine)[3]);
Assert.Equal("Coffee overheads: 9,999,999", report.Split(Environment.NewLine)[4]);
Assert.Equal("Total lines: 3", report.Split(Environment.NewLine)[6]);
Assert.Equal("Copyright 2020", report.Split(Environment.NewLine)[8]);

If the output was more complex or bigger (for example 100’s or 1000’s of lines of text) then the assert code would get even more messy and harder to maintain. Or what if the output was some binary representation such as an array of bytes representing a generated image or text to speech sound file?

It’s in these cases when dealing with complex output that Approval Tests can help to simplify the assert code as shown in the following test:

[Fact]
[UseReporter(typeof(DiffReporter))]
public void ApprovalTestsVersion()
{
    var lines = new List<string>
    {
        "Widget sales: 42",
        "Losses: 99",
        "Coffee overheads: 9,999,999"
    };

    var sut = new ReportGenerator(title: "Annual Report",
                                    footer: "Copyright 2020",
                                    lines);

    string report = sut.Generate();

    Approvals.Verify(report);
}

Notice in the preceding code the line: Approvals.Verify(report); This line calls Approval Tests and will create a new  “received” .txt file in the test project. You can examine this text file and if it is correct rename it to be an “approved” file. When the test runs in the future, Approval Tests will use the approved file (which should be added to source control) and if the generated report is the same then the test will pass, otherwise the test will fail and a new received file will be output. The [UseReporter] attribute lets you specify how to visualize approval failures, in this example by using a diff tool, and there’s a number of other reporters that come out of the box that you can use.

If you want to see Approval Tests in action and learn more about how they can make your testing life easier check out my Approval Tests for .NET Pluralsight course which you can currently start watching for free today with a Pluralsight free trial.

SHARE:

Coding with Compassion

From an engineering perspective we judge code by its complexity, test coverage, etc. These are all empirical measurements that give us an indication of the qualities that the code base possesses.

We say things like: "we have 90% code coverage and 2764 unit tests".

We may give ourselves targets such as:

  • No class longer than 100 lines of code;
  • 100% code coverage for new tests;
  • Treat all compiler warnings as errors;
  • and so on...

These are all mechanical, empirical, engineering type things and there is nothing wrong with that.

I think there may be another aspect to the way we think about software; a more human-centric way - Compassionate Coding.

All human beings are the same. We all want happiness and do not want suffering.

Dalai Lama

Compassionate Coding has two aspects: compassion for the end-user and compassion for the next developer who has to understand our code.

More...

SHARE: