In the previous article I described how the Approval Tests library can help reduce the amount of assert code that needs to be written. The second benefit of using Approval Tests is the ability to use innate human intelligence to decide if the result of the test is correct.

Imagine a scenario where you need to assert that a text-to-speech generator has generated the correct output. In this example the output could be a byte array representing a .WAV or .MP3 sound file. How would you write traditional asserts to test this output?

As another example, suppose you had to test code that applied a creative filter to an input photograph, this could be some sort of “make skin tones look nice” filter, the output in this case would be a modified image file. How would you assert that the output photo looked “nice”?

In cases like these using traditional asserts may be impossible or very time consuming to implement, there is no Assert.Speech(…) or Assert.LooksNice(…).

This is where the Approval Tests library offers great benefits. You could simply write Approvals.Verify(speechWavBytes); or Approvals.Verify(processedImageBytes); In the case of the sound file you could listen to it and decide if it sounds correct. In the case of the processed photo, you could look at it on screen and use human intelligence to decide if it “looks nice”.

Once you are happy you can approve the results and then in future tests runs if the output accidentally changes due to a bug the tests will fail.

If you want to see Approval Tests in action and learn more about how they can make your testing life easier check out my Approval Tests for .NET Pluralsight course which you can currently start watching for free today with a Pluralsight free trial.

SHARE:

Sometimes assert code in tests gets big and messy and complicated when the output we’re testing is complex.

Approval Tests is a library that can help simplify assert code. The library has other benefits/use cases which I’ll cover in future posts such as using human intelligence to judge if the output is correct; providing a safety net when refactoring legacy code that has no tests; and even testing view rendering.

In the following test code, notice the assert phase:

[Fact]
public void TraditionalAsserts()
{
    var lines = new List<string>
    {
        "Widget sales: 42",
        "Losses: 99",
        "Coffee overheads: 9,999,999"
    };

    var sut = new ReportGenerator(title: "Annual Report",
                                    footer: "Copyright 2020",
                                    lines);

    string report = sut.Generate();

    // Notice the complexity of the asserts below
    Assert.Equal("Annual Report", report.Split(Environment.NewLine)[0]);
    Assert.Equal("Widget sales: 42", report.Split(Environment.NewLine)[2]);
    Assert.Equal("Losses: 99", report.Split(Environment.NewLine)[3]);
    Assert.Equal("Coffee overheads: 9,999,999", report.Split(Environment.NewLine)[4]);
    Assert.Equal("Total lines: 3", report.Split(Environment.NewLine)[6]);
    Assert.Equal("Copyright 2020", report.Split(Environment.NewLine)[8]);
            

    // We could also have just asserted using a long expected string rather than individual line asserts
}

And for reference the ReportGenerator class looks like the following:

public class ReportGenerator
{
    public string Title { get; }
    public List<string> Lines { get; }
    public string Footer { get; }

    public ReportGenerator(string title, string footer, List<string> lines)
    {
        Title = title;
        Footer = footer;
        Lines = lines;
    }

    public string Generate()
    {
        var report = new StringBuilder();

        report.AppendLine(Title);
        report.AppendLine();

        foreach (var line in Lines)
        {
            report.AppendLine(line);
        }


        report.AppendLine();
        report.AppendLine($"Total lines: {Lines.Count}");

        report.AppendLine();
        report.AppendLine(Footer);

        return report.ToString();
    }
}

So in the test there are 6 lines of assert code:

Assert.Equal("Annual Report", report.Split(Environment.NewLine)[0]);
Assert.Equal("Widget sales: 42", report.Split(Environment.NewLine)[2]);
Assert.Equal("Losses: 99", report.Split(Environment.NewLine)[3]);
Assert.Equal("Coffee overheads: 9,999,999", report.Split(Environment.NewLine)[4]);
Assert.Equal("Total lines: 3", report.Split(Environment.NewLine)[6]);
Assert.Equal("Copyright 2020", report.Split(Environment.NewLine)[8]);

If the output was more complex or bigger (for example 100’s or 1000’s of lines of text) then the assert code would get even more messy and harder to maintain. Or what if the output was some binary representation such as an array of bytes representing a generated image or text to speech sound file?

It’s in these cases when dealing with complex output that Approval Tests can help to simplify the assert code as shown in the following test:

[Fact]
[UseReporter(typeof(DiffReporter))]
public void ApprovalTestsVersion()
{
    var lines = new List<string>
    {
        "Widget sales: 42",
        "Losses: 99",
        "Coffee overheads: 9,999,999"
    };

    var sut = new ReportGenerator(title: "Annual Report",
                                    footer: "Copyright 2020",
                                    lines);

    string report = sut.Generate();

    Approvals.Verify(report);
}

Notice in the preceding code the line: Approvals.Verify(report); This line calls Approval Tests and will create a new  “received” .txt file in the test project. You can examine this text file and if it is correct rename it to be an “approved” file. When the test runs in the future, Approval Tests will use the approved file (which should be added to source control) and if the generated report is the same then the test will pass, otherwise the test will fail and a new received file will be output. The [UseReporter] attribute lets you specify how to visualize approval failures, in this example by using a diff tool, and there’s a number of other reporters that come out of the box that you can use.

If you want to see Approval Tests in action and learn more about how they can make your testing life easier check out my Approval Tests for .NET Pluralsight course which you can currently start watching for free today with a Pluralsight free trial.

SHARE:

We can write automated tests to cover various aspects of the code we write. We can write unit/integration tests that test that the code is producing the expected outcomes. We can use ConventionTests to ensure internal code quality, for example that classes following a specified naming convention and exists in the correct namespace. We may even add the ability to create a business readable tests using tools such as SpecFlow or BDDfy.

Another aspect that we might want to ensure doesn’t change unexpectedly is the public API that our code exposes to callers.

Using PublicApiGenerator to Generate a Report of our Public API

The first step of ensuring our public API hasn’t changed is to be able to capture the public API in a readable way. The PublicApiGenerator NuGet package (from Jake Ginnivan) gives us this ability.

Suppose we have the following class defined:

public class Calculator
{
    public Calculator()
    {
        CurrentValue = 0;
    }

    public int CurrentValue { get; private set; }

    public void Clear()
    {
        CurrentValue = 0;
    }

    public void Add(int number)
    {
        CurrentValue += number;
    }
}

Notice here that this code defines the public API that consumers of the Calculator class can use. It’s this public API that we want to test to ensure it doesn’t change unexpectedly.

We might start with some unit tests as shown in the following code:

public class CalculatorTests
{
    [Fact]
    public void ShouldHaveInitialValue()
    {
        var sut = new Calculator();

        Assert.Equal(0, sut.CurrentValue);
    }

    [Fact]
    public void ShouldAdd()
    {
        var sut = new Calculator();

        sut.Add(1);

        Assert.Equal(1, sut.CurrentValue);
    }
}

These tests help us ensure the code is doing the right thing but do not offer any protection against the public API changing. We can now add a new test that uses PublicApiGenerator to generate a string “report” detailing the public members of our API. The following test code shows this in use:

[Fact]
public void ShouldHaveCorrectPublicApi()
{
    var sut = new Calculator();

    // Get the assembly that we want to generate the public API report for
    Assembly calculatorAssembly = sut.GetType().Assembly;

    // Use PublicApiGenerator to generate the API report
    string apiString = PublicApiGenerator.PublicApiGenerator.GetPublicApi(calculatorAssembly);

    // TODO: assert API has not changed
}

If we debug this test and look at the content of the apiString variable we’d see the following text:

[assembly: System.Runtime.InteropServices.ComVisibleAttribute(false)]
[assembly: System.Runtime.InteropServices.GuidAttribute("c2dc3732-a4a5-4baa-b4df-90f40aad1c6a")]
[assembly: System.Runtime.Versioning.TargetFrameworkAttribute(".NETFramework,Version=v4.5.1", FrameworkDisplayName=".NET Framework 4.5.1")]

namespace Demo
{
    
    public class Calculator
    {
        public Calculator() { }
        public int CurrentValue { get; }
        public void Add(int number) { }
        public void Clear() { }
    }
}

Using Approval Tests to Assert the API Is Correct

Now in our test we have a string that represents the public API. We can combine PublicApiGenerator with the Approval Tests library to check that this API text doesn’t change.

First off we go and install the Approval Tests NuGet Package. We can then modify the test as shown below:

public class CalculatorApiTests
{
    [Fact]
    public void ShouldHaveCorrectPublicApi()
    {
        var sut = new Calculator();

        // Get the assembly that we want to generate the public API report for
        Assembly calculatorAssembly = sut.GetType().Assembly;

        // Use PublicApiGenerator to generate the API report
        string apiString = PublicApiGenerator.PublicApiGenerator.GetPublicApi(calculatorAssembly);

        // Use Approval Tests to verify the API hasn't changed
        Approvals.Verify(apiString);
    }
}

The first time we run this it will fail with a message such as “Failed Approval: Approval File "c:\…\Demo.Tests\CalculatorApiTests.ShouldHaveCorrectPublicApi.approved.txt" Not Found”. It will also generate a file called CalculatorApiTests.ShouldHaveCorrectPublicApi.received.txt. We can rename this to CalculatorApiTests.ShouldHaveCorrectPublicApi.approved.txt, run the test again and it will pass.

If we now modify the public API by changing a method signature (e.g. to public void Clear(int someParam)) and run the test again it will fail with a message such as “Received file c:\...\Demo.Tests\CalculatorApiTests.ShouldHaveCorrectPublicApi.received.txt does not match approved file c:\...\Demo.Tests\CalculatorApiTests.ShouldHaveCorrectPublicApi.approved.txt”.

Modifying the test and adding an Approval Tests reporter attribute ([UseReporter(typeof(DiffReporter))]) and running the test will now gives us a visual diff identifying the changes to the public API as shown in the following screenshot.

To learn more about the features of Approval Tests, check out my Approval Tests for .NET Pluralsight course.

You can start watching with a Pluralsight free trial.

SHARE:

In a test, we are often testing (asserting) individual items such as an (int) age is correct or a string matches an expected value.

If we are practicing test-first development we’ll write our asserts first.

Approval tests allow us to go beyond simple asserts.

What if the thing we’re checking is not a simple value, for example that a pie chart image matches the input data? Or what if we want to use our human judgement to decide when something looks correct, something that is hard to codify in one or more basic asserts?

ApprovalTests for .NET can be install via NuGet. Once installed, it gives us a whole new world when it comes to checking the output of code.

For example, say we are developing a class to represent a stickman. We want to be able to tell an instance to raise left arm or raise right leg for example.

Example of Using Approval Tests

So lets start with a test:

[Fact]
[UseReporter(typeof(DiffReporter))]
public void ShouldHaveDefaultPosture()
{
    var sut = new StickMan();

    Approvals.Verify(sut);
}

And an empty StickMan:

public class StickMan
{        
}

Here we’re using xUnit.net (the [Fact] attribute) but you could be using NUnit for example.

The first thing to notice here is there is no traditional Assert method, instead we’re using Approval Tests to verify the state of the system under test (sut).

The other think to notice is the [UseReporter] attribute that tells Approval Tests to use a diff tool to display errors when a test fails.

If we run this test, we’ll get a diff tool opened:

More...

SHARE: