Grouping and Filtering Tests in Visual Studio Test Explorer

One way to run automated tests is to use Visual Studio’s Test Explorer. Test Explorer can be found under the Test –> Windows –> Test Explorer menu items.

In this article we’ll look at how to manage the list of tests using grouping and also how to specify custom search filter expressions.

Two test projects in Visual Studio solution

Grouping Tests

There are a number of ways to group tests in Test Explorer, at the highest structural level we can group by project.

To select a group by method, click the drop down arrow as show in the following screenshot:

Selecting a grouping in Visual Studio Test Explorer

With the grouping set to Project, the test list looks as follows:

image

The next structural grouping is Class:

Grouping by test class

The final structural grouping is by Namespace:

Grouping by namespace

There are a number of non-structural groupings.

Group by Duration:

Group by duration

Group by Outcome:

Group by outcome

…and group by Traits:

Grouping by traits

Filtering Tests

Custom filters can also be applied.

For example by file path:

Filtering tests by file path

Other search examples include:

  • Trait:"Smoke Test"
  • Message:"System.Exception"
  • Class1
  • Outcome:"Passed"
  • Outcome:"Failed"

Subsets can also be excluded by prefixing the type of filter with a -. For example to show all tests in Class1 except failed tests: Class:"TestClass1" -Outcome:"Passed".

Testing That Your Public APIs Have Not Changed Unexpectedly with PublicApiGenerator and Approval Tests

We can write automated tests to cover various aspects of the code we write. We can write unit/integration tests that test that the code is producing the expected outcomes. We can use ConventionTests to ensure internal code quality, for example that classes following a specified naming convention and exists in the correct namespace. We may even add the ability to create a business readable tests using tools such as SpecFlow or BDDfy.

Another aspect that we might want to ensure doesn’t change unexpectedly is the public API that our code exposes to callers.

Using PublicApiGenerator to Generate a Report of our Public API

The first step of ensuring our public API hasn’t changed is to be able to capture the public API in a readable way. The PublicApiGenerator NuGet package (from Jake Ginnivan) gives us this ability.

Suppose we have the following class defined:

public class Calculator
{
    public Calculator()
    {
        CurrentValue = 0;
    }

    public int CurrentValue { get; private set; }

    public void Clear()
    {
        CurrentValue = 0;
    }

    public void Add(int number)
    {
        CurrentValue += number;
    }
}

Notice here that this code defines the public API that consumers of the Calculator class can use. It’s this public API that we want to test to ensure it doesn’t change unexpectedly.

We might start with some unit tests as shown in the following code:

public class CalculatorTests
{
    [Fact]
    public void ShouldHaveInitialValue()
    {
        var sut = new Calculator();

        Assert.Equal(0, sut.CurrentValue);
    }

    [Fact]
    public void ShouldAdd()
    {
        var sut = new Calculator();

        sut.Add(1);

        Assert.Equal(1, sut.CurrentValue);
    }
}

These tests help us ensure the code is doing the right thing but do not offer any protection against the public API changing. We can now add a new test that uses PublicApiGenerator to generate a string “report” detailing the public members of our API. The following test code shows this in use:

[Fact]
public void ShouldHaveCorrectPublicApi()
{
    var sut = new Calculator();

    // Get the assembly that we want to generate the public API report for
    Assembly calculatorAssembly = sut.GetType().Assembly;

    // Use PublicApiGenerator to generate the API report
    string apiString = PublicApiGenerator.PublicApiGenerator.GetPublicApi(calculatorAssembly);

    // TODO: assert API has not changed
}

If we debug this test and look at the content of the apiString variable we’d see the following text:

[assembly: System.Runtime.InteropServices.ComVisibleAttribute(false)]
[assembly: System.Runtime.InteropServices.GuidAttribute("c2dc3732-a4a5-4baa-b4df-90f40aad1c6a")]
[assembly: System.Runtime.Versioning.TargetFrameworkAttribute(".NETFramework,Version=v4.5.1", FrameworkDisplayName=".NET Framework 4.5.1")]

namespace Demo
{
    
    public class Calculator
    {
        public Calculator() { }
        public int CurrentValue { get; }
        public void Add(int number) { }
        public void Clear() { }
    }
}

Using Approval Tests to Assert the API Is Correct

Now in our test we have a string that represents the public API. We can combine PublicApiGenerator with the Approval Tests library to check that this API text doesn’t change.

First off we go and install the Approval Tests NuGet Package. We can then modify the test as shown below:

public class CalculatorApiTests
{
    [Fact]
    public void ShouldHaveCorrectPublicApi()
    {
        var sut = new Calculator();

        // Get the assembly that we want to generate the public API report for
        Assembly calculatorAssembly = sut.GetType().Assembly;

        // Use PublicApiGenerator to generate the API report
        string apiString = PublicApiGenerator.PublicApiGenerator.GetPublicApi(calculatorAssembly);

        // Use Approval Tests to verify the API hasn't changed
        Approvals.Verify(apiString);
    }
}

The first time we run this it will fail with a message such as “Failed Approval: Approval File "c:\…\Demo.Tests\CalculatorApiTests.ShouldHaveCorrectPublicApi.approved.txt" Not Found”. It will also generate a file called CalculatorApiTests.ShouldHaveCorrectPublicApi.received.txt. We can rename this to CalculatorApiTests.ShouldHaveCorrectPublicApi.approved.txt, run the test again and it will pass.

If we now modify the public API by changing a method signature (e.g. to public void Clear(int someParam)) and run the test again it will fail with a message such as “Received file c:\...\Demo.Tests\CalculatorApiTests.ShouldHaveCorrectPublicApi.received.txt does not match approved file c:\...\Demo.Tests\CalculatorApiTests.ShouldHaveCorrectPublicApi.approved.txt”.

Modifying the test and adding an Approval Tests reporter attribute ([UseReporter(typeof(DiffReporter))]) and running the test will now gives us a visual diff identifying the changes to the public API as shown in the following screenshot.

Approval Tests Diff Screenshot

To learn more about the features of Approval Tests, check out my Approval Tests for .NET Pluralsight course.

Improving Test Code Readability and Assert Failure Messages with Shouldly

Shouldly is an open source library that aims to improve the assert phase of tests; it does this in two ways. The first is offering a more “fluent like” syntax that for the most part leverages extension methods and obviates the need to keep remembering which parameter is the expected or actual as with regular Assert.Xxxx(1,2) methods. The second benefit manifests itself when tests fail; Shouldly outputs more readable, easily digestible test failure messages.

Failure Message Examples

The following are three failure messages from tests that don’t use Shouldly and instead use the assert methods bundled with the testing framework (NUnit, xUnit.net, etc):

  • “Expected: 9  But was:  5”
  • “Assert.NotNull() Failure”
  • “Not found: Monday In value:  List<String> ["Tuesday", "Wednesday", "Thursday"]”

In each of the preceding failure messages, there is not much helpful context in the failure message.

Compare the above to the following equivalent Shouldly failure messages:

  • “schedule.TotalHours should be 9 but was 5”
  • “schedule.Title should not be null or empty”
  • “schedule.Days should contain "Monday" but does not”

Notice the additional context in these failure messages. In each case here, Shouldly is telling us the name of the variable in the test code (“schedule”) and the name of the property/field being asserted (e.g. “Total Hours”).

Test Code Readability

For the preceding failure messages, the following test assert code is used (notice the use of the Shouldly extension methods):

  • schedule.TotalHours.ShouldBe(9);
  • schedule.Title.ShouldNotBeNullOrEmpty();
  • schedule.Days.ShouldContain("Monday");

In these examples there is no mistaking an actual value parameter for an expected value parameter and the test code reads more “fluently” as well.

To find out more about Shouldly check out the project on GitHub, install via NuGet, or checkout my Better Unit Test Assertions with Shouldly Pluralsight course.

Beyond Simple Asserts with ApprovalTests

In a test, we are often testing (asserting) individual items such as an (int) age is correct or a string matches an expected value.

If we are practicing test-first development we’ll write our asserts first.

Approval tests allow us to go beyond simple asserts.

What if the thing we’re checking is not a simple value, for example that a pie chart image matches the input data? Or what if we want to use our human judgement to decide when something looks correct, something that is hard to codify in one or more basic asserts?

ApprovalTests for .NET can be install via NuGet. Once installed, it gives us a whole new world when it comes to checking the output of code.

For example, say we are developing a class to represent a stickman. We want to be able to tell an instance to raise left arm or raise right leg for example.

Example of Using Approval Tests

So lets start with a test:

[Fact]
[UseReporter(typeof(DiffReporter))]
public void ShouldHaveDefaultPosture()
{
    var sut = new StickMan();

    Approvals.Verify(sut);
}

And an empty StickMan:

public class StickMan
{        
}

Here we’re using xUnit.net (the [Fact] attribute) but you could be using NUnit for example.

The first thing to notice here is there is no traditional Assert method, instead we’re using Approval Tests to verify the state of the system under test (sut).

The other think to notice is the [UseReporter] attribute that tells Approval Tests to use a diff tool to display errors when a test fails.

If we run this test, we’ll get a diff tool opened:

More...

A Feature Based Approach to Organising Test Code in BDDfy and Other Testing Frameworks

We want our test code to be as high quality as possible, this means smaller amounts of code duplication, reasonably easy to find where things are in Visual Studio, etc.

One possible organization structure is to think in terms of the individual features in the application. The approach you take will probably depend on the complexity and size of the test suite, system under test, etc. Because BDDfy is "just code" we can use all the normal techniques of composition and inheritance that we'd use in our production code to get to the right level of code reuse and organization for the application we're testing.

Organisation by Feature

Organising by feature enables a reasonable amount of code reuse between test scenarios and it also helps to think user- or business-first rather than code/implementation first.

So for example, if we’re using BDDfy to test a banking application we might have the following features:

  • Login
  • Logout
  • Move Money
  • Pay Bills
  • View Transactions
  • etc

Each of these features contains a number of scenarios, for example Login would probably contain scenarios for successful login, bad password, locked out account, 2 factor login, etc.

In Visual Studio we could create folders in the test project to represent and organize these features as the following screenshot illustrates.

Visual Studio feature folders

 

So a (cut down) Login BDDfy story class could look like the following code:

namespace Tests.Login
{
    [TestFixture]
    [Story(AsA="As a Customer",
        IWant = "I want to login",
        SoThat = "So that I can manage my accounts and money")]
    public class CustomerLogin
    {
        [Test]
        public void LoginSuccess()
        {
            this.Given(x => GivenIAmOnTheLoginScreen())
                .And(x => x.AndIHaveEnteredMyUsername())
                .And(x => AndIHaveEnteredMyPassword())
                .When( x=> WhenIChooseLogin())
                .Then(x => ThenIShouldBeLoggedIn())
                .BDDfy<CustomerLogin>();
        }

        public void GivenIAmOnTheLoginScreen()
        {
        }

        public void AndIHaveEnteredMyUsername()
        {
        }

        public void AndIHaveEnteredMyPassword()
        {
        }

        public void WhenIChooseLogin()
        {
        }

        public void ThenIShouldBeLoggedIn()
        {
        }
    }
}

Now for arguments sake, say we have a Navigation story class that represents how the user should be able to move around the applications features.

We could reuse the individual given/when/then methods in CustomerLogin but we don’t want our navigations scenarios to be bloated, we want them to represent the essence of the scenario with the right level of detail.

So the first thing we could do is to create a “step aggregation” method in CustomerLogin as follows:

public void GivenIHaveLoggedIn()
{
    GivenIAmOnTheLoginScreen();
    AndIHaveEnteredMyUsername();
    AndIHaveEnteredMyPassword();
    WhenIChooseLogin();
}

This method simply re-uses the existing steps but aggregates them into a method we can call from the navigation tests:

using NUnit.Framework;
using TestStack.BDDfy;
using TestStack.BDDfy.Core;
using Tests.Login;
using TestStack.BDDfy.Scanners.StepScanners.Fluent;

namespace Tests
{
    [TestFixture]
    [Story(AsA="As a Customer",
        IWant = "I want to navigate around the site",
        SoThat = "So that I can get to the features I want to use")]
    public class Navigation
    {
        [Test]
        public void NavigateToMoveMoney()
        {
            var custLogin = new CustomerLogin();

            this.Given(x => custLogin.GivenIHaveLoggedIn())
                .When(x => WhenChooseGotoMoveMoney())
                .Then(x => ThenIShouldBeTakenToTheMainMoveMoneyScreen())
                .BDDfy<Navigation>();
        }

        private void ThenIShouldBeTakenToTheMainMoveMoneyScreen()
        {            
        }

        private void WhenChooseGotoMoveMoney()
        {            
        }
    }
}

So here were are making use of this aggregate method from the CustomerLogin feature class.

If we don’t need to represent the fact that the customer is logged-in in the report and all the tests in the class assume a logged-in user, we could use some (e.g. NUnit) test setup code that logs the user in but doesn’t get reported.

The HTML output of this looks as follows:

BDDfy HTML report

 

While in these examples we have a single story class for the feature (e.g. CustomerLogin) once we start adding scenarios (and steps) this single story class might become too bloated. If this is deemed a problem then we can break it out into sub features/stories or if it’s applicable we could use inheritance to hold common given/when/then steps. The individual story classes relating to the customer login feature would all inherit this base class. We probably however, do not want multiple levels of nested inheritance in our stories as this may make maintenance and discoverability harder.

To see more of what BDDfy can do check out my Building the Right Thing in .NET with TestStack Pluralsight course, head on over to the documentation, or check it out on GitHub

Testing ASP.Net MVC Controllers with FluentMVCTesting

FluentMVCTesting facilitates the writing of tests against MVC controllers in a fluent way.

FluentMVCTesting is available via NuGet: Install-Package TestStack.FluentMVCTesting

There’s a number of things that FluentMVCTesting can help to test such as:

  • A controller action returns the correct view
  • A controller action returns the correct HTTP status
  • A controller action returns an empty result
  • A controller action returns a view if there are model errors
  • A controller action returns a view with the correct model data
  • A controller action should redirect to a Url / Route / Action

It’s also not tied to a specific testing framework, so it can be used with NUnit, xUnit.net, MSTest, etc.

Examples

var sut = new ExampleController();

sut.WithCallTo(x => x.Show()).ShouldRenderView("Orders");

The preceding test code is testing the ExampleController. It is testing that when the Show() action is called then the Orders view is rendered.

The following code checks that an expected HTTP status is returned.

var sut = new ExampleController();

sut.WithCallTo(x => x.MakeAnError())
    .ShouldGiveHttpStatus(HttpStatusCode.InternalServerError);

 

To see more of what FluentMVCTesting can do check out my Building the Right Thing in .NET with TestStack Pluralsight course, check out the documentation, or check it out on GitHub.

Business-Readable, Living Documentation with BDDfy

BDDfy enables the creation of tests that, in addition to verifying that the system works correctly, also results in business-readable, living documentation.

Business-readable means that the tests are described in natural language (e.g. English) that the business can read, understand, and ensure that the correct features and functionality is being built.

Living documentation means that the report results directly from the passing or failing of the tests. It’s not a word document somewhere on a shared drive or SharePoint site that may or may not actually be in sync with what the system actually does.

The Report

Below is an example of what a typical BDDfy HTML report looks like (there is a “metro” inspired report coming in V4). There’s also the ability to output the test report in markdown.

BDDfy HTML report

Notice that it’s not code-centric output, but rather business-centric.

Here the two test scenarios are grouped into a story, but they don’t have to be.

The underlying test framework that is used to execute the test doesn’t matter – you could use NUnit, xUnit.net, MSTest, etc.

The Test Code

The tests that produce this report could be configured in BDDfy in a couple of ways. There is a reflective style that uses methods that following a specific naming convention. There is also the fluent style.

The test code to produce this report looks like the below (we are using NUnit in this example).

using NUnit.Framework;
using TestStack.BDDfy;
using TestStack.BDDfy.Core;
using TestStack.BDDfy.Scanners.StepScanners.Fluent;

namespace BDDfyDemo
{
    [TestFixture]
    [Story(AsA = "As a customer",
        IWant = "I want my order total to add up correctly",
        SoThat = "I'm not overcharged for my goods")]
    public class OrderTotalCalculatorTests
    {
        [Test]
        public void MultipleOrderedItemsTotals()
        {
            this.Given(x => GivenIHaveAddedItemsToMyCart())
                .When(x => WhenICheckout())
                .Then(x => ThenTheOrderTotalExcludingTaxShouldBeCorrect())
                .BDDfy<OrderTotalCalculatorTests>();
        }

        [Test]
        public void SalesTaxAdded()
        {
            this.Given(x => GivenIHaveAddedItemsToMyCart())
                .When(x => WhenICheckout())
                .Then(x => ThenTheSalesTaxShouldBeCorrect())
                .BDDfy<OrderTotalCalculatorTests>();
        }

        public void GivenIHaveAddedItemsToMyCart()
        {
            // test code
        }

        public void WhenICheckout()
        {
            // test code
        }

        public void ThenTheOrderTotalExcludingTaxShouldBeCorrect()
        {
            // test code
        }

        public void ThenTheSalesTaxShouldBeCorrect()
        {
            // test code
        }

    }
}

Here there are a couple of NUnit tests defined – the methods with the [Test] attribute applied to them.

Within these test methods the BDDfy fluent style is being used to define the different steps in the test scenarios.

The “given” phase sets up the initial context or state of the thing being tested. The “when” phase acts upon the system to produce some change. The “then” phase is typically where we have assert code, it’s in this phase that the resulting state of the system is checked.

Notice how the individual method names appear in the test report in a nice business-readable way.

 

There’s a lot more to BDDfy, such as data-parameterised tests and a whole heap of customisation and configuration options. To see more of what BDDfy can do check out my Building the Right Thing in .NET with TestStack Pluralsight course, head on over to the documentation, or check it out on GitHub.