New Pluralsight Course: Automated Business Readable Web Tests with Selenium and SpecFlow

SpecFlow is a tool that can translate natural language scenarios (e.g. writing in English or other spoken languages) into test code. This can allow business people, users, or other stakeholders to verify that the correct features are being built.

Selenium is a tool that allows test code (multiple programming languages supported) to automated a web browser. This allows the creation of automated UI tests that operate the web application as if an end user where doing it; for example clicking buttons and typing text into input boxes.

My new Pluralsight course shows how to integrate these two tools.

The course is organized into four modules:

  1. Introduction to Business Readable Web Testing
  2. Getting Started with Selenium
  3. Adding Business Readability with SpecFlow
  4. Creating More Maintainable Web Automation

If you’re new to SpecFlow I suggest watching this course first before moving on to Automated Business Readable Web Tests with Selenium and SpecFlow.

Hook Execution Order in SpecFlow 2

SpecFlow hooks allow additional code to be executed before and after various stages of the test execution lifecycle, for example running additional setup code before each scenario executes.

If there are multiple of the same type of hook specified, by default the execution order of the hook methods is unspecified. For example the following code has three [BeforeStep] hook methods that could be executed in any order before every step of the scenario executes:

[BeforeStep]
public void BeforeHook1()
{
}

[BeforeStep]
public void BeforeHook2()
{
}

[BeforeStep]
public void BeforeHook3()
{
}

To ensure these hook methods are executed in a specified order, the hook attributes allow an optional order to be specified. When there are multiple of the same hook methods defined, the lowest order values execute before the higher order methods:

[BeforeStep(Order = 100)]
public void BeforeHook1()
{
}

[BeforeStep(Order = 200)]
public void BeforeHook2()
{
}

[BeforeStep(Order = 300)]
public void BeforeHook3()
{
}

The values of the Order property are arbitrary, you may use whatever values you wish, though it is sensible to allow some “wriggle room” for future additional steps by working in increments of 10 or 100 for example.

The following code illustrates another example where the execution order of hooks is important; the database should be reset first before test users are added:

[Binding]
public class Hooks
{
    [BeforeScenario(Order = 100)]
    public void ResetDatabase()
    {
    }

    [BeforeScenario(Order = 200)]
    public void AddTestUsersToDatabase()
    {
    }        
}

To see hook ordering in action, check out my Pluralsight course: Business Readable Automated Tests with SpecFlow 2.0.

New Pluralsight Course: Business Readable Automated Tests with SpecFlow 2.0

My newest Pluralsight course was just published. Business Readable Automated Tests with SpecFlow 2.0 teaches how to create tests that the business can read, understand, and contribute to. These “English-like” tests (other spoken languages are supported) can be executed by writing test code that is associated with the “English-like” steps. Because the tests sit alongside the source code, they can become living (executable) documentation for the system, as opposed to an out-of-date Word document somewhere on the network for example. Check out the course here.

Testing That Your Public APIs Have Not Changed Unexpectedly with PublicApiGenerator and Approval Tests

We can write automated tests to cover various aspects of the code we write. We can write unit/integration tests that test that the code is producing the expected outcomes. We can use ConventionTests to ensure internal code quality, for example that classes following a specified naming convention and exists in the correct namespace. We may even add the ability to create a business readable tests using tools such as SpecFlow or BDDfy.

Another aspect that we might want to ensure doesn’t change unexpectedly is the public API that our code exposes to callers.

Using PublicApiGenerator to Generate a Report of our Public API

The first step of ensuring our public API hasn’t changed is to be able to capture the public API in a readable way. The PublicApiGenerator NuGet package (from Jake Ginnivan) gives us this ability.

Suppose we have the following class defined:

public class Calculator
{
    public Calculator()
    {
        CurrentValue = 0;
    }

    public int CurrentValue { get; private set; }

    public void Clear()
    {
        CurrentValue = 0;
    }

    public void Add(int number)
    {
        CurrentValue += number;
    }
}

Notice here that this code defines the public API that consumers of the Calculator class can use. It’s this public API that we want to test to ensure it doesn’t change unexpectedly.

We might start with some unit tests as shown in the following code:

public class CalculatorTests
{
    [Fact]
    public void ShouldHaveInitialValue()
    {
        var sut = new Calculator();

        Assert.Equal(0, sut.CurrentValue);
    }

    [Fact]
    public void ShouldAdd()
    {
        var sut = new Calculator();

        sut.Add(1);

        Assert.Equal(1, sut.CurrentValue);
    }
}

These tests help us ensure the code is doing the right thing but do not offer any protection against the public API changing. We can now add a new test that uses PublicApiGenerator to generate a string “report” detailing the public members of our API. The following test code shows this in use:

[Fact]
public void ShouldHaveCorrectPublicApi()
{
    var sut = new Calculator();

    // Get the assembly that we want to generate the public API report for
    Assembly calculatorAssembly = sut.GetType().Assembly;

    // Use PublicApiGenerator to generate the API report
    string apiString = PublicApiGenerator.PublicApiGenerator.GetPublicApi(calculatorAssembly);

    // TODO: assert API has not changed
}

If we debug this test and look at the content of the apiString variable we’d see the following text:

[assembly: System.Runtime.InteropServices.ComVisibleAttribute(false)]
[assembly: System.Runtime.InteropServices.GuidAttribute("c2dc3732-a4a5-4baa-b4df-90f40aad1c6a")]
[assembly: System.Runtime.Versioning.TargetFrameworkAttribute(".NETFramework,Version=v4.5.1", FrameworkDisplayName=".NET Framework 4.5.1")]

namespace Demo
{
    
    public class Calculator
    {
        public Calculator() { }
        public int CurrentValue { get; }
        public void Add(int number) { }
        public void Clear() { }
    }
}

Using Approval Tests to Assert the API Is Correct

Now in our test we have a string that represents the public API. We can combine PublicApiGenerator with the Approval Tests library to check that this API text doesn’t change.

First off we go and install the Approval Tests NuGet Package. We can then modify the test as shown below:

public class CalculatorApiTests
{
    [Fact]
    public void ShouldHaveCorrectPublicApi()
    {
        var sut = new Calculator();

        // Get the assembly that we want to generate the public API report for
        Assembly calculatorAssembly = sut.GetType().Assembly;

        // Use PublicApiGenerator to generate the API report
        string apiString = PublicApiGenerator.PublicApiGenerator.GetPublicApi(calculatorAssembly);

        // Use Approval Tests to verify the API hasn't changed
        Approvals.Verify(apiString);
    }
}

The first time we run this it will fail with a message such as “Failed Approval: Approval File "c:\…\Demo.Tests\CalculatorApiTests.ShouldHaveCorrectPublicApi.approved.txt" Not Found”. It will also generate a file called CalculatorApiTests.ShouldHaveCorrectPublicApi.received.txt. We can rename this to CalculatorApiTests.ShouldHaveCorrectPublicApi.approved.txt, run the test again and it will pass.

If we now modify the public API by changing a method signature (e.g. to public void Clear(int someParam)) and run the test again it will fail with a message such as “Received file c:\...\Demo.Tests\CalculatorApiTests.ShouldHaveCorrectPublicApi.received.txt does not match approved file c:\...\Demo.Tests\CalculatorApiTests.ShouldHaveCorrectPublicApi.approved.txt”.

Modifying the test and adding an Approval Tests reporter attribute ([UseReporter(typeof(DiffReporter))]) and running the test will now gives us a visual diff identifying the changes to the public API as shown in the following screenshot.

Approval Tests Diff Screenshot

To learn more about the features of Approval Tests, check out my Approval Tests for .NET Pluralsight course.

Improving Test Code Readability and Assert Failure Messages with Shouldly

Shouldly is an open source library that aims to improve the assert phase of tests; it does this in two ways. The first is offering a more “fluent like” syntax that for the most part leverages extension methods and obviates the need to keep remembering which parameter is the expected or actual as with regular Assert.Xxxx(1,2) methods. The second benefit manifests itself when tests fail; Shouldly outputs more readable, easily digestible test failure messages.

Failure Message Examples

The following are three failure messages from tests that don’t use Shouldly and instead use the assert methods bundled with the testing framework (NUnit, xUnit.net, etc):

  • “Expected: 9  But was:  5”
  • “Assert.NotNull() Failure”
  • “Not found: Monday In value:  List<String> ["Tuesday", "Wednesday", "Thursday"]”

In each of the preceding failure messages, there is not much helpful context in the failure message.

Compare the above to the following equivalent Shouldly failure messages:

  • “schedule.TotalHours should be 9 but was 5”
  • “schedule.Title should not be null or empty”
  • “schedule.Days should contain "Monday" but does not”

Notice the additional context in these failure messages. In each case here, Shouldly is telling us the name of the variable in the test code (“schedule”) and the name of the property/field being asserted (e.g. “Total Hours”).

Test Code Readability

For the preceding failure messages, the following test assert code is used (notice the use of the Shouldly extension methods):

  • schedule.TotalHours.ShouldBe(9);
  • schedule.Title.ShouldNotBeNullOrEmpty();
  • schedule.Days.ShouldContain("Monday");

In these examples there is no mistaking an actual value parameter for an expected value parameter and the test code reads more “fluently” as well.

To find out more about Shouldly check out the project on GitHub, install via NuGet, or checkout my Better Unit Test Assertions with Shouldly Pluralsight course.

New Pluralsight Course - Better .NET Unit Tests with AutoFixture

My new Pluralsight testing course was just released: Better .NET Unit Tests with AutoFixture: Get Started.

AutoFixture is an open source library that allows the simplification of automated .NET tests. It allows for the creation of “anonymous” test data. This is data that is required for the test to execute, but where the exact values are unimportant.

AutoFixture is not dependent upon any specific testing framework, so can be used with xUnit.net, NUnit, MSTest, etc. There are some additional extensions for specific testing frameworks such as xUnit.net that allow further reductions in unit test setup code.

For more info check out the GitHub site or the Pluralsight course.

If you like AutoFixture be sure to give open source thanks to Mark Seemann.

Using AutoFixture To Generate Anonymous Test Data in Web UI Automation With BDDfy and Seleno

I’m currently working on an AutoFixture Pluralsight course and it got me thinking about using it to generate anonymous test data when writing automated UI tests.

The basic premise is that in addition to using AutoFixture to generate unit test data, it can also be used to populate UI elements where the specific data values are unimportant.

If you are unfamiliar with BDDfy and Seleno, they are part of the TestStack project.

BDDfy allows tests to be written and to produce business readable documentation. Seleno allows the automation of web browsers using Selenium and strongly-typed page object models.

Example Scenario

Imagine that we have an (ASP.NET MVC) web site that allows the addition of members of the royal family.

image

If we wanted to test different king/queen names (but didn’t care about the Regnal number) we could start off by defining some strongly typed (Seleno) page object models:

public class HomePage : Page<AddRoyaltyModel>
{
    public CreatedPage CreateRoyalty(AddRoyaltyModel royal)
    {
        Input.Model(royal);

        return Navigate.To<CreatedPage>(By.Id("Create"));
    }
}

public class CreatedPage : Page
{
}

 

(all the examples in this post are fairly quick-and-dirty to demonstrate the AutoFixture involvement)

 

Next we can write some example in BDDfy (using xUnit.net as the test framework):

public class AddRoyaltyTests
{
    private HomePage _home;
    private CreatedPage _confirmationPage;
    public string Name { get; set; }
    public string Number { get; set; }

    [Fact]
    public void ShouldAddRoyaltiesWithDifferentNames()
    {
        this.Given(x => GivenIAmStartingANewRoyalAddition())
            .And("And I have entered <name> as the royalty name")
            .And("And I have entered <number> as the regnal number")
            .When(x => WhenIChooseToAddTheNewRoyal())
            .Then(x => ThenIShouldSeeAConfirmationOfTheNewRoyalHavingBeenAdded())
            .WithExamples(new ExampleTable("name", "number")
                          {
                              {"Richard", "I"},
                              {"Henry", "I"},
                              {"Elizabeth", "I"}
                          })
            .BDDfy();            
    }



    private void GivenIAmStartingANewRoyalAddition()
    {
        _home = Host.Instance.NavigateToInitialPage<HomeController, HomePage>(x => x.Index());
    }        

    private void WhenIChooseToAddTheNewRoyal()
    {
        var royal = new AddRoyaltyModel
        {
            Name = this.Name,
            RegnalRomanNumeral = this.Number
        };

        _confirmationPage = _home.CreateRoyalty(royal);
    }

    private void ThenIShouldSeeAConfirmationOfTheNewRoyalHavingBeenAdded()
    {
        Assert.Equal("Created ok", _confirmationPage.Title);
    }
}

Notice in the preceding code that even though we don’t care about the Regnal number we are still supplying it in the examples. We could just set it manually to a hardcoded value, and in this simple example that might be ok, but if we had a form with many fields then this  will introduce extra work and may make the test less “refactor-safe”.

The following code shows the removal of the Regnal name/number:

public class AddRoyaltyTestsUsingAutoFixture
{
    private HomePage _home;
    private CreatedPage _confirmationPage;
    public string Name { get; set; }

    [Fact]
    public void ShouldAddRoyaltiesWithDifferentNames()
    {
        this.Given(x => GivenIAmStartingANewRoyalAddition())
            .And("And I have entered <name> as the royalty name")
            .When(x => WhenIChooseToAddTheNewRoyal())
            .Then(x => ThenIShouldSeeAConfirmationOfTheNewRoyalHavingBeenAdded())
            .WithExamples(new ExampleTable("name")
                          {
                              "Richard",
                              "Henry",
                              "Elizabeth"
                          })
            .BDDfy();            
    }

    private void GivenIAmStartingANewRoyalAddition()
    {
        _home = Host.Instance.NavigateToInitialPage<HomeController, HomePage>(x => x.Index());
    }        

    private void WhenIChooseToAddTheNewRoyal()
    {
        var fixture = new Fixture();

        // Use AutoFixture to create anonymous data for all properties except
        // name which is set to the BDDfy example value
        var royal = fixture.Build<AddRoyaltyModel>()
            .With(x => x.Name, this.Name)
            .Create();

        _confirmationPage = _home.CreateRoyalty(royal);
    }

    private void ThenIShouldSeeAConfirmationOfTheNewRoyalHavingBeenAdded()
    {
        Assert.Equal("Created ok", _confirmationPage.Title);
    }
}

In this version, AutoFixture’s Build method is being used to automatically generate test data for all the fields, except the royal name which is set to the name(s) specified in the BDDfy examples.

Running this test results in the following automation:

SelenoAutofixture

 

And produces the following HTML BDDfy report:

image

Fixie - A Convention-based .NET Testing Framework

Fixie is a relative newcomer to the .NET testing framework space. It throws away the idea of marking elements of test code with attributes in favour of a conventions based approach.

At a high level, what this means is simply naming things in test projects following a defined default (customized conventions are also supported) convention.

A Fixie test class with a single test inside it would look like the following, notice the lack of attributes and even using statements.

namespace MyApplication.Tests
{
    public class SomeClassTests
    {
        public void ShouldDoSomething()
        {
        }
    }
}

After building the test will now show up in Visual Studio Test Explorer as the following screenshot shows.

Visual Studio Test Explorer showing Fixie Test

Fixie knows this is a test because it matches the default conventions that come out of the box.

Fixie knows that this is a test class because it ends with “Tests” and it knows that the method is a test because it’s a public void method.

If these default conventions are not suitable for your project you can create your own custom conventions to customise the test discovery. There’s also a lot more to custom conventions, such as customising the test execution lifecycle and creating data-driven tests.

To learn more about Fixie, check out the docs or my Introduction to Fixie Pluralsight course.