Business-Readable, Living Documentation with BDDfy

BDDfy enables the creation of tests that, in addition to verifying that the system works correctly, also results in business-readable, living documentation.

Business-readable means that the tests are described in natural language (e.g. English) that the business can read, understand, and ensure that the correct features and functionality is being built.

Living documentation means that the report results directly from the passing or failing of the tests. It’s not a word document somewhere on a shared drive or SharePoint site that may or may not actually be in sync with what the system actually does.

The Report

Below is an example of what a typical BDDfy HTML report looks like (there is a “metro” inspired report coming in V4). There’s also the ability to output the test report in markdown.

BDDfy HTML report

Notice that it’s not code-centric output, but rather business-centric.

Here the two test scenarios are grouped into a story, but they don’t have to be.

The underlying test framework that is used to execute the test doesn’t matter – you could use NUnit, xUnit.net, MSTest, etc.

The Test Code

The tests that produce this report could be configured in BDDfy in a couple of ways. There is a reflective style that uses methods that following a specific naming convention. There is also the fluent style.

The test code to produce this report looks like the below (we are using NUnit in this example).

using NUnit.Framework;
using TestStack.BDDfy;
using TestStack.BDDfy.Core;
using TestStack.BDDfy.Scanners.StepScanners.Fluent;

namespace BDDfyDemo
{
    [TestFixture]
    [Story(AsA = "As a customer",
        IWant = "I want my order total to add up correctly",
        SoThat = "I'm not overcharged for my goods")]
    public class OrderTotalCalculatorTests
    {
        [Test]
        public void MultipleOrderedItemsTotals()
        {
            this.Given(x => GivenIHaveAddedItemsToMyCart())
                .When(x => WhenICheckout())
                .Then(x => ThenTheOrderTotalExcludingTaxShouldBeCorrect())
                .BDDfy<OrderTotalCalculatorTests>();
        }

        [Test]
        public void SalesTaxAdded()
        {
            this.Given(x => GivenIHaveAddedItemsToMyCart())
                .When(x => WhenICheckout())
                .Then(x => ThenTheSalesTaxShouldBeCorrect())
                .BDDfy<OrderTotalCalculatorTests>();
        }

        public void GivenIHaveAddedItemsToMyCart()
        {
            // test code
        }

        public void WhenICheckout()
        {
            // test code
        }

        public void ThenTheOrderTotalExcludingTaxShouldBeCorrect()
        {
            // test code
        }

        public void ThenTheSalesTaxShouldBeCorrect()
        {
            // test code
        }

    }
}

Here there are a couple of NUnit tests defined – the methods with the [Test] attribute applied to them.

Within these test methods the BDDfy fluent style is being used to define the different steps in the test scenarios.

The “given” phase sets up the initial context or state of the thing being tested. The “when” phase acts upon the system to produce some change. The “then” phase is typically where we have assert code, it’s in this phase that the resulting state of the system is checked.

Notice how the individual method names appear in the test report in a nice business-readable way.

 

There’s a lot more to BDDfy, such as data-parameterised tests and a whole heap of customisation and configuration options. To see more of what BDDfy can do check out my Building the Right Thing in .NET with TestStack Pluralsight course, head on over to the documentation, or check it out on GitHub.

SHARE:

Beyond the Compiler with ConventionTests

We often have conventions in our code. For example all DTOs/entity/etc. classes should be in a specific namespace, or the name of certain types of class should end with a given word. Another example, we may need to make sure that all classes (or those in a specific namespace) contain all-virtual members.

These kind of things are conventions that the team may agree to but the compiler won’t pick up as build errors or warnings. The code may still be completely valid C# even though it violates a convention.

ConventionTests is a brilliant idea (and it’s ideas like this that keep me loving programming) allows us to write tests to verify these conventions, conventions that the compiler cannot verify.

Installation

Simple installation via NuGet into your test project: PM> Install-Package TestStack.ConventionTests

Simple Usage

Using whatever testing framework you prefer (xUnit.net, NUnit, etc.) create a test method.

these kind of things are conventions that the team may agree to but the compiler won’t pick up

In this method, the first thing to do is to select all the types in the production code that we want to check against a convention.

We can do this with code such as:

var typesToCheck = Types.InAssemblyOf<SomeClass>();   

Here the Types.InAssemblyOf method is being used to find which assembly contains the class called SomeClass. Using this overload, all of the types in the assembly will be selected, however there are other overloads that allow the returning of types in a given namespace.

The next step is to create an instance of one of the out-of-the-box conventions.

var convention = new AllClassesHaveDefaultConstructor();

This convention checks that all the selected types have a default constructor defined.

Now we have the list of types we want to check, and a convention to check them against.

To actually perform the check, we use the Convention.Is method:

Convention.Is(convention, typesToCheck);

If any of the classes selected in typesToCheck don’t have a default constructor then an exception will be thrown (with some useful info in it) and the test will fail.

You can see some of the other out-of-the-box conventions on the GitHub site, you can also write your own custom conventions.

 

ConventionTests is one of the tools in the TestStack suite. To learn more about ConventionTests and the other TestStack tools, checkout out my Building the Right Thing in .NET with TestStack Plurasight course.

SHARE:

Using Page Object Models in UI Test Automation

When using UI automation technologies, changes to the UI can break our tests. For example, if a textbox is being located by its ID in numerous tests, if this ID changes then all these tests will break. They won’t be able to find the ID.

Another problem that can arise with automated UI testing is code duplication. Imagine a system that had an color drop down list and an apply button in multiple places in the application. Each UI test that exercises these color choices must first locate the color choice box, select a color, locate the apply button, and then click it.

The 3 Levels Of User Interface Testing

3 Levels Of User Interface Testing

One way to think of the “evolution” of UI testing is that of: test script, page object model (POM), and logical functional model (LFM).

Test Script

Test scripts are self contained tests. They contain the whole test from starting the application through to the asserts at the end.

Test scripts can be lengthy as they have to include all the UI control finding and interaction code for every single step of the particular test. This can make them hard to read and understand exactly what behaviour is being tested.

Test scripts also don’t share any code with other tests, so code duplication can create additional maintenance effort over time.

Test scripts are an easy way to get started with automated UI testing, especially if you are learning the UI automation framework at the same time.

Pretty quickly, test scripts become unwieldy and so page object models can be introduced.

Page Object Models

Page object models (POM) represent a screen or page or group of controls in the UI that’s being tested.

For example, a screen with a color chooser could be represented by a POM class that exposes the drop down control and buttons as properties.

These properties encapsulate the locating of the UI dropdown and button (perhaps by ID). The test code itself then uses an instance of the POM to interact with the dropdown and button – it doesn’t have to worry about the UI IDs in the test itself.

There’s a couple of levels of abstraction within POMs. The property that represents a UI control can either expose the underlying automation framework object that represents the control, or a simple type such as string or int.

When first getting started with POMs you might want to go with the former approach until you get the idea of using POMs, though there’s not much additional work to implement the latter option.

By returning simple types rather than automation framework types, the actual test code can become more readable; they also help to abstract away the chosen automation framework.

Logical Functional Models

Logical function models (LFM) are an extension to POMs and may not deserve their own distinction in practice, however the distinction helps us to think how we can further enhance the readability of test code and reduce code duplication.

LFMs add methods to POMs. These methods represent logical actions that the end-user can perform on a given page.

The LFM methods can use the properties already defined on the POM.

For example, the logical user function of “I want to choose a color” can be represented as a single method in the POM, for example: ChooseColor(string color);

Within this single method, the existing POM properties representing the dropdown list of colors and button can be used.

Now, test code can simply write: myScreen.ChooseColor(“red”);

The LFM “style” further increases the ability to discern exactly what behaviour a given test is testing.

Building Page Object Models

When building a POM it’s a good idea to evolve it over time, adding properties and logical functional methods as needed by the tests.

The alternative is to model every control and user interaction on the page before starting writing test that use that POM. This increases the up-front cost of writing automated UI tests, you may not end up interacting with every UI control in your tests so it makes sense to evolve POMs in an iterative fashion.

When choosing a UI automation framework, investigate what built-in support it has for POMs/LFMs. It’s possible to hand-code them from scratch, but a little extra help from our chosen framework helps to reduce the cost of building UI automation tests.

 

To see POMs/LFMs in action in both WPF and Web UI automation, check out my Building the Right Thing in .NET with TestStack Pluralsight course.

SHARE:

3 Ways to Pass State Between SpecFlow Step Definitions

Sometimes we need one step definition to know about parameter data that was passed to previous steps.

There’s 3 ways main ways to do this, each with various considerations.

Private Field in Binding Class

This is the simplest approach. Simply add a private field to your [Binding] class containing your step definitions.

In a step, just grab the parameter data and store it in this private field.

In later steps, retrieve the data from this private field.

This approach is simple but breaks down when we start to refactor our step definitions into multiple step definition classes; the private field data is not accessible between steps in different binding classes.

ScenarioContext

We can store data for the current executing scenario using the ScenarioContext class.

There's a number of ways to use it, but essentially it’s a dictionary that we can store arbitrary data and retrieve it by a key:

ScenarioContext.Current["age"] = 42;

This approach can be used even if we refactor step definitions out across multiple binding classes. The scenario context is shared between all steps for the lifetime that a scenario is executing.

This approach can start to get a little ugly, we also have to make sure our dictionary keys are correct in all the steps where we set or retrieve data.

By default, it’s also weakly-typed – the dictionary just stores objects so we can end up with casts every time we retrieve data.

Context Injection

The third method (which I wrote about previously) lets us take a dependency injection style approach, and define POCOs to represent the data (in a strongly typed way) that automatically gets injected into each of our binding classes.

Like ScenarioContext, this approach also works across multiple binding classes. Unlike ScenarioContext, we don’t have to worry about dictionary keys or casting.

 

If you’re new to SpecFlow and Gherkin, check out my Pluralsight course:  Automated Acceptance Testing with SpecFlow and Gherkin or if you understand the basics or have been using it for a while, check out  SpecFlow Tips and Tricks course to learn how to create more maintainable SpecFlow test automation solutions.

SHARE:

New Pluralsight Course: SpecFlow Tips and Tricks

My latest Pluralsight course was released today: SpecFlow Tips and Tricks is a short course to help those who are relatively new to SpecFlow and those who have been using it for a while and want to create more maintainable testing solutions.

The course focuses on 3 main areas:

  • Steps and Bindings
  • Hooks and Scoped Bindings
  • Step Parameters and Data Tables

This course is a loose follow-up to my previous Automated Acceptance Testing with SpecFlow and Gherkin course that covers the Gherkin DSL and other fundamental SpecFlow topics.

You can check out the course on the Pluralsight website or on my author page.

SHARE:

SpecFlow Cheat Sheet

SpecFlow is a tool that allows the writing of business-readable tests that can then be automated in code. If you’re new to SpecFlow check out my Pluralsight course or the resources page.

The cheat sheet below shows the basics of different step binding styles, the hooks that can be used to execute additional code, and scoped bindings.

More...

SHARE:

Advanced SpecFlow: Custom Conversion of Step Parameters and Data Tables with [StepArgumentTransformation]

SpecFlow is a tool that allows the writing of business-readable tests that can then be automated in code. If you’re new to SpecFlow check out my Pluralsight course or the resources page to get up to speed before looking at these more advanced topics.

SpecFlow will attempt to perform some conversions of step arguments as specified in the documentation.

We can also tell SpecFlow to use our own custom conversions. Take the feature below:

Feature: SomeFeature

Scenario: Update User Password
    Given The original user was created over 10 days ago
    When I update the users password to "some password"
    Then I should see the following errors
        | ErrorCode         | ErrorDescription          |
        | 442               | Out of date user          |
        | 28                | Changes were not saved    |

From this we generate some basic step definitions:

[Given]
public void Given_The_original_user_was_created_over_DAYS_days_ago(int days)
{
}

[When]
public void When_I_update_the_users_password_to_NEWPASSWORD(string newPassword)
{
}
       
[Then]
public void Then_I_should_see_the_following_errors(Table table)
{            
}

Automatically Parsing 10 Days Ago into a DateTime

In the Given step we get an int specifying how many days ago the user was created. As it’s the Given step we need to put the system into a state where the user creation date, for example held as a DateTime in the user database, is 10 days old. If this is just a single step then we can simply write the code to subtract a number of days from todays date in the step itself. Note: this kind of date based testing where we rely on “now” may prove brittle, imagine the a Given ran just before midnight, but the Then gets executed after midnight, i.e. the next day.

More...

SHARE:

Gherkin Cheat Sheet

Gherkin is a Business Readable Domain Specific Language. It allows the documentation of what a system should do in natural language (multiple languages are supported) and allows test automation code to be written for the system’s features.

This cheat sheet outlines the language basics.

More...

SHARE:

Advanced SpecFlow: Restricting Step Definition and Hook Execution with Scoped Bindings

SpecFlow is a tool that allows the writing of business-readable tests that can then be automated in code. If you’re new to SpecFlow check out my Pluralsight course or the resources page to get up to speed before looking at these more advanced topics.

Step definitions and hooks by default have global scope within the test project.

For example, take the following feature:

Feature: SomeFeature

Scenario: Scenario 1
    Given A
    When B
    Then C

Scenario: Scenario 2
    Given A
    When X
    Then Z

Here the “Given A” step shares a step definition, note in the following code there is only one step that matches “Given A”:

[Binding]
public class SomeFeatureSteps
{
    [Given]
    public void Given_A()
    {
        // etc.
    }       

    [When]
    public void When_B()
    {
        // etc.
    }
    
    [When]
    public void When_X()
    {
        // etc.
    }
    
    [Then]
    public void Then_C()
    {
        // etc.
    }
    
    [Then]
    public void Then_Z()
    {
        // etc.
    }
}

If for some reason we want different automation code to run for each of the “Given A” steps, we could simply reword one of the steps, then we would have two different step definitions.

More...

SHARE:

Advanced SpecFlow: Using Hooks to Run Additional Automation Code

SpecFlow is a tool that allows the writing of business-readable tests that can then be automated in code. If you’re new to SpecFlow check out my Pluralsight course to get up to speed before looking at these more advanced topics.

In addition to executing test automation code in step definitions, SpecFlow provides a number of additional “hooks” to allow additional code execution at various points during the test execution lifecycle.

For this example, consider the following feature:

Feature: SomeFeature

Scenario: Add New Contact Name
    Given I have entered Sarah
    When I choose add
    Then the contact list should show the new contact

With the following step definitions:

using TechTalk.SpecFlow;

namespace SpecFlowHooks
{
    [Binding]
    public class SomeFeatureSteps
    {
        [Given]
        public void Given_I_have_entered_NEWNAME(string newName)
        {
            // etc.
        }
        
        [When]
        public void When_I_choose_add()
        {
            // etc.   
        }
        
        [Then]
        public void Then_the_contact_list_should_show_the_new_contact()
        {
            // etc.
        }
    }
}

When the scenario is executed we get the following test output:

More...

SHARE: