Lifelike Test Data Generation with Bogus

Bogus is a lovely library from Brian Chavez to use in automated tests to automatically generate test data of different kinds.

As an example suppose the following class is involved in a unit test:

public class Review
{
    public int Id { get; set; }
    public string Title { get; set; }
    public string Body { get; set; }
    public int Rating { get; set; }
    public DateTimeOffset Created { get; set; }

    public override string ToString()
    {
        return $"{Id} '{Title}'";
    }
}

In a test, a Review instance may need properties populating with values. This could be done manually, for example to check the ToString() implementation:

[Fact]
public void BeRepresentedAsAString()
{
    var sut = new Review
    {
        Id = 42,
        Title = "blah blah"
    };

    Assert.Equal("42 'blah blah'", sut.ToString());
}

Notice in the preceding test, the actual values and title don’t really matter, only the fact that they’re joined as part of the ToString() call. In this example the values for Id and Title could be considered anonymous variable / values in that we don’t really care about them.

The following test uses the Bogus NuGet package and uses its non-fluent facade syntax:

[Fact]
public void BeRepresentedAsAString_BogusFacadeSyntax()
{
    var faker = new Faker("en"); // default en

    var sut = new Review
    {
        Id = faker.Random.Number(),
        Title = faker.Random.String()
    };

    Assert.Equal($"{sut.Id} '{sut.Title}'", sut.ToString());
}

Bogus also has a powerful fluent syntax to define what a test object will look like. To use the fluent version, a Faker<T> instance is created where T is the test object to be configured and created, for example:

[Fact]
public void BeRepresentedAsAString_BogusFluentSyntax()
{
    var reviewFaker = new Faker<Review>()
        .RuleFor(x => x.Id, f => f.Random.Number(1, 10))
        .RuleFor(x => x.Title, f => f.Lorem.Sentence());

    var sut = reviewFaker.Generate(); 

    Assert.Equal($"{sut.Id} '{sut.Title}'", sut.ToString());
}

The first argument to the RuleFor() methods allows the property of the Review object to be selected and the second argument specifies how the property value should be generated. There is a huge range of test data types supported. In the preceding code the Random API is used as well as the Lorem API.

Some examples of the types of auto generated data include:

  • Addresses: ZipCode, City, Country, Latitude, etc.
  • Commerce: Department name, ProductName, ProductAdjective, Price, etc.
  • Company: CompanyName, CatchPhrase, Bs, etc.
  • Date: Past, Soon, Between, etc.
  • Finance: Account number, TransactionType, Currency, CreditCardNumber, etc.
  • Image URL: Random image, Animals image, Nature image, etc.
  • Internet: Email, DomainName, Ipv6, Password, etc.
  • Lorem: single word, Words, Sentence, Paragraphs, etc.
  • Name: FirstName, LastName, etc.
  • Rant: Random user review, etc.
  • System: FileName, MimeType, FileExt, etc.

Some of the random generated values are quite entertaining, for example Rant.Review() may produce "My co-worker Fate has one of these. He says it looks tall."; Company.Bs() may produce "transition cross-media users", and Company.CatchPhrase() may produce "Face to face object-oriented focus group".

Bogus configuration is quite powerful and allows fairly complex setup as the following code demonstrates:

[Fact]
public void CalculateAverageRatingWhenMultipleReviews()
{
    int rating = 0;

    var reviewFaker = new Faker<Review>()
        .RuleFor(x => x.Id, f => f.Random.Number(1, 10))
        .RuleFor(x => x.Rating, f => rating++);

    var productFaker = new Faker<Product>()
        .RuleFor(x => x.PricePerUnit, f => f.Finance.Amount())
        .RuleFor(x => x.Description, f => f.WaffleText(3))
        .FinishWith((f, x) =>
            {
                reviewFaker.Generate(3).ForEach(r => x.Reviews.Add(r));
            });

    var sut = productFaker.Generate();

    Assert.Equal(1, sut.AverageRating); // (0 + 1 + 2) / 3
}

The WaffleText() API is provided by one of the extensions to Bogus (WaffleGenerator.Bogus) that produces inane looking waffle text such as the following:

The Quality Of Hypothetical Aesthetic

"The parallel personal hardware cannot explain all the problems in maximizing the efficacy of any fundamental dichotomies of the logical psychic principle. Generally the requirements of unequivocal reciprocal individuality is strictly significant. On the other hand the characteristic organizational change reinforces the weaknesses in the evolution of metaphysical terminology over a given time limit. The objective of the explicit heuristic discordance is to delineate the truly global on-going flexibility or the preliminary qualification limit. A priority should be established based on a combination of functional baseline and inevitability of amelioration The Quality Of Hypothetical Aesthetic"

 - Michael Stringer in The Journal of the Proactive Directive Dichotomy (20174U)

structure plan.

To make the main points more explicit, it is fair to say that;
  * the value of the optical continuous reconstruction is reciprocated by what should be termed the sanctioned major issue.
  * The core drivers poses problems and challenges for both the heuristic non-referent spirituality and any discrete or Philosophical configuration mode.
  * an anticipation of the effects of any interpersonal fragmentation reinforces the weaknesses in the explicit deterministic service. This may be due to a lack of a doctrine of the interpersonal quality..
  * any significant enhancements in the strategic plan probably expresses the strategic personal theme. This trend may dissipate due to the personal milieu.

 firm assumptions about ideal major monologism evinces the universe of attitude.

The Flexible Implicit Aspiration.

Within current constraints on manpower resources, any consideration of the lessons learnt can fully utilize what should be termed the two-phase multi-media program.

For example, the assertion of the importance of the integration of doctrine of the prime remediation with strategic initiatives cannot be shown to be relevant. This is in contrast to the strategic fit.

To learn more about Bogus head over to the documentation.

MSTest V2

In the (relatively) distant past, MSTest was often used by organizations because it was provided by Microsoft “in the box” with Visual Studio/.NET. Because of this, some organizations trusted MSTest over open source testing frameworks such as NUnit. This was at a time when the .NET open source ecosystem was not as advanced as it is today and before Microsoft began open sourcing some of their own products.

Nowadays MSTest is cross-platform and open source and is known as MSTest V2, and as the documentation states: “is a fully supported, open source and cross-platform implementation of the MSTest test framework with which to write tests targeting .NET Framework, .NET Core and ASP.NET Core on Windows, Linux, and Mac.”.

MSTest V2 provides typical assert functionality such as asserting on the values of: strings, numbers, collections, thrown exceptions, etc. Also like other testing frameworks, MSTest V2 allows the customization of the test execution lifecycle such as the running of additional setup code before each test executes. The framework also allows the creation of data driven tests (a single test method executing  multiple times with different input test data) and the ability to extend the framework with custom asserts and custom test attributes.

You can find out more about MSTest V2 at the GitHub repository, the documentation, or check out my Pluralsight course: Automated Testing with MSTest V2.

Getting Started Testing .NET Core Code with xUnit.net

xUnit.net is a testing framework that can be used to write automated tests for .NET (full) framework and also .NET Core.

To get started, first create a .NET Core application, in the following example a .NET Core console app.

Creating a .NET core console project

A testing project can now be added to the solution:

Adding an xUnit test project in Visual Studio 2017

This test project will come pre-configured with the relevant NuGet packages installed to start writing test code, though you may want to update the pre-configured packages to the newest NuGet versions.

The xUnit Test Project template will also create the following default test class:

using System;
using Xunit;

namespace ConsoleCalculator.Tests
{
    public class UnitTest1
    {
        [Fact]
        public void Test1()
        {

        }
    }
}

Notice in the preceding code, the Test1 method is decorated with the [Fact] attribute. This is an xUnit.net attribute that tells a test runner that it should execute the method, treat it as a test, and report on if the test passed or not.

Next add a project reference from the test project to the project that contains the code that is to be tested, this gives the test project access to the production code.

In the production project, the following class can be added:

namespace ConsoleCalculator
{
    public class Calculator
    {
        public int Add(int a, int b)
        {            
            return a + b;
        }
    }
}

Now the test class can be renamed (for example to “CalculatorTests”) and the test method changed to create a test:

using Xunit;

namespace ConsoleCalculator.Tests
{
    public class CalculatorTests
    {
        [Fact]
        public void ShouldAddTwoNumbers()
        {
            Calculator calculator = new Calculator();

            int result = calculator.Add(7, 3);

            Assert.Equal(10, result);
        }
    }
}

In the preceding code, once again the [Fact] attribute is being used, then the thing being tested is created (the Calculator class instance). The next step is to perform some kind of action on the thing being tested, in this example calling the Add method. The final step is to signal to the test runner if the test has passed or not, this is done by using one of the many xUnit.net Assert methods; in the preceding code the Assert.Equal method is being used. The first parameter is the expected value of 10, the second parameter is the actual value produced by the code being tested. So if  result is 10 the test will pass, otherwise it will fail.

One way to execute tests is to use Visual Studio’s Test Explorer which can be found under the Test –> Windows –> Test Explorer menu item. Once the test project is built, the test will show up and can be executed as the following screenshot shows:

Running xUnit tests in Visual Studio Test Explorer

To learn more about how to get started testing .NET Core code check out my Testing .NET Core Code with xUnit.net: Getting Started Pluralsight course or check out the docs.

Testing Automation: The Big Picture

It’s often useful to take a step back and look at the bigger picture, this is true in different aspects of life such as health or wealth or relationships, and is also true of software development.

When it comes to creating automated tests (as with other aspects of software development) dogmatism and absolutist schools of though can exist.

As with all things, the decision to write tests (and how many tests, what type of tests, test coverage aims, etc.) ultimately should boil down to one question: do they add value to what you are doing?

To be clear, I absolutely believe in the creation of automated tests in many cases, however it is good to not be dogmatic. For example if there is a niche market that is ripe for capitalizing on, and time-to-market is the most important thing to capture this market, then an extensive suite of automated tests may slow down getting to that initial release. This of course assumes that the potential market will have some tolerance for software defects. It also depends on what the product is; medical life-critical software is probably going to have a higher quality requirement than a social media app for example. This can be a trade-off however with shorter term delivery speeds being quicker but at the expense of delivery speed in the long term, if you’re overwhelmed fixing production outages you have very little time to add new features/value.

Another aspect to consider is that of risk. What are the risks associated with defects in the software making their way into production? Different features/application areas may also have different risk profiles; for example  the “share on social media” feature may not be deemed as important as a working shopping cart. It’s also important to remember that risk is not just monetary, in the previous example a broken “share on social media” feature may bring the business into disrepute, aka “reputational risk”.

When it comes to the myriad of different types of tests (unit, integration, subcutaneous, etc.) the testing pyramid is an oft-quoted model of how many of each type of tests to have in the test suite. While the testing pyramid may be of great help for someone new to automated testing to help them learn and navigate their initial steps, as experience grows the model may no longer be optimal to some of the projects that are being worked on. Beyond the testing pyramid the different aspects of test types can be considered such as execution speed, breadth/depth, reliability/brittleness etc.

Automated tests also do not exist in and of themselves, they are part of a bigger set of processes/considerations such as pair programming, code reviews, good management, well-understood requirements, good environment management/DevOps, etc.

If you want to take a step back and look at the big picture, or know a developer or manager who wants to understand the trade-offs/benefits check out my Testing Automation: The Big Picture Pluralsight course.

New Pluralsight Course: Testing C# Code in Production with Scientist.NET

My latest Pluralsight course is now available for viewing. It demonstrates how to use the Scientist.NET library to execute candidate code in production alongside the existing production code. This allows the new candidate code to be additionally verified as able to work in production (for example with production data that may be of variable quality) and offers an additional check in addition to automated tests that have been executed in the development/QA environment.

From the course description: “Errors in production code can be costly to fix, more stressful for the development team, and frustrating for the end-user. In this course, Testing C# Code in Production with Scientist.NET, you will get to see how Scientist.NET allows sections of the application to be changed more safely by running both the existing code alongside the new code in production. You'll begin with an introduction to Scientist.NET before installing it and writing your first experiment. Next, you'll learn how to customize the configuration of experiments. Finally, you'll learn how to publish experiment data to SQL Server and analyze experiment results…”

You can check out the new course here.

Refactoring Production Code With Experiments and Scientist.NET

When refactoring a part of an application we can use the existing tests to give a level of confidence that the refactored code still produces the same result, i.e. the existing tests still pass with the new implementations.

A system that has been in production use for some time is likely to have amassed a lot of data that is flowing through the “legacy” code. This means that although the existing tests may still pass, when used in production the new refactored code may produce errors or unexpected results.

It would be helpful as an experiment to run the existing legacy code alongside the new code to see if the results differ, but still continue to use the result of the existing legacy code. Scientist.NET allows exactly this kind of experimentation to take place.

Scientist.NET is a port of the Scientist library for Ruby. It is currently in pre-release and has a NuGet package we can use.

An Example Experiment

Suppose there is an interface as shown in the following code that describes the ability to sum a list if integer values and return the result.

interface ISequenceSummer
{
    int Sum(int numberOfTerms);
}

This interface is currently implemented in the legacy code as follows:

class SequenceSummer : ISequenceSummer
{
    public int Sum(int numberOfTerms)
    {
        var terms = new int[numberOfTerms];


        // generate sequence of terms
        var currentTerm = 0;
        for (int i = 0; i < terms.Length; i++)
        {
            terms[i] = currentTerm;
            currentTerm++;
        }


        // Sum
        int sum = 0;
        for (int i = 0; i < terms.Length; i++)
        {
            sum += terms[i];
        }

        return sum;
    }
}

As part of the refactoring of the legacy code, this implementation is to be replaced with a version that utilizes LINQ as shown in the following code:

class SequenceSummerLinq : ISequenceSummer
{
    public int Sum(int numberOfTerms)
    {
        // generate sequence of terms
        var terms = Enumerable.Range(0, 5).ToArray();
            
        // sum
        return terms.Sum();
    }
}

After installing the Scientist.NET NuGet package, an experiment can be created using the following code:

int result;            

result = Scientist.Science<int>("sequence sum", experiment =>
{
    experiment.Use(() => new SequenceSummer().Sum(5)); // current production method

    experiment.Try("Candidate using LINQ", () => new SequenceSummerLinq().Sum(5)); // new proposed production method

}); // return the control value (result from SequenceSummer)

This code will run the .Use(…) code that contains the existing legacy implementation. It will also run the .Try(…) code that contains the new implementation. Scientist.NET will store both results for reporting on and then return the result from the .Use(…) code for use by the rest of the program. This allows any differences to be reported on but without actually changing the implementation of the production code. At some point in the future, if the results of the legacy code (the control) match that of the new code (the candidate), the refactoring can be completed by removing the old implementation (and the experiment code) and simply calling the new implementation.

To get the results of the experiment, a reporter can be written and configured. The following code shows a custom reporter that simply reports to the Console:

public class ConsoleResultPublisher : IResultPublisher
{
    public Task Publish<T>(Result<T> result)
    {
        Console.ForegroundColor = result.Mismatched ? ConsoleColor.Red : ConsoleColor.Green;

        Console.WriteLine($"Experiment name '{result.ExperimentName}'");
        Console.WriteLine($"Result: {(result.Matched ? "Control value MATCHED candidate value" : "Control value DID NOT MATCH candidate value")}");
        Console.WriteLine($"Control value: {result.Control.Value}");
        Console.WriteLine($"Control execution duration: {result.Control.Duration}");
        foreach (var observation in result.Candidates)
        {
            Console.WriteLine($"Candidate name: {observation.Name}");
            Console.WriteLine($"Candidate value: {observation.Value}");
            Console.WriteLine($"Candidate execution duration: {observation.Duration}");
        }

        if (result.Mismatched)
        {
            // saved mismatched experiments to event log, file, database, etc for alerting/comparison
        }

        Console.ForegroundColor = ConsoleColor.White;

        return Task.FromResult(0);
    }  
}

To plug this in, before the experiment code is executed:

Scientist.ResultPublisher = new ConsoleResultPublisher();

The output of the experiment (and the custom reporter) look like the following screenshot:

screenshot of console application using Scentist.NET with custom reporter

To learn more about Scientist.NET check out my Pluralsight course: Testing C# Code in Production with Scientist.NET.

New Pluralsight Course: Automated Business Readable Web Tests with Selenium and SpecFlow

SpecFlow is a tool that can translate natural language scenarios (e.g. writing in English or other spoken languages) into test code. This can allow business people, users, or other stakeholders to verify that the correct features are being built.

Selenium is a tool that allows test code (multiple programming languages supported) to automated a web browser. This allows the creation of automated UI tests that operate the web application as if an end user where doing it; for example clicking buttons and typing text into input boxes.

My new Pluralsight course shows how to integrate these two tools.

The course is organized into four modules:

  1. Introduction to Business Readable Web Testing
  2. Getting Started with Selenium
  3. Adding Business Readability with SpecFlow
  4. Creating More Maintainable Web Automation

If you’re new to SpecFlow I suggest watching this course first before moving on to Automated Business Readable Web Tests with Selenium and SpecFlow.

Hook Execution Order in SpecFlow 2

SpecFlow hooks allow additional code to be executed before and after various stages of the test execution lifecycle, for example running additional setup code before each scenario executes.

If there are multiple of the same type of hook specified, by default the execution order of the hook methods is unspecified. For example the following code has three [BeforeStep] hook methods that could be executed in any order before every step of the scenario executes:

[BeforeStep]
public void BeforeHook1()
{
}

[BeforeStep]
public void BeforeHook2()
{
}

[BeforeStep]
public void BeforeHook3()
{
}

To ensure these hook methods are executed in a specified order, the hook attributes allow an optional order to be specified. When there are multiple of the same hook methods defined, the lowest order values execute before the higher order methods:

[BeforeStep(Order = 100)]
public void BeforeHook1()
{
}

[BeforeStep(Order = 200)]
public void BeforeHook2()
{
}

[BeforeStep(Order = 300)]
public void BeforeHook3()
{
}

The values of the Order property are arbitrary, you may use whatever values you wish, though it is sensible to allow some “wriggle room” for future additional steps by working in increments of 10 or 100 for example.

The following code illustrates another example where the execution order of hooks is important; the database should be reset first before test users are added:

[Binding]
public class Hooks
{
    [BeforeScenario(Order = 100)]
    public void ResetDatabase()
    {
    }

    [BeforeScenario(Order = 200)]
    public void AddTestUsersToDatabase()
    {
    }        
}

To see hook ordering in action, check out my Pluralsight course: Business Readable Automated Tests with SpecFlow 2.0.

New Pluralsight Course: Business Readable Automated Tests with SpecFlow 2.0

My newest Pluralsight course was just published. Business Readable Automated Tests with SpecFlow 2.0 teaches how to create tests that the business can read, understand, and contribute to. These “English-like” tests (other spoken languages are supported) can be executed by writing test code that is associated with the “English-like” steps. Because the tests sit alongside the source code, they can become living (executable) documentation for the system, as opposed to an out-of-date Word document somewhere on the network for example. Check out the course here.