Data-Driven tests in MS-Test for unit and acceptance testing

  • Tutorial
I would immediately like to emphasize the fact that the world of unit tests and the world of acceptance tests through the user interface are very different worlds: with their own laws, different capabilities and limitations. And if the world of unit tests works to cover each part of the application separately and in isolation, then tests through the user interface are an emulation of the user’s work with the system, for the most part through pressing buttons and typing, which eventually merge into larger business scenarios .

It often happens that even if the tool provides very good features for unit tests, these features turn out to be practically inapplicable for UI tests.

It happened in my practice when I decided to use data-driven tests in the Ms-Test framework. In this article, I will describe in more detail the problem and my solution, for which I still do not understand - I need to be proud or ashamed.

Data-driven approach in the world of unit tests


Suppose we have an int ConvertToNumber (string input) method that takes a string with a number as input and returns the same number, only in an integer type.

Let us write one positive test ( which brings joy ), for example, for input = "3" and check that the result is also 3.
What else? Of course, boundary values ​​and equivalence classes: "-1", "0", "MaxInt", "- (MaxInt)".
Well, what if the decimal fraction hits the input? And if the line starts with a number followed by characters?

What should be the right decision in such different situations?
But there is no right decision.

Please note that different programming languages ​​handle these situations differently. For example, JavaScript as a result of operation {"10" + 1} will return the string "101", and as a result of {"10" - 1} it will be the number 9. And Perl will say: 11 and 9, respectively. But the lack of a “true” answer does not prevent us from independently devising the right behaviors and saying that from now on this is the truth, and carving this truth with stone tests in stone.

But, are you really going to save and paste a dozen unit tests for one unfortunate method with a single parameter?
And why not create a table of input and expected values, and pass each row of such a table as a parameter into one single test.
At first, I myself was very surprised that Ms-Test provides quite rich opportunities for this.

Sample data-driven unit test on Ms-Test


For illustrative purposes, in the implementation of ConvertToNumber, I will not use Convert.ToInt32 () —this would be too simple. Instead, I will offer my implementation:
static int ConvertToNumber(string input)
{
    int result = 0;
    while (input.Length > 0)
        result = (input[0] >= '0'
        || input[0] <= '9') ? (result
        * 10) + input[0] - '0' + (((input
        = input.Remove(0, 1)).Length
        > 0) ? 0 : 0) : 0;
    return result;
}

Unfortunately, in Ms-Test there is no way to set the input and expected values ​​directly in the code using attributes. But there is an alternative - to use data sources. And as such a source, you will not believe we will use the Excel file.

Although, in fact, you can use any data source that can be accessed from .NET. Including CSV, XML and others.
But working with Excel is awesome! For example, our table may look like this:
The main thing is to remember to select the desired fragment and format it as “Table with title”. Otherwise, magic will not work.

Then the unit test with implementations of the connection to the Excel table and reading the data will look like this:

const string dataDriver = "System.Data.OleDb";
const string connectionStr = "Provider=Microsoft.ACE.OLEDB.12.0;Data Source=C:\\matrix.xlsx;Extended Properties=\"Excel 12.0 Xml;HDR=YES\";";

[TestMethod]
[DataSource(dataDriver, connectionStr, "Shit1$", DataAccessMethod.Sequential)]
public void TestMe()
{
    string rowInput     = TestContext.DataRow["Input"].ToString();
    string rowExpected  = TestContext.DataRow["Expected Result"].ToString();
    string rowException = TestContext.DataRow["Exception"].ToString();
    string rowComment   = TestContext.DataRow["Comment"].ToString();

    int actualResult = ConvertToNumber(rowInput);
    Assert.AreEqual(rowExpected, actualResult.ToString());
}


Yes, I had to spend a little effort on setting up the connection and converting the input data, but look at what visual results we got.



And to add a new test - just add a new row in Excel. And you don’t need to climb into the code to see what exactly is covered by tests, and what is not.

In addition, you can go into any of the tests to see the details of the error or the log.

Please note that even on my low-performance machine, all 14 tests passed in less than 1 second.

I don’t know about you, but I was just impressed by such opportunities and visual results.
Note: To make everything work, do not forget to copy the matrix.xlsx file to the root of the C: \ drive

Data-driven approach in the world of testing through the user interface


To my great regret, after I took up the UI tests using Selenium WebDriver - my admiration for the mega-cool feature Ms-Test came to an end.

The fact is that UI tests in any implementation are, by their nature, very slow compared to modular ones. Here you can’t just take and call a function from the core of the system, easily juggling with context and parameters. No ... in order to implement a complex user scenario - you need to click a lot of buttons.

Therefore, taking into account all possible optimizations, one UI test can go from 30 seconds to tens of minutes. It all depends on the complexity of the script.

For example, if we assume that 1 out of 14 data-driven tests takes one minute, then the whole set will pass in 14 minutes.
Now imagine that one test will fall for some reason, for example, because the button on the page did not have time to appear ...
It doesn’t matter, you say, you can only fix and run fallen tests.
Not. Ms-Test considers a pack of data-driven tests as one test. Therefore, to overtake 1 test - you have to run all 14 again. And it’s not a fact that those tests that passed before will suddenly not fall. And in the case of debugging, you have to not just set a breakpoint, but set up a conditional breakpoint: Visual Studio allows this ... but, it requires additional effort and time.

It was necessary to find a way that would allow to overtake only the fallen test. And I found such a way.

Solution: “cycle through inheritance”


To solve the problem, you need to add 2 files to the project: TestBase.cs and TestRows.cs .

Then in the namespace "MsTestRows.Rows. *", The classes TestRows_ 01 ... TestRows_ 100 will appear , which contain from one to one hundred generated methods.

Inherit your TestClass from TestRows_NN with the required number of methods (NN).

Next, Visual Studio will ask you to implement two methods:
  • GetNextDataRow () - which will return data for the next test
  • TestMethod () - which will be called from the test with the data received from GetNextDataRow ().

Full example code
using System;
using System.Text;
using System.Collections.Generic;
using System.Linq;
using Microsoft.VisualStudio.TestTools.UnitTesting;

namespace HabraDrivenTests
{

    // Custom Data Row 
    public class HabrDataRow
    {
        public string Input { get; set; }
        public int Expected { get; set; }
        public Type ExpectedException { get; set; }
        public string Comment { get; set; }

    }
    
    [TestClass]
    public class HabrTestInherienceLoop : MsTestRows.Rows.TestRows_04<HabrDataRow>
    {
        // Production-ready method
        static int ConvertToNumber(string input)
        {
            int result = 0;
            while (input.Length > 0)
                result = (input[0] >= '0'
                || input[0] <= '9') ? (result
                * 10) + input[0] - '0' + (((input
                = input.Remove(0, 1)).Length
                > 0) ? 0 : 0) : 0;
            return result;
        }

        // Test Data
        static HabrDataRow[] testData = new HabrDataRow[]
        {
            #region Data
            new HabrDataRow()
            {
                 Input    = "0",
                 Expected = 0,
                 ExpectedException = null,
                 Comment = "Граничное значение",
            },
            new HabrDataRow()
            {
                 Input    = "1",
                 Expected = 1,
                 ExpectedException = null,
                 Comment = "Граничное значение",
            },
            new HabrDataRow()
            {
                 Input    = "-1",
                 Expected = -1,
                 ExpectedException = null,
                 Comment = "Граничное значение",
            },

            new HabrDataRow()
            {
                 Input    = "2147483647",
                 Expected = 2147483647,
                 ExpectedException = null,
                 Comment = "int.MaxValue",
            },
            #endregion
        };

        // Data Generator
        public override HabrDataRow GetNextDataRow(int rowIndex)
        {
            return testData[rowIndex];
        }

        // Test Implementation
        public override void TestMethod(HabrDataRow dataRow, int rowIndex)
        {
            int actualResult = ConvertToNumber(dataRow.Input);
            Assert.AreEqual(dataRow.Expected, actualResult);
        }
    }
}


As a result, using such a perverted maneuver, I managed to get the opportunity to restart only the fallen test.
image

Source code and links



Other related materials:



Note 1: The file TestRows.cs contains 30,000 lines of code. If your salary bonus depends on the number of lines of code written, and the system automatically credits you a bonus of $ 1 million, then I think it will be fair if you send me at least 5% of this amount.

Note 2: Try to implement ConvertToNumber so that it passes all the tests.
Attention: stick to the "author's style."