Writing a suit of unit tests that exhaustively exercise and validate the logic of the code under test is not easy. It can even be considered too expensive to do at all. Fortunately, the IntelliTest feature shipping in the Visual Studio 2015 Enterprise Edition addresses both concerns: it helps you achieve high code coverage at a fraction of the cost.
When you run IntelliTest on your .NET code, it generates test cases by dynamically analyzing the code under test as it is running on the CLR. For every statement in the code, it crafts an input that will reach that statement, doing a case analysis for every conditional branch—if statements, assertions, and all operations that can throw exceptions—much like white box testing. The goal is to generate a test suite covering all branches of your code, and so every time it crafts an input that increases coverage, it emits that concrete value as a test case using C# and one of MSTest, xUnit.net, or NUnit as the test framework. The result is a compact suite of tests with high coverage that you didn’t have to write from scratch.
Note: IntelliTest draws from the Pex project and has had several incarnations, starting off as extensions to Visual Studio. It was also called “Smart Unit Tests” prior to Visual Studio 2015 RC. Through all these changes, however, the underlying APIs and namespaces have been retained, so that expertise, and extensions, you may have built around these earlier incarnations can be leveraged; indeed, any Pex extensions you might have written in the past will likely work with IntelliTest.
Making it easier to write and maintain unit tests
Here are the problems that IntelliTest is trying to solve.
- Problem 1: Writing an exhaustive test suite for complex code can take as much effort as writing the code under test itself. Consequently, there is temptation to avoid writing the tests, leading to test holes and a reliance on integration testing that is done after the development activity. Bugs get caught later in the development cycle (if at all), and are removed in time and space from the line of code causing the bug, which means the cost of fixing those bugs is dramatically higher than when they’re caught earlier in the cycle.
- Problem 2: It’s challenging to maintain tests when the code under test is constantly evolving. This leads to a reluctance to create tests in the first place, leading to test holes and integration issues (Problem 1).
IntelliTest addresses these problems. It addresses code-complexity by taking a white-box approach to testing—that is, it analyzes your code as it executes and synthesizes precise test inputs to achieve high coverage. It reduces the effort in writing tests by automatically generating tests cases using these inputs and capturing any observed output from the code under test. It eliminates the challenge of maintaining a body of tests by automatically evolving the suite as the code under test evolves (when you re-run IntelliTest). It enables early detection of bugs by integrating into Visual Studio and being just a right click away, so that you can invoke it in your regular development workflow.
These problems show up as much in existing and legacy code as they do in new code. A combination of integration tests, and perhaps a few hand written unit tests (likely following the happy path), are inadequate in these contexts, and that is where IntelliTest can augment your existing testing practice:
- Evolving existing/legacy code: Use IntelliTest to generate a safety net of tests before commencing refactoring.
- Exploring existing/new code: Use IntelliTest to understand the input/output behaviour of the code against various data values. This can serve as a low friction introduction to unit testing.
There is much more to IntelliTest than can be covered in this single blog post, so here are additional resources where you can learn more:
- Visual Studio Toolbox: IntelliTest: a recent Channel 9 video.
- IntelliTest – hands on: a walkthrough of using IntelliTest on a real world example.
- Introducing Smart Unit Tests, Smart Unit Tests – a mental model, and Visual Studio 2015 – Build Better Software with Smart Unit Tests: high level overviews of what the feature is about, and how it works under the hood.
- IntelliTest – One test to rule them all: driving test case generation using parameterized unit tests embodying a partial specification of the code-under-test.
- Unit Test Generators Extensibility – hats off to our community: participation of the NUnit and xUnit.net community in enabling support for the corresponding test frameworks. These are available through open source plugins that you can install from here:
- Smart Unit Tests – Test to Code Binding, Test Case Management: how IntelliTest is able to evolve the generated tests on-demand.
- IntelliTest documentation: documentation on MSDN.
As ever, we welcome your feedback. You can leave comments on this or any of the other posts linked above, send us feedback through the Visual Studio Send a Smile feature, UserVoice, and Twitter, and pose your queries on the MSDN forum or over on Stack Overflow with the tag “IntelliTest”.
Pratap Lakshman, Senior Program Manager, Visual Studio Testing Tools
Pratap works on Testing tools in the areas of IntelliTest, Fakes, Unit Testing, and CodeCoverage.