As part of the overhaul of everything Visual Studio and .Net Core, there is an overhauled testing framework. Microsoft has been informally calling the MSTest V2. The most exciting part of this is the fact that we’ve finally have Assert.ThrowsException and its async counterpart Assert.ThrowsExceptionAsync as part of the framework. Unfortunately, MS didn’t completely abandon ExpectedException, which was a major disappointment as its use is almost always an anti-pattern. From what I can tell, MS didn’t invent this anti-pattern, they just brought it in when they created MSTest using other frameworks as a guideline.

ExpectException’s Problem

Why do I (and many1 others) cringe when we see ExpectedException? It’s simple: it usually doesn’t fulfill the purpose of a well written test. Tests must provide assurance that a given part of the system is able to produce the correct output. This isn’t just when the code is first written, but, more importantly, also as additional features are added to the system. ExpectedException fails at this because it is unable to tell you WHERE the exception is thrown. Imagine the following:

[TestMethod]
[ExpectedException(typeof(ArgumentNullException))]
public void DoWork_Should_Throw_ArgumentNullException_When_Parameter_bar_Is_Null()
{
    // arrange
    var instanceUnderTest = new Foo();

    // act
    instanceUnderTest.DoWork(null);
    
    // assert
    // should not get this far
}

This is simplistic so that I can illustrate the issue. Let’s suppose later that we refactor that Foo() constructor to take a parameter that isn’t directly related to whatever is happening in DoWork. The developer refactoring the tests thinks “This particular method on Foo doesn’t actually use it, so I won’t deal with it, I’ll just pass null”. Later, Foo changes such that the constructor parameter is now required. That developer adds a check for null that throws ArgumentNullException, but for whatever reason, doesn’t see that this test exists. Now the test is not doing anything. Why? Look at the test now:

[TestMethod]
[ExpectedException(typeof(ArgumentNullException))]
public void DoWork_Should_Throw_ArgumentNullException_When_Parameter_bar_Is_Null()
{
    // arrange        
    var instanceUnderTest = new Foo(null); // ArgumentNullException here

    // act
    instanceUnderTest.DoWork(null);
    
    // assert
    // should not get this far
}

It no longer is telling me whether instanceUnderTest.DoWork(null) is producing the expected output because it never gets there. I have seen this sort of problem happen when ExpectedException is used, this isn’t a hypothetical case. Your code will change as your application grows, the main point of writing tests is to let us make those changes with confidence that we’ll have few regression defects from those changes.

What’s also bothersome is that it doesn’t communicate clearly the intention. I’m diligent about inluding the //arrange // act // assert comments in my tests, but not everyone does. If the comments were left out and the test were a little more complicated, then I may not be sure where that should be coming form.

Using ThrowsException

Microsoft finally got around to incorporating a static assertion for exceptions on the Assert class after literally years of people saying not to use the attribute and providing samples for how to wrap up the exception in an Assert type of construct. The xUnit framework introduced the assertion at latest by 2008. How to use the new assertion? The original test from above now becomes:

[TestMethod]
public void DoWork_Should_Throw_ArgumentNullException_When_Parameter_bar_Is_Null()
{
    // arrange        
    var instanceUnderTest = new Foo();

    // act
    Assert.ThrowsException<ArgumentNull>(() => instanceUnderTest.DoWork(null));
    
    // assert
}

What happens if we don’t throw the exception anywhere? You get the test failed with the following message:

Assert.ThrowsException failed. No exception thrown. ArgumentNullException exception was expected.

If the exception were thrown in //arrange, then I would have seen the test fail with this message instead:

Test method 
MyTestProject.MyTestClass.DoWork_Should_Throw_ArgumentNullException_When_Parameter_bar_Is_Null threw exception: 
System.ArgumentNullException: Value cannot be null.

As a result, I’m very confident that this test is now doing what I intend. It tells me that instanceUnderTest.DoWork(null) will result in an ArgumentNullException being thrown. Nothing will hide that fact from this test now.

Extra Assurance

I won’t go into detail here, but there is a way to cover your bases on tests becoming useless by having the code change from underneath them. In most cases, it results in code that never gets executed during the test run. To find code that never gets run in test, you run test coverage analysis. This is a separate tool that instruments the paths through your code and then measures how many of those paths got executed. You then need to analyze the output to see if the output meets expected values. There’s a fair bit to the topic, but tools that will do most or all of this for you include NCover, dotCover, TeamCity, TFS, and Visual Studio Enterprise2.

  1. Don’t take just my word. Many electrons have been spilled writing about how to properly test for exceptions and provided effectively the same solution. Other frameworks introduced the assertion style years ago. If you can’t use the new framework, then look at some of the samples of what others have done. Just a few links:

    ↩

  2. It infuriates me that Microsoft does not include this tooling in lower levels of VS. Writing tests and measuring coverage before committing is a measure of professionalism and even VS Professional does not provide the full set of tools. It downplays the utility of coverage analysis and it encourages a mindset that coded tests are optional.

    It is also a bit distressing to see how many tools that provide coverage output don’t provide a simple command to compare that output to a targets file. It’s trivial code, so it’s not that hard. I’ll put together a future post on that when time allows. ↩