Today I was setting up a nightly build for a new project. After creating the Team Build Type and writing the script to trigger the build on the build machine, I ran the script to ensure it would work. As usual Team Build retrieved the sources, compiled the projects and began running the tests. Half an hour passed and the tests were still running.

I knew the tests couldn't take that long so I stopped the build and opened the solution in Visual Studio to run the tests locally. One of the tests managed to create just the right combination of values in a project class to violate what should otherwise be a class invariant. This particular invariant was verified by a call to Debug.Assert inside the production code.

It has been a very long time since I have written a Debug.Assert statement with an expression that failed the assertion. Today I was reminded that it does not simply throw some kind of AssertException when the expression evaluates to False. Instead a rather hefty dialog box is presented to the user with a full stack trace of the code that failed and a choice of options to proceed.

The problem is that tests running under the service of the build server don't have a visible window station to display the dialog box and there is nothing to automatically click the buttons so the build just hangs indefinitely.

For now I'm just replacing any Debug.Assert statements I find with exception throwing or additional unit tests as appropriate for each case. It may be possible to modify the app.config file to use a different listener to redirect the Debug.Assert output to something non-blocking but that is going to require some investigation.

Yesterday I managed to crash vstesthost.exe, the program that runs Visual Studio Unit Tests when you press Ctrl+Shift+X. A few tests in my suite would run and pass, then BAM, fatal error. I ran the tests again, this time in debug mode and the tests broke into the debugger with a StackOverflowException, due to a poorly implemented interface in a mock object.

According to some searching, some exceptions including StackOverflowException, just cannot be caught in .NET 2.0 and I'm not sure it would be smart to try. Unfortunately when it happens in testing, the whole suite fails. In my case, I was able to pave over my mistake in the code and move on with my life.

The vstesthost crashing problem also happens when an exception is through on a secondary thread though. Testing asynchronous UI or background processing code could also kill the process when any exceptions occur, and these might be expected exceptions for a particular tests.

I haven't needed to use it yet, but there is a workaround to avoid the whole test suite falling over. It involves editing the vstesthost.exe.config file to enable the legacy unhandled exception policy as used by the earlier CLR versions. You will need to be careful to synchronise your threads though to ensure any exceptions are assigned to the correct test in the results.

As Jim has recently explained, ReSharper 3.0 is now available in beta and for the first time supports VB.NET. As the majority of the code I work with each day is VB, I have, also for the first time, installed ReSharper. I am enjoying the experience so far and it works well across both VB and C# projects in the same solution.

However, as I've mentioned before, I use C# projects to test VB production code and ReSharper was failing to consider InternalsVisibleTo attributes and incorrectly marking code in my unit tests as erroneous. Thankfully Visual Studio still built and executed the code fine. Also very impressive was how quickly the guys at Jetbrains had solved the bug once I had emailed them. Try the nightly builds here for the latest version.

As a side note, if you're working with the InternalsVisibleTo attribute, you may find fellow Australian developer David Kean's generator very useful for extracting the rather long assembly PublicKey.

Now, if I can just convince ReSharper to give back my F12 key.

We are entering a new stage of development at work, we've released betas of three modules and are now beginning the initial iterations for two new modules. We are taking this opportunity to push unit testing into our standard development cycle, which is something I've wanted to do for a long time.

With friend and work colleague, Jim Burger, leading the charge to unit test and document our common libraries, some "best practices" that suit our project model are starting to reveal themselves.

For each project under test, we create a corresponding test project with the same name suffixed by "Tests". This keeps the projects paired together in the Solution Explorer and creates default namespaces for the projects that are just different enough to avoid clashes.

Within the test project we then create a folder for each class under test and within those folders, we create a unit test class for each function under test. We've found that each function usually requires at least three tests (null guard test, common case, edge case) and giving each function it's own file keeps our tests manageable.

Most importantly, while all our production code is in VB, we write our unit tests in C#. One of the main reasons is that the C# compiler respects the InternalsVisibleTo attribute and allows us to test code marked as Friend without resorting to reflection in our tests or putting test-only code in our production projects.

We also get the nice benefit that C# will automatically put classes in namespaces that match the folder structure in the project. This provides a simple mechanism to categorise our tests for running them and viewing the results. C# also tends to produce slightly more succinct code which just feels better as the number of tests grows.

Although both VB and C# have their pros and cons, the majority of the .NET world is C# and so are the job opportunities. As a VB programmer, by using C# for at least some of your work, you make sure you are learning more aspects of .NET and when you get involved in the .NET community you're more likely to appreciate what they're talking about.

This post brought to you by Windows Live Writer Beta 2.

I have been wanting to make use of Unit Testing in my personal projects and my projects at work. So far I've read a lot about unit testing concepts and implementations and even written a few tests to get a feel for it. The majority of the material that exists about unit testing in .NET refers to the NUnit testing framework. The is presumably because it was one of the first .NET unit testing frameworks and based on the well established JUnit. However, now that Visual Studio 2005 includes it's own unit testing framework, why wouldn't you want to use that?

Jean-Paul Boodhoo once blogged¹ that when a new developer joins your team they should be able to Get Latest from source control, build, and be ready to work. I agree and using tools included in Visual Studio means one less dependency for a new developer to install on their workstation. However, there is the possibility that Visual Studio's Unit Testing, as a version 1.0 product, might be lacking important functionality to be found in established testing tools. A quick Google for "visual studio unit testing vs nunit" leads to a few notable issues.

Jason Anderson explains that only Visual Studio Team Suite includes the Unit Testing tools and that Visual Studio Professional misses out. We all use Team Suite at work so this isn't a problem but I can understand that unit testing should be available developers in all team sizes. Thankfully, Naysawn Naderi says that we can expect Unit Testing in Visual Studio Pro when Orcas is released.

Roy Osherove also has two posts about choosing between Visual Studio Unit Testing and the alternatives. Roy's comments makes VS Unit Testing seem completely useless and claims some parts are broken and even says "how this one went by into the release is a mystery to me". However, I recently had the pleasure of working with VS Unit Testing briefly when contributing to the BlogML project and some of Roy's claims just felt wrong. Surely it can't be that bad. Considering we will want to decide on a testing framework for projects at work soon, I decided to verify these problems.

Roy's first complaint is that there is a bug with the ExpectedException Attribute checking the type of the Exception only and not checking the Message. This is in fact a misinterpretation of the VS Unit Testing ExpectedException Attribute and assuming it was designed the same as NUnit's counterpart. NUnit takes an Exception type and message and verifies that the right Exception is thrown and that is has the matching message. VS treats the message parameter that same as an Assert statement and uses it as the display message to the user when the test fails, admittedly the MSDN documentation is very poorly worded. However, achieving the same behaviour as NUnit is often desired and there is a quick solution:

[TestMethod()]
[ExpectedException(typeof(InvalidOperationException))]
public void ExceptionWithMessage()
{
    try
    {
        MethodToTest();
    }
    catch (InvalidOperationException ex)
    {
        Assert.AreEqual("Expected message.", ex.Message);
        throw;
    }
}

Roy is also disappointed that ExpectedException doesn't work with the base Exception type. Sure it's something the unit testing framework doesn't support but you shouldn't be throwing Exception objects and you shouldn't be catching them either. You can also use the Test Results window to group by Result (ie Passed/Failed), Class Name, and Full Class Name (ie Namespace too), this makes locating the tests your interested in much easier and also negates Roy's corresponding observations with regard to Test Driven Development.

VS Unit Testing isn't perfect though and that's to be expected for a first release. As Roy points out, you can't easily get to the failing production code from a failed test result, it doesn't have quite as many included Asserts as NUnit does and you do need to have Visual Studio installed to use the MSTest command line test runner. For the time being, our requirements don't push VS beyond it's capabilities, Orcas should improve on the weaker points when it's released later this year (?) and it is really easy to convert between NUnit and VS Unit Testing if we need to.

So, if you're looking to choose a unit testing framework, don't be put off by some of the articles flying around about VS. Give NUnit and VS Unit Testing a go and maybe MbUnit and some of the others too. Ultimately you'll find something that you and the rest of your team are comfortable with and fits your process. I can't say that any of them are perfect, nor are any flawed beyond useful. And, if you've managed to read this entire post, thanks for putting up with my ramblings ;).