VS2010, TFS 2008 and Unit Tests

For the last couple of days I have been battling with TFS2008 and getting our build server to run the unit tests defined in a Visual Studio 2010 solution.

Previously we installed .Net 4.0 and updated the MsBuild path in the tfsbuild.proj file so our VS2010 solutions were building without a problem on the TFS2008 based build server. But the minute I tried to get the build server to run the unit tests in my solution, the build would partially fail and if looked in the BuildLog.txt file, this is the error I got:

Task "TestToolsTask"
MSBUILD : warning : Visual Studio Team System for Software Testers or Visual Studio Team System for Software Developers is required to run tests as part of a Team Build.  [Path\TFSBuild.proj]
  The previous error was converted to a warning because the task was called with ContinueOnError=true.
  Build continuing because "ContinueOnError" on the task "TestToolsTask" is set to "true".

Googling the error and you find out that you need Visual Studio 2010 Premium or Ultimate edition to be able to run unit tests as part of your build. So I went ahead and installed VS2010 ultimate and I still got the same damn error shown above.

It turns out that in addition to VS2010 (premium or ultimate) you also need VS2008 Development edition on the build machine. (In addition is the operative word here – you need both!). It turns out that the TestToolsTask is looking for VS2008 dev edition before running even a VS2010 based solution (and I am not sure if this for some required dlls or for licensing purposes – either ways – it sucks) (Obviously the reason I had to battle with this issue was that the build machine was never previously setup to run MsTests – even with Vs2008, else someone would have figured out that they needed VS2008 dev edition on that box).

Once I got VS2008 dev edition on that machine I was able to get the build a little further. HOORAY!

Some other things to note:

Once you get both VS2008 dev and VS2010 premium on that machine, you should delete the workspace used by the TFS service on the build machine (the easiest way to do this is to use the TFS SideKicks tool and search for all the workspaces on that machine and then delete the one that is being used for your build). This allows TFS to rebuild the workspace and all the files, but this time with MsBuild 4.0.

Now that I got one problem fixed, TFS decides to throw another one at me and the error that I get is as follows:

Task "TestToolsTask"
  Command:
  d:\Program Files\Microsoft Visual Studio 10.0\Common7\IDE\MSTest.exe /nologo /searchpathroot:"path\Binaries" /resultsfileroot:"path\TestResults" /testcontainer:"path\Binaries\\\\Tests.dll" /publish:"TFSURL" /publishbuild:"vstfs:///Build/Build/26026" /teamproject:"Project" 
  The "TestToolsTask" task is using "MSTest.exe" from "d:\Program Files\Microsoft Visual Studio 10.0\Common7\IDE\MSTest.exe".
  No platform specified for publish operation.
  For switch syntax, type "MSTest /help"
MSBUILD : warning MSB6006: "MSTest.exe" exited with code 1. [path\TFSBuild.proj]
  The previous error was converted to a warning because the task was called with ContinueOnError=true.
  Build continuing because "ContinueOnError" on the task "TestToolsTask" is set to "true".
Done executing task "TestToolsTask" -- FAILED.

[Update 3/14/2010]

The problem with MsTest is that it always use the generic error code of 1 when it fails. So it makes it extremely hard to figure out what is going on when it fails. I looked through a long list of online articles that had me doing a whole bunch of things and at the end the following is what fixed it for me:

1. Need to Microsoft.TeamFoundation.Build.targets file to use the VS2010 dll instead of the VS2008 dll for the testToolsTask
   In the folder: “C:\Program Files\MSBuild\Microsoft\VisualStudio\TeamBuild\” on the build machine you will find the file “Microsoft.TeamFoundation.Build.targets”. Edit the following lines:
   Comment out the original line for the TesToolsTask:
   
<UsingTask TaskName="TestToolsTask"
             AssemblyFile="$(MSTestRefPath)\Microsoft.VisualStudio.QualityTools.MSBuildTasks.dll"
             Condition=" '$(ProjectFileVersion)' == '2' "/>

Insert the following line right after it

<UsingTask TaskName="Microsoft.TeamFoundation.Build.Tasks.TestToolsTask"
             AssemblyFile="c:\Program Files\Microsoft Visual Studio 10.0\Common7\IDE\PrivateAssemblies\Microsoft.TeamFoundation.Build.ProcessComponents.dll"
             Condition=" '$(ProjectFileVersion)' == '2' "/>

Make sure that the path to “Microsoft.TeamFoundation.Build.ProcessComponents.dll” is correct

2. Next make sure that you have specified the platform in your TfsBuild.proj file:
   
   This is how it looks like in my TfsBuild.proj:
   
   <ItemGroup>
       <SolutionToBuild Include="$(SolutionRoot)/mySolution.sln" >
         <Targets></Targets>
         <Properties>Configuration=Release;Platform=Mixed Platforms</Properties>
       </SolutionToBuild>
     </ItemGroup>
     <ItemGroup>
           <ConfigurationToBuild Include="Release|Any CPU">
               <FlavorToBuild>Release</FlavorToBuild>
               <PlatformToBuild>Any CPU</PlatformToBuild>
           </ConfigurationToBuild>
     </ItemGroup>
   
   If the PlatformToBuild is not specified the “No platform specified for publish operation” error can occur.

With the above 2 fixes in place, the final thing that I needed to specify were which tests should be executed. This is done using one of the following 2 methods:

Specify the VSMDI file using the metadataFile node as follows:

<MetaDataFile Include="$(SolutionRoot)/mySolution.vsmdi">
            <TestList>TestListToExecute</TestList>
</MetaDataFile>

Or use the following node to define a wildcard to select the files that contain the tests:

<TestContainer Include="$(OutDir)\Tests.dll"/>

With that your TeamBuild should be executing all your unit tests.

Code-Coverage:

The only thing that I havent been able to get going is Code-Coverage on my solution. Here is what I have tried:

I have setup the VSMDI to use a testConfiguration for the unit tests. As part of the test-configuration, Code-Coverage is enabled and I have selected the dlls on which code-coverage should be run. With the above settings, some files (like data.coverage) get created in the drop folder, but TFS reports that code-coverage was not run.

After spending quite some time on this problem, I am just about ready to give up (especially because this post seems to state that code-coverage is not possible using TFS 2008 and VS2010 and .Net 4 built modules - http://social.msdn.microsoft.com/Forums/en-US/tfsbuild/thread/39aaaf43-0e3a-41c7-8b72-c1cabc9438e1/).