I had another interesting struggle today playing around with a TFS build server and unit tests. Ostensibly this was to work through TFS and ALM matters with a Xamarin app, but this particular issue isn't specific to Xamarin.
I'd set up a team project on the TFS server, created and checked in the app, and then added a unit test project to the solution. On the local machine the tests ran fine.
In the build definition for the TFS server, I had the default setting to run tests in any assembly with "test" in the name, and the Unit Test project I added to the solution fit that criteria. I also set the build definition for Continuous Integration to build (and therefore test) on any checkin. I then checked in the unit test project along with a bug in the app code to fail the test, and a build was queued automatically as expected.
However, no tests were run. Hmmm. I checked the log of the build and didn't see my test assembly anywhere. I played around with settings in the build definitions and searched for answers for a while, to no avail.
So what was the problem? Turned out to be another really simple thing: I had neglected to check in the .sln file along with everything else. I'm not sure why that was, but that's what happened. As a result, the TFS build server had the test assembly source code, but because that project wasn't in the solution in its copy of the code, it didn't build it. Therefore it didn't find any test code to run, and thus the build succeeded because it wasn't blocked by a failed unit test.
Once I checked in the solution file, the TFS build included the test assembly and ran those tests as expected upon a checkin, failing the build with my intentional bug in the app code. Correcting that code and checking in again queued another build, which worked.
As an addendum, here are my raw notes on reproducing the problem I'd encountered, setting up the Xamarin project to work with a TFS build server, continuous integration with unit tests. I didn't bother to edit these; I'm only including them as a quick-and-dirty tutorial on a minimal setup to evaluate TFS and CI functionality, because it's not easy to find something short and simple along these lines.
I have a TFS server installed. Made sure that the build agent account can get to necessary resources (important for Xamarin and Android SDK install, but this isn't about that–see previous post).
- Running Visual Studio, Connect to team server and do Create Team Project…
- Map the team project to a local workspace (the folder that will sync)
- In Team Explorer – Home > Solutions > Workspace <my machine>, select New… and in the new project dialog, create a blank Xamarin app, MathTestB.
- Checked the code in and ran a local build to verify it works. Make sure the solution file is also checked in.
- Created a build definition using defaults. Team Explorer > Builds > New Build Definition, trigger build on CI, only change is sending builds to another file share.
- Queued a new build to test. Did build definition (right click) > View Controller Queue… to see results.
- Got the Android SDK Directory could not be found, set /p:AndroidSdkDirectory=c:android-sdk in the build definition under Process > Build > Advances > MSBuildArguments. Also set MSBuild platform to x86. Saved definition and build again.
- Removed Windows Phone 8 project as Silverlight 4 SDK is not on the server. Checked in .sln file which queued a rebuild on the server. This built successfully.
- In MathTestB (PCL) > App.cs, add a method called AddOne to return a+1. This is what we'll unit test.
- Added a Unit Test project to the solution. Right click solution > Other Languages > Visual C# > Test > Unit Test Project call it AdditionTests. Add reference to MathTestB (required).
In UnitTest1.cs, add
- using MathTestB.
- Change TestMethod1 to TestAddOne.
- Add code to test 124+1 = 125
- Rebuilt solution. Then did run tests and had success.
- Changed MathTestB.App.AddOne to be +2. Ran tests again, they failed.
OK, so we're cool now. Let me now check in AdditionTest as and the change to app.cs. We're cool, right? This queues some builds and should run the test, which should fail on the server.
- However, it works on the server. Examining the log, we find that no tests were run. Why is that? It's not because the build definition is wrong, it's because I neglected to check in the .sln file that includes AdditionTests. Thus the server isn't building the test assembly, and therefore the build definition isn't finding it.
- Checked in the solution file now. New build queued. And it fails as expected.
- Restore proper code to app.cs. Check in. Build succeeds as expected.