Running Windows Store Tests in TeamCity
Based on Visual Studio user experience, one would think running unit tests for Windows Store apps is not all that different from running standard .NET framework unit tests using MSTest testing framework. When you try running them on a build server, it turns out there are a lot of differences.
I'll describe them with a series of error messages and the corresponding changes that were required to resolve them. My starting point was a build step based on Visual Studio Tests runner using MSTest engine type, pointing to the assembly containing test (identical configuration can be used for running .NET framework unit tests).
No tests to execute.
MSTest.exe doesn't recognize Windows Store unit tests; vstest.console.exe test runner must be used instead. In TeamCity this means VSTest must be chosen as Test engine type.
Warning: Unit tests for Windows Store and Windows Phone apps cannot be run outside appcontainer. Create an app package and run tests in appcontainer mode. [http://go.microsoft.com/fwlink/?LinkId=254169]
Since Windows Store tests can't be run directly from the assembly, the corresponding .appx package must be specified as the Test file name instead of the assembly. By default, the build drops them in AppPackages folder, not in bin folder. Relative path from the solution folder will be similar to:
WinStore.Tests\AppPackages\WinStore.Tests_1.0.0.0_x86_Debug_Test\WinStore.Tests_1.0.0.0_x86_Debug.appx
Error: Could not start test run for unit tests for Windows Store app: No valid developer license found for running unit tests for Windows Store apps. Please install/renew your developer license..
In Windows 8/8.1 a developer license is required to run Windows Store unit tests. Visual Studio will automatically show the dialog to enter your Microsoft account credentials when you attempt to run a Windows Store app or test for the first time. If you don't want to do that on the build server, you can use a PowerShell cmdlet instead (admin privileges are required):
Show-WindowsDeveloperLicenseRegistration
In Windows 10 this step is not required any more. You will still need to enable your device for development, though.
Error: Could not start test run for unit tests for Windows Store app: Unit tests for Windows Store apps cannot be run from a service or non interactive process. Please run unit tests from an interactive process..
Windows Store tests require desktop access, therefore the build agent must not be set to run as a Windows service. If you already have it configured like that, first stop the service and uninstall it by running the following commands as administrator from the build agent home directory (C:\BuildAgent by default):
./bin/service.stop.bat
./bin/service.uninstall.bat
Now create a new batch file to start the build agent as an executable:
C:\BuildAgent\bin\agent.bat start
Put a shortcut to this file into %AppData%\Microsoft\Windows\Start Menu\Programs\Startup to make it run automatically at log on. Log off and log on to check that it works. Enable automatic logon on the build agent machine to make sure the agent starts automatically after a reboot.
error 0x800B0109: The root certificate of the signature in the app package or bundle must be trusted..
App packages which are not associated with the store, are by default signed with a self signed certificate, generated on the developer's machine from inside Visual Studio. To make this certificate trusted on the build server, you need to import it into Local Machine > Trusted People store. Just double click the certificate (.pfx file) which you can find in the test project folder (by default inside C:\BuildAgent\work), select Local Machine as Store Location and choose Trusted People as Certificate Store:
After performing all these steps, the build agent finally successfully ran the Windows Store tests. It's quite a bit more complicated to get them running than the standard .NET unit tests.
