The Windows App Certification Kit or WACK, which is part of the Store Certification process, is something you should be running every day during app development to see if it comes up with any problems.
Typically it warns about startup time and suspend time, almost invariably, but it’s good to really watch these conditions because of the automatic deadlines built into Windows. To review, if an app takes more than 15 seconds to create its first window (and thus tear down the default splash screen), and the user switches away, Windows will terminate that app. If the user doesn’t switch away, mind you, the app will continue to launch. With suspend, if the app doesn’t return from its suspending event handler in five seconds, it will also be terminated.
Under optimal conditions, you typically don’t need to worry about these things, but if you’ve learned anything about testing you’ll know that apps seldom run under optimal conditions!
On such non-optimal condition is when there is a lot of disk and/or network activity going on at the same time, such that your app will experience slower than normal response from disk/network APIs. For example, the app might be trying to load file data from its package during startup, or might be trying to make various XmlHttpRequests on suspend. When there’s a bunch of activity going on, an operation that takes half a second normally might take much longer and perhaps cause the app to exceed the time limits.
This means two things. First, be sure to test app launch and suspend when other stuff is going on. Try, for instance, to launch your app as soon as you can after booting the system, when there’s still other disk activity. Or do some intensive operation on the file system (like run the Disk Cleanup tool) and see what happens when you launch or suspend. Try the same things after you’ve started a few large downloads and a few uploads, so your app has to compete with those transfers.
Second, implement your app to be resilient to high latency. If in the above tests you find that startup can take a long time, be sure to implement an extended splash screen to give yourself as much time as you need. For suspend, you can first take the approach to write/transfer as much state data to disk before suspend occurs (i.e. when the data changes), thereby minimizing the work that’s necessary during the event itself. Also avoid making the assumption that you can do the full load of possible work when suspending. If there’s doubt, create a strategy wherein you save the most critical data first, then attempt additional stages of secondary work if there’s still time. The suspending event tells you the deadline for completing your work, so you can keep checking elapsed time after each operation to see if you want to start another.
In any case, running your app under high-latency conditions will also show you where you might increase performance and/or show progress indicators where you haven’t before, because tests under optimal conditions never showed the need for such.