For all the time you’ve been working on your app, everything’s ran beautifully. But then you submit the app to the Windows Store for certification and it comes back with a failure on requirement 3.2, “Your app must not stop responding, end unexpectedly, or contain programming errors” (see Store certification requirements). You’re utterly confused by this, because so far as you’ve seen, the app has never, in fact, crashed or failed to respond. What can you do about it?
First of all, the person who tested the app has hopefully given you some indication of what they were doing when the crash or hang happened. But what if they didn’t? Or what if you have trouble reproducing the problem?
What this really boils down to is how you test your own apps prior to onboarding them to the Store. I point this out because the developers I’ve worked with during the developer previews of Windows 8 had widely varying testing methodologies–or a complete lack of them! The ones who had a culture of testing within their organizations generally didn’t run into this certification failure–it was much more common with partners who didn’t have a good sense of what testing meant for their app.
Let me make it clear as I did in Chapter 17 of my book: testing an app is orders of magnitude more important than testing a website! Testing a site is often done one page at a time, and changes to one page typically don’t affect the rest of the experience. Therefore, if there’s a bug on that page, its impact is isolated. Furthermore, once you identify and fix the problem, you can release an update (by uploading the new page) in a matter of minutes.
In contrast, a problem in an app has a much greater impact, because the app is a single package as opposed to multiple hosted pages. Issuing an update to the app (assuming it’s already in the Store), also takes at least a week. Therefore it behooves us to spend much more time really testing apps.
Basic testing–and this is all I really want to go into here–primarily means exercising every code path you can identify in your app: leave no feature untouched! To check this, set breakpoints in your code at the start of every feature-related function, and make a checklist. Run and exercise the app, and check off those functions that have been hit and for which you’re walked through the code. Take off the breakpoint and continue testing. At the end of the process, if you haven’t hit certain functions, figure out how to create the conditions where they will be.
Once you’ve identified all the code paths, you need to exercise them under a variety of real-world conditions that the app might encounter. These conditions include, but are not limited to:
- First time the app runs after installation, meaning an entirely clean installation and that there are no existing caches, no existing app data, no saved state, etc. For this purpose, try to have a separate device where you only install the .appx for your app and not run it in Visual Studio.
- First time an app is run on one device where the same user has installed it on another device, meaning that there may be roaming state present on first run.
- Subsequent runs with previously saved app data and new roaming state from another device.
- Running after being terminated by the user, both within 10 seconds of closing (during which time the app is just suspended), and after 10 seconds (when the app has been fully unloaded).
- Running after being terminated by the system, where the app needs to rehydrate itself from saved state. (Visual Studio helps to simulate this.)
- Variations in network connectivity: network, no network, airplane mode, a slow connection (requests go through but might be timing out on occasion), metered networks (which affect background transfers and should affect your own data transfers).
- Variations in screen sizes and pixel densities. (The Visual Studio simulator helps with this.) This can include plugging in external monitors or projectors.
- Exercising all the view states: snapped, filled, fullscreen-landscape, and fullscreen-portrait. These exercise your resize methods and media queries.
- Exercising the app with all input methods: mouse, touch, and keyboard, both independently and simultaneously.
- Exercising the app on a variety of different hardware with different performance characteristics, e.g. from a slow ARM machine to a fast multi-core gaming box. This can help uncover race conditions and latency issues that might occur on some devices but not on others (like the machine you’re working with). That is, some problems can arise when code executes too slowly, or when it runs too quickly. Remember for this purpose that Microsoft has set up Windows App Labs where you can go test with lots of real-world devices.
- Purposely abuse the app in ways that you don’t think it should be used. Try invoking different command when they should be (especially via the keyboard). Open all your charms (search, share, settings) and play with them on every page of the app.
- Navigate between all your pages slowly and quickly.
- Pan/scroll around within your app slowly and quickly, especially in list views and other layout that has lots of rendering involved.
- Simulate failures with any web services you employ. You might have great connectivity, but what if the service itself is down? What if it returns bad data? What if you’re hitting a server in another country that you didn’t realize? I speak of the latter because the manual testers for the Store are located in different places around the world, so while you might be hitting a server near your home, the tester might be hitting one in Eastern Europe.
- Ask yourself also if your code is robust to handle bad data coming back from a service.
There are probably more things you can try, for which I refer you to other resources on software testing. The point is, testing is a real art that needs your attention with apps. With some focus on testing–hopefully a strong focus–you should avoid certification failures for crashes and hangs. And even if you still get such a failure, doing testing along the lines I’ve described here will likely uncover the problem.