I try to release ConnectStats regularly. Once in a while, as today, I push an update with an issue. I have been receiving a lot of bug reports today from users who can’t start the app after last night release.

I know it’s quite frustrating for users, and it is for me as well. In this case so far it has been impossible for me to reproduce the issue.

So as I try to sort it out, I decided to write a bit about the testing and debugging of ConnectStats

Before the problem happens

Automatted Testing

First I wrote a bunch of tests that verify the very basic functionality of the app at the lowest level. For example it checks that all the unit conversions continue to be correct, that the statistics computation continue as expected, etc, etc

I have also built a testing sister app to ConnectStats that share most of the code, but run a bunch of test replicating the download process, or statistics on saved samples of activities.

Device Testing

While the automated testing is critical, most of the issues, like the one today I suspect are due to subtle interaction between different aspect of the app, when run with different set of activities or different devices.

Given I develop ConnectStats alone in my free time, this kind of testing is fairly limited for me. I mostly test with my phone and my activities, as well as a secondary device. For example I do not use an iPad, so issues on iPad are more likely to appear as I don’t have as much opportunity to try these.

Once a problem happens

Bug Reports

Typically, the first thing that happens is I’ll start receiving some bug report. The bug report contains workflow information and diagnostic collected as the app executes. It’s only as good as what I anticipated to record. But quite a few bug can be understood this way, especially when people email me and answer a few questions about what they were doing or how to reproduce the problem.

Apple Crash Reports

When as today, a large amount of people email me that the app crash on startup it’s much harder, as the log does not contain any useful information. I can also see some crash report collected by apple, but they take a while to appear. So right now I just need to wait and hope that the information in apple’s report will help me.

Bad Reviews

Of course the next thing that happens, is that ConnectStats will start receiving bad reviews. It’s basically a race, fix it before too many bad reviews accumulate…