How to track the non reproducible bug in version 2.1

So, version 2.1 was crashing for quite a few users on startup. Obviously the app wasn’t crashing for me in any of my tests.

First line of defence when crashes happens is to ask the user what they were doing and try to replicate the same action to see if I can recreate the crash. But here they weren’t doing anything it wasn’t starting.

Second is to look for patterns in the crash. The first few users reporting the issue were all using an iPad. There are a few differences between the iPhone and the iPad, mainly that the iPad displays an activity detail at the same time as the activity list. This can result in some difference in logic and makes the startup on iPad marginally more prone to issues. This can lead to hard to reproduce bugs too as it is a function of the type of activity the user would load on startup (the last one downloaded)

Another help is the crash report from apple. In this case, they were very few and reporting crash in location that are not triggered at start up or inside apple libraries. So not helpful to pinpoint the problem.

Some users affected by the issue kindly sent me their activity data so I could try to see if the problem was linked to their specific activities data. It wasn’t, I could start the app with their data fine.

This leaves one other possibility, a thread concurrency issue. This can be really random, and a function of both the speed of the hardware and the exact activities you have. ConnectStats is quite multi-threaded to try to keep the app responsive as many calculations are performed in parallel. So it is possible that some of this parallel activity result in a collision or some instructions run too early (when required data is not ready).

I try to be very careful on collision conditions and to make the code robust to data not being ready. In version 2.1, in order to optimise the startup, I had done a clean up of all the notifications between events in the app to limit them to the minimum needed. So likely the problem was linked to that: issue affecting random users, not a logic issue (users data didn’t let me reproduce it), but the issue was systematic for some users: so this gives another hint. The one issue that can exhibit these symptoms is if you trigger a User Interface event not from the thread dedicated to the UI (main thread).

I did a review of all the events that happen on start up and found one that, in the clean up of 2.1, wasn’t not forcefully directed to that main thread…

So while I couldn’t reproduce, I feel strongly that this must be the issue:

  • linked to some change I made in version 2.1,
  • random: depends of when precisely event get triggered
  • systematic for a given user: for a set of hardware speed and size of data, always happens or not

I submitted a fix for that event issue to apple, let’s cross fingers until version 2.1.1 is approved by apple and hope the issue is indeed resolved. I requested an expedited review, but it’s not always granted.

Frustrations of a botched release

I try to release ConnectStats regularly. Once in a while, as today, I push an update with an issue. I have been receiving a lot of bug reports today from users who can’t start the app after last night release.

I know it’s quite frustrating for users, and it is for me as well. In this case so far it has been impossible for me to reproduce the issue.

So as I try to sort it out, I decided to write a bit about the testing and debugging of ConnectStats

Before the problem happens

Automatted Testing

First I wrote a bunch of tests that verify the very basic functionality of the app at the lowest level. For example it checks that all the unit conversions continue to be correct, that the statistics computation continue as expected, etc, etc

I have also built a testing sister app to ConnectStats that share most of the code, but run a bunch of test replicating the download process, or statistics on saved samples of activities.

Device Testing

While the automated testing is critical, most of the issues, like the one today I suspect are due to subtle interaction between different aspect of the app, when run with different set of activities or different devices.

Given I develop ConnectStats alone in my free time, this kind of testing is fairly limited for me. I mostly test with my phone and my activities, as well as a secondary device. For example I do not use an iPad, so issues on iPad are more likely to appear as I don’t have as much opportunity to try these.

Once a problem happens

Bug Reports

Typically, the first thing that happens is I’ll start receiving some bug report. The bug report contains workflow information and diagnostic collected as the app executes. It’s only as good as what I anticipated to record. But quite a few bug can be understood this way, especially when people email me and answer a few questions about what they were doing or how to reproduce the problem.

Apple Crash Reports

When as today, a large amount of people email me that the app crash on startup it’s much harder, as the log does not contain any useful information. I can also see some crash report collected by apple, but they take a while to appear. So right now I just need to wait and hope that the information in apple’s report will help me.

Bad Reviews

Of course the next thing that happens, is that ConnectStats will start receiving bad reviews. It’s basically a race, fix it before too many bad reviews accumulate…

 

Year to date or Month to date statistics

Version 2.1 of ConnectStats introduces the ability to see year to date or month to date statistics. The yearly cumulative graph was already giving a good idea of how well you were doing this year versus the last, but now you can see the full stats you reached at the same point of last year.

Below for example you can see that in 2015, so far I ran 758km, while on the full year of 2014 I ran 1356km. If you press on the button All, you get to the YTD screen, that shows that in 2014 at the same point in the year as today (August), I had run 913km.

IMG_1664 IMG_1665

In the monthly view, the graph is slightly different, the full month bar is shown, but the blue shows how much for each month was reached at the same point in the month, here around august 15, or about half way through the month.

 

IMG_1666 IMG_1667

HealthStats, ConnectStats new sister app…

HealthAppLaunchJust released a new app HealthStats. It has the same core as ConnectStats, but it’s geared towards analysing data in the Health app of your iPhone.

It’s aimed at data collected either via the motion coprocessor (M7 or M8) of the latest iPhone, or collected via the Apple Watch.

If you have an Apple Watch and collects workouts, they should also appear in HealthStats.

My wife just got an Apple Watch two months ago, so I could only do limited testing, but on my initial tries the data collected by the watch is pretty limited. But with HealthStats, you will have ability to dig into that data much more than you can with the native Health App from iOS.

As with ConnectStats, it’s all about data, so the look may not be fantastic… But I intend to add as many ways as I can think of to look at the details of the data.

 

Best Rolling Graphs over Time

ConnectStats can now maintain best rolling graph over time. I find it quite interesting to see how one month compare to the other. You access the feature by going to the statistics screen and it’s the graph at the bottom of the summary page.

You can also find information about best rolling plots for an activity here and how to interpret them here

Initial Computation

Computing these graphs can be quite expensive. Every time you do a download, ConnectStats will process a few activities. It requires the full activity details, so only the activities for which you will have looked at the detail screen will be used.

You can force to recalculate older activities by going to the settings, profiles, and select Compute Best for more old Activities. This processes quite a few existing activities. Each time to press it, it will look for activities not already processed.

Interpreting the graphs

Heart Rate

This is the graph showing the best Heart Rate I achieved for any given period of time.
iOS_Simulator_Screen_Shot_8_Aug_2015_09_34_03The arrow 1 below shows the graph for July. You can see here that it was a month for which the best of the year was achieved except for the area pointed out by the arrow 2. The arrow 3 potints to the yellow graph showing that the last year (2014) was not as good as this year, I definitely pushed myself more this year, especially in july, when I completed a half marathon…

If you tap once on the graph it will rotate through the last few months, to show you how you performed then.

 

 

Pace or speed

This is the graph with the best speed or pace I achieve for a given distance. Not the heart rate is for time, not distance as I think it makes more sense.

iOS_Simulator_Screen_Shot_8_Aug_2015_09_34_15
If you pan to the left on the graph it will switch to the best pace or speed. Arrow 1 here shows you that the best speed for the year on most distances was achieved in july. Note that for pace a graph lower is better, but for speed higher is better. Again, if you tap on the graph, it will rotate through the last few months.

 

 

 

ConnectStats version 2.0

ConnectStats 2.0 brings quite a few exciting new features. The major change is that now ConnectStats will start maintaining some measures over time, while until now it really only acted as a cache for the data collected on either Strava or Garmin.

This opens the door to a lot of new features. So I upgraded the version and I celebrated by redesigning the icon…

BestHRThe first data I was really interested in looking at over time, was how the best rolling heart rate graph compared from month to month and year to year. Similar to the best pace. These graphs are not necessarily very common to see,  but I find they are a fascinating way to see how I am performing on a run or over the month. I will dedicate a post soon to these. Some snapshots here

BestPace

CalendarAllThe other feature is an improved calendar view. It now has more control to only show some activities, and a new visual circle to indicate how much each day fared compared to the maximum for the month. The best day is highlighted with by a darker color in the circle.

 

CalendarRun

App Video Preview for ConnectStats

I have build an app preview to showcase the screens in ConnectStats. Hopefully it gives a good overview of the variety of screens and stats available. This was captured with version 2.0 of ConnectStats so older versions won’t have necessary all the screens. I had to compress as many views as possible in the maximum 30 seconds allowed…