Important release to correct data import problems

A new version 4.4 is now available.

This fixes the issue around missing data on import, as well as incorrect units for some fields like min or max elevation.

Bad Release Problem Solved!

As a bonus, I added support for Garmin Device Power while running, as least for the devices for which users sent me sample fit files.

Missing Data

Because of my own silly bug, ConnectStats was not trying hard enough to look for all activities on the initial download, and was stopping too early, resulting in missing older activities. This is now fixed in 4.4. But it’s hard for the app to know if it missed old activities or not. So if you think you are in that situation, the best is to force the app to reload all activities, by going to “settings”, selecting “profile” and “Force Reload Old Activities” as below. This will start a download that will try to reload everything, it will also help fix data previously imported with incorrect units.

 

Incorrect Units

Even though I have build a lot of tests over time, it is always difficult to make sure when a new API changes that the app is behaving properly. This is due to the fact that the API is undocumented and that I have limited scope for testing of new features, given that I do it on my own. I basically only test on my activities and the activities people have sent me over time in the past. Though we should not complain about the lack of documentation given as mentioned in the past, this API is not intended for external use, but just for garmin own website… So I am just grateful Garmin lets ConnectStats continue to use it, even if we just have to reverse engineer everything.

To illustrate the issue and give people a feel for the reason behind some of the bugs, the bad units were due to inconsistencies in the scale of the number depending on which data you were receiving: summary or detail for an activity.

In the summary the elapsed duration data comes as milliseconds:

While in the detail file it comes as seconds…Beside the elapsedDuration, maxElevation and minElevation everything else is consistent… Go figure… These are hard to anticipate and catch when you test on your own, so I want to thank all the users who have the patience to report details about such bugs and also the patience to wait for me to push fixes.

 

What the Alternative API mean for ConnectStats

Most users experienced new activities from Garmin not syncing up with ConnectStats since last week (March 23rd or so). As described in the last post, it can be fixed by using an alternative API.

It looks like it’s going to be a permanent change. It also has some minor consequences for users.

What is the alternative API?

It’s probably obvious to most of you, but the API is the mechanism that third party (ConnectStats) can use to access the data stored by a provider (Garmin or Strava).

Garmin a long time ago had a public API to access the activities recorded in Garmin Connect. Back in 2014, Garmin decided to change it’s approach to third party. While the old API allowed a third party to access directly the running or cycling data from Garmin servers, they decided to switch to a push approach. The key consequence is that a third party now has to build and maintain an infrastructure to store the data in the cloud after it was pushed by Garmin. This is great for a platform like Strava, it makes for a great user experience. But it’s impractical for small independent hobbyist developers like myself with ConnectStats like app.

While Garmin announced they would deprecate the old API in 2014, in practice until last week it continued to work more or less. It just had regular tiny changes, which beside the fact that the change came as a surprise outage to developer via lots of sudden bug reports, they were usually easy to keep up with.

The old API continued to work and was used by their own Garmin Connect website. since they upgraded their website to the new look shortly after the API announcement, they also slowly started to upgrade the site to a very different API. This is what the settings “Alternative API” refers to.

Practical Difference between the two API

ConnectStats downloads data in two stages. The first is to get a summary of all recent activities, which is intended to be faster. Then when you go and look at an activity in details, it will downloads all the individually recorded data point to create the graphs and the track on the map. The details can be a lot of data, so it’s an optimisation to only download the summary first.

The main difference between the two APIs, is that the new one (alternative) contains a lot less information in the summary. Using the new API, a lot of the extra information (for example: running dynamics, min and max, etc) will only be downloaded while getting the full detail of the activity. It’s not a major issue, as in practice when you select an activity, you need to wait a bit for the graphs and map data anyway.

Where it can impact the user experience is in some of the historical statistics pages. Some historical statistics on the extra information will be missing until each activity detail is downloaded. To mitigate the issue, ConnectStats tries to automatically download the last dozen or so activities that are missing detail.

Will the old API come back?

Only Garmin knows. But at this point I would anticipate it won’t. ConnectStats lived on borrowed time since 2014 when they announced they would deprecate the API.

Also the behaviour of the old API is not that it errors, but it contains a lot less data (almost none). In the past changes, it always was either some error, or some subtle change that broke ConnectStats use of the API. This time feels different.

What’s next?

I am releasing a new version of ConnectStats (4.3) that will by default use the new API. I keep the setting to switch back, in case things change in the short term and to keep optionality.

As in all the previous discussion on API change, I also continue to prepare for a day when Garmin may completely shutdown the API ConnectStats uses.

At such time the only option to continue using ConnectStats will be to switch to another service provider like Strava.

Using Strava data though has the drawback that it contains a lot less details on each activities (no running dynamics, etc).

Alternatively, now the the app is open source, if it comes to that we may link it to other service and hopefully the app will continue to live.

 

ConnectStats is now open source

I lately have not been very active in ConnectStats development. I have been quite busy, which didn’t help, but also I do not have a lot of new ideas, beside a few little features requests people sent me.

So I have decided to open source the code for ConnectStats (and my other apps). Maybe some people will want to help, or the code could help others who’d want to build similar apps. Hopefully more people will help think of new ideas or make the app better.

I definitely plan to continue maintaining the app. I released a new version (4.0) which will not have many new features but will be in sync with the refactoring and little cleanup I made before pushing the code to GitHub.

You can find the code here https://github.com/roznet/connectstats

The repository also contains the code for HealthStats, which I need to fix to work with the latest version of iOS, FitFileExplorer, an utility for macOS to open files, as well as TennisStats, an experiment to record and analyse tennis matches.

I also open sourced a few more of my apps:

My other iOS app MacroDial can also be found here https://github.com/roznet/macrodial.

And finally Simulator Data Finder, an utility to access iOS simulator files conveniently on macOS, is also available here https://github.com/roznet/iossimfinder.

 

2016 year review and double counted activities

It’s time again to look at the past year summary. How did I do?

2016 was a fine year, I ran 1387 km, it is 100km behind 2015. But one aspect bothered me, looking at the distance, there is a clear increase on the graph in December 2015, which put me behind in 2016. It worried me a bit. I couldn’t remember an especially good December last year. Did I have another bug in ConnectStats?

I went to check on Garmin connect and it reported the same distance as connectstats: 1478km for 2015…

Hum. Next step was to go and look at the activities in December last year. The search feature in ConnectStats made it easy, just enter december 2015  in the search box of the activity list. Sure enough, I found some duplicate activities. This started to be very worrying. ConnectStats imported the same activity twice? New bug? After some more investigation, it turned out the problem was in Garmin itself… Quite a few activities in 2015 appear twice on the website as well…

They have a different activity identification number, but they are clearly the same activity except for the altitude gain as you can see on the snapshot above.

Next steps was to add a new feature to ConnectStats to double check for such duplicate activities and ignore them… Well, my stats are now lower only 1310km but correct. And 2016 is ahead of 2015 by 50km! Yeah!

I finally decided to check what Strava reported. My account is linked to import activities automatically. I was curious: would strava have the total including the duplicate as well? It actually didn’t… The total is 5km off from my new corrected total, but I suspect it is simply due to the slightly different way Strava computes the total distance from the gps file.

The fix to search and eliminate duplicate will be included in a new release early next year.

Update on development and new API issues

There is shutterstock_120037903currently quite a few issues with ConnectStats following some changes in the Garmin API. I am working on it but it is quite more involved than I initially thought.

As of last release, the laps are not downloaded anymore, swimming activities do not download properly, nor do the multi sport activities. This is all due to an API access that disappeared from Garmin last week. I released an emergency fix that let people going for the main types of activities (running and biking) but I am aware there are still lot more issues. To fix the rest, I need to upgrade to a newer access point that is significantly different from the old one. I am also trying to future proof the app by removing use of all older API similar to the one that disappeared that ConnectStats is still using. I want in addition to improve the ability to seamlessly switch to Strava should Garmin decide to shut down everything for ConnectStats.

Also because the changes required currently are quite fundamental (completely new APIs), it requires quite a bit of testing and even then the testing is limited by the fact I only have limited set of activities to test (mine plus a few people sent me over the years). So it is possible that after the next release some issues will appear when others try it, but I will do my best to fix as soon as possible.

As I have explained in the past, I work on ConnectStats on the side of a demanding day job and a family, so development can be quite slow, given I can only work on it a few hours a week at best.

I am sorry about this. The most frustrating aspect for me is that I spent the precious few hours I have for development of ConnectStats on pure maintenance and not on new features. It has limited benefit for the users beside keeping the app running…

Currently I estimate it will take another week or two to complete the upgrade.

ConnectStats 2.5 rejected…

shutterstock_310884725So as per last week post, I had submitted to the apple review process a new version of connectstats with the 3D flyover feature…

Unfortunately, Apple rejected this version. The reason is that the code is using some of the HealthKit API. Actually it is appears to be using it because I am currently working on a version that will let you look at the steps and other information recorded via the Apple Health App or your Apple Watch. It is not yet enabled as I am still working on it and I am waiting until it’s ready for prime time. But the Apple review team rejected it on the basis that my code should not appear to potentially use health data if it’s not clear to the user why.

So I will wait until the next version is ready to release the 3D flyover, at this point would be too painful to patch an old version or  remove that code that I know I will need soon.

How to track the non reproducible bug in version 2.1

So, version 2.1 was crashing for quite a few users on startup. Obviously the app wasn’t crashing for me in any of my tests.

First line of defence when crashes happens is to ask the user what they were doing and try to replicate the same action to see if I can recreate the crash. But here they weren’t doing anything it wasn’t starting.

Second is to look for patterns in the crash. The first few users reporting the issue were all using an iPad. There are a few differences between the iPhone and the iPad, mainly that the iPad displays an activity detail at the same time as the activity list. This can result in some difference in logic and makes the startup on iPad marginally more prone to issues. This can lead to hard to reproduce bugs too as it is a function of the type of activity the user would load on startup (the last one downloaded)

Another help is the crash report from apple. In this case, they were very few and reporting crash in location that are not triggered at start up or inside apple libraries. So not helpful to pinpoint the problem.

Some users affected by the issue kindly sent me their activity data so I could try to see if the problem was linked to their specific activities data. It wasn’t, I could start the app with their data fine.

This leaves one other possibility, a thread concurrency issue. This can be really random, and a function of both the speed of the hardware and the exact activities you have. ConnectStats is quite multi-threaded to try to keep the app responsive as many calculations are performed in parallel. So it is possible that some of this parallel activity result in a collision or some instructions run too early (when required data is not ready).

I try to be very careful on collision conditions and to make the code robust to data not being ready. In version 2.1, in order to optimise the startup, I had done a clean up of all the notifications between events in the app to limit them to the minimum needed. So likely the problem was linked to that: issue affecting random users, not a logic issue (users data didn’t let me reproduce it), but the issue was systematic for some users: so this gives another hint. The one issue that can exhibit these symptoms is if you trigger a User Interface event not from the thread dedicated to the UI (main thread).

I did a review of all the events that happen on start up and found one that, in the clean up of 2.1, wasn’t not forcefully directed to that main thread…

So while I couldn’t reproduce, I feel strongly that this must be the issue:

  • linked to some change I made in version 2.1,
  • random: depends of when precisely event get triggered
  • systematic for a given user: for a set of hardware speed and size of data, always happens or not

I submitted a fix for that event issue to apple, let’s cross fingers until version 2.1.1 is approved by apple and hope the issue is indeed resolved. I requested an expedited review, but it’s not always granted.

Frustrations of a botched release

I try to release ConnectStats regularly. Once in a while, as today, I push an update with an issue. I have been receiving a lot of bug reports today from users who can’t start the app after last night release.

I know it’s quite frustrating for users, and it is for me as well. In this case so far it has been impossible for me to reproduce the issue.

So as I try to sort it out, I decided to write a bit about the testing and debugging of ConnectStats

Before the problem happens

Automatted Testing

First I wrote a bunch of tests that verify the very basic functionality of the app at the lowest level. For example it checks that all the unit conversions continue to be correct, that the statistics computation continue as expected, etc, etc

I have also built a testing sister app to ConnectStats that share most of the code, but run a bunch of test replicating the download process, or statistics on saved samples of activities.

Device Testing

While the automated testing is critical, most of the issues, like the one today I suspect are due to subtle interaction between different aspect of the app, when run with different set of activities or different devices.

Given I develop ConnectStats alone in my free time, this kind of testing is fairly limited for me. I mostly test with my phone and my activities, as well as a secondary device. For example I do not use an iPad, so issues on iPad are more likely to appear as I don’t have as much opportunity to try these.

Once a problem happens

Bug Reports

Typically, the first thing that happens is I’ll start receiving some bug report. The bug report contains workflow information and diagnostic collected as the app executes. It’s only as good as what I anticipated to record. But quite a few bug can be understood this way, especially when people email me and answer a few questions about what they were doing or how to reproduce the problem.

Apple Crash Reports

When as today, a large amount of people email me that the app crash on startup it’s much harder, as the log does not contain any useful information. I can also see some crash report collected by apple, but they take a while to appear. So right now I just need to wait and hope that the information in apple’s report will help me.

Bad Reviews

Of course the next thing that happens, is that ConnectStats will start receiving bad reviews. It’s basically a race, fix it before too many bad reviews accumulate…

 

Working with iOS Simulator files

Simulator Data Finder greatly improves my workflow in working and debugging apps in Xcode 6. You can download it here. My apps relies on many downloaded json and xml files, as well as keeping all its data as sqlite files. I use Simulator Data Finder to easily access these files, copy them from one simulator to another or bring into the simulator the files from a device.

Accessing the files

The first use is to access the files, either by the finder or terminal. The terminal is actually quite handy to work with sqlite files and sqlite3 directly, use grep or any kind of other shell tools for old school developers… Simulator Data finder has a clipboard button so when terminal’s tool are not enough, it’s easy to paste the directory and access it from emacs for example.

OneSimulator

Copying file between iOS Simulator versions

Another typical use is when you have a lot of files and you want to keep the same setup into a new iOS Simulator version. For example you have everything setup on the iPhone 6 simulator, but want to see how it looks on iPhone 5. Simulator Data Finder makes that a breeze. You can easily choose the two simulators from the table, open the two Documents folder with the finder and copy the files over.

This makes it very easy when a new version of iOS or Xcode comes out to move all the files you need. The older version will be easily accessible and clearly labeled.

If you work across two device simulators, the modified date also lets you see at a glance which of the two simulator has the latest modified files.

BetweenSimulator

Copying file from device to the simulator

The other task that Simulator Data Finder makes easy is to move around of files between the device container and the simulators.

When a developer device is connected, download the application container to your download folder using the standard name. Simulator Data Finder will then present all the container it finds in the Download folder that match the app you have currently selected. If you pop up the finder for the container and for the simulator it becomes very easy to copy files. The containers will also be conveniently sorted with the most recent on the top of the table.

DeviceAppData

Adapting and sharing code for iOS and OS X

So now that I have started writing OS X application for the mac and there are obviously a lot of similarity with iOS I decided to write some companion app on OS X for my new project TennisStats.

Most of the code is quite abstracted between the UI and the data logic behind, which helped a lot, but I embarked upon sharing my main plotting library inherited from ConnectStats. What will it take to share that library between iOS and OS X?

Compiler Macros

The first thing was to figure out the right macros to predicate the code between the two OS. TARGET_OS_IPHONE and TARGET_OS_MAC are define and the one to use here. Though one thing I quickly learned was the really I should use TARGET_OS_IPHONE as TARGET_OS_MAC is defined both for OS X and iOS.

The key library can then be imported with #if. Note that CoreGraphics seems to be the same across both.

UI and NS prefix

A lot of the classes prefixed by UI provided by apple on iOS have equivalent named with NS. Was it going to be as easy as using one for the other?

Brief look at the class reference showed a lot of similarity and iOS was derived from OS X, so why not… So I started to add the following typedefs

And the replace each instance of UIColor, UIView, etc with the equivalent version prefixed with RZ. Most of it compiled! NSBezierPath has the most differences, and I had to write an extension to provide the following functions

So now it compiles… Will it just work and is that easy?

Ooops

Oops… A new issue to worry about, the coordinates in OS X and iOS have their Y axis inverted… Luckily I had quite nicely abstracted the CGPoint calculation in a Geometry class… So a key target predicate

And a few more around the title and axis location, et voila!

Better

In Summary

It was much easier than I expected. And definitely makes the thought of building an OS X companion app very realistic…