Getting the weather for an activity in ConnectStats

As of version 5.1 you can obtain weather information for Connectstats activities if you use the new service I implemented using Garmin Health API and DarkSky. But in order to access it there are a few things to know and do.

The weather will only be added to new activities, once you have started using the new service, which is not currently the default in the app.

To enable the new service, you should go to your config page in Service and accounts and change the Source for the Garmin information to be All as below.

Note that if you do that it should preserve the history you have already downloaded from the Garmin Connect website and only start downloading new activities from the new ConnectStats service. Note that in case of issue you can always revert to only using the Garmin Website as a source.

New activities after you have completed the first login will start having the weather information added.

The reason I didn’t back populate the weather on older activities is both technical, as trying to back populate the older activities was a bit tricky to handle and cost, as I was worried it would make too many request to the weather API and start costing too much.

New services and Withings working again

Finally I got withings and the new service for ConnectStats to work again with version 5.1! At the time of this writing version 5.1 is pending apple approval for release.

Withings

The Withings authentication process has been broken for a while…

It took me trying a whole new approach to do the authentification process for the withings API, a few exchanges with withings support and learning way more than I ever wanted to know about the OAuth 2.0 protocol to get it all in the right place again.

The key change I believe is that withings started to have the OAuth 2.0 authentification token to expire and the way I was using the authentification library was slightly incorrect. But before I tried to use that library correctly, I tried to upgrade to a more modern approach than the one I was using that had been declared obsolete. These though required Withings to make adjustment on their side (non http scheme), which they kindly did. I never got it to work. Ultimately, after studying the deprecated library, using the ever more useful proxy app Charles I finally figured out how to get things working again! In the process I also streamlined the way the setup screen works, fix the ability to logout and back in, and synchronised the interface into Strava to use the same new code.

New ConnectStats Service

To use the new Garmin Health API, I had to implement a server that would receive, process and save all the activities of ConnectStats users when their device uploaded them to Garmin.

I knew that processing these requests as they came in realtime was a bad shortcut in the first implementation of the service for the new Garmin API. I had not expected how bad the deadlock and timeout would be when used in real life as often many files would be sent to my server at once. I had adopted the “make it work, optimise later” approach, but here the optimise step was immediately needed…

While I had postponed implementing it, I knew what the right approach was: I had to add all the incoming requests from the service to a queue and process sequentially a few at a time.

To host my server, I am using an entry level service from GoDaddy, which is a great relatively cheap hosting but with fairly limited optionality in configuration. Which meant it wasn’t an option to bring some pre-built queuing system. I had to build it myself with the tools available…

It’s of course not advisable to implement such a common system yourself, but, well, I do all this to learn anyway… It was definitely interesting to figure out how to make sure it behaved in a robust manner in all the edge cases I could think of. As with Connectstats itself, I open sourced the code of the server. Hopefully it can be useful to someone interested in having a lightweight server to work quickly with the Garmin Health API or can help people struggling to implement their own.

I am releasing this now in version 5.1. In that version only new users will use the new service by default, but existing user will have to switch manually in the configuration page. Crossing finger that this time it will work better and scale…

Bonus with the new service

Beside resiliency to the next API change from Garmin on their web site, the new service allowed me to bring in weather information from DarkSky, so that’s an added bonus for using the new service. Because I now fully control the service that serves activities to Connectstats, it maybe possible in the future to add more such new features.

Thought on servers, hosting, services etc

This experience made me appreciate better the differences in services. I use a service from GoDaddy that is very simple, turn-key and fairly cheap. While it forces you to be a bit more nibble in the implementation, it was a bit interesting to compare it to using more scalable and more configurable services like DigitalOcean or AWS. I briefly considered, but it’s clear that it would require a non-significant learning process and higher costs. So I am sticking with GoDaddy for now, and hope the system will handle an increase of users without requiring me to upgrade to a fancier hosting service.

The weather service from DarkSKy is very nicely done and reasonably priced and free for the first 1000 calls per day. I anticipate based on the current number of users that the cost will remain small.

New Service not working well

A few days after releasing my new version that uses the new Garmin Health API and my own server, it’s not really a success story… My server is overwhelmed by the synchronisation and backfill requests…

I had initially planned to turn on the new service only for new users to avoid stressing the server too much, but still it does. So I have issued a new version that only doesn’t enable the new service by default.

For those who have issues with the service, just switch the Source in the settings for Garmin to be Garmin Website

I am going to do two things, try to optimise the way the server processes backfill, and add a way for ConnectStats to download the history from the website, and only process new activities from the new service. I hope it’s just that I did something silly in my implementation (very likely) and not that I will need to upgrade to a bigger infrastructure…

I feel bad for the new users of the app this weekend as this is not a great first experience of the app, but hopefully, soon it will be in a better place..

Dark Mode and New Service

I just pushed version 5.0 of ConnectStats to Apple for review. It will likely take a few days for the version to be approved. Hopefully it will be done in time for the iOS 13 release.

This version has two major changes. First the app has been updated with a new look for iOS 13 and support for dark mode. Second it can now use the new service I have been testing with a few users. Before you switch, please read the details below to make sure it’s for you.

Dark Mode

Version 5.0 has a major revamp of the way the interface is driven internally to enable much more customisation of the look of the app. This was driven first to be able to support the new dark mode from iOS13, but it may later help support more customisations.

In the settings you can now choose a theme. You’ll find a few so far:

  • Original: it just make the app look like it has been in the last 8 years…
  • Dark Mode: it is a mode that changes the look to a dark mode on iOS 12
  • Dynamic: it is a mode that changes between the above two on iOS 13 depending on the general mode of the app (light or dark)
  • Native iOS 13: This is a mode only available on iOS 13 that uses the standard colors of iOS 13. The main difference being in dark mode and some minor changes in the light mode. This is probably my preferred one so far, and when the app runs for the first time on iOS 13 it will change to this mode (after iOS update)

You can see a few previews of the iOS 13 dark mode here. If you find any issues with it, send a bug report or comment here.

New Service

I have been implementing a new service in the last few months using the new Health API from Garmin. The main benefit is that it is supported by garmin, as opposed to the current way ConnectStats access your activities by using the unsupported API that the garmin connect website uses. It will present a few benefit now or later:

  • we should not have unexpected failure of the app like this one or this one. This service is fully supported by Garmin
  • It give more control on the data exchange and provide the ability to implement new features like adding weather information (will come soon after this release) or notification (maybe, if it’s deemed useful)

The flip side is that currently it has a few shortcomings, so I do not recommend that everyone switches to it just yet, unless you understand the below issues:

  • The initial synchronisation of the account can be slow : about 2 min for every 2 months of activities you want to synchronise. There is nothing I can do about that, it is a limitation of the API.
  • Details of older activities that were recorded on an older device no longer associated with your garmin account will not be available. So only a summary but now maps or details graph. Garmin told me they are working on fixing this issue.
  • Extra information you have on the Garmin Connect Website, for example activity names, or edited activity type after the initial upload will not be seen by the new service.
  • It’s a new service I implemented myself, so it is possible, instead of unexpected failure because of a garmin change, it will could have some issues initially because of my own mistakes… But I’ll do my best to fix.

In light of the above limitations, I currently set up the app to only use the new service for brand new users. Existing users will continue on the old API and would have to switch manually to the new service in the settings and as in the below screen, switch the source to ConnectStats.

The new service is implemented such that will will understand the equivalence with activities downloaded from garmin connect. So you can switch back and forth between the two services and it should work and not be confused. You can also create a new profile to test it out.

I intend to implement soon a new version that will allow to more seamlessly get the data from the two services, which will be a bit of the best of both world…

Note that I encourage you to read the privacy policy to make sure you understand what happens to your data. The key point being, the new service does not keep any information whatsoever about who you are. At no point do I save your name, email, etc (the service actually never asks for it and never obtains it), but the service will save your fit files.

You can read a bit more about the new service here.

New Garmin Service Integration

Garmin recently announced a new Health API for third party developer. Given the issues I have been having extracting activities from the web service with regular changes to the API breaking the app, I jumped on the opportunity to try to have a more robust integration.

The new service does not let you access the activities from your account directly as ConnectStats has been doing via the web site API, instead it is a mechanism to provide the activities into your own service. Which meant that to use it, I would have to host on a server my own service that receives the data, stores it and make it available to the app.

The service is now integrated, and I will have to start rolling it out. I will initially roll it out via Test Flight and ask volunteer to do some testing to make sure the service works. It is quite scary as I could only test it with my set of activities and my Garmin account.

Principles of the integration

It is possible either to start a new profile where all the activities are downloaded from this new service or to only start downloading from this service new activities when you have older activities already downloaded from Garmin. It should handle past duplicate properly.

The data will be synchronised into a database online, but the app will not ask you any identification via email or others. All the data will be referenced via hashed keys linked to Garmin account, and I intend to prevent any access to your data if you didn’t successfully login to the Garmin service. Any way around this is a bug, and I will fix them as they are reported. The web service is also open source so for transparency people can verify I don’t do anything bad or point the bugs out.

There are a couple of short comings from the new service to understand.

History requires slow first synchronisation

When you first enable the service, it will contain no history of your activities. Upon first login, it will trigger a synchronisation of your previous activities with Garmin which is time consuming. It happens at a rate of 90 days per two minutes. So synchronising one year of activities it will takes about 8 minutes.

Also because it is impossible for the app to know how many years to ask in the past, when you setup the account, you’ll need to specify from which year you want to synchronise your activities, which will default from the previous year. If you select many years you’ll need to wait a about 8 minutes per year until the synchronisation is finished.

This is a rate limitation imposed intentionally by the Garmin service, so there is nothing I can do about it.

Only details for activities from the devices currently associated with your account

Currently the service will only provide details (from fit files) for activities that were recorded with a device that is currently associated with your Garmin account. So if you changed devices and unregistered the old device, you will no longer be able to download the detailed fit file and only have access to a summary of the activity.

This is an unintentional issue with the Garmin service, and Garmin support told me they will work on removing the limitation. I’ll update the post when the limitation is removed.

Testing volunteers

I will need some volunteer to test the service. I can’t release widely after just testing on my account. It is likely that there will be some issues initially, but it should be easy to maintain two profiles in the app: one from the current Garmin service and one from the new one for testing. It is also intended to let you switch back an forth between services, if you delete some activities from the app and trigger a synchronisation from the other services, only the missing activities should come from the previous service.

The activities on the Garmin Connect service will NEVER be touched or deleted by anything you do on the ConnectStats app.

You’ll find instruction to test the new build here

Quick update

Many spurious activities continue to be sent by the Garmin Service that connectstats currently uses.

I will likely be able to push a fix for this during the weekend. The good news I anticipate the app will be able to delete all the spurious activities that are sent from the Garmin API.

Garmin is also now providing a new service, which I am working on integrating and will avoid all the Garmin API issues in the future, but this takes time.

Issue with many extra activities downloaded

Just a quick heads up, that many users are reporting that ConnectStats downloading a lot of activities unrelated to their own.

While I initially thought it had to do with Strava, it’s actually an issue with the Garmin service. Currently not consistent and not affecting everyone (it may soon), but I think I understand what the change is and will need to push a new version when I can, hopefully within a week.

Will update further here.

ConnectStats 4.6 released

ConnectStats is now approved and available in the store for update. With version 4.6, the connectivity with Garmin should work again.

On a separate note, I have applied to the new Garmin API and just got approved, so I’ll be working on getting that available for a later version, and hopefully, this should prevent similar unexpected outage in the future.

Quick update on fix timing

I am currently travelling, and somehow, I forgot to copy the key/certificate I need to submit to the apple store on my laptop… So won’t be able to submit the fix until I get back home this weekend and access my iMac… Sorry for the delay.

On a separate news, someone pointed out to me that Garmin is now offering a new API to access data on their service. I just submitted a form and applied to get access. Hopefully ConnectStats will be approved to use that API and it should simplify the app maintenance in the future. Will keep you posted.