Improved Statistics Page

Version 1.19 includes an improved and better organised statistics page.

The statistics page is quite simple to find on the iPhone. A few users on iPad sometimes miss it as it’s a bit less obvious, you need to press the stats icon pointed by the blue arrow below.

IpadStatsIcon

The New Fields Stats Page

The new stats page now have the fields pre-organised rather than trying to dynamically work out the ordering based on all the fields found on garmin connect data. This is both so that it works better with other services than garmin and due to the fact that the data include now a lot of new fields that confuses the old logic resulting in quite a messy page. Note that any fields not known by the app will still appear systematically at the very end of the list under the category Other.

NewStatsPage

One new feature is that you can now have a quick filter to see the summary stats for either the week or the month of the most recent activity. For that press the Sum Icon, which will then turn into a week or month icon as pointed by the arrow below.

Screen Shot 2014 04 12 at 11 38 58

Weekly or Monthly Summary

As before if you press the All button on the right it rotates through weekly, monthly or yearly summary. There is now also a quick filter for these pages. If you press the All button pointed by the arrow below you’ll enter to filtered page for the last 3m, 6m or year.

WeeklySummaryAll2TimeFilter

Note that if you want to get more detail on an activity you can now tap on the line for the period you are interested in and it will bring up the page with that full stats on the activities. Below you will see all the statistics for the week starting on the 23rd of march.

WeeklySummaryFor3m

It achieves that with the search feature, the activity list will also contains only the relevant activities. You can from the activity clear the search to see the full list of activities again. When the stats are about the current search it then replace the activity type icon. Note that you can this way get stats on any subset of activity you can define with a search in the activity list screen.

WeeklySummaryTapToDetail

Another subtle change in the Garmin API

UPDATE: thanks to a tip from @GViewerPro I have now a fix which will be a next release. Thanks a lot.

Since the login change, another less obvious change to the data service of garmin has resulted in a few issues with ConnectStats.

The track data contains now significantly less points. It used to provide points as often as recorded, but now the track data service returns a lot less points.

Here is an example of an activity downloaded before and after the service change. Notice the scatter point contains a lot less points.

BeforeMorePoints

AfterLessPoints

This has a few consequences in the app:

Smoothing

The app smoothing logic is not working as well anymore. We now have a point every 10 to 20 seconds, so it’s harder to figure out how to smooth. Intervals for example may not be plotted properly if they are less than a minute. If that happens turn off the smoothing, by selecting the graph full screen and pressing the Slider Icon to show the smoothing options.

Smoothingoptions

Auto Lap

The logic in auto lap was trying to account for times where the device was on pause. This logic gets mixed up with the new sparse points. A fix and a better logic is on the way for next update.

What I will do about it

I have somewhat tweaked the logic in the smoothing and auto lap feature and this will be incorporated in the new release 1.18. I am also investigating to use the tcx file available. This file seems to still contains all the data point. ConnectStats used to get its data from that file, but it was changed to a different api that contains more information, like the running dynamics of the new ForeRunner 620. I will probably see if I can do a merge of the files, use the tcx file for main data and the sparse data for running dynamics and other fields.

Multiple Profiles in ConnectStats

ConnectStats allows you to maintain multiple profiles. This can be useful to either track several users or now several services independently.

You access the profile setup from the settings view, the blue arrow 2 below

ServiceAndProfiles

You will then see the list of profiles existing in the app. It will start with only one named Default. Each profile will maintain its own list of activity and service setup.

You create a new profile by using the New Profile line. It will the prompt you for the name of the profile. Please avoid using special characters in the name. You can then simply tap on the profile you want to activate at any time.

Here is the list of profile in my current setup. You can see I created profile for different service setup and other accounts in garmin connect I use, as well as for the activity of friends.

Screenshot 29 03 2014 10 14

If you use the Delete Activities or Profile, this will let you either completely delete the current profile or some activity in the profile. Note that deleting activities here, is only locally in the app, not on the remote service, at the next refresh the activity will be downloaded again. It’s mostly useful if you want to make sure you refresh many activities from the remote services at once.

View or manage cache is simply a way to see how much disk space the app uses across the different files it keeps.

Garmin Connect New Policy Impact on ConnectStats and 3rd party apps

First I want to be clear that I have been a huge fan of Garmin’s for years. I own 6 devices and think they are extremely well done. I have also encouraged a lot of friends, colleagues and family to purchase Garmin devices over the years.

Monday March 2nd, Garmin announced they would stop making their API freely accessible and that 3rd party would have to pay $5,000 to access the data in Garmin connect.

I understand their infrastructure is costly to maintain but I feel such a steep fee risk killing the independent 3rd party applications which I believe adds value to Garmin customers and therefore Garmin as well.

I would propose a multi-tier system where 3rd party apps register and get a first limit of daily access for free, say 50,000 access per day. Heavy user breaching this limit would have to pay the 5,000$ fee.

I believe this system would be to both Garmin’s and its users benefit.

The Policy Change

On wednesday February 19th, Garmin updated its website which resulted in the approach used by most apps and website to access garmin connect data via an API to fail. The api used to have a license saying the following


This is the license file for GarminDeveloper.com. You are free
to access our API as long as you agree to create great things.

The link to that license has now been disabled http://connect.garmin.com/proxy/activity-search-service-1.2/downloads.html#License

On monday March 2nd, Garmin has notified developers that they are changing their policy and will charge $5,000 dollar fee for access to the API. Here is an extract of the message about the new policy.

Garmin has instituted a new policy regarding the accessibility of our developer programs. The previous strategy of freely available APIs quickly became unsustainable due to increasingly high demand. In response, we have established a new pathway for our Connect API. Firstly, there is a vetting process. We are strategically limiting the scope of this program to specific developers that will enhance the user experience. Secondly, those who are approved for inclusion will be charged one-time $5,000 administrative fee to cover the extensive engineering and server support required for the Connect program.

Consequence of the decision

This decision will likely force the eco system of independently developed 3rd party app to partially or completely disappear. This is a consequence which will affect customers. A lot of innovation can come from independent developers. Niche functionality can make its way to customers. It is likely not worth Garmin’s engineers’ time to cater to all customers software needs or custom data analysis needs. Enabling third party to provide for them helps make the Garmin devices more attractive. The size of the fee will dissuade a lot of experimentation for small players. The cost of entry is too high.

Most disturbingly it means it prevents people to use their own data in more creative ways than just using the data from the website. ConnectStats started as purely a tool for myself to look and slice my own data. I decided to share it as it could be useful to others, but right now it means I would not be able to use my own data in my code without paying $5,000, even if I pull the app from the store. This sounds wrong to me. Savvy users wanting to use a script to import their data efficiently for processing won’t be able to do that either.

In a world where the fitness market is exploding, I feel it would be to Garmin’s and its customers’ interest not to exclude this source of innovation for functionality.

My choices

I have the following choices for ConnectStats:

  1. kill the app. It would really pain me as part of the initial motivation was to provide a service for myself I couldn’t otherwise get: a more advanced viewer for my data on iPhone and iPad that what Garmin offers.
  2. swallow the cost. It’s a lot of money. I intentionally kept the purchase price at the minimum as to cover development cost and devices (money going back to Garmin by the way…) but my key motivation beyond my own use was sharing the app and not really generating large profits.
  3. increase the sell price of the app and hope people will continue to buy it. It somehow feels wrong to charge the users more just for the sake of them accessing the data they should own already. Right now if I increase the price from $.99 to $2.99 for ConnectStats, I am hoping to recover the Garmin fee over time. People pay several hundred dollars for their device. I hope they will accept the few extra dollars to pay Garmin for the right to access their data.
  4. switch ConnectStats to rely on another service. I am already working on that to use strava as primary download service and hope strava will pay the fee to get the data from garmin. This has some downside as some data won’t be available and Strava does not support swimming and skiing which I personally rely on.

Comparison to others services

I also wanted to comment on the fee structure proposed. A fee of $5000 is a lot of money by any standard. Niche independent Apps do not generate much revenue, mostly cover the costs. Here are some other services I use as comparison.

  • Apple charges $99 a year plus 30% of every sale to access their development tools and app distribution infrastructure. And this infrastructure and marketing power is of extremely high quality and adds a lot of value to independent developers like myself and to its customer. It wouldn’t occur to me to complain about this cost.
  • Strava provide an access to its data for free up to a certain rate limit once you are registered. Their API is extremely well documented, modern and their support and help to my questions have been great. I applaud Strava would provide a sophisticated website and API for free when their business model does not include revenue from any hardware.
  • Garmin support and documentation was so far inexistent. There are no or little documentation. I have never received an answer to any of my inquiries. Everything had to be reverse engineered. It doesn’t bother me at all given it was free but for $5,000 it’s a different issue.

I have been inquiring with Garmin about what exactly is provided when I pay the fee. What exactly do I get for that money: same API? Do I need to adapt my code to a new API? Do I get more documentation? Better support? Some marketing/promotion of the app as Apple provides?

Proposal

I believe the independent and small 3rd party partners add a lot of value to Garmin and the ability for users to access freely their own data is important. I understand the potential infrastructure cost and need to control usage for Garmin.

I suggest they offer a tiered system. Any user of the API would have to register and get some basic number of API access per day, say 50,000. Users above that limit would be required to pay the $5,000 fee to use the service. This would allow them to recoup some infrastructure cost from heavy user but still allow their customers to benefit from 3rd party functionality and access to their data.

What I plan to do for ConnectStats

I am doing 2 things:

  1. I have written to Garmin to explain why I feel a tier system would be better for all parties involved. I hope they will agree.
  2. ConnectStats is somewhat successful, receives good reviews and regular download. I hope by raising the price by an additional $2 this should help sponsor and recover the fee from Garmin. I have applied to be part of the program. Assuming Garmin approves my application, I plan to pay the fee myself and hope people will continue buying the app at the higher price to recover the money over time

Thanks for reading.

Comparing activities in ConnectStats

My activities are often on the same paths, so I wanted a way to compare activities. Of course segment in Strava are really useful in finding out how fast you can run or bike certain route, but I wanted a bit more control in what I was comparing. I also wanted a way that is not over complicated within the UI flow of connectstats. Here is what I came up with.

To compare an activity you need to first mark the activity that will compared to any other activity you will then look at. For that, you slide an activity in the list to the left, which reveals the Mark button.

Markactivity 6

After you press the button and the activity is then marked, note the mark icon on top of the activity icon.

Markedactivity 4

You can then look at any other activity and the graphs of the marked activity will then appear in the background of the current activity. Here I am looking at activities where I skied up the same slope with skins. It’s clear the first one my heart rate was much higher than the compared one. It was pretty much the same path up, so really just show how I adapted to the altitude over the course of the week, I think… Look at the HR histogram

SkiupHRHistogram 3

Or the rolling best plot

Skiupbestrolling 3

Now of course I claim I took the same path up, or did I? Hit the map and the two paths will appear. The compared path is slightly less opaque

Comparepaths 2

Here you see though large part of the ski up was the same, I didn’t start from the same exact same place, but arrived at the same destination.

Data relationships in a single activity

Here we’ll explore different ways to look at two different attribute together in connectstats.

Let’s say we want to look at how the speed and heart rate relate to each other. The first approach is simply to plot the two series together on the same graph as below.

2fieldsplots 12

The arrows 1 and 2 shows that on the two short interval when the running speed increased, the heart rate increased as well.

One approach to relate the information a bit better is to use a color overlay to show the one information on the a single plot

2fieldsasline

The arrow 1, shows where the line becomes darker and red as the heart rate increased with the speed, the arrow 2 shows the heart rate slowing down as the color goes back to the green

The last and, in my opinion, most interesting way to relate the two series of data is the scatter plot.

Scatterplot 17

Here you can see the points (arrow 1) where the speed/heart rate increased together. Array 2 is the beginning of the run with lower heart rate until it stabilised in the region of arrow 3 for most of the run. The regression line shows how the main relationship between the 2 data goes, here slightly to the downward slopping as expected.

It is also interesting in the lap view to see where in the larger scatter plot a given lap appear as below

Scatterlap 9

The green points show for this mile lap that the beginning was quite fast/high heart rate (arrow 1) and then slow down of both speed and heart rate (arrow 2), and clearly at the lower end of the overall speed of the run.

Data and smoothing used in connectstats

ConnectStats let us apply basics statistics technics to analyse our fitness data. We use these techniques to answer questions about the activities. How well did I do on this run? Is my stamina improving? Did I push myself more today that last week?

Raw Data

Before doing any analyses, let’s review the raw data we have access to. ConnectStats relies on the collection of the data is done with a garmin device and a collection of sensors.

The raw data varies by activity type and device.
For outdoor activities, the raw data will be gps coordinates, a timestamp, speed and values from different sensors: Heart Rate beat per minutes, cadence (legs or pedal), Power delivered on the bike, and sometimes more advance running dynamics like Vertical Oscillation, ground contact time. For swimming the data collected will be the time for each length and the number of stroke.

Here is an example of what the raw data looks like Blog txt

Depending on the activity or the device the raw data will be collected at fixed intervals or at different intervals trying to reduce the amount of data saved. If you move in a straight line for 5s at constant speed, it isn’t necessary to collect the points every seconds as they can easily be interpolated.

From this raw data, there are two main directions for the analysis we can do. The first is analysing a single activities and the second is analysing summary of each activities over time.

Focus on a single activity

The summary statistics are quite simple. Mostly we will want to look at the average of the different measure, maximum and minimum. Where it gets more interesting is to look at the series of data. As for most statistics analysis a first phase will be cleaning the data.

Data Cleaning

Let’s look at the graph of the data collected by the gps. You can see it is quite noisy, there is also large part of the data that is not really interesting to look at to. Some is gps noise, area with bad reception, pause due to stop lights,

Rawdatabad

To clean the data, we can first use assumption on expected constraints on the data. For example, define a minimum speed considered valid running speed, or a minimum valid heart rate to remove unreasonable spikes. Another technique used by connectstats is to filter points that have unrealistics acceleration.

FilterBadValues

The last smoothing technic we can apply is a moving average of a few points. Usually the app will use 5 points moving average for data, but it can be controlled in the sliding left settings window.

Filterbadnormalsmoothing

In the case below the graph start to be more readable now, especially with the 2 minutes moving average overlay, which shows the two short faster run intervals around 30min and 40min.

Historical Scatter Plots I like to use

One of my initial motivation to write the app was to look at my activities using scatter plots. I was especially interested in looking at the relationship of heart rate and speed.

Here you can find information how to access them.

scatter-hr-pace

The first thing I look is where is my last activity in my overall history. Here you can see that I was in the middle of the pack, a bit on the high heart rate side.

Screen_Shot_2014-01-18_at_17_44_54-6

 

It’s also interesting to check the pattern overtime. You can see here that the more recent colors are on the higher HR, slower pace area. Not good, I need to improve.

Screenshot_19_01_2014_22_25

Sometimes it can also be useful to check only the recent history, using this button to rotate between all, 1m, 3m, 1y

Screenshot_19_01_2014_22_31-4

Other times, I also want to have a more sophisticated filter for the graph, in that case I can use the search feature. If I define a search in the activity list I can then get the scatter plot only for those activities. The statistic tab will have an extra button beside running, cycling swimming and all: Search.

Screenshot_19_01_2014_22_34-2 Screenshot_19_01_2014_22_35-2 Screenshot_19_01_2014_22_36

Some other interesting historical relationship to look at for bikers: Power and Cadence or Power and Heart Rate

Screen Shot 2014-01-18 at 22.57.28 Screen Shot 2014-01-18 at 22.57.46

 

Best Rolling Plots

Critical Power Plots

I discovered recently the concept of Critical Power Plot, after a user left a comment on the site. Critical Power is the maximum power you can maintain for a given period of time. The concept seems mostly used by cyclists. But it felt to me it could be extended to other data: What is the best pace I maintained for any given period of time? What is the maximum heart rate I maintained for a given period of time?

So I decided to give it a go and implement it in ConnectStats. All I had to do was maintain a rolling average every X seconds and keep track of its maximum. It’s quite an expensive computation: for N points, I need to do NxN computations, and the devices processing power is a bit low, so I chose 5 seconds for X, which seems to work well in practice even on my old iPhone 4.

Of course, the original data in the file produce by the watch isn’t nicely produced every 5 seconds and for the algorithm to work efficiently, I have to resample the original data at the average on that exact interval, which can result in a bit of loss from the original file. But in practice it didn’t appear too bad on most of the ride I looked at.

I end up with a new time serie evenly spaced every 5 seconds: the maximum average heart rate over 5 seconds, over 10 seconds, over 15 seconds, etc, etc.

Just had to come up with a name, cyclist refer to it for the power field as Critical Power, but “Critical Power” or “Critical Pace” felt a bit confusing to me, so I named it “Best Rolling Heart Rate” or “Best Rolling Pace”

Here is the example from my best 10k race for the heart rate.

doc-detail-bestrolling-hrplot

 

This is quite nice and works also for power or pace.

Best Rolling Laps

Of course the plots are nice, but the next thing I wanted to know was for that best HR what were my others statistics for each of that given period of time where I reached that maximum?

So I enhanced the auto lap feature to show the best rolling lap corresponding to a few arbitrary periods of the plot. Below you can see the details for the Best Rolling Power or Critical Power of a friend, along with the distance, speed, HR for each of the correspond section of the ride.

doc-detail-bestrolling-lapsdoc-detail-bestrolling-lapchoices

 

 

 

The last missing piece was to plot on the map and highlight on the plot of another statistic where that best average was reached as below

doc-detail-bestrolling-lapdetail