Select to expand quoteK888 said..
The images below are from a wing session last year. Of course, the Motions had the benefit of being on my biceps, rather than wrists. The differences can be attributed to measurement errors (several possible causes) and arm movements
Did you have the watches on the same arm, or on different arms?
Select to expand quotesailquik said..
Can we respectfully request that we refrain from doing this sort of individual file analysis online in the forum please? It is better done slowly and carefully, with time for team consultation, and when not feeling under pressure to reply immediately.
Please just send the files to info@gpsteamchallenge.com.au and we will do our best to analyse and compile the results among the team thoroughly, and then reply/report on it as appropriate.
I disagree. I find this kind of analysis quite interesting, and would very much prefer to see it here on the forum. The analysis in a closely knit group of experts has some serious limitations (which are reduced by K888's detailed posting of results). And it completely excludes others who might be interested.
One example of these limitations is the handling of missing points. The vast majority of "repeated" points in Garmin are in fact missing points. They show up in lots of sessions, and sometimes at a very high frequency. Just a few years ago, a much lower occurrence of missing points was quoted as a reason to remove a device from the list of approved devices.
I have seen at least one instance where missing points increased the observed different to Motion data for the top 2 second run by about 0.4 knots. Interpolating missing points pretty much eliminates this error increase in this case, and is expected to do so in many (but not all) cases.
But, of course, you can just keep all your analysis "private", and then bless us with your final decisions. That, however, reduces my motivation to adapt Speedreader to just about zero. Maybe I should make Speedreader private again, too.
Select to expand quotesailquik said..
Can we respectfully request that we refrain from doing this sort of individual file analysis online in the forum please? It is better done slowly and carefully, with time for team consultation, and when not feeling under pressure to reply immediately.
Please just send the files to info@gpsteamchallenge.com.au and we will do our best to analyse and compile the results among the team thoroughly, and then reply/report on it as appropriate.
No rush from me. I sent the files through and was surprised to see such a quick response. I did post here with the KA72 results for each, but I was not trying to rush you guys. I just looked at it myself and found it interesting so posted it while they were on ym screen. For me the end game is getting the 965 approved. Take your time with going through the individual files I send.
It looks like currently the 965 is performing similarly to the 255 in speeds and position, but the 965 performs better in regards to the timing issue that causes repeated position points. The activity that I sent through seems to have picked up a couple of edge cases (alpha on the border of the circle limit) which may have confused the results. In another thread @tonyd said he had sent through some 965 comparison files some time ago. Have those files been investigated and were the results similar to mine?
Regarding the jitter. It seems much worse on my bike than your example (up to around 1 knot). But I can't say that was not due to mounting on the bars causing vibration so will put it down to that. Does the doppler speed include vertical components as well as horizontal. If it is picking up vertical speed, that would not be helpful for use in windsurfing but would average out if averaged over several samples... but would make the graphs look as I have observed with jitter. The Garmin graph I referred in red shows no decelerations over the accelerating down a hill. So perhaps Garmin has some filtering or somehow isolates only the horizontal component.
Select to expand quoteK888 said..
Agreed, both devices have unknown errors but the Motion has greater precision. As way of an illustation, I've previously compared two Motions and two Garmins to see how the reported speeds differ throughout a session. The images below are from a wing session last year. Of course, the Motions had the benefit of being on my biceps, rather than wrists. The differences can be attributed to measurement errors (several possible causes) and arm movements.
The graphs you had posted seem to support your statement that "the Motion has greater precision". However, there are multiple problems with your post (even if we assume that the reader understands the difference between precision and accuracy). The primary issue with the data you show is that it's from a winging session, with the Motions being on your upper arms, and the Garmins being on your wrists. Anyone who has ever pumped or jibed while winging knows that arm movements are often independent when winging, and that the wrists move a lot more than the upper arm. While not explicitly stating it, y\our statement is leading towards the differences being from "measurement errors (multiple causes)" rather than arm movements.
I have repeated your experiment in a way that completely eliminates any arm movements - driving around. I compared two Garmin 255 watches (one Music, one not) and two ESP32 loggers, all placed right next to each other on the dash board. Here are the results, in a similar graph to the one you showed:
In this example, the curves look quite similar. The Garmin 255 curve has a slightly broader center (also seen in the 1st-3rd quartile range), while the ESP32 data show more outliers and this a bigger range. Here's an example of a 2-second region with a difference of > 0.4 knots in speed over 2 seconds from the ESP32 units:
As is typical for regions where the two units deviate, the error is directional (all speeds are lower in one unit), meaning that a higher sampling rate would not reduce the actual error (but it would report a lower error estimate!).
When doing "all point" comparisons of wing data, one has to be very careful, since winging often includes rapid arm movements. For those who have not seen this in GPS tracks, here's an example:
This is from a Motion recording at 5 hz. The spikes are from pumping to get on the foil. Most likely, the Motion was on the upper arm, and recordings from a wrist-mounted unit would show even larger spikes than the 2-3 knot drops in 0.2 seconds seen here.
There was a fair bit of discussion prior to me being offline yesterday. I will share some thoughts in seperate posts, including some genuine windsurfing data from my last speed session in Portland Harbour.
Regarding the question about 2D speeds, everything is initially calculated in 3D (relative to the earth's centre of mass), then converted into 2D (lat + lon + speed) using an ellipsoid model (typically WGS-84) and a primitive geoid for elevation / altitude. Some activities on Garmin watches have a 3D distance + 3D speed option which also factors in changes in altitude. The GNSS chipset is sometimes capable of outputting vertical speeds, either via their binary protocol, or custom NMEA sentences. The 2D speed is referred to as speed over ground (SOG), which is tangential with the ellipsoid.
Regarding filtering. Noisy sensors (including GNSS) are typically passed through a Kalman filter to smooth the output. There are lots of videos (and books) describing the topic but the general principle is to combine actual measurements with predictions. Given an understanding of the current state (time, position, velocity, acceleration, etc) make a prediction for the next epoch (e.g. 1 second into the future). When you get to that point in time, use the latest satellite observations to calculate the current position + velocity + time (PVT) and then combine them with the previous prediction to produce your best estimate. In principle the combined results are less noisy and more trustworthy than either the calculation from raw measurements, or the prior prediction alone. This is a massive over-simplification and I've not described dynamic models, underlying stats or maths, but hopefully it gives some idea as to what the Kalman filter is doing. It's a combination of measurements and predictions. This is all done natively within the GNSS chipset.
On a related note, I once read a post by a GNSS engineer describing the internals of the SiRFstar III (used in GT-31) as updating the tracking of doppler shift @ 10 Hz internally. The Kalman filter could only output a PVT solution at 1 Hz, which is what we got on the GT-31. The SiRFstar IV chipset also updates the Doppler observable at 100 ms, but the Kalman filter can output every 200 ms (5 Hz) as we saw on the GW-52 and GW-60. The Airoha (and MediaTek ancestors) are able to output PVT solutions at 10 Hz. Based on all of the aliasing tests that I've performed, 1 Hz outputs appear to be using a split-second speed and not using interim Doppler measurements. Despite the Kalman filter you can still see aliasing artefacts because only one Doppler measurement is used per second. I tried to describe aliasing in simple terms at logiqx.github.io/gps-details/general/aliasing/
The likes of COROS and Garmin also apply their own filtering / smoothing they deem suitable for the sport / activity. I'm 99.9% sure the windsurfing / kitesurfing / other speeds being saved in Garmin FIT files are the Doppler-derived speeds produced by the GNSS chipset. There is however a Garmin filter which snaps speeds under 0.2 m/s (circa 0.4 knots) to zero.
The post of winging data wasn't meant to diminish the significance of arm movement. The original analysis of that specific dataset was simply to give an insight into the real-world performance of the Garmin watches when winging, recognising that a large part of any differences will be due to arm movements.
This happens to a much lesser degree when windsurfing but it is still significant. I'll show some actual windsuring data in the next post.
Whilst testing + evaluating the mini motions for Weymouth Speed Week, I always sailed with 4 mini motions - 2 on each bicep. I was very impressed with just how consistent they were with each other on the same arm and how they could pick up the arm differences when winging (see screenshot).
You can see how the motions on the same arm track each other very closely, yet the two arms differ in places. The differences between the two arms appear to be due to independent arm movements. These tests were from minis worn on the biceps, so a much smaller range of movement than the wrists.
Looking at some SUP data was also quite interesting. These charts were used as baselines for some aliasing investigations - wearing on the bicep vs wrist. 0815 (red) was on the board and 0815 (blue) was on my arm. Initially 0815 was on my bicep then moved to the wrist, just above my watch.
Whilst on the bicep the speed was comparable to the board speed, but with a slight difference in phase.
When the motion was on my wrist you can even spot the difference between paddling on different sides (upper hand vs lower hand).
All of these charts are simply to illustrate how sensitive our GNSS receivers are to movements. This is data that I have to hand and may be interesting to some people. The data in this post is not supposed to represent the dynamics of windsurfing, just share some general observations.
Now for some actual windsurfing data...
Regarding testing, I tend to do 3 types of testing where possible - static testing (leave devices in an open area for several hours), driving (non-challenging dynamic testing) and real-world testing (windsurf / windfoil / wing). Real-world testing uses the most ideal (but still practical) method of mounting / wearing the device. Secure on the bicep produces great results with the Motion, but the wrist is obviously most practical for a watch. I have no doubt that two watches would produce more consistent results if they were worn on the bicep, but that isn't very practical.
I always wear two motion minis as my references devices. This makes it easy to spot any outliers and allow for an evaluation as to whether the Motion results can be trusted as the baseline. Visual inspection of the data always usually shows bigger differences in my watches than the Motions. A visual inspection of the entire track is par for course, along with a comparison of the top 5 results for all significant categories. Visual inspections will usually cause me to pick up any anomalies (and occasional spikes) which don't appear in top 5 results.
My previous reference to errors includes the standard GPS / GNSS error budgets and things like an upside down receiver. A receiver may be capable of measuring the true speed at any specific moment (including split-second jerky movements), but in an ideal world you really need multiple measurements per second if outputting PVT solutions at 1 Hz. Recall the previous post about what happens internally in the SiRF chipsets. That is very likely to be the reason we don't see aliasing effects on 1 Hz devices from Locosys.
Peter's driving tests show little difference between the ESP and Garmin devices that were tested, although there was a slight bias in the watches (not centered around zero). This is consistent with my own testing but sadly it's not representative of the real-world performance we see on the water. It provides useful insights into the innate precision of the watches, so I appreciate seeing Peter's data and think it provides useful insight to anyone interested in the nuances of GNSS performance.
I've taken my last windsuring session (pre-Christmas) and created similar plots to my earlier winging example. This was a typical speed session in Portland Harbour. The charts aren't measuring the innate precision, but they show how the live speeds (smoothed using a 2s rolling average) differ between 2 equally good watches worn on the wrist, and 2 motions worn on the bicep.
The percentile figures say that during this session, whilst sailing > 25 kts:
- watches within 0.13 kts of each other for 50% of the time, 0.44 kts for 95% of the time and 0.81 kts for 99.7% of the time
- motions within 0.03 kts of each other for 50% of the time, 0.09 kts for 95% of the time and 0.18 kts for 99.7% of the time
Roughly speaking the differences between the motions are 4 to 5 times smaller than the watches across a range of speeds. These figures are not meant to represent "errors" or give an absolute measure of precision, they simply show how much two watches / motions vary throughout a typical windsurfing session when worn in the recommended manner.
These stats are consistent with what I observe when comparing top 5 results using Speedreader, or comparing 2s results as I finish a run, which I have done many hundreds of times. A lot of the differences are likely due to how the watches are being worn on the wrists, but that is unavoidable. Most runs show results within 0.1 or 0.2 kts of each other on the watches, but 0.5 to 0.7 kts is not a rare occurrence.
Accuracy and precision are different things from a scientific perspective, and this post does not even touch on the differencea or attempt to quantify them for these devices. The ISO definition of accuracy works quite nicely for GNSS, which is a combination of trueness and precision. One aspect of Peter's earlier post related to trueness, when discussing errors being high or low for sustained periods.
I guess the main goal of this specific post is to show what can be achieved with modern Garmin watches in real-world conditions and when wearing the devices in the most optimal (but practical) way. It isn't giving a measure of accuracy in the ISO sense (combination of trueness + precision) but observing the consistency of data from two devices provides some useful insights imho.
Select to expand quoteK888 said..
Regarding the question about 2D speeds, everything is initially calculated in 3D (relative to the earth's centre of mass), then converted into 2D (lat + lon + speed) using an ellipsoid model (typically WGS-84) and a primitive geoid for elevation / altitude. Some activities on Garmin watches have a 3D distance + 3D speed option which also factors in changes in altitude. The GNSS chipset is sometimes capable of outputting vertical speeds, either via their binary protocol, or custom NMEA sentences. The 2D speed is referred to as speed over ground (SOG), which is tangential with the ellipsoid.
So do both the Garmin and Motion use the 2D speed (SOG) for the reported doppler speed? Or is it possible that one or both use the 3D version. My previous post showed the Motion with some significant noise on my bike rolling down a hill. Noise that was showing a 1 knot deceleration during a sustained average acceleration. Something that an 80kg rider+bike could not do without significant violent vibration being felt by the rider. The Garmin shows nothing similar, but it is sampling 1Hz (you suggest the garmin ignores the other 9 doppler samples and does not average them), so perhaps the increase in speed in 1s was enough to mask any individual sample short term reduction. Motion mounted to handle bars quite tight with the strap, but the strap may have allowed it to move up and down at high frequency on small bumps by a few mm. Of course going down a hill there is a small horizontal component of a movement perpendicular to the road surface. I don't believe it was vibrating parallel with road surface because the bump forces are perpendicular and I could not see any movement looking from above it.
I'm just trying to work out why the Motion looks different to what I expect in the doppler speed plot, but the Garmin looks exactly like what I expect for a projectile rolling down a hill on a smooth surface accelerated by gravity. The bike example is much easier to consider than windsurfing as a road surface is much smoother than water and gravity is a more constant acceleration than the wind.
If the Garmin is recording the correct doppler speed at 1Hz (and ignoring the other 9 samples), and the Motion is recording every sample, it stands to reason that if the speed signal has noise like this, the Garmin and Motion will rarely have identical 2S. The Garmin averages 2 samples, and the Motion does 20 samples. Often the noise will average to zero over that time, but depending on the frequency of the noise, the aliasing effects could cause differences. If the Garmin happens to sample on the higher parts of the noise (if the noise frequency is close to an integer multiple of the 1s sample rate), it will read higher than the average recorded at 10Hz and vice versa. So even with both devices reading highly accurate doppler, with a noisy signal (made worse with vertical components), they could both get quite different results over 2 seconds due to aliasing artifacts of different sample rates. This effect would be much more if they include 3D velocity as speed surfing is about ground speed.... not about the vertical speed.
Another effect, is the presence of 10 times the samples means there are 10 times as many positions the analysis software could start the 2S calcluation. In the presence of noise, if the noise is such that the 2S period starts at a max amplitude peak of the noise and finishs at a max amplitued peak of the noise, it will get a higher result than a 1Hz sample rate that did not have access to those peaks as they were between samples. This may explain why on average it seems the Motion typically reads more 2S higher than Garmin even if both are reading identical doppler speeds... just sampled at different rates. In my tests so far, the Motion had higher 2S on both activities, and I think some analysis of many 2S speeds in the same activity showed the Motion was higher most of the time.
If you run a test of two of the same devices (Motion or Garmin), are the sample rates synced between the units? Do the Garmin 1Hz doppler signals sync to the same timing events sent by the satelites? Or could they be out of sync which may explain why two Garmin devices have less precision even if reading the same identical doppler readings. Are they making readings half a second apart and therefore the noise effects them differently?
Both devices are using SOG. The Motion + ESP we know for sure (u-blox documentation) and Garmin will simply be using the standard NMEA sentences that are output by all GNSS chipsets. Those sentences also use SOG and do not include the vertical component.
I'll use your cycling data to describe some general environmental challenges. It should't be regarded as a review of the device, or assessment of a specific session. The intention is simply to share some thoughts about this type of cycling data.
The ride goes up and down hills and through gorges, through some tunnels and deep into some valleys with (small) mountains on each side. In terms of tracking the GNSS signals, it presents a far more challenging environment than we encounter when speed sailing.
When you are in areas with clear sky the Motion performs like normal and the sAcc value (bottom chart) are low.
The noisiest sections in the Motion data are when you are riding through wooded (and / or urban) areas. The sAcc values are elevated, which tells us there is more uncertainty in the calculated speeds.
That particular section of the data was whilst you were in a wooded area.
When we render speed data in apps such as GPSResults and GPS Speedreader we simply join the dots. The 1 Hz data will always produce a smoother looking line, whilst 5 Hz / 10 Hz data will show the good, bad and ugly sub-second measurements.
In the wooded sections of your ride, the Garmin is also going to benefit from the multi-band feature. This is somewhere that multi-band really is beneficial because the L5 signals are more powerful and have greater penetration of foliage than the L1 signals.
There are other benefits to multi-band such as improved multipath characteristics due to the higher chipping rate. Essentially, multipath reflections that increase the pseudorange of the L5 signal by more than 29 meters can be ignored, whereas it's 290 meters for the L1 signals with a chipping rate that is 10 times slower.
Regarding sampling the Garmin isn't quite ignoring 9 of 10 PVT solutions. The navigation solutions (PVT) require power, so I doubt they are calculating them at 10 Hz and outputting at 1 Hz. I need to go back and check my article to ensure I haven't misworded anything. Nevertheless, what you say about the difference between 2 samples and 20 samples holds true.
The navigation solutions from the u-blox are aligned to nice offsets - e.g. 0, 100, 200, ... 900 ms relative to UTC (thanks to GNSS). Sometimes they will be off by a few ms, but the chip is trying to align the PVT solutions to those nice offsets. I'd imagine the Airoha is doing much the same and if we had the raw NMEA it would be possible to confirm.
Just to clarify, I'm not part of the GPSTC advisory committee. I simply aim to share my insights where they may be helpful; whether it be the GPSTC, GP3S, or curious individuals. I hope you appreciate the technical discussion, since I know you works as an electronic engineer.
Select to expand quoteK888 said..
Regarding sampling the Garmin isn't quite ignoring 9 of 10 PVT solutions. The navigation solutions (PVT) require power, so I doubt they are calculating them at 10 Hz and outputting at 1 Hz. I need to go back and check my article to ensure I haven't misworded anything. Nevertheless, what you say about the difference between 2 samples and 20 samples holds true.
The navigation solutions from the u-blox are aligned to nice offsets - e.g. 0, 100, 200, ... 900 ms relative to UTC (thanks to GNSS). Sometimes they will be off by a few ms, but the chip is trying to align the PVT solutions to those nice offsets. I'd imagine the Airoha is doing much the same and if we had the raw NMEA it would be possible to confirm.
Thanks for the great explanation. The take home I get is that the noise I see is not actually the device vibrating from bumps. It is actually error in the GNSS observations caused by compromised satelite signal due to the environment and that a multi-band copes with this better. It is likely the Garmin is giving a more accurate speed signal in this case, but this is not typical in a windsurfing environment.
I got the idea it was ignoring 9 out of 10 samples from you quote in a previous post "Based on all of the aliasing tests that I've performed, 1 Hz outputs appear to be using a split-second speed and not using interim Doppler measurements. " but now that I re-read that, I may have interpreted it wrong.
I am interested in if there is a way to confirm that two Garmin watchs being used in a comparative test are sampling at the same time. I think it could explain why there is a larger variation in 2S between two Garmins than two Motions. It they are sampling out of phase by half a sample period and the 2S speed graph shows a typical acceleration to a peak and then deceleration, then even if they both are perfectly accurate, you will get a different 2S results because of the sampling offset. Obviously this is much reduced for 10Hz. I would expect this 2S peak would typically be lower than for a 10Hz device since it is more likely to miss the peak speed and have it between two samples.
So going forward towards in the hope of GPSTC approval of the 965 device, it appears further bike rides are not very helpful. Probably driving in a car would be similar in my area since it is mostly wooded in hilly. I just have to get more sailing in. Let me know if anyone can think of any non-windsurfing tests I can do before I return the Motion to its owner.
Select to expand quotevosadrian said..
I am interested in if there is a way to confirm that two Garmin watchs being used in a comparative test are sampling at the same time. I think it could explain why there is a larger variation in 2S between two Garmins than two Motions. It they are sampling out of phase by half a sample period and the 2S speed graph shows a typical acceleration to a peak and then deceleration, then even if they both are perfectly accurate, you will get a different 2S results because of the sampling offset.
The "sampling out of phase" could certainly contribute to observed differences between 2 Garmin watches. In the extreme case, duplicated ("missing") points in Garmin data can put the data out of alignment. Here is an example:
This is from 2 sections of a wing session, about 5 minutes apart. In the first section, the speeds of the Garmin (red) are well aligned with the speeds of the Motion (in blue). A few minutes later, the Garmin speeds lack about a second behind. If you calculate the difference between the two tracks, it would increase a lot in the second section; but if you look at the top speeds in the sections, the difference would not increase. The same thing could happen between 2 Garmin watches - perhaps even on a sub-second level (although that would require a time difference on when the GPS observations were generated by the GPS, rather than when they were processed by the watch).
Select to expand quoteboardsurfr said..vosadrian said..
I am interested in if there is a way to confirm that two Garmin watchs being used in a comparative test are sampling at the same time. I think it could explain why there is a larger variation in 2S between two Garmins than two Motions. It they are sampling out of phase by half a sample period and the 2S speed graph shows a typical acceleration to a peak and then deceleration, then even if they both are perfectly accurate, you will get a different 2S results because of the sampling offset.
The "sampling out of phase" could certainly contribute to observed differences between 2 Garmin watches. In the extreme case, duplicated ("missing") points in Garmin data can put the data out of alignment. Here is an example:
This is from 2 sections of a wing session, about 5 minutes apart. In the first section, the speeds of the Garmin (red) are well aligned with the speeds of the Motion (in blue). A few minutes later, the Garmin speeds lack about a second behind. If you calculate the difference between the two tracks, it would increase a lot in the second section; but if you look at the top speeds in the sections, the difference would not increase. The same thing could happen between 2 Garmin watches - perhaps even on a sub-second level (although that would require a time difference on when the GPS observations were generated by the GPS, rather than when they were processed by the watch).
I'm more talking about the difference in calculated 2S speed caused if the samples were out of phase by half a sample period on the same speed waveform. Looking at the second of your waveforms the peak in the red (near the beginning) seems to be sampled at near optimal time to get the highest speed. Imagine if it had been sampled half a sample each side of that peak. The peak would then have been lower but more sustained. The 2S calulcation from post processing software would almost certainly have been different, but perhaps by not that much depending on the shape of the speed waveform. I am just suggesting that if we compare Garmins and we get lots of small differences with not much spread in the differences, perhaps the GPS signal of both is very good, but if the sampling is out of phase we expect lots of small errors.
If the chipset is returning 10Hz speed... or is capable of that, but it is recorded at 1Hz.... what defines when the samples are made... is it simply when the record button is pressed? Obviously it would be great if all devices sampled exactly on the second changeover for the global GPS clock, but do we know that to be the case? We already know some Garmins have a weird timing issue that seems to be related to the sample clock being different to the GPS chipset clock causing delays such as you show above. I am suggesting this as a reason why 2 Garmins may have higher speed delta than 2 Motions. I am not suggesting this would cause enough error to make the data too erronous to use for GPSTC. So far my 2 uses of both Garmin and 965 have had the data on KA72 read mostly lower on the 965 and that seems to be common from people who use both. It seems that it would be rare that a Garmin would report a large positive difference in results and is more likely to be a small negative difference which would have little effect on the accuracy of the GPSTC database.
Select to expand quoteK888 said..
Both devices are using SOG. The Motion + ESP we know for sure (u-blox documentation) and Garmin will simply be using the standard NMEA sentences that are output by all GNSS chipsets. Those sentences also use SOG and do not include the vertical component.
It is correct that SOG should not include a vertical component. But this requires that the GPS can separate vertical and horizontal components of the doppler speed with perfect precision. So the real question is - how much are vertical movements reflected in the reported SOG?
A simple little experiment is to let GPS units fall to the ground, and see if they report "SOG" speed during the fall (I recommend letting them fall on pillows, not cement
). Here's a graph from one such experiment:
Blues is an ESP32 logger at 10 hz, red is a Garmin 255 watch. The tall spikes, with "SOG" speeds up to 4.5 knots, are from the GPS falling. This is sometimes followed by a small bounce off the pillow, and then by lifting the GPS back up into starting position.
By definition, the fall is completely vertical (in the absence of wind), so there should not be any SOG speed recorded. With the u-blox GPS in the ESP logger, we can look at the speeds in each of the three dimensions (velN for N-S, velE for E-W, and velD for up-down). Here's what this looks like for the first fall:
The down speed (velD) increases over 0.5 seconds, which is about the expected time for a 1.5 m fall (see www.omnicalculator.com/physics/free-fall). But it does not increase to the expected end speed, which should be around 10 knots. Instead, we see speeds of 2.6 and 2.89 knots reported for the two horizontal dimensions, which give a calculated SOG off 3.89 knots. This SOG is entirely error resulting from incorrect separation of horizontal and vertical speed.
How big the "3D to SOG" error is varies a lot over time. Here's another set of about 20 falls to show the range:
For the 10 hz data, the range of false SOG speeds is between about 2 and almost 6 knots. For the 1 hz data, the range is between 0 and 4 knots. Note that we expect to see both lower values and complete misses in the 1 hz data, since the fall itself is just about 0.6 seconds, so the watch will sometimes not measure any speed while it is falling.
The bottom line is that there is significant "bleed through" of vertical speed into the reported "SOG" speed, both for u-blox based GPS units and for Garmin watches. For positional data, it's quite well known that that vertical accuracy is much worse than horizontal accuracy, and separating vertical and horizontal speed components seems to suffer from similar problems.
Nice. I'd not considered the "pillow test" but had tried a variety of things in the past to determine whether it really represented SOG.
I've looked at some of my old snowboarding data (GT-31), which aside from a few dropped points shows doppler-derived speeds to be in line with position-derived speeds.
Also some more recent u-blox data from Daffy whilst skiing:
The same is true for some MediaTek data from the prototype GT-35. Based on all of those data sets, DD speeds can be seen to represent SOG (due to their closeness to the PD speeds) but your results really put the cat amongst the pidgeons! It may go some way to explaining why sailing in chop can result in the odd mini spike and sawtooth effects.
Select to expand quotevosadrian said..
If the chipset is returning 10Hz speed... or is capable of that, but it is recorded at 1Hz.... what defines when the samples are made... is it simply when the record button is pressed? Obviously it would be great if all devices sampled exactly on the second changeover for the global GPS clock, but do we know that to be the case? We already know some Garmins have a weird timing issue that seems to be related to the sample clock being different to the GPS chipset clock causing delays such as you show above.
Once the GNSS signals are being tracked, the navigation solution aligns itself with nice offsets (relative to UTC) - 1Hz (0 ms), 5 Hz (0 / 200 / 400 ms), 10 Hz (0 / 100 / 200 ms). This is true for SiRF, UBX, and MTK chipsets that I have studied over the years. The Airoha is the direct descendent of the MTK 3333, so I'm 99.9% sure it will be doing the same thing. Essentially the PRNs repeat every 1 ms and GPS data bits last for 20 ms, so it's relatively straightforward to calculate the navigation solution using nice time offsets.
The Garmin issue is something else, independent of the GNSS chip. Imagine two Garmin activities "A" (process the GNSS data) and "B" (write new record FIT - time, lat, lon, altitude, speed, hr, cadence, etc). Ordinarily the order goes A B A B A B A B, but every so often on the FR 255 it goes A A B B A B A B (second A B pair out of order), or A B B A B A B A (missing A), or A A B A B A B A (missing B). This can be seen in the FIT file at the same time as frozen positions, double-jump positions and changes in time offset. I've over-simplified by only describing it as A and B, but that's the gist of what I've observed in the data.
The observable order issues in the FIT (records + GPS metadata) can potentially be used to inform workarounds such as interpolation, etc. I've just had too much on recently to consider it in any more detail, or document my observations and provide actual examples to Peter for comments + feedback. It's quite an involved topic, so perhaps one for offline discussion and sharing of ideas.
Select to expand quoteboardsurfr said..
The "sampling out of phase" could certainly contribute to observed differences between 2 Garmin watches. In the extreme case, duplicated ("missing") points in Garmin data can put the data out of alignment. Here is an example:
Just a quick response as I'm out of time.
I failed to mention that the histograms were for sessions where the watch alignments did not change. The comparison was also based on aligned data, which was initially offset by 1 second for the entire session. The final stats seem to be pretty consistent with what I've seen when comparing individual sessions and runs on the 2 watches.
Select to expand quoteK888 said..vosadrian said..
If the chipset is returning 10Hz speed... or is capable of that, but it is recorded at 1Hz.... what defines when the samples are made... is it simply when the record button is pressed? Obviously it would be great if all devices sampled exactly on the second changeover for the global GPS clock, but do we know that to be the case? We already know some Garmins have a weird timing issue that seems to be related to the sample clock being different to the GPS chipset clock causing delays such as you show above.
Once the GNSS signals are being tracked, the navigation solution aligns itself with nice offsets (relative to UTC) - 1Hz (0 ms), 5 Hz (0 / 200 / 400 ms), 10 Hz (0 / 100 / 200 ms). This is true for SiRF, UBX, and MTK chipsets that I have studied over the years. The Airoha is the direct descendent of the MTK 3333, so I'm 99.9% sure it will be doing the same thing. Essentially the PRNs repeat every 1 ms and GPS data bits last for 20 ms, so it's relatively straightforward to calculate the navigation solution using nice time offsets.
Have you considered the option where the Airoha is setup by Garmin to report at 10Hz (if it is capable of that and I understand this is not energy efficient), but the Garmin only writes the data to the FIT file at 1Hz and further to that, the time base for the Airoha is indeed aligned with GPS time, but the Garmin doing the logging has a different internal time reference (I believe Garmin watchs set time when they sync to the Connect app. This is certianly the case when I change time zones or daylight saving time changes).
So even though the Airoha chip may be consistent in its logging time across all devices.... if the Garmin is writing the data at different randomly phased time references, the data in the FIT files may not reflect the Airoha time reference.
Select to expand quotevosadrian said..
Have you considered the option where the Airoha is setup by Garmin to report at 10Hz (if it is capable of that and I understand this is not energy efficient), but the Garmin only writes the data to the FIT file at 1Hz and further to that, the time base for the Airoha is indeed aligned with GPS time, but the Garmin doing the logging has a different internal time reference (I believe Garmin watchs set time when they sync to the Connect app. This is certianly the case when I change time zones or daylight saving time changes).
So even though the Airoha chip may be consistent in its logging time across all devices.... if the Garmin is writing the data at different randomly phased time references, the data in the FIT files may not reflect the Airoha time reference.
I have considered the possibility, but don't think it is likely. The Airoha has a setting where you specify the fix rate in milliseconds (100-1000, default = 1000), and a setting where you set the NMEA output rate for once every N fixes. Garmin could of course be setting the fix rate to 100 / 200 / 500 ms and N to 1 / 2 / 5 / 10, but my recent investigations don't suggest it is the case.
Thanks to the additional GNSS data being logged by my copy of APPro, I can see that lat + lon + speed are only being refreshed once per second. I know this because the timing issues affect the core Garmin functionality and Connect IQ apps in different ways at any point in time. I sometimes see freezes in only one or the other (more commonly in the Garmin core), and big jumps or changes in lag between the two across FIT records and whole second boundaries.
Throughout all of that the lat + lon + speed data never varies between the Garmin core and the Connect IQ app (APPro). They are sometimes offset by a second when the Garmin timing goes awry, but never different values. It is of course possible the Garmin have set up 2 / 5 / 10 Hz processing but that would mean consuming all of the NMEA but only using every nth sample. That in itself would be quite a challenge because the various NMEA messages are expected to start and stop when going in and out of tunnels, etc.
The Garmin timestamps in the FIT records don't come from GNSS. The watch simply writes a record to the FIT once every second using whatever data was last provided by the core Garmin code, or the Connect IQ app / datafield. So far as I can tell the FIT timestamp only ever increase, but will sometimes skip a second.
All that aside, I guess regardless of why it's happening at the GNSS level there are regularly differences between 2 watches that are somewhat larger than the typical 0.1 - 0.3 knots. I've said it before but sometimes this can be up to 0.7 - 0.8 kts windsurfing, and even up to 1.5 kts when winging. The true board speed is likely to be somewhere inbetween the two results, but not 100% guaranteed.
After developing a gut feeling for the differences that I see on the water, I decided to produce the histograms to look more closely at the distributions and that's what were posted above. They aren't a measure of the inherent precision, just the differences you might see when wearing two near-identical watches on the water. Peter's test in a car showed much smaller differences.
I just sent more FR965/Motion files for my session this afternoon (to both GPSTC and Mike). Motion on left bicep, FR965 on left wrist. FR965 running tbwonders datafield. KA72 stats for each:
Motion:2 Second Peak (kts):31.76
5x10 Average (kts):30.602
Top 5 5x10 speeds: (1)31.142
Top 5 5x10 speeds: (2)30.923
Top 5 5x10 speeds: (3)30.466
Top 5 5x10 speeds: (4)30.255
Top 5 5x10 speeds: (5)30.225
1 Hr (kts):14.689
Alpha 500 (kts):20.272
Nautical Mile (kts):23.856
100m peak (kts):31.338
Total Distance (km):60.712
FR965:
2 Second Peak (kts):31.912
5x10 Average (kts):30.532
Top 5 5x10 speeds: (1)31.001
Top 5 5x10 speeds: (2)30.984
Top 5 5x10 speeds: (3)30.408
Top 5 5x10 speeds: (4)30.155
Top 5 5x10 speeds: (5)30.112
1 Hr (kts):14.7
Alpha 500 (kts):20.134
Nautical Mile (kts):23.314
100m peak (kts):31.312
Total Distance (km):60.836
Select to expand quoteK888 said..
The Airoha has a setting where you specify the fix rate in milliseconds (100-1000, default = 1000), and a setting where you set the NMEA output rate for once every N fixes.
The u-blox chips have the option you describe (which the ESP logger uses when logging detailed satellite information), but it also has the option to specify measurement rate and navigation solution rate separately. It would be perfectly possible to measure the GPS signal at 10 hz, but get a navigation solution only every 10th measurement. The documentation is not totally clear, however, if the navigation solution would then use all 10 measurements ("averaging"), or if it would just use one measurement, and simply ignore the other 9 ("sub-sampling"). What the documentation does state is that if raw data are written, they would be written at the measurement rate. Theoretically, this would allow raw data rates that are higher than the rate at which the GPS can calculate solutions, although that would require post-processing of the raw data. My interpretation of the u-blox documentation is that using a lower navigation rate would follow the "sub-sampling" approach. I did a few experiments with this a few years back, and the results supported the conclusion (no indication of improved or smoothed results when measuring at higher rates for a given solution rate).
I have no clue what the Airoha chips do, but I'd find it surprising of they'd actually use all measurements or navigation solutions when set to output only once every N fixes. The "pillow drop" data, where some drops were completely missed if they fell in between measurements, indicate that the Garmin data are definitely not derived this way. Would be nice, but does not seem to be the case.
Select to expand quoteboardsurfr said..
I did a few experiments with this a few years back, and the results supported the conclusion (no indication of improved or smoothed results when measuring at higher rates for a given solution rate).
That is a test that I've wanted to see done for quite some time, so great to hear the result. When I asked Julien about it some time back, he thought the result would be as you've reported.Select to expand quoteboardsurfr said..
I have no clue what the Airoha chips do, but I'd find it surprising of they'd actually use all measurements or navigation solutions when set to output only once every N fixes. The "pillow drop" data, where some drops were completely missed if they fell in between measurements, indicate that the Garmin data are definitely not derived this way. Would be nice, but does not seem to be the case.
That's consistent with all of the aliasing tests that I've done on the Garmin watches. It looks like the nav solutions are reporting near-instantaneous speeds (relating to just a few 10's of milliseconds) and ignoring the rest of the second.
Aliasing used to be evident on the Airoha-based COROS watches, but disappeared in 2024 in the working firmwares that are very close to the motion. I half wonder whether they are consuming 10 Hz data and then applying their own low-pass filter. Alyernatively, there might also be a setting in the Airoha (although not in the docs I have read) and perhaps worth noting the Garmin watches using the MT3333 didn't exhibit any aliasing.
It goes without saying that once aliasing has occurred, it's hard / impossible to recover the true data. Fortunately, I don't think that aliasing is a show-stopper, just a minor annoyance.
