My understanding of the current GPSTC device approval procedure is that the device under test is simply compared to an already approved device ideally whilst sailing. The logs are then studied by the advisory panel. As far as I know this is a fairly qualitative study. It works ok as the panel is made up of people with a great deal of experience. This system has been fine as there has only been a hand full of devices submitted for test, however I think the advisory panel should move towards a quantitative statement of what the accuracy requirement is for GPSTC.

Select to expand quote

tbwonder said..
>>>>I think the advisory panel should move towards a quantitative statement of what the accuracy requirement is for GPSTC.

First you have to define accuracy, that's rather complicated.

...and the device MUST collect and report "precision" numbers, such as HDOP and such. My Timex and Garmin watches do not. Their results are very close to those of the GW-60, but you can't prove that without the numbers.

Select to expand quote

segler said..
...and the device MUST collect and report "precision" numbers, such as HDOP and such. My Timex and Garmin watches do not. Their results are very close to those of the GW-60, but you can't prove that without the numbers.

Exactly!
But, HDOP is pretty useless for validating Dopper speed.

Select to expand quote

decrepit said..
OK, Here's one of mine from 2018, gw52 and DIYV3 at 10hz, I think both on my head.

Files and colors:
Liptons GW52 25 oct.sbp - blue
Liptons V3 25 oct.ubx - red


So I think this supports what Peter is saying, the red ubx is more even, the blue sbp is much more varied.

So the average values of both are similar, it does look like the ubx numbers have some sort of smoothing going on.

I would be extremely surprised if any filtering or smoothing is being applied to the sAcc data. That just does not make sense in the light of what sort of data it is. But it is entirely possible that the scaling range could be different.

I think it's likely that the data is less variable because it is multi GNSS with 2 to 3 times the number of sats being used compared with the GPS only data, but, as mentioned, there may be a scaling difference as well.

I have never paid much attention to the data outside the actual runs when comparing the accuracy of devices, although it is important to keep a bit of an eye on it to see other behaviours. Without studying this particular file I am guessing that most of those high spikes in the GW data are during manoeuvres or stops where satellite reception is more compromised for the GPS only device. The graph I posted shows only the run, during which the two devices have relative stable sky view, and the error was in the same order of magnitude. That is, well under the 0.5Kts per individual data point.

It seems fundamental that SDOP and sAcc would be calculated on the same principles. And in fact, the same basic principle as HDOP but looking at the variation in Doppler shift instead of Position. The more satellites you have for your calculation, the greater the certainty of your position. It should be the same with Doppler speed. More satellites added to the equation would add higher resolution and less variablility. The loss of one or two satellites should have far less affect on the calculation if you have 18-20 sats compared with 6 or 7 sats. It's likely that the calculation may take into account the constellation geometry as well. With more satellites, you will likely have at better core number of satellites in the better geometry. these things could easily partly explain the less variable error observation.

Of course, it is also possible that there is an integer in the calculation that compresses the scale of the sAcc so that the SDOP range is expanded in comparison by that alone. That may explain why SDOP sometimes goes lower and well as higher. Perhaps the run 5 graph below is evidence of this where the GW-52's were using 11 satellites (pretty rarely seen by me with GPS system alone!) Run 1 shows a similar difference as well with 10 sats. GPS satellite availability may also explain differences between Mikes results and mine in this session.

So, just looking at a run, this graph with the GW-60, shows a very different range of SDOP and characteristic from the 2 x GW-52's. Since they use the same base engine, this is most likely because of the different antenna size and orientation. That is the point I was making. And this is something I saw in almost all the on the water comparisons I did.

Red and blue lines are the two GW-52's in helmet. Green line is the GW-60 on wrist. All the other runs in this session showed the same pattern.


And in comparison, here are the same two GW-52's compared with the Motion in that same session. Not identical error level, but very low and similar.

Motion LCD USB on arm: Blue
GW-52's in helmet: Red and Green

run #1:


Run #2

error numbers very close but slightly more variability in GW's



Run #3

Again, very similar number level, but more variability in GW's


run #4

Like run #1 with GW's slightly lower, but they are all very low.




Run #5

11 GPS satellites! That must have been a very good GPS visibility day!




Note that the 5 x 10 sec average one the 3 devices has a max range of 0.02 Knots! Roughly half of that the max error prediction is.

All the 10 sec runs are well within the error range by a similar magnitude.

When comparing devices, this is the main thing we are looking for.

Select to expand quote

segler said..
...and the device MUST collect and report "precision" numbers, such as HDOP and such. My Timex and Garmin watches do not. Their results are very close to those of the GW-60, but you can't prove that without the numbers.

Accuracy and precision are different things.

Is it fair to say that Tom Chalko wrote a paper on the accuracy of Doppler GPS units with self reporting speed error in conjunction with Locosys in the noughties. This paper is the basis of the approved GPS unit standard for GPSTC. All other approved GPS units have been measured against this standard.

Select to expand quote

sailquik said.. >>> I would be extremely surprised if any filtering or smoothing is being applied to the sAcc data. That just does not make sense in the light of what sort of data it is. >>>>>>

Well seems you didn't read Peter's post very well.
He shows an example of a sudden crash, where the sbp file SDoP jumps up very quickly, but the sAcc data from a ubx file rises a lot slower, and the speed numbers are repeated for several readings. So in case you missed it, here it is again. Numbers identically repeated, (horizontal line)

And a slow rise time, interestingly there's no filter on the drop, I guess because when recption comes good you want to report it instantly.

Peter explains this, by saying that SDoP was developed specifically for windsurfing, where a clear sky view is expected. But sAcc was developed for motor racing, where a constant clear sky view isn't expected.
So the log isn't interrupted when cars go under a bridge, or heavy overhanging trees etc, sAcc is filtered to give it a slow rise time, and missing data is filled in from the previous numbers. So this means for short times, the sAcc numbers can be understated. Looking for repeat speeds between points and a slower rise time in the sAcc data may be a way around this. But achieving that, means the need to identify ubx data that has been reformatted into another protocol

I read it. I just think it highly unlikely.

"One swallow, does not a summer make."

But it may turn out to be......... More info needed I think. I will see what I can find out from Ublox.

And I think Julien suggested there was a particular fw mod that affected this particular case?

Select to expand quote

sailquik said..
I read it. I just think it highly unlikely. "One swallow, does not a summer make." >>>

So I went looking for more examples, and you may well be correct, Here's the first comparison I found.
red DIY2 ubx blue GW60



So the GW60 goes higher than DIY2, but the slope is very similar.
OK here's another one, this is typical of a few I've found, looks very similar to above.
Trouble is the watch goes completely underwater but the DIY on my head, doesn't.

Maybe I'll do some tests with a GW52 and DIY2, mount them together on a screened plate, then flip them over a few times. So they both get good reception, then none, at the same time.

Select to expand quote

decrepit said..
Maybe I'll do some tests with a GW52 and DIY2, mount them together on a screened plate, then flip them over a few times. So they both get good reception, then none, at the same time.

I think that is a good idea Mike. I was thinking to do the same with the Motions and GW-52.

Too hot to do anything at the moment, I'll wait till it cools off a bit

Select to expand quote

decrepit said..
Too hot to do anything at the moment, I'll wait till it cools off a bit

Nor really any decent wind here. But at leasts it comfortably mild.

Here is an interesting example from Nic foiling today.


I believe the crash happened at the vertical red line. This was a breeching foil crash at a relatively low speed, 15 kts. I know this was the position because Nic drifted some considerable position downwind whilst sitting on the board afterwards. It is clear looking at the error data when the problem started, however 4 seconds later the software records the 2 sec peak (yellow bar). The log continues for 14 seconds to show the incorrect speed.
This was the 2 sec peak for the session, reported at 16.7 by KA72 and Speedreader.
It is a joke that the current allowable error for 5hz and 10hz data is set at 4kts. This can easily be fixed with software changes.

I am not criticising the Motion device or the Speedreader software. The error data is not under Juliens control (as far as I know)
The problem can be fixed by adjusting the custom filters in Speedreader. But it is up to GPSTC to make sure that all permitted posting software is following the same filter rules

Select to expand quote

tbwonder said..>>>It is a joke that the current allowable error for 5hz and 10hz data is set at 4kts. This can easily be fixed with software changes.
>>

Totally agree. Peter is looking at dropping the default filters to 2kts, but even that is probably high for the motion.
I think this all came about, when Manfred adjusted GPSResults to 4kts, or was it 6? To allow for the bad data from the GW60 in an underhand gybe.
Unfortunately we still have that, although I can't find an example over 2.
We could easily set 10hz data to 1. But getting all software developers to agree is a problem.

Select to expand quote

tbwonder said..
Here is an interesting example from Nic foiling today.


I believe the crash happened at the vertical red line. This was a breeching foil crash at a relatively low speed, 15 kts. I know this was the position because Nic drifted some considerable position downwind whilst sitting on the board afterwards. It is clear looking at the error data when the problem started, however 4 seconds later the software records the 2 sec peak (yellow bar). The log continues for 14 seconds to show the incorrect speed.
This was the 2 sec peak for the session, reported at 16.7 by KA72 and Speedreader.
It is a joke that the current allowable error for 5hz and 10hz data is set at 4kts. This can easily be fixed with software changes.

I am not criticising the Motion device or the Speedreader software. The error data is not under Juliens control (as far as I know)
The problem can be fixed by adjusting the custom filters in Speedreader. But it is up to GPSTC to make sure that all permitted posting software is following the same filter rules

That's a very typical example. I see these a lot in foil sessions (because I crash more often than when windsurfing), but have seen examples in windsurfing sessions, too.

In GW-60 data, similar examples can be found, but are very rare. It's much more common to see the SDoP values go up very rapidly in a crash - but the often still stay just below the 4 knot threshold. The bigger difference is in the number of satellites tracked. In Locosys data, that number drops very quickly, often from one point to the next. In u-blox data, the drop often is more gradual, and can go over several seconds. That means that the GW-60 "crash spikes" are much more likely to get filtered out than Motion spikes - but by the satellite filter, not the accuracy filter.

When I initially wrote GPS Speedreader, I pretty much had to use the filter thresholds and methods that other software was using, with GPSResults as the "gold standard". There were a few very small deviations, for example for the minimum speed filter, but these do not affect the top speed results in the vast majority of cases.

Since Speedreader has become reasonably well known and accepted now, I can now consider doing things differently from other software if it makes more sense. But changing any default thresholds requires determining what the best threshold would be, which in turn requires examining many files from many different spots and units. I have started to gather some files to do that, and will probably release a new version of Speedreader with changed filter settings in the future. I can't say when that will be, though, since that depends on when I have time to work on this, and on what I find. I have a hunch that this will also include a higher threshold for the satellite filter for u-blox data; when a GPS like the Motion can use 2 or 3 satellite systems but gets only 5 satellites, something is definitely wrong. But again, that needs to be looked at with a sufficient amount of real-world data.

Until then, by all means feel free to change the filter thresholds to stricter values. If you're only analyzing Motion data, an SDOP threshold between 1 and 2, and a min. satellite threshold of 10-15, will make it much more likely to catch artifacts like the one you are showing, without removing good data. When you do this, you may want to display the "Filter" column, so that you can see which filters are used for which points ("+" stands for SDOP (+/-), "S" for satellites, "V" for velocity (min. speed), "A" for acceleration).

Select to expand quote

tbwonder said..
Here is an interesting example from Nic foiling today.


I believe the crash happened at the vertical red line. This was a breeching foil crash at a relatively low speed, 15 kts. I know this was the position because Nic drifted some considerable position downwind whilst sitting on the board afterwards. It is clear looking at the error data when the problem started, however 4 seconds later the software records the 2 sec peak (yellow bar). The log continues for 14 seconds to show the incorrect speed.
This was the 2 sec peak for the session, reported at 16.7 by KA72 and Speedreader.
It is a joke that the current allowable error for 5hz and 10hz data is set at 4kts. This can easily be fixed with software changes.

I am not criticising the Motion device or the Speedreader software. The error data is not under Juliens control (as far as I know)
The problem can be fixed by adjusting the custom filters in Speedreader. But it is up to GPSTC to make sure that all permitted posting software is following the same filter rules

The same thing happened to me a couple of months ago. KA72 filtered the anomaly, but GPSResults didn't.

Select to expand quote

John340 said..

tbwonder said..
The same thing happened to me a couple of months ago. KA72 filtered the anomaly, but GPSResults didn't.

That's surprising, considering that ka72.com has a higher threshold for the SDoP filter. Any chance you can post a link to the file on ka72.com? I'd love to look at it.

Remember KA72's oao speed accuracy import issue highlighted in another topic, so the threshold for oao back then was 0.4 instead of 4.0.

Select to expand quote

JulienLe said..
Remember KA72's oao speed accuracy import issue highlighted in another topic, so the threshold for oao back then was 0.4 instead of 4.0.

You are right, the bug in the ka72.com reading may have been the reason for what John340 saw.

Your numbers are a bit off, though. According to the data that Dylan posted in the other thread, ka72.com uses an SDOP threshold of 3 m/s. That's meters per second, not knots, and corresponds to 5.83 knots. Which means that the accuracy values from GT-31s can never trigger the filters, since the maximum SDOP value in GT-31 data is 4.95 knots.

The effective accuracy threshold for data in .oao files was 0.583 knots until the issue was fixed due to a bug that read the error estimates 10 x higher. That has been fixed, but it seems that the SDOP filter threshold on ka72.com is still at 5.8 knots, which is too high.

This is a very good example of why GPSTC does not allow posting from Phones or other non "Dopplar-Error" devices:

3 Motions all agreed well within the reported error margins. GPS-Logit, using the Android Phones internal GPS, was more than 1 knot out on the two best 2 second runs!!!

There would have been no way to tell how wrong Logit was if not for the other devices worn. The satellite numbers were high for GPS unit (8 sats) and, HDoP was 0.2, which does not indicate any problems.

And yes! I use this phone set up this way in every session and most of the time it agrees pretty closely with the Motions, or the GW's I used before. This means NOTHING when you get just one session like this where it s WAY off. I have seen it being in error a few times in the 0.5Kt range, but not this big and not on the best two runs in a single session!

Normally, even when Logit is 0.3 to 0.5Kt out, it really does not make much difference as the on the water feedback, and especially the speed talk is still extremely useful. But in this session I got all excited when I felt I had snagged good gusts and was hearing the speed talk saying '41, 41, 41' for a few consecutive seconds. It wasn't until I got back the the beach and looked at the LCD Motion display on my arm that I got a bit disappointed. The mini's in my helmet confirmed it when I downloaded the data.


Mini #30 v's logit:



Mini's #28 and #30 v's Logit(green)



The 3 Motions superimposed. (Green LCD on Arm)

Select to expand quote

sailquik said..
This is a very good example of why GPSTC does not allow posting from Phones or other non "Dopplar-Error" devices

Well, here's an example of one Motion giving results way outside the estimated accuracy range when compared to three other Motions:

Here's a closer look at the second fastest 10 second run:

The first Motion (in blue) has a speed that's about 0.7 knots lower than the three others, despite tracking 15-16 satellites. The error estimate for this unit, while higher than for the others, always stayed below 1 knot, so it never got anywhere close to triggering the accuracy filters.

This is one of two examples with large errors that I noticed by chance in about 50 Motion files I downloaded from ka72.com. They are from different units and different people sailing at different spots. That is a surprisingly high rate of problems.

In this particular example, Gaussian error estimates clearly are not seem appropriate. The observed error is close to the average sAcc value of 0.671, and more than 5-fold higher than the estimate that assumes errors are random. This can be seen quite often in less-than-perfect u-blox data. It means that going to ever-higher data rates will not improve accuracy. Sure, the +/- numbers given for 10 seconds etc. by GPSResults will keep going down, but they will become more and more meaningless. A comparison of the ranges for nautical miles already often shows that these accuracy estimates are already to optimistic.

I can't force 2683/blue's user to remove it from its pouch. I tried. He has his reasons.

I found the files and I wonder if 2683/blue being an outlier is that subtle a filter couldn't catch it:

To put some perspective in this, the difference between his three correct devices on the 5 best 10s are:

0.02kn
0.03kn
0.01kn
0.01kn
0.01kn

And on a competition-used average, 500m, differences are:

0.01kn
0.01kn
0.01kn
0.01kn
0.01kn

Select to expand quote

boardsurfr said..
Well, here's an example of one Motion giving results way outside the estimated accuracy range when compared to three other Motions:

I dont suppose you thought to wonder why this sailor was wearing, comparing and uploading 4 motions in one session to the KA72 page, or god forbid, even ask him??

This particular unit is one we are very suspicious of, and has been under study and review for some months. The owner flagged it immediately and has been sharing session studies with me all of that time. No sessions have been actually posted to GPSTC from this unit after it was flagged if they deviated from the mini's worn at the same time. (There has always been at least one control unit)

It might be more productive and useful if you actually talked directly to the people involved in a cooperative and constructive way when you go on trawling expeditions, find things that bother you and then get online and cast dispersions.

Do you think it is ethical to hack the KA72 website to download files that are not made public by their owner, and them publish them on this forum?

In any case, testing and comparison if that unit with the latest firmware is now looking consistent with the other devices we are comparing it with.

Select to expand quote

JulienLe said..
To put some perspective in this, the difference between his three correct devices on the 5 best 10s are:

0.02kn
0.03kn
0.01kn
0.01kn
0.01kn

And on a competition-used average, 500m, differences are:

0.01kn
0.01kn
0.01kn
0.01kn
0.01kn

The motion is great and highly regarded, we just need more screen motions available so we can all ditch the other lesser options.
Many of us have our fingers crossed 2022 will see this happen.

Select to expand quote

JulienLe said..
To put some perspective in this, the difference between his three correct devices on the 5 best 10s are:

0.02kn
0.03kn
0.01kn
0.01kn
0.01kn

And on a competition-used average, 500m, differences are:

0.01kn
0.01kn
0.01kn
0.01kn
0.01kn

And to emphasise that perspective: 0.01 knots, said another way, is 1 hundredth of a Knot!!!

Are even the Camera Gates used in 500m WSSRC record attempts and competitions capable of a better order of accuracy?

Can some mathematical prodigy please work out the distance difference for us of 1 hundredth of a knot traveling at at 40 and 50 knots?

It is true that using the least squares error calculation, we sometimes see two side by side 10Hz Motion/Ublox based devices giving results for the NM that fall outside the reported error margin by a few thousandths of a knot! I admit this bothered me at first as it could indicate that there was actually some random error in the data that was not accounted for by the sAcc computation.

Two things eased my mind:

1. The difference in the speed calculation was very very small. In the order of 5-10 thousandths of a Knot. If this was an indication of some unaccounted for 'random' error, then it is still very small and insignificant for our results anyhow.

2. In a recent conversation with Mal Wright, he pointed out to me that this may well easily be accounted for by rounding in the data . The mention of that bought back a lot of distant memories of this very thing we struggled with when we first started using positional data from the original Garmin Foretrex and Etrex devices around 2003-2005. Those devices could only report position to a limited number of decimal places. This in effect meant that when you zoomed right in the meter level on a positional speed track, we found that each point was on the corner of a grid with dimension in the order of (from memory approximately) 1.6m by 2.4m at our latitude (38?S) This 'Grid Effect' as I labeled it, added speed to the calculated speeds by making the plotted speed path slightly zig-Zagged. (although to be fair, that component of error was not of great significance compared with other inherent errors of using positional data to calculate speed). i cant easily find an example of that for the Garmin Devices, but I found this one on this computer from a 5Hz chip we tested some time later. In this example, the dotted grid lines are 1m. The grid effect here, from a geostationary test, is in the order of 20cm's.


When the GT-11 came along, we gained something like double the number decimal places in the positional calculation and the 'Grid Effect' diminished to the order of cm, or sub cm better than the illustration above, and into relative insignificance.
So this is a long winded way to illustrate 'rounding'. Mal reasons that, in all the thousands of computer calculations that go into every 1 or 2 tenths of a second data point, there is bound to be somewhere, some limited resolution (limited number of decimal places) in each figure used, which necessarily means that there is a rounding error that can accumulate. Of course, these rounding errors themselves can tend to cancel each other out over time, but there remains the possibility of a certain variable amount of rounding error.


Two interesting observations from my recent session illustrated here:




This NM comparison is actually well within the least squares calculated error margins of 15/1000ths Knots and 18/1000ths Knots. The difference between the two results is only 6/1000ths Knots. For all our practical purposes, that is an absolutely insignificant difference, and I might add, it shows just how potentially accurate this device can be.

But then I noticed the 1Hr results. The difference between the two calculations is 0.001 Kts. One thousandth of a Knot.
The calculated error margin is reported in the software as 0.003 Kts. But this seems to be the limit of what can be reported by this system. I cant remember ever seeing a calculated error number below 0.003 Knots. That seems to be the limit of the resolution available to us.
I hasten to add that I have not always seen 1Hr side by side results this small. You may find some, but I would be surprised if they exceed 100th of a knot. I'll look through a few more sessions to see if I can find some.

And just for interest. Have a gander at the distance calculation in this example. Over 70.96 KM, there is only 3 meters difference. That one actually surprised me.

I'm not going to give any specific conclusions from these observations, except to say that even using least squares error calculation and sometimes seeing a difference between the NM results outside the error difference margin, the magnitude of that difference is so small as to be of insignificant consequence to us. And does not seem to support an argument that there is something inherently wrong with using the least squares error calculation.

The important thing is to get low error numbers for the individual data points

If my maths is correct-
.01knot= .00514 m/sec= app 5mm/sec
That's app 1cm in a 2sec run.
( 1knot= app .5 m/sec)

Select to expand quote

mikey100 said..
If my maths is correct-
.01knot= .00514 m/sec= app 5mm/sec
That's app 1cm in a 2sec run.
( 1knot= app .5 m/sec)

So 500m over 20 seconds (high 40's), 0,01 Knots would be about 125mm difference at the end? Hmmm? Slightly more than I anticipated. I think the camera gates could possibly resolve that difference? My understanding is that they can now run at either 50 or possibly 100 frames/sec, which should enable resolution of 0.02 or 0.01 sec respectively. And interpolation may improve that a bit.

My head hurts.