Some on-the-water data (from a foil session) show the same characteristics as the test drive:

The actual observed differences in 2 and 10 second speeds is below 0.09 knots, and typically below 0.05 knots. Error estimates for the u-blox data fall into a very narrow range, while the Locosys data show large variations.
Only the SDoP values for the Locosys unit show a correlation to speed, with higher error estimates at higher speed. It's very obvious that the sAcc values for the u-blox GPS are calculated using a rather different method.

Select to expand quote

boardsurfr said..

sailquik said..
Tell me if I am wrong, but in the example you gave, I am assuming that the GW-60 watch was worn on the wrist and was subject to the usual poor orientation to the Sky as they are almost always are when worn this way.

Yes indeed, you are wrong. The example I gave was from driving test, with all antennas oriented towards the sky. One of the Locosys units was a GW-60, the other one a GW-52. Both show the same characteristics.

All I can say is that your 'test' in the car, shows markedly different characteristics from my on the water test. I have years of on the water data from two GW-52's side by side and they show nothing like the results you are reporting or the type of error results I get from actual on the water runs from the GW-60 (which is all over the place)

OK, Here's one of mine from 2018, gw52 and DIYV3 at 10hz, I think both on my head.

Files and colors:
Liptons GW52 25 oct.sbp - blue
Liptons V3 25 oct.ubx - red


So I think this supports what Peter is saying, the red ubx is more even, the blue sbp is much more varied.

So the average values of both are similar, it does look like the ubx numbers have some sort of smoothing going on.

Sailquik, boardsurfr & JulienLe. A question that has been bugging me for a while. Why isn't an available standard (on the net) used to characterise the accuracy/stability of the GPS loggers we use for our sport ?. The JCGM 100:2008 standard is for characterising standard uncertainties of test equipment (eg RF network analysers, multimeters, vernier calipers, strain gauges). I think the GPS loggers we use would be a good fit to this standard. My knowledge of this standard is that is it possible to compare oranges and apples, because in metrology, it's not possible to control product conformity (nobody tells our business - 'you have to use XYZ brand multimeter !'. We use whatever we want, as long as it has a standard uncertainty spec). For example (I'm bum plucking numbers here), a GW-60 watch could be characterised as having a +/- 1 knot 95% confidence interval in accordance with this standard (ie 95% confidence that the true value falls between +/-1 knot of the mean). A Garmin Felix 6 Pro may have a +/-3 knot confidence interval. So how you could make these two products co-exist in our sport is to take the worse case scenario: If a GW60 rider logged 30kt, you subtract 1kt from their result (29kt peak). For the Garmin rider who also logged 30kt, 3kt is subtracted (27kt peak). Orange and apple can now be compared.

My understanding is you are taking error data from various GPS receivers that are closed architectures ? (ie you don't have access to the algorithms or the input data to verify these error calcs ?). How do you know these receivers are calculating their error results properly ?

I know this could be a sore topic. My fingers are barleese

I care about two things: proximity to video timing/RTK and repeatability.

So one Motion user can trust his training runs before an official event.
So one Motion user can trust another Motion user that the difference between them isn't caused by the devices.

The rest is outside my reach.

Jn,
The problem is the way the units are used, I think inherent accuracy differences are swamped by external factors, centred around satellite reception. there's so many ways, signal can be degraded, and adversely affect gps accuracy. That's why we rely on SDoP numbers, that is the GPSs' calculation of inaccuracy due to external factors.

So applying an inherent accuracy number to individual units becomes a bit meaningless.

Select to expand quote

decrepit said..
The problem is the way the units are used, I think inherent accuracy differences are swamped by external factors, centred around satellite reception. there's so many ways, signal can be degraded, and adversely affect gps accuracy. That's why we rely on SDoP numbers, that is the GPSs' calculation of inaccuracy due to external factors.

So applying an inherent accuracy number to individual units becomes a bit meaningless.

I certainly agree with the first part - that external factors can be the major determinant of actual accuracy while windsurfing. But how GPS units react to external factors is actually anintrinsicproperty of the GPS units, and therefore could be characterized.

Select to expand quote

jn1 said..
Why isn't an available standard (on the net) used to characterise the accuracy/stability of the GPS loggers we use for our sport?

Supposedly, there is a standard that the GPSTC uses to decide if a device should be approved. You will have to ask sailquik why this standard is not available on the net.

Select to expand quote

jn1 said..
My understanding is you are taking error data from various GPS receivers that are closed architectures ? (ie you don't have access to the algorithms or the input data to verify these error calcs ?)

Correct. Speed error calculation is done in the GPS firmware, which in intellectual property of the companies making the units. This is different from other accuracy measures that GPS units can give, like HDoP or VDoP, for which the formulas are public and can easily be found. AFAIK, the only GPS chip manufacturer who offers speed accuracy estimates (called sAcc) on all their GPS chips is u-blox. The only other chip company that offers speed accuracy estimates (called SDoP) on some of their units is Locosys.

Select to expand quote

jn1 said..
How do you know these receivers are calculating their error results properly ?

A very good question.

Originally, the accuracy of the Locosys SDoP numbers was validated by Tom Chalko, who published a paper with his methods and results. The paper was sufficiently detailed that it is easy to reproduce what he did. From what I understand, Tom and (or?) others were working closely with Locosys in the development of the SDoP numbers, specifically for speedsurfing.

For u-blox GPS units, I am quite certain that Manfred Fuchs has done a lot of validation a decade or two ago, but that was never made publicly available. One important difference for u-blox units is that they do not contain any code specifically developed for speedsurfing. The primary market for u-blox GPS chips is the automotive sector, which has quite different accuracy requirements. The most relevant is that car speeds usually change relatively slowly, but impaired reception happens quite often - under bridges, next to sky scrapers, and so on. Often, these are short changes - you get good reception again once you pass the bridge. Slow speed changes and short periods of poor reception means that there is no need for sAcc values to change quickly for car navigation, and it appears that u-blox units filter sAcc values quite heavily, restricting how fast the error estimates can rise. Here's an example from a crash:

Note that when the GPS looses reception (satellites go to 0), it freezes the speed at the previous speed, and the +/- numbers increase very slowly. That's quite reasonable when driving in a car, but quite wrong in windsurfing crash.
In this example, the error estimate rises quite slowly: from the first lost of satellite reception, it takes 12.9 seconds to reach a maximum of 3.285 knots. Note that this is still below the default filter threshold of 4.0 knots, so it never triggers the accuracy filter! The only filter triggered in this region is the satellite filter (which probably should use a higher threshold than 5 for Motion data).

For comparison, here's how a crash looks like with a Locosys GW-60:
Note that the error estimates jump from 1.963 to 3.998 within 0.2 seconds as soon as the GPS uses satellite reception. Interestingly, the SDoP value does not rise to the maximum of 4.957 knots, but remains just below 4 knots, where it also does not trigger the SDoP filter.

These frames have an invalid fix field and should be ignored. Why uBlox decided to keep gibberish in the frame, I don't know.

Select to expand quote

boardsurfr said..
Supposedly, there is a standard that the GPSTC uses to decide if a device should be approved. You will have to ask sailquik why this standard is not available on the net.

This is a very disingenuous statement Peter, and you know it!

The is no 'supposedly' about it, because you know very well that the GPS-TC has a policy procedure document for this, as it was sent to you, and it contents discussed with you, when you asked for it during the building of your home made ublox GPS devices. And those devices were subsequently approved for use in the GPS-TC as formulated in that Policy.

And it has been sent to every other interested party who had asked for it who was building, or proposing to build a Ublox based GPS device. .

Anyone who has ever read anything on these forums about GPSTC approved devices would know that only devices that produce Doppler error values are considered for approval. And the only GNSS engine readily available at consumer level friendly prices at the moment that produce this data are the Ublox based chipsets.

You have highlighted some very important points that we have been discussing for some time about the usefulness of the current agreed filter values between the software analysis authors. The advent of the Motion, and other ublox based devices has changed the situation for sure, as you have also rightly pointed out.

Select to expand quote

decrepit said..
Jn,
The problem is the way the units are used, I think inherent accuracy differences are swamped by external factors, centred around satellite reception. there's so many ways, signal can be degraded, and adversely affect gps accuracy. That's why we rely on SDoP numbers, that is the GPSs' calculation of inaccuracy due to external factors.

So applying an inherent accuracy number to individual units becomes a bit meaningless.

So what you're saying is if uncertainty estimates were used, they would be so ** (due to poor sat reception), that they would be useless ?. Ok, fair comment.

Still, it would be interesting to calculate the U estimates for a GW60 and a Garmin Felex watches using this standard under worst case scenarios (eg watch antenna pointing down at the water).

I guess the key difference in mindset is: the GPSTC/KA72 throws away bad data. Where the standard I mentioned above lives/coexists with bad data.

Select to expand quote

jn1 said..

decrepit said..
Jn,
The problem is the way the units are used, I think inherent accuracy differences are swamped by external factors, centred around satellite reception. there's so many ways, signal can be degraded, and adversely affect gps accuracy. That's why we rely on SDoP numbers, that is the GPSs' calculation of inaccuracy due to external factors.

So applying an inherent accuracy number to individual units becomes a bit meaningless.

So what you're saying is if uncertainty estimates were used, they would be so ** (due to poor sat reception), that they would be useless ?. Ok, fair comment.

Still, it would be interesting to calculate the U estimates for a GW60 and a Garmin Felex watches using this standard under worst case scenarios (eg watch antenna pointing down at the water).

I guess the key difference in mindset is: the GPSTC/KA72 throws away bad data. Where the standard I mentioned above lives/coexists with bad data.

Yes. the existence and inclusion of the Doppler error data is what actually enables us to quantify the error. This firstly enables us to assess the consistent accuracy of a device, and then in use, to flag bad data and to reject it. How you do it with something that does not tell you this, has not been practically shown.

Yes, your last comment hits the nail right on the head. The decision was made long ago, that a competition with random errors, (and often large errors at that), was not going to be fair, (or even very interesting really).

In an ideal world, we would also subtract the reported error from the speeds. This was in fact done of the WGSPRRC Records, (for a claimed speed with which we could be 99.9% certain has been AT LEAST achieved.) But it was, and still is considered to complex for the GPSTC.

Select to expand quote

boardsurfr said..jn1 said..

How do you know these receivers are calculating their error results properly ?

Note that the error estimates jump from 1.963 to 3.998 within 0.2 seconds as soon as the GPS uses satellite reception. Interestingly, the SDoP value does not rise to the maximum of 4.957 knots, but remains just below 4 knots, where it also does not trigger the SDoP filter.

Hence oranges and apples ?

Select to expand quote

jn1 said..

boardsurfr said..jn1 said..

How do you know these receivers are calculating their error results properly ?

Note that the error estimates jump from 1.963 to 3.998 within 0.2 seconds as soon as the GPS uses satellite reception. Interestingly, the SDoP value does not rise to the maximum of 4.957 knots, but remains just below 4 knots, where it also does not trigger the SDoP filter.

Hence oranges and apples ?

What is "properly"?

We know they calculate error consistently, and can show that when it is reported as higher, the speeds are less accurate (side by side units differ by more)

As we are learning more about the Doppler error reporting characteristics of the Ublox based, multi-GNSS devices, and seeing more anomalies like those reported by Boardsurfr, we are seeing that some adjustments may have to be made to our filtering to better deal with bad data on those particular devices. Learning and development is a continuum.

Select to expand quote

sailquik said..

Yes, your last comment hits the nail right on the head. The decision was made long ago, that a competition with random errors, (and often large errors at that), was not going to be fair, (or even very interesting really).

GPSTC is in a unique situation that it can regulate measurement equipment. Many professions/industries don't have this luxury

I hate this. The post is based on erronous premises and now it will linger here forever sowing doubt.

Now, why is Sirf's SDOP all over the place in decrepit's example? If we exclude the periods at rest, he's cruising along nicely with both devices on his head and without any obstacles in sight. So why the jitter?

And why, in these very nice conditions, are you surprised uBlox's is stable at what could be an accuracy floor but not surprised by Sirf's jitter?

There's no comparisons to be done between the two. They are different results out of different computations. Likely in different units. I have absolutely no idea how Sirf does its thing. There's no documentation available and Chalko doesn't mention internals.

Select to expand quote

sailquik said..

boardsurfr said..
Supposedly, there is a standard that the GPSTC uses to decide if a device should be approved. You will have to ask sailquik why this standard is not available on the net.

This is a very disingenuous statement Peter, and you know it!

The is no 'supposedly' about it, because you know very well that the GPS-TC has a policy procedure document for this, as it was sent to you, and it contents discussed with you, when you asked for it during the building of your home made ublox GPS devices. And those devices were subsequently approved for use in the GPS-TC as formulated in that Policy.

And it has been sent to every other interested party who had asked for it who was building, or proposing to build a Ublox based GPS device. .

Anyone who has ever read anything on these forums about GPSTC approved devices would know that only devices that produce Doppler error values are considered for approval. And the only GNSS engine readily available at consumer level friendly prices at the moment that produce this data are the Ublox based chipsets.

You have highlighted some very important points that we have been discussing for some time about the usefulness of the current agreed filter values between the software analysis authors. The advent of the Motion, and other ublox based devices has changed the situation for sure, as you have also rightly pointed out.

You are blatantly lying. If that standard is available on the net, why do you not post the link?

Select to expand quote

sailquik said..
The is no 'supposedly' about it, because you know very well that the GPS-TC has a policy procedure document for this, as it was sent to you, and it contents discussed with you, when you asked for it during the building of your home made ublox GPS devices.

Once again, a false statement. I indeed asked what it would take to get a device approved in May 2018. The answer you sent on May 29 was:

Select to expand quote

sailquik said..
Our only task now is to work out some (preferably simple) procedure to validate other custom devices built on the ublox M7 and/or M8 gps chipsets. I have been discussing it with Decrepit.

I am thinking along these lines:

1. User gathers some data in use in side by side with an approved GPS, preferably a GW-52/60, but I think a GT-31 would probably do the job, and sends it to us. This is just to confirm the device is working as designed and has no obvious faults. Hopefully, this will enable us to pick up if the GPS chip is not a substandard fake as well.

2. Provisional approval to post on condition they still use another approved device side by side for continued validation and data study. Not sure how long this should go on for yet and will depend on if any glitches that come up I think.

3. Full approval for that particular device.

Any other devices made with the same design and components should be able to be approved after just step 1 above, especially if made by, or under the supervision of the original builder, but maybe we would have a short Provisional approval for the first few and/or if they are built by another person to the same plans.

Any device put into series production should probably only have to have a sample of the early production verified. We should be able to trust the manufacturer to maintain the quality.

We are interested in any other thoughts you have on this.

In any case, you have already pretty well met the first point with your BT M8 GPS.:-)

The next steps will be to finalise a policy and submit our recommendations to the advisory group for formal approval. I am optimistic that it should not take long. :-)

That's not a "procedure document", that is one sentence "User gathers some data in use in side by side with an approved GPS .. and sends it to us". It does not specify any details about criteria that would need to be met. Neither does the GPSTC web site provide any details. It only states "Other GPS units can be approved by the technical advisers of the challenge, including Ublox GPS chipset based, custom or home made devices."

Almost a year later, in February 2019, I did some tests of the original Motion GPS, together with Mike (decrepit) at Lake George. There still did not seem any "finalized policy" then, and Mike would have known.

Once again - if you have a document that describes the standards used in deciding whether or not to approve a GPS device, why do you not post it?

Select to expand quote

JulienLe said..
I have absolutely no idea how Sirf does its thing. There's no documentation available and Chalko doesn't mention internals.

You're attacking the Sirf, implying that it's unique in "no documentation".

There also is no public documentation available about the u-blox chips that describes how sAcc is calculated, or even what the numbers are supposed to mean. If I'm wrong and you found something, please share.

It's available under NDA.

Newer uBlox public datasheets now mention "Velocity accuracy: n.nn m/s, 50% at 30 m/s for dynamic operation".

I'm not implying that, I'm stating clearly that I don't know how Sirf does it, why I can't know and that I won't hazard some guesses.

On the other hand, there's a lot to learn from rtklib's opensource receiver.

Select to expand quote

JulienLe said..
And why, in these very nice conditions, are you surprised uBlox's is stable at what could be an accuracy floor but not surprised by Sirf's jitter?

Do not put words in my mouth. At no point did I say I was "surprised". I was describing what can be observed when taking a close look at the SDoP data, and explaining why sailquik's assertion that error estimates were "very comparable when I was comparing GW-52's with a Ublox logger" are incorrect.

I look at error estimates from two perspectives: comparing different GPS units, and calculating accurate speed results for the GPSTC.

For comparing different units, I have shown above that just comparing error estimates is not good enough, since they are obviously calculated very differently.

For accurate speed results, I have shown the the currently used "single point" accuracy filters can not catch artifacts from crashes in u-blox data, since the error estimates are heavily filtered.

I'd start by checking if the frame is valid.

Select to expand quote

JulienLe said..
These frames have an invalid fix field and should be ignored. Why uBlox decided to keep gibberish in the frame, I don't know.

Since the fix flag requires 4 satellites to be valid, and a minimum of 5 satellites are required to pass the satellite filter, records with invalid fix are ignored in all speed calculations.

The interesting thing is that the sAcc values always increase very slowly - even when satellite reception is lost completely. That raises serious doubts that it would be possible to catch speed errors from sudden problems, like crashes, using accuracy estimates in u-blox data. That's very different from Locosys data, where SDoP values can increase much more rapidly.

Please give me one use-case of this.

The first sentence of your second paragraph is a great example of you not ignoring the solution being invalid by the way.

Select to expand quote

JulienLe said..
Give me one use-case of this.

Since you asked for it: the use case is accurate GPSTC postings and PBs.

Here is one example where the SDOP/SACC filter failed because sAcc rose too slowly:


The session was posted to GPSTC (gpsteamchallenge.com.au/sailor_session/show?date=2021-12-18&team=37). The track is at www.ka72.com/Track/t/484031

This is a crash where the 2 seconds before the GPS lost all reception were picked as the top speed. Error estimates were quite high, but below the filter thresholds in GPSResults, GPS Speedreader, and ka72.

This region also affected the alpha. ka72 reported it as 16.067 knots, as does GPSResults, which would be a PB improvement of about 3 knots. Filtering with a 4.0 knot SDOP threshold reduces this to 13.498 knots (ka72 uses 3 m/s, about 5.8 knots), but it still includes the questionable 2 second region before reception loss. It also affects the top 10 second speed on ka72, which reported as 22.32 knots for the crash region, but that's a ka72.com problem.

So that's wrong results in 3 GPSTC categories when posting from ka72.com, and 2 categories when analyzed with Speedreader/GPSResults.

I found this example while looking at 18 Motion files from 12/18/2021 downloaded from ka72.com. From the small sample size, it's impossible to say how often problems like this occur, but they do occur.

Note that this a description of observations. It is not a critique of the Motion, which definitely is a very accurate GPS.

But understanding the actual nature of the data is important if we want to analyze them correctly. There are multiple ways how the filters could be improved to handle data such as this track correctly. The easiest, but perhaps not the best, is to use different thresholds for u-blox and Locosys data.

Ho my, this is junk. I remember 3034 being a special fix for Luderitz. And then I had to revert it. I'll contact the two possible owners (based on country and firmware #). Thank you.

Yes, the filters need to change anyway. Anything above 1 is junk.

I had a suspicion there might have been something wrong with this particular unit, so thanks for confirming it may have been the firmware.

I'll have to do a bit more research on filter values for different units. First glance indicates that the current GT-31 values are too strict, since the GT-31 SDoPs go up a lot in slingshots. To make the analysis robust, I'll have to gather a few hundred files from recent sessions.

This will likely result in a change of the filter thresholds, but that in turn may require other changes. For example, I can't just rely on the file format, since Motion files can be exported to Locosys formats from Speedreader and GPSResults, and exporting should not affect the speed results. So it may be a while before thing change.

This is a really good discussion. Thanks, guys.

For us duffers who just want to sail, and maybe take speed measurements that are believable, a discussion like this will help shake out the best hardware, best wearing methods, and best application of it all. So far it looks like the GW-60 has been improved upon by the Motion/ublox.

Keep going, guys. This is good stuff.

Select to expand quote

segler said.. <<<<a discussion like this will help shake out the best hardware, best wearing methods, and best application of it all. So far it looks like the GW-60 has been improved upon by the Motion/ublox.

Keep going, guys. This is good stuff.

There's no doubt about it, at the moment a motion worn on the head will give best accuracy for our purposes.

So. my Apple Watch + Waterspeed app ain't gonna cut it? (I'm honestly afraid to seriously ask!)

Just visiting this forum. Haven't read for years. You're all as hard-core as I remember! : )))

I've learnt a lot from this thread.

Select to expand quote

boardsurfr said..

sailquik said..

boardsurfr said..
Supposedly, there is a standard that the GPSTC uses to decide if a device should be approved. You will have to ask sailquik why this standard is not available on the net.

This is a very disingenuous statement Peter, and you know it!

The is no 'supposedly' about it, because you know very well that the GPS-TC has a policy procedure document for this, as it was sent to you, and it contents discussed with you, when you asked for it during the building of your home made ublox GPS devices. And those devices were subsequently approved for use in the GPS-TC as formulated in that Policy.

And it has been sent to every other interested party who had asked for it who was building, or proposing to build a Ublox based GPS device. .

Anyone who has ever read anything on these forums about GPSTC approved devices would know that only devices that produce Doppler error values are considered for approval. And the only GNSS engine readily available at consumer level friendly prices at the moment that produce this data are the Ublox based chipsets.

You have highlighted some very important points that we have been discussing for some time about the usefulness of the current agreed filter values between the software analysis authors. The advent of the Motion, and other ublox based devices has changed the situation for sure, as you have also rightly pointed out.

You are blatantly lying. If that standard is available on the net, why do you not post the link?

sailquik said..
The is no 'supposedly' about it, because you know very well that the GPS-TC has a policy procedure document for this, as it was sent to you, and it contents discussed with you, when you asked for it during the building of your home made ublox GPS devices.

Once again, a false statement. I indeed asked what it would take to get a device approved in May 2018. The answer you sent on May 29 was:

sailquik said..
Our only task now is to work out some (preferably simple) procedure to validate other custom devices built on the ublox M7 and/or M8 gps chipsets. I have been discussing it with Decrepit.

I am thinking along these lines:

1. User gathers some data in use in side by side with an approved GPS, preferably a GW-52/60, but I think a GT-31 would probably do the job, and sends it to us. This is just to confirm the device is working as designed and has no obvious faults. Hopefully, this will enable us to pick up if the GPS chip is not a substandard fake as well.

2. Provisional approval to post on condition they still use another approved device side by side for continued validation and data study. Not sure how long this should go on for yet and will depend on if any glitches that come up I think.

3. Full approval for that particular device.

Any other devices made with the same design and components should be able to be approved after just step 1 above, especially if made by, or under the supervision of the original builder, but maybe we would have a short Provisional approval for the first few and/or if they are built by another person to the same plans.

Any device put into series production should probably only have to have a sample of the early production verified. We should be able to trust the manufacturer to maintain the quality.

We are interested in any other thoughts you have on this.

In any case, you have already pretty well met the first point with your BT M8 GPS.:-)

The next steps will be to finalise a policy and submit our recommendations to the advisory group for formal approval. I am optimistic that it should not take long. :-)

That's not a "procedure document", that is one sentence "User gathers some data in use in side by side with an approved GPS .. and sends it to us". It does not specify any details about criteria that would need to be met. Neither does the GPSTC web site provide any details. It only states "Other GPS units can be approved by the technical advisers of the challenge, including Ublox GPS chipset based, custom or home made devices."

Almost a year later, in February 2019, I did some tests of the original Motion GPS, together with Mike (decrepit) at Lake George. There still did not seem any "finalized policy" then, and Mike would have known.

Once again - if you have a document that describes the standards used in deciding whether or not to approve a GPS device, why do you not post it?

Yep. I was wrong. My memory of the fine details failed me. I remember now that it was at that time that we were actually formulating the procedure and policy statement for approval by the Advisory Panel, partly as a response to your request (and a few others).

So the policy and procedure was in the making at that time but we did inform you of the requirements at the time as I stated as the document was being written.

I dont see any practical reason to have that "published on the net somewhere". It is not a set in stone document. As we learn more it will be updated. All we need to say is that it is available on request, for those very, very few people that it applies to. Adding those words on the website has not been, and is still not a trivial quick task in the case of our amateur/voluntary administered organisation, but it will probably happen sooner or late. What is there now as you quoted seems all that is practically needed.

Select to expand quote

evlPanda said..
So. my Apple Watch + Waterspeed app ain't gonna cut it? (I'm honestly afraid to seriously ask!)

Correct!