Select to expand quote

sailquik said..

remery said..
I recently did a couple sessions with a new GW60 one my wrist and Mini Motion on my bicep if that's a help. The top speeds were extremely close.

I would like o see the data files to add to my collection of research if you are happy to send them to me.

I just sent a link to a Dropbox folder with a couple of logs in. I have several others but would have to download the Motion file from KA72. I continued to wear both but only send the Motion logs. The GW60 was just for on the water reading.

Example of last two sessions with side by side Mini Motion loggers. Note that this is a random sample. I didn't pick any particular files with bias. Just the last two session.



Row by row, this is the difference between each result: (warning, there may be silly arithmatic and typo errors in my sums)
2 sec- (This and the Alpha is where we would expect the highest differences if there are to be any).
0.058
0.021
0.054
0.051
0.046

10 second
0.024
0.008
0.020
0.048
0.019

5 x 10
0.016


I hour
0.003

NM
0.010
0.002
0.003
0.005
0.001

Distance
Difference: 25meters

Alpha (Subject to higher errors due to use of positional proximity)

0.022
0.025
0.003
0.015
0.038

if you want to do the sums on this second one - go for it. but a casual glance says most of these are even closer.


If anyone is serious about doing more analysis of every session I have, let me know.

Select to expand quote

sailquik said..

User error!

Select to expand quote

sailquik said..
but I have also proven to myself that two devices worn in a good, stable, waterproof upper arm bag, positioned properly is 99% as good. 99% of sailors can do this quite easily and happily.

I still maintain, that if you have results outside the error margin from two devices worn at the same time, it is because they are not positioned side by side in the same plane and one or the other is compromised.

You seem to contradicting yourself, you said there was a "user error" by using an aquapac and then stated that a bag on the upper arm is 99% as good.
Is it ok to put the motion in an aquapac (or similar stable bag) on the upper arm or should it always be mounted using the provided strap?

Select to expand quote

decrepit said..

I'm afraid if we want the very best accuracy we have to wear at least two devices.
and in the case above maybe average them?

I like the idea of averaging results, but the GPSTC rules do not allow it. The rules state that you must post from the device with the lowest reported error.

Select to expand quote

sailquik said..
Example of last two sessions with side by side Mini Motion loggers. Note that this is a random sample. I didn't pick any particular files with bias. Just the last two session.

I agree with what you say Andrew, nearly every file comparison I look at with the Motions are extremely accurate, the Motions are without doubt the best commercially available devices we have seen. What's more, price wise they are a bargain and the after sales support is fantastic.
But they are not perfect, errors do happen. What I am trying to learn is how to pick up these very rare discrepancies by looking at the log data.

Select to expand quote

tbwonder said..

My understanding was that if the device was not getting a good view of the satellites, for what ever reason, then this would be reflected in the error data or in the number of satellites being tracked.

Julien you state that you are disappointed with the accuracy and there is noise in the satellites and spikes in the data, however most users do not have the skills to analyze this, they tend to look at the reported error figures which in these two cases are both below 0.2 kts.

If I was using a single device and the reported error was better then +_ 0.2 kts I would be happy to post it.

How can a user identify easily that there are other issues with the file?

I don't understand the above. Motion made it clear that speed accuracy was +/-0.4-0.5, sats around 14 and HDOP around 1.6 which are all poor:

Without taking the spikes on this run or in the turns into account, the average accuracy/sat/hdop of both devices were disappointing and the Aquapac makes a likely culprit. To remove a factor in the next test, it would be great if the user could upgrade both to firmware 3010 before too.

For the sake of clarity, here's a 2s made in The Netherlands a month ago, other 2s of that log are similar:

That's what I expect from a good device in good conditions.

I tested some new devices last week and HDOP reached 0.30 on some to my great pleasure. Because increasing positional accuracy and thus lowering HDOP is the whole point of using that bigass antenna in-lieu of a screen.

I don't think reception quality is equal on every location of the world, so comparing a HDOP value from a session in Australia and HDOP values from a location in Western Europe doesn't explain anything....have read some papers where in a specific situation reception of GPS data in area with a higher latitude is defined worse than location with a lower latitude....

Then here is a 2s from an Australian log made in Lake Cootharaba a month before the problematic one above, with a device built in the same batch as the device above, received from the user for reasons completely unrelated to accuracy or tests:

Select to expand quote

tbwonder said..


You seem to contradicting yourself, you said there was a "user error" by using an aquapac and then stated that a bag on the upper arm is 99% as good.
Is it ok to put the motion in an aquapac (or similar stable bag) on the upper arm or should it always be mounted using the provided strap?

Select to expand quote

sailquik said..
I still maintain, that if you have results outside the error margin from two devices worn at the same time, it is because they are not positioned side by side in the same plane and one or the other is compromised.

No contradiction.

I didnt mention a 'bag.'

I meant that they must be in the same plane, stable and facing the sky. On the upper arm works very well in this respect if the other conditions are met. Most people I have seen with the Mini Motion seem to be wearing it on the upper arm using the supplied arm strap or similar. That is excellent positioning.

If the bag is worn loose and slips around the arm, one device can easily be facing a different direction and have compromised satellite view. The devices could also move within the bag and even have one overlapping the other. I didn't even know where the bag in question was worn. Was it on the upper arm or somewhere else? Were the two GPS rigidly positioned in the same plane? etc.

But I certanly agree that I would also like to learn more of why, in this case, the error didn't show up clearly in the +/- figures.

That's what I don't get: where do you get this +/- figure from?

The biggest issue with bags, pockets and the likes is having stagnant water over the antenna. Worse if the antenna makes yet another layer stick to the antenna. The top of the device is glossy so water slides off. Water is the enemy.

Select to expand quote

JulienLe said..
That's what I don't get, where do you get this +/- figure from?

tbwonder said: I have seen files from 2 mini motions worn on the same arm, with 2 sec peak results 1 kt different, even though the reported "accuracy" was only around 0.2 kts.

Select to expand quote

JulienLe said..
That's what I don't get: where do you get this +/- figure from?

The biggest issue with bags, pockets and the likes is having stagnant water over the antenna. Worse if the antenna makes yet another layer stick to the antenna. The top of the device is glossy so water slides off. Water is the enemy.

Julien the +/- figure I am quoting comes from the GPS Speedreader software, it is described as "Error estimate (SDOP/sAcc)"

Then we'll have to wait for boardsurfr to chime in.

Let's not confuse the accuracy as reported by the device and any other computation. The accuracy reported by the device is a byproduct of the position/velocity computation.

So I work a lot with various laboratory equipment. My understanding or error may be different to what the actual error means, but my understanding says that if the device is reporting an error that the actual accurate measurement should be within the error range. The device should be accounting for "user error" by recognising less than ideal antenna positioning or containment (in a bag) as a poor satelite signal and this should be within the error reported. The data presented earlier shows that this is not the case. Two devices travelling at the same speed reporting a difference in speed greater than the sum of each error. Should not be possible if errors is accurate.

Saying user error does not cut it. It proves the error value is not all it has been claimed to be. My understanding is a large component of device selection in the GPSTC is the presence of this error paramter. But it is proven here that it is not reliable.

Accuracy and precision are not the same.

Select to expand quote

JulienLe said..
Then we'll have to wait for boardsurfr to chime in.

Let's not confuse the accuracy as reported by the device and any other computation. The accuracy reported by the device is a byproduct of the position/velocity computation.

Peter's +/- figures agree with Manfred's, GPSResults has been doing this for a long while. I assumed it's another way of expressing the SDoP/sAcc numbers. In the case of the ublox8 chip this is included in the NAVPVT sentence

Select to expand quote

vosadrian said..
So I work a lot with various laboratory equipment. My understanding or error may be different to what the actual error means, but my understanding says that if the device is reporting an error that the actual accurate measurement should be within the error range. The device should be accounting for "user error" by recognising less than ideal antenna positioning or containment (in a bag) as a poor satelite signal and this should be within the error reported. The data presented earlier shows that this is not the case. Two devices travelling at the same speed reporting a difference in speed greater than the sum of each error. Should not be possible if errors is accurate.

That's a very sloppy and misleading statement.

In the end, error estimates are always linked to p-values. For raw errors like the sAcc values in ublox (Motion) data, the error is typically the standard deviation, or a closely related value. Which means that a lot of single data points will NOT overlap.
The +/- values for results are typically based on 2 or 3 standard deviations, with the error propagation deemed appropriate for the given data (e.g. GT-31 data cannot be assumed to have independent errors due to heavy filtering). At 2 standard deviations, you'd expect to several percent of the measurements to disagree; at 3, you'd still expect a disagreement about 1 in 300 times. Those numbers are for normal-distributed errors. With skewed or other non-normal distributions, non-overlaps would be even more common.

I have compared GPS data from a variety of different approved devices and prototypes, and the observed agreements were generally within expectations for the short distances (2 seconds, 10 seconds, up to 500 m). You have to go to nautical miles or 1-hour runs to see disagreements more often than expected - or to alphas when 2 devices are worn in different positions, like right hand and left hand, or arm and helmet. In alphas, you can actually measure the real differences between the inside and outside arm in a turn - the accuracy of the Motion (and similar) is that good. When you go to hours, the estimated error values get so ridiculously low (e.g. 0.006 knots in a random session that I pulled up). Yes, you might see an actual disagreement that is more than twice the estimated error - but it will still be less than 0.02 knots. And if you see it, chances are that the two devices were not right next to each other, and that the measured difference can actually be explained with different movements of body parts in a direction perpendicular to the travel direction.

I do happen to have a few decades of experience in scientific error analysis, along with plenty of formal education.

Select to expand quote

tbwonder said..
The Blue line device had a lock on 17 satellites throughout the 2 sec peak of 41.276 and reported an error of +- 0.163
The Red line device had a lock on 15 satellites throughout the 2 sec peak of 41.728 and reported an error of +- 0.199

The blue speed range goes up to 41.439 knots. The red speed range goes down to 41.529 knots. So the ranges do not overlap.

The plus/minus numbers are two sigma errors. The actual results is expected to be within the range 95% of the time (check news.mit.edu/2012/explained-sigma-0209 for graphs). So in rough approximation, you'd expect this about 1 in 20 times when you compare two devices worn right next to each other.

In your case, your raw data error was around 0.5 - 0.6. That's assumed to be a one sigma error. You would expect a lot of points to differ by 1 knot, which is exactly what you are seeing.

For a ublox chip, that's pretty bad data, which the low number of satellites confirms. Could be the arm band moved and the Motions were pointing down at the water, so they picked up a lot of secondary path noise.
---
Edit for clarification:
The "1 in 20 times" refers to how often you'd expect the value from one device to be outside of the range of the other device, assuming normal distribution of the error. Seeing a non-overlap as in this example would be rarer, but it would still occur occasionally.With a non-gaussian error distribution, a non-overlap could be even more likely. We actually know that errors are not completely independent in the short range (< 1 second or so) if the reception is poor, as it was in this example. This effect becomes negligible if reception is better, or if you look at time frames of 10 seconds or longer. This knowledge is used to some extend in the code by using filters, especially for maximum SDoP/sAcc values and for min. satellites. It's not used as much as would be ideal (from an accuracy standpoint) for straight-line speed since that would "throw out" too many alphas, and too many points in nautical mile and hour results. It would be throwing out the child with the bath water.

that's all very nice but what about the 300$ watch ????

decrepit > the NAV-PVT speed accuracy is the +/- in my screenshots above. It's high as expected.

vosadrian > my very first post, minutes after receiving the logs, mentions that the accuracy of both logs is poor, probably due to the Aquapac. You're purposefully misleading. Again.

I don't get what everybody is on about. Both devices reported poor accuracy all logs long and on that run, one device reported very poor accuracy. It has all red lights flashing. If anything, raise filtering standards. 3-GNSS 10Hz devices shouldn't share the standard of old 1-GNSS 1Hz devices. In the same way 3-GNSS 20Hz devices shouldn't. The gap is only going to get wider.

Select to expand quote

JulienLe said..
decrepit > the NAV-PVT speed accuracy is the +/- in my screenshots above. It's high as expected.

Sorry julien, I don't get, what you don't get. We know both data are bad. But most of us expect the difference between them to be within the sum of both the +/- values. In this case they aren't. And I'm not smart enough to follow Peter's explanation of why that doesn't matter.

In other words the error of one or both files must be worse than the +/- values state. (These values are in the same units as the speed results surely??)

400ms of divergence. Let's raise filters so runs/logs so far from the norm are thrown out.

Thanks Boardsurfr for that great explanation. I recall all the 1,2 and 3 SD from the mean stuff from school. So GPSSpeedreader errors are for 2 Standard deviations or approx 95%. This is good to know.

My original point was to try and raise awareness in the community that Motion devices do not always give perfect results. You cannot just wear them wherever you like and guarantee that your measured speeds will be within +/- 0.1 kts.

High ranking sessions should be checked with analysis software and not just uploaded blindly through KA72. Either of the two files in question would both pass KA72 as valid files for posting to GPSTC.

I am not criticizing the Motion in any way. It is great to be arguing about 0.5 kts as a really bad error.
I agree with Julien the filter standards should be raised. But as GPSTC does not differentiate between types of doppler device used this would be difficult to implement.

Select to expand quote

tbwonder said..
>>> But as GPSTC does not differentiate between types of doppler device used this would be difficult to implement.

Not just the GPSTC, KA72 caters for all sorts of users, that would mean having special filters for GPSTC posts

Select to expand quote

tbwonder said..
My original point was to try and raise awareness in the community that Motion devices do not always give perfect results. You cannot just wear them wherever you like and guarantee that your measured speeds will be within +/- 0.1 kts.

Very good point.

Select to expand quote

tbwonder said..
But as GPSTC does not differentiate between types of doppler device used this would be difficult to implement.

Actually, at least some of the software already uses different filter settings for different devices, although currently, this is linked to the recording rate.

It is a somewhere between unrealistic and nonsensical to raise filter standards. If you'd just raise filter standards for all devices, you'd throw out a lot of runs that have been deemed "good enough" in the past (or, depending how the filters are implemented, lower the calculated speeds). Even with current settings, GPSTC accepts data with much lower accuracy than what a properly worn Motion gives; GT-31 data (which always distort alphas) and underhand grip GW-60 data are common example. The Motion will give higher accuracy data than those devices virtually all of the time. The error estimates of +/- 0.6 knots for the individual data points that we call "horrible" in this example are typical for GT-31 data. Underhand grip GW-60 data are worse.

If you look at enough data (which we do collectively), you will find examples like this one. But, like tbwonder and others have said, such data are quite rare with Motion (and similar) devices. Let's keep this in perspective. The 2-second results from the two devices in this particularly bad example were less than 0.5 knots apart. It's more likely than not that both numbers are within 0.4 knots of the correct result. For Locosys data, it's quite common to get +/- estimates of 0.4-0.6 knots for 2 second results; with underhand grip or low satellite numbers, it can be even worse.

The 2 second category always has been the least accurate of the GPSTC categories. But it's nice to have a "top speed" category, and it makes much more sense than to use single-point top speeds, which have much higher typical errors (often in the 1-knot range or even higher). For 2 seconds, a 0.2 knot difference may or may not be real; as soon as you go to 5x10, such a difference is very likely to be real.

I have no issue at all with the Motion range of GPSs or any other that have this error parameter. But this error parameter is one of the selection criteria for acceptance as an approved GPSTC device, and it seems most do not even know what it means. If I use a voltmeter with accuracy of 1% to measure 10V, I can have certainty that the measured value is within 1% of the true value. My observations of discussions of this error value have been treated the same, but if the discussion above indicates this is not the case, and instead it is based on a statistical distribution and is not guaranteed to be in that range. Users here expected the error value to account for "user error", but it seems it does not. So now it appears that errors of 1 knot are to be expected if the device is not used in a certain way.

My issue here is well known I believe. GPSTC claims as its philosphy "inclusiveness" yet it excludes all available GPS devices in the market place except the Motion. Apparently changes were coming to this over a year ago, but lets just ignore that promise. One model of Motion could not be manufactured reliably and I believe is obsolete. The other has no display for on water feedback so you need a second device for feedback. Both models are not easily available and dependant on a single supply source that is struggling to remain profitable enough to stay in business.

Great example, you forgot to mention the "accuracy only valid under specific operating conditions" warning of any voltmeter manufacturer or calibration lab. I'll add my own then: "accuracy only valid when worn correctly".

The error values did account for the Aquapac has it jumped to my eye in seconds that the accuracy of both logs was terrible from start to finish. If this device had shown such accuracy here during tests, it would have become a paperweight.

The rest is wrong, made up, boring, disappointing. Why do you even write such stuff.

Originally put my logger in Aquapac but quickly ditched that idea and moved it to upper arm with provided strap- comfortable and secure and easy to keep aligned skyward... as recommended. Great files.
I suppose if we just follow the manufacturer's instructions, all will be good.

Mikey I am disappointed to learn that you have moved to the upper arm mount. I assumed you were still using this position.

Received another australian log yesterday for unrelated reasons and just to drive the point home, this is what you should expect:

If you have a doubt, hit me up.

Select to expand quote

tbwonder said..

Mikey I am disappointed to learn that you have moved to the upper arm mount. I assumed you were still using this position.

Left an odd tan mark so had to change.