Today I went out with one boom unit on the default "portable" setting, and the other one changed to the "sea" setting. So red is sea, a 2D calculation of speed, Blue is portable a 3D calculation of speed.
For me this proves once and for all that the sawtooth in 10hz waveforms represent vertical speeds, not noise or artefacts.


Jan's esp config file say the see model cuts out at 40kts, but I'm sure I've had it up to 60kts on the open road, recoding speed faithfully . Any way I left them on for the drive home and pushed it to 43kts.






So it's clear sea and portable record the same speed at 43kts. You can also see on a flat road neither trace has sawtooth.

There doesn't seem to be any great advantage in using either mode, as the sawtooths average out. But if you want a nice clean trace the sea mode will give it to you.

portable mode may be useful to sort out where you go fastest, in chop or smooth, there's arguments for and against either.

Select to expand quote

decrepit said..
For me this proves once and for all that the sawtooth in 10hz waveforms represent vertical speeds, not noise or artefacts.

That's one possible explanation, but not the only one, and not necessarily the correct one. Keep in mind that (a) the speed field we are using (gSpeed) is described in the u-blox documentation as "Ground Speed (2-D)", not as 3D speed. Ground speed should not include any vertical movements; if it does, that's pretty much an artifact by definition.

Another possible explanation is that the firmware uses different filtering for the different modes, and that the sea mode filter smoothes data more than the portable mode. Here is an example of a track smoothed using 5-point averages (blue: original, red: filtered):

The filter in the firmware would probably something a bit more sophisticated than an average filter, maybe a Kalman filter variation; but the effect would be similar.

Select to expand quote

decrepit said..
There doesn't seem to be any great advantage in using either mode, as the sawtooths average out.

That is certainly true if you look at larger ranges, like nautis. But for "top speed" categories, there will be a noticeable difference. That is obvious for "top speed" from single points; in your graphs, the spiky portable model gives a single-point top speed that is about 1.5 knots higher than the sea model.

For 2 seconds, the effect will be reduced, but not completely eliminated: the found top 2 second range is likely to include spikes at the left and right side, so it will overstate the speed. Here's a little example with some artificial data that look similar to your graph:


The "true speed" here is 0. It's overlayed with a regular up and down of +/- 1 knot, and some random noise of up to +/- 0.5 knots. The region shown would be about 3 seconds of 10 Hz data. The single-point speed is 1.49 knots too high for the unfiltered data, and 0.36 knots too high for the smoothed data. Over 2 seconds, the error goes down to 0.19 and 0.13 knots, respectively.

Note that:
(a) the speed is overstated, which is a result of looking for the maximum in noisy data.
(b) the error for the unfiltered data is about 50% higher than the error for the filtered data.

Select to expand quote

decrepit said..
Jan's esp config file say the see model cuts out at 40kts, but I'm sure I've had it up to 60kts on the open road, recoding speed faithfully . Any way I left them on for the drive home and pushed it to 43kts.

It should be very sad that you miss your pb with speeds exceeding 40 knots due to a bad dynamic model.......
To avoid loss of data, the dynamic model will switch back to "portable" as the speed exceeds 20 m/s. As the speed goes lower then 15 m/s, it will switch back to "sea" (see line 21 github.com/RP6conrad/ESP-GPS-Logger/blob/master/GPS_data.cpp).
If you have the logging.txt file active, you could see this in the .txt file.
Configuration setting "sea" is only possible with the M8 Ublox, did not extend the ESP-sw to the newer M9/M10.

Greetings, Jan.

Select to expand quote

rp6conrad said..
Configuration setting "sea" is only possible with the M8 Ublox, did not extend the ESP-sw to the newer M9/M10.

Many (and perhaps all) configuration commands for the M8 still work for the M10. They may be "deprecated", and their use may be discouraged, but even the u-Center still seems to use the old commands, at least for some of the commands that I looked at. It usually takes quite a while for deprecated things to be removed, since doing so may break existing software.

I would assume that setting the model to "sea" using the same commands as for M8 chips will work on M10 chips, but have not tested it.

Select to expand quote

decrepit said..
Today I went out with one boom unit on the default "portable" setting, and the other one changed to the "sea" setting.

You made me curious, so I set out to see if I can find any evidence of filtering. I did a "rapid acceleration/decelleration" test by mounting two units next to each other on a piece of wood, one in portable and one in sea mode. I then moved the wood rapidly in front of my body from side to side, coming to a rapid and complete stop for ~ a second between movements. Here is what the speeds look like in Speedreader:

The speed shown by the unit set to "sea" is consistently lower. We do not know the true speed here, but can look at positional speeds for comparison. The positional speeds for both units are relatively close to the doppler speed for the "portable" unit; only the doppler speed for the "sea" unit is much lower (4.7 vs. 8.3-10.6 knots).
Here is a closeup of the doppler speeds for the different models:


The short (but probably real) spike in the doppler speeds is reduced in the sea model speeds by a factor of about 2 every time. For comparison, here are the positional speeds for the same section:

So, I'm still confused, sea is better for accuracy under 40kts, but the switch from sea to portable at 40kts may loose you your PB.
So for safety we stay with portable?????

Select to expand quote

boardsurfr said..

You made me curious, so I set out to see if I can find any evidence of filtering. I did a "rapid acceleration/decelleration" test by mounting two units next to each other on a piece of wood, one in portable and one in sea mode. I then moved the wood rapidly in front of my body from side to side, coming to a rapid and complete stop for ~ a second between movements. Here is what the speeds look like in Speedreader:

The speed shown by the unit set to "sea" is consistently lower. We do not know the true speed here, but can look at positional speeds for comparison. The positional speeds for both units are relatively close to the doppler speed for the "portable" unit; only the doppler speed for the "sea" unit is much lower (4.7 vs. 8.3-10.6 knots).
Here is a closeup of the doppler speeds for the different models:


The short (but probably real) spike in the doppler speeds is reduced in the sea model speeds by a factor of about 2 every time. For comparison, here are the positional speeds for the same section:

Good to see you trying to get some data.

Some Questions that arise:

1. Can you be sure there was no vertical movement in your side by side experiment?

2. Can you try the same experiment with only vertical movement (or as close as possible to)?

3. Is positional data filtered for vertical movement?

Select to expand quote

rp6conrad said..

decrepit said..
Jan's esp config file say the see model cuts out at 40kts, but I'm sure I've had it up to 60kts on the open road, recoding speed faithfully . Any way I left them on for the drive home and pushed it to 43kts.

It should be very sad that you miss your pb with speeds exceeding 40 knots due to a bad dynamic model.......
To avoid loss of data, the dynamic model will switch back to "portable" as the speed exceeds 20 m/s. As the speed goes lower then 15 m/s, it will switch back to "sea" (see line 21 github.com/RP6conrad/ESP-GPS-Logger/blob/master/GPS_data.cpp).
If you have the logging.txt file active, you could see this in the .txt file.
Configuration setting "sea" is only possible with the M8 Ublox, did not extend the ESP-sw to the newer M9/M10.

Greetings, Jan.

So too clarify for me Jan please.

1. Are you referring to the 'sea (2D)' mode as 'Dynamic'?

2. Is this change something you deliberately programmed into the ESP software?

3. If the above is correct, is it because it is true that the M8 chip won't log speeds above 20m/s in 'sea 2D' mode - as has been suggested before?

Thanks.

OK. Looking up the M8 specifications we get this:





Confirming again that the 'At Sea' mode has max horizontal velocity of 25m/s.

Perhaps this confirms that there is a 'filter' on vertical movement in sea mode, as it can't exceed 5m/s. ? I wonder if this would actually be what we are seeing on Decrepits data?

Select to expand quote

sailquik said..
1. Are you referring to the 'sea (2D)' mode as 'Dynamic'?

2. Is this change something you deliberately programmed into the ESP software?

3. If the above is correct, is it because it is true that the M8 chip won't log speeds above 20m/s in 'sea 2D' mode - as has been suggested before?
Thanks.

1. Correct, "sea" or "portable" is the type of dynamic model that is set in the ublox module. Default @ startup for M8/M9/M10, the dynamic model "portable" is used.
2. If you want another dynamic model ("sea" or "automotive" or...), the configuration of the ublox has to be changed. This is normally done @ startup. But even when logging is active, the dynamic model can be changed on the fly !! This is indeed programmed into the ESP-software (speed exceeds 20m/s -> dynamic model "portable" is set again, speed drops to 15m/s -> back to "sea" model.)
3. I did some tests with the "sea" model @ higher speeds, and indeed, above 25 m/s the reported speed goes to zero !! As the manual stated : If a sanity check against a limit of the dynamic platform model fails, then the position solution becomes invalid.

I will extend the ESP-sw for the M9/M10, so the dynamic model can be set for these modules as well.
Greetings, Jan.

decrep@decreps-ideacenter:~$ units 25m/s knots
* 48.596112
/ 0.020577778
decrep@decreps-ideacenter:~$

decrep@decreps-ideacenter:~$ units 45knots m/s
* 23.15
/ 0.043196544
decrep@decreps-ideacenter:~$ units 23m/s knots
* 44.708423
/ 0.02236715
decrep@decreps-ideacenter:~$

I'd be happy with the change at 23knots, that's way above my skill level. but others of course can get to the 25m/s.

Or is the change to slow in a rapidly accelerating speed surfer to miss out on changing before it goes to zero?

Maybe I'll just stick to portable.

Select to expand quote

rp6conrad said..

sailquik said..
1. Are you referring to the 'sea (2D)' mode as 'Dynamic'?

2. Is this change something you deliberately programmed into the ESP software?

3. If the above is correct, is it because it is true that the M8 chip won't log speeds above 20m/s in 'sea 2D' mode - as has been suggested before?
Thanks.

1. Correct, "sea" or "portable" is the type of dynamic model that is set in the ublox module. Default @ startup for M8/M9/M10, the dynamic model "portable" is used.
2. If you want another dynamic model ("sea" or "automotive" or...), the configuration of the ublox has to be changed. This is normally done @ startup. But even when logging is active, the dynamic model can be changed on the fly !! This is indeed programmed into the ESP-software (speed exceeds 20m/s -> dynamic model "portable" is set again, speed drops to 15m/s -> back to "sea" model.)
3. I did some tests with the "sea" model @ higher speeds, and indeed, above 25 m/s the reported speed goes to zero !! As the manual stated : If a sanity check against a limit of the dynamic platform model fails, then the position solution becomes invalid.

I will extend the ESP-sw for the M9/M10, so the dynamic model can be set for these modules as well.
Greetings, Jan.

Many thanks Jan.

As Decrepit says, can you change the switch over to 25m/s, or maybe 24.5m/s? Or would that create a problem with some missed points?

Select to expand quote

decrepit said..
Or is the change to slow in a rapidly accelerating speed surfer to miss out on changing before it goes to zero?

I did a little test drive to answer that question with two ESP loggers,one set to "sea", one to "portable". I added an Openlog
base logger also set to "sea" to see how the 25 m/s filter is applied. Here's a section of this drive:


Red is the Openlog based unit, the ESPs are green and blue. The "sea" cutoff is a hard limit - the reported speed cannot exceed 29 knots. After the first point that hits the speed limit, the number of satellites is reported as zero. The sAcc increases slowly. After 10 seconds, the reported speed drops to 0, until the actual speed drops below the threshold for a while. Jan has seen this in his tests, and therefore added the automatic switch to portable mode. Nice work.

As for dropped points: the track where the switch from sea to portable and back happened did not have a single dropped point, so this may not be a problem. The unit was set to 5 Hz, but Mike probably has some 10 Hz points in "sea" mode where he can look for dropped points.

Increasing the switch threshold certainly makes sense. For safety, I'd stay a bit below 25 knots, otherwise it could happen that 2 points are recorded above the threshold before the change is effective. In this case, the second point would report 0 satellites, so it would be filtered out. Anyone expecting to go above 48 knots should probably just stick to portable mode.

A couple more observations from the test drive. Here is an example where the "sea" mode definitely reduced artifacts and thus produced better data:


The drive was partly on streets with trees on both sides, and sometimes overhanging the street, so the spikes in the blue ("portable") track are from secondary path (and similar) noise problems. Both "sea" units provide smoother traces that are closer to reality. As can be seen here, the u-blox chips have a tendency to err on the low speed side if the signal is bad, so the "sea" model would give both higher and more accurate speeds in most affected regions. For the see-saw pattern Mike observed on the boom, which may well be connected to up-down or similar movements as Mike suggested, the "higher" speed part may not apply, though.

Another observation in the tracks was that the "sea" model tracks are just slightly delayed relative to the "portable" tracks when the speed changes:

This is similar to what we've seen in heavily filtered watch data, but the delay is smaller (about 200 ms in 5 Hz data).

Select to expand quote

boardsurfr said..
That's one possible explanation, but not the only one, and not necessarily the correct one. Keep in mind that (a) the speed field we are using (gSpeed) is described in the u-blox documentation as "Ground Speed (2-D)", not as 3D speed. Ground speed should not include any vertical movements; if it does, that's pretty much an artifact by definition.

In the nav-pvt message, next speeds are included :
Ground speed
Velocity Nord
Velocity East
Velocity Down
I did some checks, and although the gSpeed is referred as 2-D, it is definitely the 3D-vector speed of vNord, vEast and vDown !
Ground_speed*Ground_speed = vNord*vNord + vEast*vEast + vDown*vDown
So the "sawtooth" in the speed-graph (portable mode) could be caused due the vertical movement. Analysing the ubx file should give the answer directly !

I will change the switch-over point sea-> portable if the speed exceeds 24 m/s (46,6 knots), switch back portable-> sea if the speed drops below 20 m/s (only if the configuration is set to "sea").

Greetings, Jan.

It's only my surmising here.
The overhead sats are only good for vert movement, the best sats for horizontal movement are in front and behind, on the horizon. But sats on the horizon have too much atmospheric distortion, so sats at a higher elevation are used. These sats have both vertical and horizontal components. Can the pure vertical component from overhead be used to subtract the vertical component from the elevated sats?
This could give horizontal movement with no/minimal filtering?

Select to expand quote

rp6conrad said..
I did some checks, and although the gSpeed is referred as 2-D, it is definitely the 3D-vector speed of vNord, vEast and vDown !
Ground_speed*Ground_speed = vNord*vNord + vEast*vEast + vDown*vDown

You may want to check again, and show your data. I think your statement is incorrect.

Here is an example from a .ubx file:

The highlighted row has a non-zero vertical speed (velD), but 0 doppler speed and horizontal speed vectors (velN and velE). This is consistent with 2D speed (gSpeed = sqrt(velN^2 + velD^2), but not with 3D speed. Point #14689 is similar.
These are just examples, the same holds true for all points in this file:


The 2D speed, calculated from the vector addition of velN and velE, is identical to the doppler speed (gSpeed) reported in the ubx file to within +- 0.003 knots (the small differences are probably from multiple conversions between the integer and real values, and number of digits used in the table).

The 3D speed, calculated by also including velD is up to 0.348 knots higher than the doppler speed.

The data in this file are from a driving test, so the reported vertical speeds are artifacts. The vertical speeds are clearly not included in the gSpeed field in the ubx files.

here's that sea portable overlay again with peak speed at 30kts. I don't see any signs of the 29kt transition.

and here's a section of the car drive dropping down from 40kts. The curser is at 29.04kts
This is where the sea model (red) diverges from portable blue, coincidence, or due to the change back to sea?
Also in the car there is virtually no saw tooth in the portable trace.

I believe the 29 knots is mistype, as 25 m/s is 49 knots (ublox limit) !
In the ESP sw, the switch over to "portable" was set at 20 m/s (2 s avg speed), this is 39 knots ! Switch over back to "sea" was set at 15 m/s, this is then 29 knots.
The 2-D / 3-D groundspeed needs more investigating, but a dataset with real velocity down speeds should be interesting.... Is to be continued....

Select to expand quote

decrepit said..
here's that sea portable overlay again with peak speed at 30kts. I don't see any signs of the 29kt transition.

and here's a section of the car drive dropping down from 40kts. The curser is at 29.04kts
This is where the sea model (red) diverges from portable blue, coincidence, or due to the change back to sea?
Also in the car there is virtually no saw tooth in the portable trace.

Mike, keep in mind that you will not see single dropped points in the speed tracks. You have to check the "Messages" window for missing points, or/and look in the track point table where missing points and the first point after are shown on grey background. But I'd expect that you won't find any. Interesting to see the tracks starting to diverge just where the switch to the "sea" model happens. Could just be chance, but it may also be caused by the heavier filtering coming into play.

I checked the tracks from my drive, and it shows a similar divergence after switching back to the sea model at 29 knots:

Are those filters / algorithms somewhere documented by the supplier ?

Select to expand quote

CarlosSainz said..
Are those filters / algorithms somewhere documented by the supplier ?

Never found some documentation about internal filtering, I guess this is a company secret. I did some more checks for the 2D groundspeed, and indeed the groundspeed is the 2D speed (northVelocity and eastVelocity). The vertical component (velocityDown) is not present !
I checked a ubx file from a bikeride with some bridges (heigth diff max 8 m), and I extract the mm/s values (full resolution of the speeds).
You can see that the difference groundSpeed and the 2D vector is always near to 0 mm/s, as the difference with the 3D vector is going up with every heigth change.



Greetings, Jan.

Select to expand quote

sailquik said..
OK. Looking up the M8 specifications we get this:





Confirming again that the 'At Sea' mode has max horizontal velocity of 25m/s.

Perhaps this confirms that there is a 'filter' on vertical movement in sea mode, as it can't exceed 5m/s. ? I wonder if this would actually be what we are seeing on Decrepits data?

OK. I have always wondered why there are limitations for speed (horizontal velocity) and max altitude in GPS systems.

Just found the answer:

"The speed limit is somewhere in the supersonic range. And there is also a height limit. Both won't restrict any commercial planes, but are intended to make commercial GPS unuseable for ICBMs and other rockets"

Apparently, there are Military systems that are not restricted.....