Select to expand quote

decrepit said..
Ubuntu 14 "disks" benchmarks
old class 4
Average read speed 18.1 MB/s, Average write speed 3.2 MB/s, access timr 1.89msec

I know my English sucks at times, but this is what I do to make a living :)
You tested in Ubuntu with large continues filestreams and big blocks, how was your scheduler configured?
A GNSS chips writes very (small font) small (/small font) messages, did you test your card with that?
You tried a PC which supports SDIO, did you reconfigure the controller to just use SPI?

Don't bother to answer above, I tried a lot of things before I got a working setup. (normally I do between 2 and 13 GigaByte/sec)
Standard SD cards are to slow, the PRO+ has a dualcore cpu with SLC cache, this is a great benefit to handle data.
Every 512 bytes your SD has to search for an empty block and skips on recording GNSS messages.
So every 0.512 sec you get a +-0.08/0.1 sec delay and data is lost (depends on SD card)

So you need a large buffer and/or a continues file.

Regarding logging:
NEO-M8N logs 3 GNSS networks at 3hz, it looks like a hard limit. (firmware limit??)
I recorded a lot of data last days, Galileo is not ready to be used :(

/edit
Believe me if you lived nearby I would be more than happy to offer you beer and discuss this in person
There is NO financial gain for me in this, I earn money in a daytime job, windsurfing is a hobby (nah.. lifestyle )

Select to expand quote

decrepit said..


I've ordered another logger, based on the open logger, but with a bigger buffer, a full size HC SD card support, higher baud rate and usb download.
Fingers crossed, it will do 10hz.

More later.

What is the new type of logger? I think buffering will solve the problem as it seems to be a problem while the card is trying to allocate free blocks, not necessarily the speed at which it can accept data.

The class of the card relates to speed, except I think they all have the same problem allocating the next block with which it is to write the file.

Select to expand quote

raymondw said..
Regarding logging:
NEO-M8N logs 3 GNSS networks at 3hz, it looks like a hard limit. (firmware limit??)
I recorded a lot of data last days, Galileo is not ready to be used :(

Hi Raymond. just to be clear, did you find that you got better accuracy with 10Hz / 2 GNSS or 3Hz / 3 GNSS?

Select to expand quote

raymondw said..
Regarding logging:
NEO-M8N logs 3 GNSS networks at 3hz, it looks like a hard limit. (firmware limit??)

Maybe it's the NEO-M8N. I just looked at a windsurfing test run. At 5 Hz, the NEO-M8N has 19 missing points in ~53400 points. The Beitian BN280 had no dropouts. All are with 3 GPS systems. I also recorded on a BN880 with the added sat info once per second, which increased file size by 50%, and caused drops in 6 areas (1-3 missing points). Satellite numbers are between ~16 and 25, and similar between the 3 units.

The Drotek was connected via USB to an Android phone. The connection and the phone should be fast enough and not the cause of the dropouts.

Could be that the extra front end filters etc. on the NEO-M8N are causing the problems. The Beitian units are u-blox 8 based, but (probably) don't have some of the extra circuitry typical for the NEO-M8N.

Select to expand quote

raymondw said..
I recorded a lot of data last days, Galileo is not ready to be used :(

Are you saying that because of the missing data points, or did you see other issues? If so, what exactly?

I just checked a couple more windsurfing data files from the BN280 and 880 Openlog units. They don't show any dropped points at 5 Hz with 3 GPS systems, and around 20 sats tracked, when the extra satellite info is not recorded. That's also what I recall from earlier Beitian/Openlog unit tests - no missing data points.

As I have reported before. I have no dropped points in normal data running the NeoM8's (2 BT GPS tested) @10hz on 2 GNSS ( GPS and Glonass).
So I would be surprised if it is in the GPS module that is the bottleneck. We are also recording NAV-SAT @1Hz and both PVT and DOP @10Hz so there is a LOT of data recorded. Perhaps Manfred has inplemented some buffers in GPS-Logit/BT to write to the SD card in the phones? I am just using normal low cost MicroSD cards, nothing special. But I do run into regular missed points (roughly one every second) when I try to log at 18Hz from just the GPS/GNSS. It is also interesting that Roo and Manfred do not report these missing points @18Hz with their more expensive, higher spec phones.

Perhaps I should test @18 Hz with a faster SD card and see if it makes a difference?

Select to expand quote

sailquik said.. >>>>> Perhaps I should test @18 Hz with a faster SD card and see if it makes a difference?

Go for it Andrew! Why a question mark?

I just assumed, the phone is in control of writing to the SD card, it's probably got a huge buffer. A faster card may well work at 18hz, it would be interesting to find out.

Select to expand quote

raymondw said..>>>>> Believe me if you lived nearby I would be more than happy to offer you beer and discuss this in person
There is NO financial gain for me in this, I earn money in a daytime job, windsurfing is a hobby (nah.. lifestyle )

That would be nice Raymond, visiting your neck of the woods is on the bucket list, but it could be a while before it happens.

Select to expand quote

FormulaNova said..
What is the new type of logger? I think buffering will solve the problem as it seems to be a problem while the card is trying to allocate free blocks, not necessarily the speed at which it can accept data.

The class of the card relates to speed, except I think they all have the same problem allocating the next block with which it is to write the file.

It's the, SDLogger - Open Hardware Data Logger (Seeed Studio). Quite expensive at $57.00 but in the interests of science possibly worth it.
I was toying with the spark fun,

Logomatic v2 Serial SD Datalogger (FAT32)
But that's even more expensive, I was tricked by looking at the US$ price. it's $84 + postage. But has two buffers, (one writes while the other fills up), and onboard battery management, just connect a LiPo cell directly to it.

I wish I could remember where I read about the open logger not benefiting from high speed cards, probably on the open logger forum.
But I think he said, he got best results from a class 4

I just put my logger back to 5hz, (there's wind tomorrow, I don't want missed points), on the "rates" menu there's also a choice between "gps time" and "UTC time", the default was "gps time", so I've changed that to UTC, I'm hoping that will get rid of the annoying 18s time difference between the logger and the the GW52.
If the forecast eventuates I'll find out tomorrow.

Select to expand quote

raymondw said.. >>>>>> Don't bother to answer above,

No worries Raymond, I couldn't if I tried, no idea really what you were talking about.

Select to expand quote

boardsurfr said..
I just checked a couple more windsurfing data files from the BN280 and 880 Openlog units. They don't show any dropped points at 5 Hz with 3 GPS systems, and around 20 sats tracked, when the extra satellite info is not recorded. That's also what I recall from earlier Beitian/Openlog unit tests - no missing data points.

Peter, I look forward to getting my BN280 going. I don't think I want to pull my working M8N apart, so looks like I'll have to wait until the new logger gets here around the end of this month.

Select to expand quote

sailquik said..Hi Raymond. just to be clear, did you find that you got better accuracy with 10Hz / 2 GNSS or 3Hz / 3 GNSS?

When I tested last week, the 5Hz/2 GNSS data was better than 3Hz/3 GNSS file.
3Hz gave SDOP values around 0.1, +-13 sats in GPSResults
5Hz gave SDOP values around 0.01, +-19 sats in GPSResults

Both devices where on my right upper arm in the same Paqua as always.

I use this antenna from Taoglas/Mouser connected to the NEO-M8N
nl.mouser.com/datasheet/2/398/CGGBP.18.4.A.02-1218625.pdf

Select to expand quote

boardsurfr said..
Maybe it's the NEO-M8N. I just looked at a windsurfing test run. At 5 Hz, the NEO-M8N has 19 missing points in ~53400 points. The Beitian BN280 had no dropouts. All are with 3 GPS systems. I also recorded on a BN880 with the added sat info once per second, which increased file size by 50%, and caused drops in 6 areas (1-3 missing points). Satellite numbers are between ~16 and 25, and similar between the 3 units.

The Drotek was connected via USB to an Android phone. The connection and the phone should be fast enough and not the cause of the dropouts.

Could be that the extra front end filters etc. on the NEO-M8N are causing the problems. The Beitian units are u-blox 8 based, but (probably) don't have some of the extra circuitry typical for the NEO-M8N.

The original Ublox is rated at 3Hz, when I set Ucenter at 5 or 10Hz, data is around 3 or 4Hz.

Select to expand quote

Are you saying that because of the missing data points, or did you see other issues? If so, what exactly?

Low amount of Galileo satellites, when connected to a sat received data was fine. Tested with Ucenter and my Gyro in pass-through mode.

Select to expand quote

sailquik said..
As I have reported before. I have no dropped points in normal data running the NeoM8's (2 BT GPS tested) @10hz on 2 GNSS ( GPS and Glonass).
So I would be surprised if it is in the GPS module that is the bottleneck. We are also recording NAV-SAT @1Hz and both PVT and DOP @10Hz so there is a LOT of data recorded. Perhaps Manfred has inplemented some buffers in GPS-Logit/BT to write to the SD card in the phones? I am just using normal low cost MicroSD cards, nothing special. But I do run into regular missed points (roughly one every second) when I try to log at 18Hz from just the GPS/GNSS. It is also interesting that Roo and Manfred do not report these missing points @18Hz with their more expensive, higher spec phones.

Perhaps I should test @18 Hz with a faster SD card and see if it makes a difference?

NEO-M8 is the family name.
www.u-blox.com/sites/default/files/NEO-M8_DataSheet_%28UBX-13003366%29.pdf
Copy/paste gives a challenge, please look at page 6 :)

My REV-C uses the Q, every newer release uses the N.

As soon as I configure the datarate ABOVE what Ublox is advising I get missing data.
When I look into detail the issue is not dropped points, but data comes malformed from the unit.
Every GNSS update from the unit comes in sentences NAV-PVT, NAV-DOP etc etc etc
NAV-PVT comes before NAV-DOP, the first sentence is cut off when the second sentence starts.
Same when enabling NAV-SVINFO output.
PVT is cut when DOP starts and DOP is cut when SVINFO starts

And as you mention you have a dropped point every second, and you record NAV-SAT@1hz ;)

It would be nice to see how the Beitian handles this overclock as it is against the Ublox firmware recommendations.

For my units I recommend to follow the Ublox guidelines, although the user can configure to go above this, at their own risk.

Just a FYI.... if it is the openlogger that is causing the issue, there are different versions of firmware that could be tried if you want to mess with Arduino?

The different official firmware versions are found here: github.com/sparkfun/OpenLog

The standard (comes with openlog) has a 512byte rx buffer, light version has a 850byte rx buffer, minimal version has 1024byte. I was wanting to play with these and report back if there was any news, but I went down the rabbit hole of buying a china copy of the openlog that had no boot loader....so tried to add boot loader but bricked the atmega, so replace atmega only to notice that pads were damaged....hmm, it was said before, but don't bother with the china copied openlog hardware....get a sparkfun one if you don't want any hassle!

How much data is a UBLOX packet?
If I set my ebay ublox (firmware ROM 2.01 so tracks only 1 gps system) to NAV-PVT ubx, I get 100bytes per GPS packet.

Does the UBLOX tracking 2 or 3 systems have more data per packet?

Select to expand quote

srtgumbee said.. Just a FYI.... if it is the openlogger that is causing the issue, there are different versions of firmware that could be tried if you want to mess with Arduino?
>>>>

I've zero experience with arduino, but it may come to that if the SDlogger is no improvement.

Select to expand quote

srtgumbee said..
How much data is a UBLOX packet?
If I set my ebay ublox (firmware ROM 2.01 so tracks only 1 gps system) to NAV-PVT ubx, I get 100bytes per GPS packet.

Does the UBLOX tracking 2 or 3 systems have more data per packet?

www.u-blox.com/sites/default/files/products/documents/u-blox8-M8_ReceiverDescrProtSpec_%28UBX-13003221%29_Public.pdf

V2 is also fine, which model do you have?

Yes, it is 100 bytes per packet when only using NAV-PVT.
GPSResults expects NAV-DOP also, it is an advise to use those two as a minimum.
Although you can work with just PVT for display values.

Tracking more GNSS networks gives a lower SDOP value. Lower is better.

Select to expand quote

decrepit said..

sailquik said.. >>>>> Perhaps I should test @18 Hz with a faster SD card and see if it makes a difference?

Go for it Andrew! Why a question mark?

I just assumed, the phone is in control of writing to the SD card, it's probably got a huge buffer. A faster card may well work at 18hz, it would be interesting to find out.

Well, I would have to go and buy one.......

Select to expand quote

raymondw said.. >>> GPSResults expects NAV-DOP also, it is an advise to use those two as a minimum.
Although you can work with just PVT for display values.

Tracking more GNSS networks gives a lower SDOP value. Lower is better.

Raymond, I'm only using NAV-DOP with GPSResults, and that looks like enough. I get both dop values and SDoP numbers.
What else is needed?

Select to expand quote

raymondw said..

sailquik said..
As I have reported before. I have no dropped points in normal data running the NeoM8's (2 BT GPS tested) @10hz on 2 GNSS ( GPS and Glonass).
So I would be surprised if it is in the GPS module that is the bottleneck. We are also recording NAV-SAT @1Hz and both PVT and DOP @10Hz so there is a LOT of data recorded. Perhaps Manfred has inplemented some buffers in GPS-Logit/BT to write to the SD card in the phones? I am just using normal low cost MicroSD cards, nothing special. But I do run into regular missed points (roughly one every second) when I try to log at 18Hz from just the GPS/GNSS. It is also interesting that Roo and Manfred do not report these missing points @18Hz with their more expensive, higher spec phones.

Perhaps I should test @18 Hz with a faster SD card and see if it makes a difference?

NEO-M8 is the family name.
www.u-blox.com/sites/default/files/NEO-M8_DataSheet_%28UBX-13003366%29.pdf
Copy/paste gives a challenge, please look at page 6 :)

My REV-C uses the Q, every newer release uses the N.

As soon as I configure the datarate ABOVE what Ublox is advising I get missing data.
When I look into detail the issue is not dropped points, but data comes malformed from the unit.
Every GNSS update from the unit comes in sentences NAV-PVT, NAV-DOP etc etc etc
NAV-PVT comes before NAV-DOP, the first sentence is cut off when the second sentence starts.
Same when enabling NAV-SVINFO output.
PVT is cut when DOP starts and DOP is cut when SVINFO starts

And as you mention you have a dropped point every second, and you record NAV-SAT@1hz ;)

It would be nice to see how the Beitian handles this overclock as it is against the Ublox firmware recommendations.

For my units I recommend to follow the Ublox guidelines, although the user can configure to go above this, at their own risk.

Opps. I should have said M8N

Select to expand quote

decrepit said..
Raymond, I'm only using NAV-DOP with GPSResults, and that looks like enough. I get both dop values and SDoP numbers.

Mike, I believe you are mistaken here. NAV-PVT is 100% sufficient for analysis with GPSResults. NAV-DOP only adds antiquated DOP numbers that are always above the thresholds when there is even a halfway reasonable reception.

Select to expand quote

raymondw said..

When I tested last week, the 5Hz/2 GNSS data was better than 3Hz/3 GNSS file.
3Hz gave SDOP values around 0.1, +-13 sats in GPSResults
5Hz gave SDOP values around 0.01, +-19 sats in GPSResults

That makes no sense. Why would you have fewer sats with 3 GNSS systems? This looks like you had 2 GNSS at 3 Hz, and 3 GNSS at 5 Hz. Maybe you can share the files?

Select to expand quote

sailquik said..
So I would be surprised if it is in the GPS module that is the bottleneck.

The idea that the NEO-M8N is the bottleneck is based on the recommendations that u-blox, the company that makes the chips, has made in the PDF file that Raymond has given a link for. Note that the document also states "when executed from Flash", so this may not apply to all NEO-M8 modules.

Select to expand quote

sailquik said..
As I have reported before. I have no dropped points in normal data running the NeoM8's (2 BT GPS tested) @10hz on 2 GNSS ( GPS and Glonass).

How exactly do you look for missed points? In GPSResults, it's quite difficult to know if there are any missed points, if they are rare. My last 5 Hz / 3 GNSS Drotek data, for example, had one missing point every 10 minutes on average.

It's easy to see missing points in GPSAR Pro if you look at the trackpoint table, and sort the data by time difference in descending order. The little program I wrote to look at data gives a summary of time differences, and I think Raymond has something similar.

Select to expand quote

sailquik said..
Perhaps Manfred has implemented some buffers in GPS-Logit/BT to write to the SD card in the phones?

That's not really necessary on phones, since they have vastly more resources than Arduino-based solutions. A unit based on a small Arduino like the Openlog will have buffers in the range of 500 bytes to a few thousand bytes, which can overflow easily; a phone can easily expand buffers to megabytes, and store the speed data for the entire session in memory. Nor are phones likely to run into CPU bottlenecks, since they are typically quad-core nowadays.

The thing that is confusing here is that there are multiple causes for dropped points. They include (1) SD card limitations, (2) small buffer sizes in Arduino units, (3) speed limits when executing u-blox firmware from Flash, and probably others. Any one of these alone can cause dropped points, which makes it hard to pinpoint the exact cause of any problems. There is also additional variation that comes from changing numbers of satellites during reception, which changes the size of NAV-SAT messages.

Select to expand quote

boardsurfr said.. Mike, I believe you are mistaken here. NAV-PVT is 100% sufficient for analysis with GPSResults. NAV-DOP only adds antiquated DOP numbers that are always above the thresholds when there is even a halfway reasonable reception.

Woops, oldtimers disease! I meant to say NAV-PVT

Since there was some confusion about whether or not the u-blox 8 can use 3 GNSS systems at 10 Hz: Julien has confirmed that the Motion GPS can indeed record at 10 Hz using 3 GNSS systems, and that the 3-Hz limitation "only applies when running from external flash memory." (www.seabreeze.com.au/forums/Windsurfing/Gps/Motion-aqua-GPS-from-Belgium-seems-to-be-interesting-or-?page=1#22)

Select to expand quote

decrepit said.. >>>on the "rates" menu there's also a choice between "gps time" and "UTC time", the default was "gps time", so I've changed that to UTC, I'm hoping that will get rid of the annoying 18s time difference between the logger and the the GW52.
If the forecast eventuates I'll find out tomorrow.

Bugger that didn't work, I still have the 18s difference.

Select to expand quote

decrepit said..

decrepit said.. >>>on the "rates" menu there's also a choice between "gps time" and "UTC time", the default was "gps time", so I've changed that to UTC, I'm hoping that will get rid of the annoying 18s time difference between the logger and the the GW52.
If the forecast eventuates I'll find out tomorrow.

Bugger that didn't work, I still have the 18s difference.

Came across this
"The GPS navigation message includes the difference between GPS time and UTC. As of January 2017, GPS time is 18 seconds ahead of UTC because of the leap second added to UTC December 31, 2016. Receivers subtract this offset from GPS time to calculate UTC and specific timezone values. New GPS units may not show the correct UTC time until after receiving the UTC offset message. The GPS-UTC offset field can accommodate 255 leap seconds (eight bits)."
Perhaps, Mike, by restricting the sentences you are not getting that vital correction?

That's a possibility Dave, guess I'll have to put up with it then, unless I can come up with a way to write faster.