What i've driven so far, (blue are the roads where i've been)
Somewhere in 2023 i tought it was a great idea to ride trough EVERY street in the city of Groningen.
Since i have a bike with the Bosch smart system the app for my bike tracks all rides. It has wondered me for years already how manny streets do i actually visit of the city where i live.
So after a long time thinking about it i finally decided to import all rides into a map and see where i've written. To my disapointment i discoverd that i've bearly seen any of the city i live in.
Since then i've started to ride trough the city and try to visit as manny streets as possible.
At first i used the Bosch Flow App but i've found a few problems with it tracking my rides.
The first problem is that exporting the rides is very annoying.. I need to export every ride one by one and it takes around 10 taps to export a single ride to my Google Drive. I would like to export rides to my phone's storage but the Bosch Flow App only allows me to export rides to a Cloud Storage like why 😠.
Another problem is that i'm limited to tracking when i'm riding my bike, as my end goal is to have visited all of Groningen i must walk some part and i cannot do that with the Bosch Flow App.
Beside these practical problems i also discoverd that the tracking rate is quite slow (I think it records a tracking point every 5 seconds) and for small streets i might be quicker trought the street than the App adds a tracking location.
So after some time trying other apps i discoverd Geo Tracker the most generic name every i know but the app seems to respect my privacy and have all the options i ever need for tracking my rides. I've bought their the pro subscription for 0.60 euro's a month so i can use the MapBox maps within the app over Google their maps as i like their style more.
I use Android's split screen function to place a screenshot of where i've been and the Geo Tracker below to see where i'm currently riding.
This is not an ideal setup as apps get very very very small when using split screen. I might have a big phone Google Pixel 7 but still it kinda works.
I don't use a phone mount on my bike i rather just have my phone in my pocket and pull it out every second street to see where i should go.
There is also the problem that i'm a cheap boii and i have a pre paid sim card so internet on the road is basicly not a thing for me. So maps need to be available offline. Mapbox seems to not have offline support but when pre zooming in at home with wifi the map seems to be cached and i can use it offline.
This was a challenge and futer below i will get to the technical details but first i will explain in a broad way how i do it.
I firstly export all my rides in the Geo Tracker app to GPX files.
Then i move the files to my desktop using LocalSend.
On my PC i run a program over all the GPX files and convert them to a single csv file with all location points i've ever been. This program also remove points that are close very close to each other for example 2 points that are within 1 meter distance of each other do not really make a difference on a map so close together points get removed.
I've made a website that uses MapBox to show a map and overlay the points using their APIs.
As the points are so close together they will quickly start to look like lines when zoomed out, see the image above for an example. I would like to one day change convert these into lines but when you've driven over 1000 times trough 1 street they get a bit messy with GPS points so converting those to lines is a bit of a challenge that i'm still thinking about.
My goals here are a few tings
That one is mostly easy. It basicly comes down to reading a hunders of .GPX files with thousands of points, collecting these points and writing them to a single file.
For some reason it's "slow" to read xml files and it usually took around 30 seconds to read all the .GPX files and this will only become slower once i start riding my bike more because of this project so this was a no go for me.
I've done a few things to speedup the GPX file reading process.
Switch from a Deno (javascript) script to a Rust script tough nowadays i would more call it a program.
This wiredly did not improve the speed of reading the GPX file much tough i was using a specialized lib for reading GPX files.
So the next thing i tried was repacing the gpx rust lib with a fast xml reader named quick-xml and that did indeed improve the speed of reading the GPX files by i would say 50%.
Tough still i was not happy why should reading about 1 million (more like 3 nowadays) lines of xml take 15 seconds??
So i started looking at multithreading using a thread pool and holyshit that made a difference. After this changed it took about 350ms on my ryzen 16 core cpu to read all the GPX files and around another 50ms to write the results to disk.
TL;DR:
This one requires some thinking from the start. This is what i'm currently doing.
Firstly Lets chunk all points in a grid so when doing any sort of comparing we can reduce the amount of points we need to compare and we can probably multithread this.
The area size is about 100m * 100m but don't pin me on this it was a bit of trail and error to find the best size compared to the number of chunks. Thankfully i live around 53 degrees away from the equator so using squares based on the lat lon system to chunk cordinates "looks" quite square on a sphere earth, the lat and log cordinates system projects nicely on a rectangle map but when moving this system to a sphere the distance between 2 points is not the same everywhere hence why things as the Haversine formula exists.
So now that i've placed all the points in a grid i can start to optimize every chunk.
I'm doing the following things to optimize a chunk.
For every chunk i'm making a new list of points for the chunk.
For every point in the chunk i check i compare them to all other points in the new list and if the distance between the 2 points is less than 5 meter i skip the point.
This process is executed in a thread pool within 300ms on my ryzen 16 core cpu.
Filtering out dots
There is currently no way to see what percentage of Groningen i have completed and i'm not yet sure how i should track this. Should i even track it is looking at the map enough? When do i even consider a street as completed?
When having long streets it should just convert all the points to a single line to reduce the amound of points shown on the map but as said earlier i'm not yet sure how i should do this.
Quicker way to export rides from the Geo Tracker app then move them to my PC and run a program to optimize them and commit them to a repo.