Skip to content

Read each GPX file once per run, and warn about it once - #912

Merged
caglarpir merged 1 commit into
mapillary:mainfrom
caglarpir:read-track-files-once
Oct 24, 2026
Merged

caglarpir merged 1 commit into
mapillary:mainfrom
caglarpir:read-track-files-once

Conversation

@caglarpir

Copy link
Copy Markdown
Contributor

Since #898, interval sampling with --geotag_source gpx reads the GPX
again for each video, to tell which samples fall outside its track.
Sampling 11 BlackVue videos every 30 s against a 163 MB GPX of 498
tracks, video_process took 293 s instead of 161 s before #898, and
printed "Found 498 tracks in the GPX file" 12 times instead of once.
The other readers parsed such a file once per video too: distance
sampling, in a pool of processes started for each video, and the
geotagging stage of videos, in each process of its pool. An NMEA file
is read once per process since #909.

Parse a GPX file once per process, and share its tracks among the
readers: interval and distance sampling, GPX sync of --geotag_source
and --video_geotag_source, and the geotagging stage of images. The
tracks are kept by the path and content of the file, so a file that
changes is parsed again, and only those of the last four files parsed
are kept, in the main process only. The geotagging stage of videos
reads a GPX that geotags several videos once, before its pool, as it
reads an NMEA file; the GPX of each video is still parsed in the pool.
Distance sampling reads its video in its own process, as interval
sampling does. GPX sync moves copies of the points. The warnings about
the tracks of a file, and about its points before 1980, are given once
per file.

That video_process run now takes 82 s and prints the warning once. Its
22 samples and their descriptions are the same, and its peak memory is
2.68 GB (2.84 GB on main). On one of the videos it takes 15.9 s instead
of 25.3 s, and process 10.1 s with 2.69 GB, as on main. Sampling the 11
videos by distance with the GPX as --video_geotag_source takes 25 s
instead of 107 s. process on them, with the GPX as --geotag_source or
--video_geotag_source, takes 22 s instead of 26 to 29 s, and its
processes peak at 6.4 GB together instead of 14 to 16 GB, as each of
them parsed the file. Compared with main on process, video_process and
sample_video, with GPX and NMEA sources and with the default sources,
on BlackVue, GoPro MAX, Sony and 70mai videos and on images, the
descriptions and samples are the same, and only the repeated warnings
are gone.

#898 said that with the legacy sources the frames before a late GPS
start were kept and geotagged as before: they are kept, and fail as
before.

Unit and integration tests, doctests, mypy --python-version 3.9, ruff check, ruff format --check and usort diff are clean.

Since mapillary#898, interval sampling with --geotag_source gpx reads the GPX
again for each video, to tell which samples fall outside its track.
Sampling 11 BlackVue videos every 30 s against a 163 MB GPX of 498
tracks, video_process took 293 s instead of 161 s before mapillary#898, and
printed "Found 498 tracks in the GPX file" 12 times instead of once.
The other readers parsed such a file once per video too: distance
sampling, in a pool of processes started for each video, and the
geotagging stage of videos, in each process of its pool. An NMEA file
is read once per process since mapillary#909.

Parse a GPX file once per process, and share its tracks among the
readers: interval and distance sampling, GPX sync of --geotag_source
and --video_geotag_source, and the geotagging stage of images. The
tracks are kept by the path and content of the file, so a file that
changes is parsed again, and only those of the last four files parsed
are kept, in the main process only. The geotagging stage of videos
reads a GPX that geotags several videos once, before its pool, as it
reads an NMEA file; the GPX of each video is still parsed in the pool.
Distance sampling reads its video in its own process, as interval
sampling does. GPX sync moves copies of the points. The warnings about
the tracks of a file, and about its points before 1980, are given once
per file.

That video_process run now takes 82 s and prints the warning once. Its
22 samples and their descriptions are the same, and its peak memory is
2.68 GB (2.84 GB on main). On one of the videos it takes 15.9 s instead
of 25.3 s, and process 10.1 s with 2.69 GB, as on main. Sampling the 11
videos by distance with the GPX as --video_geotag_source takes 25 s
instead of 107 s. process on them, with the GPX as --geotag_source or
--video_geotag_source, takes 22 s instead of 26 to 29 s, and its
processes peak at 6.4 GB together instead of 14 to 16 GB, as each of
them parsed the file. Compared with main on process, video_process and
sample_video, with GPX and NMEA sources and with the default sources,
on BlackVue, GoPro MAX, Sony and 70mai videos and on images, the
descriptions and samples are the same, and only the repeated warnings
are gone.

mapillary#898 said that with the legacy sources the frames before a late GPS
start were kept and geotagged as before: they are kept, and fail as
before.
@meta-cla meta-cla Bot added the cla signed label Oct 24, 2026
@caglarpir
caglarpir merged commit e050e97 into mapillary:main Oct 24, 2026
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant