Submodule-Tracking

One interface to four 2D point trackers (each linked as a git submodule, unmodified) plus the CoWTracker + TAPNext combination we use in the XM4D pipeline. Give it a video, get tracks [N,T,2] and visible [N,T].

--method	Tracker	Kind	Notes
tapnext	TAPNext / TAPNext++ (PyTorch)	online, 256x256	accurate short-to-mid term, queries can start at any frame
cotracker	CoTracker3 offline	offline, joint	queries tracked together with a support grid
spatracker	SpatialTrackerV2	offline, 3D-aware	also estimates depth + camera; slowest
cowtracker	CoWTracker	dense, reference-anchored	long-range; gated + jump-pruned
cow_tapnext	CoWTracker skeleton + TAPNext tracklets	hybrid	long tracks with TAPNext accuracy (see below)

By default the queries are a regular grid over the whole image at the query frame.

Install

git clone --recursive https://github.com/ComputationalRobotics/Submodule-Tracking.git
cd Submodule-Tracking            # (already cloned? git submodule update --init --recursive)

conda create -n tracking python=3.11 -y && conda activate tracking
pip install torch==2.4.1 torchvision==0.19.1 --index-url https://download.pytorch.org/whl/cu124
pip install -r third_party/SpaTrackerV2/requirements.txt
pip install -r requirements.txt
bash scripts/download_checkpoints.sh   # TAPNext++ (2.5 GB) and CoTracker3; SpaTrackerV2 / CoWTracker weights come from HF

All four trackers run in this single environment. TAPNext is imported without tapnet/__init__.py (which pulls in the JAX TAPIR model), so no JAX is needed.

Usage

python track_video.py --video clip.mp4 --method tapnext                        # 400-point grid on frame 0
python track_video.py --video frames/ --method cow_tapnext --n-points 800
python track_video.py --video clip.mp4 --method cotracker --spacing 16 --query-frame 30 --max-side 640
python track_video.py --video clip.mp4 --method spatracker --mask object.png   # grid restricted to a mask

Outputs results/<name>_<method>.npz (tracks x,y pixels, visible, queries (t,x,y), plus method extras) and an mp4 with the trails. From Python:

from tracking import METHODS, load_video, grid_queries
video = load_video("clip.mp4", max_side=640)                 # uint8 [T,H,W,3]
q = grid_queries(*video.shape[1:3], n_points=400, t=0)       # [N,3] (t, x, y)
res = METHODS["cow_tapnext"](video, q)                        # res.tracks [N,T,2], res.visible [N,T]

What each wrapper takes care of

Lessons from running these trackers on DyCheck / NVIDIA / DAVIS in XM4D (all handled outside the submodules):

• TAPNext. Queries are (t, y, x) in 256x256 pixels and tracks come back (y, x). Visibility uses the paper's rule sigmoid(visible_logit) * certainty > 0.5; frames before a query's start are invisible. Default checkpoint is TAPNext++ (tapnextpp_ckpt.pt); the original JAX .npz checkpoints also load.
• CoTracker3. CoTrackerPredictor thresholds visibility at 0.9 internally; we call the network directly and threshold the raw probability at 0.5 (much higher recall, same precision on our scenes). Raw probabilities are saved as vis_prob / conf.
• SpatialTrackerV2. The VGGT4Track front end attends over all frames at 518 px and runs out of memory beyond ~128 frames on 48 GB, so it runs in 64-frame chunks that share one frame (depth scale and poses stitched). The tracker's 200-frame windows also OOM, so S_wind=60. In inference it silently drops queries with depth confidence < 0.5 and pads the output with duplicates; outputs are matched back to the queries by nearest neighbour (dropped queries are NaN / invisible, count in n_dropped).
• CoWTracker. Input sides must be multiples of 56, not just 14: the decoder adds a 2x-upsampled 1/8 map to the 1/4 map, so H/4 and W/4 must be even (e.g. 448x252 fails, 448x224 works). The long side is resized to 448. Visibility = vis > 0.5 & conf > 0.5 (the demo's vis*conf > 0.1 keeps drifted points: they stay at vis ~0.5 with conf ~0.15). Large 2D jumps and drifted reappearances after occlusion are pruned (common.prune_large_moves). It is reference-anchored, not online: chaining it with frequent reference resets accumulates drift, so a mid-video query frame is tracked forward and backward from that frame instead.
• CoWTracker + TAPNext. CoWTracker gives a long-range but gappy skeleton; TAPNext tracklets started every restart frames on a denser grid are attached to a skeleton point when, over >= 5 co-visible frames, the 90th percentile of their distance is <= 12 px (at 480 px), then offset-corrected and averaged. Candidate pairs come from per-frame KD-tree radius queries (exact necessary condition), so attaching is fast even with thousands of tracklets.

Observations from our comparisons (masked dynamic objects, DyCheck)

• Fast-spinning objects (paper-windmill) are hard for every tracker: on-object visibility at the last frame was CoWTracker 100%, SpaTrackerV2 86%, CoTracker3 76%, TAPNext 2% (TAPNext loses points when it can't see them for long).
• TAPNext is the most accurate short-term tracker (zoomed / windowed it tracks well within a window); CoWTracker is the long-range one. Neither alone keeps blade identity on the windmill; attaching TAPNext tracklets to a CoWTracker skeleton gave the longest consistent tracks, which is why the XM4D pipeline uses it.
• When an object leaves and re-enters the frame (DyCheck wheel), CoWTracker and CoTracker3 recover ~87% of points.

Layout

third_party/   tapnet, co-tracker, SpaTrackerV2, cowtracker   (git submodules, upstream code untouched)
tracking/      common.py, tapnext.py, cotracker.py, spatracker.py, cowtracker.py, cow_tapnext.py
track_video.py example CLI
scripts/       download_checkpoints.sh

Each submodule keeps its own license; see the respective repositories.