Skip to content

Import music tracks from a library's web page - #48

Merged
davior merged 2 commits into
mainfrom
claude/admiring-planck-ggdijx
Oct 26, 2026
Merged

davior merged 2 commits into
mainfrom
claude/admiring-planck-ggdijx

Conversation

@davior

@davior davior commented Oct 26, 2026 •

Copy link
Copy Markdown
Owner

You can now paste a page from a music library into Add → "Or paste a page of music tracks" and press Find tracks. GAM lists the tracks on the page with their details. You filter the list, tick the tracks you want (up to 50 at a time) and import them. Each track becomes its own background job: it is downloaded, catalogued like an upload and given the page's details. The tempo, key and energy analysis from davior/gam#47 then fills in whatever the page left out.

How a page is read (app/ingest/page_tracks.py)

Sources are tried in order, most reliable first, and the review list says which one was used:

Source Gives
incompetech Its page builds the track list in the browser from pieces.json, so GAM reads that file directly. Title, genre and album (names taken from the lookup tables in the page's own script), feel (as moods), instruments, BPM, length, description, date, and the licence's exact CC BY attribution text as the credit line.
schema.org MusicRecording JSON-LD Title, artist, genre, duration, date, licence. This is what sites publish for search engines.
mixkit (on top of its JSON-LD) The moods, instruments and tags linked from each track's card, matched to the track by MP3 URL, plus the mood or instrument the listing is filtered to.
AI provider (fallback, can be turned off) Reads the page text. It may only name audio URLs that actually appear on the page; any other URL it returns is dropped. Costs one completion, recorded in Usage.
bare links Every linked or embedded audio file, titled by its link text.

Duplicates

Check When Catches Then
URL In the review list, and again in the job A track already imported from this address Shown as In library and can't be ticked. If an import of it runs anyway, the job points at the existing copy.
Checksum (SHA-256) In the job, after downloading The same file at another URL, or a file uploaded by hand The download is discarded. The page's details only fill the existing copy's blanks, so a person's corrections stay. Its source URL is recorded, so the URL check catches it next time.
Title + artist In the review list The same piece downloaded elsewhere as a different file Shown as Maybe in library. It can still be ticked, because titles collide, but Tick shown skips it.

A track ticked twice in one import becomes a single job.

Downloading (app/ingest/web.py)

  • An SSRF check runs on every redirect hop.
  • Audio is accepted only uncompressed and is capped at 200 MB while it streams in.
  • A response that isn't audio (a login wall, an interstitial page) is refused rather than catalogued.
  • Requests identify themselves as GAM, and downloads run one at a time on the enrichment queue.
  • Audio URLs that come back from the browser are checked again before any job is queued, and dates are normalised.

What lands on the asset

The page's details are stamped embedded:

  • creator, publisher, album, licence and credit line;
  • source URL and retrieval time;
  • genres, moods, instruments and BPM;
  • the page's tags, attached as library tags.

Checked against your saved pages

  • Mixkit: "Carefree" (13 tracks) and "Strings" (10 tracks), each with artist, duration, genre, moods, instruments, tags and licence.
  • incompetech: the saved page confirms it loads pieces.json. I couldn't reach the file itself, so the adapter reads exactly the fields the page's own script uses, and the test fixture follows that shape.
  • The committed fixtures are trimmed, hand-made copies of each page's structure. No third-party page is committed.

Design notes are in docs/page-import.md.

Testing

  • pytest -q: 1370 passed. tests/test_page_import.py covers:
    • each adapter;
    • the AI fallback, including that an invented URL is dropped;
    • bare links;
    • the import job;
    • all three duplicate checks, including that corrections on the existing copy survive;
    • SSRF on a redirect, a non-audio response, the size cap, and the per-import limit.
  • npm run format:check, npm run lint, npm test (425 passed) and npm run build all clean.

🤖 Generated with Claude Code

https://claude.ai/code/session_01DCL4jm6ogZBBhnjEfBBaXU

Paste a page (incompetech, Mixkit, or any page listing tracks), review
what it holds, tick what you want, and import each as its own job with
the page's details on it.

- Read best source first: incompetech's pieces.json (the file its page
  loads), schema.org MusicRecording JSON-LD, Mixkit's card links for
  moods, instruments and tags, then the AI provider (held to the audio
  URLs actually on the page), then bare audio links.
- ingest/web.py: SSRF-checked on every redirect, capped while streaming,
  uncompressed audio only, non-audio responses refused.
- Page metadata lands stamped "embedded": credits (incompetech's exact
  CC BY wording), genres, moods, instruments, BPM and tags. Tempo, key
  and energy the page left out are then measured from the sound.
- Tracks already in the library are marked and not downloaded twice;
  at most 50 per import.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DCL4jm6ogZBBhnjEfBBaXU
The URL alone missed the same file under another address and files
uploaded by hand. Now:

- Checksum: after downloading, the file's SHA-256 is matched against the
  library. A copy already here is kept, the download discarded, and the
  page's details only fill its blanks (corrections stay); its source URL
  is recorded so the URL check catches it next time.
- A track ticked twice in one import is one job.
- The review list flags "Maybe in library" when the title (and artist,
  when both are known) matches an audio file already here: a different
  file of the same piece. Still tickable; "Tick shown" skips it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DCL4jm6ogZBBhnjEfBBaXU
@davior
davior marked this pull request as ready for review October 26, 2026 11:33
@davior
davior merged commit ea6d60e into main Oct 26, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants