[chinyeungli]

Issues

• Closes: nixpkgs-clarity: Design and implement Scancode pipeline for single nixpkg #1938

Changes

This PR creates a dedicated pipeline for Nix packages.

Requirements

• This pipeline only works on packages that found in the official nixpkgs.
• It takes a single Nix PURL as an input (i.e. pkg:nix/nixpkgs/...)
• A 'version' or a 'commit' qualifier is required in the input PURL as this is needed to fetch the correct source and binary for deployment to development mapping scan.
• It also requires the input PURL to have a "system" qualifier to resolve system-specific binaries.
• Docker is required. The code use docker to build sources by applying patches/configs that's defined in the upstream .nix files.
• If the PURL has no output qualifier, the pipeline tries debug first (dwarf symbols are needed for D2D mapping) and falls back to out. If the PURL specifies an output, only that one is used.

Summary

• The code will get metadata from https://search.devbox.sh and find and fetch the binary package from https://cache.nixos.org/
• If a Nix package cannot be found in https://cache.nixos.org/ and a commit hash is provided, the code will try to fetch and build the source from https://github.com/NixOS/nixpkgs and patch the sources using the exact configuration defined in the upstream .nix files.
• The pipeline will check if there is any license mismatch between the codebase and the declared license from the detected packages

Notes for deployment to development mapping

As Nix is not tied to a single programming language ecosystem, this pipeline executes all currently supported D2D steps across multiple languages.

Handling Missing Debug Symbols:

For pkg:nix/nixpkgs/SDL_mixer@1.2.12?system=x86_64-linux , as seen in https://search.devbox.sh/v2/pkg?name=SDL_mixer , there is no debug build found under the "outputs" section. Therefore, even the ELF binary is not stripped:

there is no dwarf symbols collected and no development to deployment mapping can be performed

Template File Mapping (.in files):

For pkg:nix/nixpkgs/openssl@3.6.0?out=debug&system=x86_64-linux, following are some of the dwarf symbols collected:

include/openssl/asn1.h
include/openssl/asn1t.h

However, these exact header files do not exist in the source codebase; they are generated from GNU Autoconf template files. To complete the mapping, the code has been updated to map to the corresponding .in template files:

Source-Only Scans:
For pkg:nix/nixpkgs/haskellPackages.aasam@0.2.0.0?system=x86_64-linux&commit=a3ae4cdd64f675cf2580affecdee01c401b43638 , this package does not exist on https://search.devbox.sh/, so no binary can be fetched, and the local Nix build fails because the package declares dependencies incompatible with the GHC version in this nixpkgs commit. The pipeline still fetches the patched source tree and runs the license scan; only the D2D mapping is skipped.:

Design notes

• Custom Nix image: Built on first use with zstd, xz, bzip2, and gzip installed. Without this, nix-shell -p fetches the decompressors on every extraction, which hits the CDN's rate limit.
• Local nixpkgs cache: A bare git clone of nixpkgs lives on the host. Each scan shallow-fetches its commit and creates a temporary worktree, bind-mounted into the container read-only. Avoids GitHub rate limits.
• Concurrent scans: RQ runs each scan in its own worker process. File locks serialize image builds, git operations, and GC.
• Nix store GC: The nix-eval-cache volume is pruned with nix-collect-garbage --delete-older-than 12h, at most once every 3 hours, so disk usage stays bounded.

Notes

Because the pipeline relies on a Linux-based Docker container (nixos/nix) to build sources, providing a PURL for a non-Linux system (e.g., darwin) triggers a system barrier warning.
When this happens, the pipeline safely falls back to evaluating the source using the container's native Linux environment. The extracted source tree will contain Linux-specific patches rather than macOS patches. The impact on the d2d mapping is expected to be minimal, though a small number of files may remain unmapped due to missing OS-specific structural patches.

Also, there is no Nix Package detected until aboutcode-org/scancode-toolkit#2823 is resolved.

Checklist

• I have read the contributing guidelines
• I have linked an existing issue above
• I have added unit tests covering the new code
• I have reviewed and understood every line of this PR

[chinyeungli]

pipes/utils.py is still a generic module, with Docker helpers only used by nix.py.

I'd prefer to keep them in utils.py rather than move them to nix.py, because other pipelines (e.g. Rust) are likely to need the same "run a build in a container" pattern.

I've also introduced uncovered_licenses to calculate which licenses are detected but not declared.

so nearly every package will get the "License Mismatch" issue and the flag becomes noise

I understand the flag might become noise, but without it, users may not notice the license differences.

[tdruez]

@chinyeungli No need for a new PR: since you merged main, the "Files changed" tab only shows the 26 files of this PR, and there is no conflict.

Most of the previous points are addressed, see my new comments.

I'd prefer to keep them in utils.py rather than move them to nix.py, because other pipelines (e.g. Rust) are likely to need the same "run a build in a container" pattern.

Then let's name the module after what it does, pipes/container.py for example. A generic utils.py ends up collecting unrelated helpers.

[tdruez]

@chinyeungli Let's take a step back on this PR. It has grown too big for a single pipeline, and I should have raised it earlier instead of reviewing it line by line.

To run one pipeline, the PR currently adds:

• An opt-in privileged Docker-in-Docker service in docker-compose.yml, with TLS certificates and an extra service to fix their permissions.
• The Docker CLI in every ScanCode.io image, new Makefile targets, and changes in docker.env.
• 5 new settings.
• A local clone of nixpkgs, file locks, and a Nix store garbage collection run by the pipeline.
• Local builds of the binary when it is not available in the Nix binary cache.
• A new DiscoveredPackage field and a migration, for a check only this pipeline uses.

This is overkill for a pipeline that few users will run. It makes the whole stack heavier for everyone, and it's a lot of code to maintain in the long run.

Part of this work will be reused (PURL validation, metadata retrieval, D2D, tests). But rather than adding more rounds on this branch, I'd keep it open for reference and restart with a smaller, clean PR.

Before writing code, let's agree on the plan in #1938. There is no defined approach there yet, only the initial scenario. A first version could be:

• Input: a nix PURL.
• Download the binary from the Nix binary cache, and the source. No local build.
• Run D2D between them, scan the source, and compare the detected licenses with the declared license.

Questions to answer first:

1. Do we need Docker at all, or can the pipeline call nix directly?
2. Is a local nixpkgs clone needed for a first version?
3. Which settings are really required?
4. Where do we store the license check result without changing the data model?

The other points of the scenario (source repository check, MatchCode matching) can come later.