[pbuncic]

Orchestration: certification by bits-orchestrator, SBOM scanning, build plan, several stores

This merges orchestration into main: 148 commits since 35b7773, with contributions from Juan Miguel Carceller (#134, #136, #138), Maria Folina (#139, #142) and Akrit Kumar Behera (#131). CHANGELOG.md (Unreleased) lists every change by area; the main lines are below.

bits-orchestrator (console-backend)

• A worker beside the console backend, with its own SQLite database: builds reconciled from GitLab and the S3 build index, failures deduplicated into issues, typed actions with approvals, and a hash-chained event log with an outside copy (audit-log service).
• Certification: the worker replays and then performs certification itself from the console pre-approval, as soon as a build is in the store. With BITS_ORCH_OWNS_SET=1 it owns the certified set: it writes numbered per-architecture manifests to MANIFESTS/, mirrors the set to bits-manifests and marks the store (MANIFESTS/orchestrator.json). Admins take builds or packages out with decertify and decertify-package; the worker then withdraws them from the signed manifests. bits then defers: bits certify stops after the approval and bits sign refuses the store.
• Build queue: console builds wait in the orchestrator, not in GitLab (where they counted against the job limit), and start one per free build host. Hosts with offline runners are skipped, and one passkey approves a whole batch's certifications.
• Supply chain: every published build's SBOM is scanned (bits-services sbom-scanner, grype), and rescanned when the vulnerability database updates. Community admins triage findings (VEX, including package-wide rules). Upgrade plans and licence and dependency analyses are computed from the scans. Separately, bits trace reports who built, published and approved a file on CVMFS.
• Credentials can go through a second security proxy, so neither the backend nor the worker holds a GitLab token or store key.

bits

• Build plan before every build: build-only dependencies of reused packages are neither fetched nor built (skip downloading only build_requires packages if a package is found. #141; --fetch-build-requires restores the old behaviour). The store is listed in parallel, and certified listings are cached in <work dir>/.store-cache/. An end-of-build summary says why each package was built.
• Stores: several read and write stores, each bucket with its own credentials (Multiple endpoints --remote-store and --write-store can we support multiple S3 buckets? #130). Builds work offline when a read store can't be reached, and fetch_cache caches a recipe's own downloads (Caching Support for additional sources. #129).
• Trust: manifests are fetched afresh each run (revalidated with their ETag). Entries are per architecture and hash, trust_manifest_prefix accepts several prefixes, and manifest sequence numbers prevent rollback to an older manifest.
• SBOM: recipe references: (CPEs and purls), versions of system packages, sbom: tool for host tools, and version pins.
• bits upgrade sets new versions in the recipes from a console upgrade plan.
• --symlink-packages-from (Adding an feature that help us create symlink of packages that are already deployed on cvmfs. #131, Akrit Kumar Behera): links packages from a deployed tree without modulefiles (CMS) into the work directory. Strict mode needs a matching .build-hash and, with signed reuse on, a certified one; a SYMLINK_STAGE recipe-repository hook handles SCRAM. The companion cms.bits change is a separate PR.
• Environment commands (q, enter, load, setenv) work from CVMFS without Python or a work directory, replacing bitsenv. A release with a complete view is set up through the view's setup.sh.
• Also: a macOS build sandbox, site configuration in site.d/NN-<name>.yaml, bits status hashes that now match the build's (--set, --docker, source form), regrouped bits -h, and CVMFS publish fixes.

Compatibility

• No package hash changes for existing recipes, and BUILD_EPOCH stays 0. The new hash-excluded inputs are references:, sbom:, system_package: and the SYMLINK_STAGE hook.
• Behaviour changes:
  • A defaults chain with a compiler but no build type now builds opt (e.g. key4hep::gcc15 → key4hep::gcc15::opt). This puts those builds under the -opt architecture, where the certified tarballs are.
  • Build-only dependencies of reused packages are skipped.
  • CERN builds read lcgapp-bits-testing.
  • bitsenv is removed.
• The build manifest gains additive fields (rebuilt_because, references, pinned_by, sbom, symlinked_from, symlink_policy); the schema version is unchanged.
• The orchestrator features are opt-in through environment variables (see console-backend/README.md). Without them the backend behaves as before.

Merging

main has d3ea1a6 (grep -R → -r in build_template.sh), which this branch has as cfa3bbe. The merge is clean. On main alone, test_build_epoch fails because that commit didn't update COVERED_SHA256; the merge fixes it.

Testing

• bits: 2568 passed, 37 skipped.
• console-backend: 573 passed, 2 skipped.
• test_build_epoch passes on the merge result.