From cf164189665d2af6cbd2b2e35e012806026e729e Mon Sep 17 00:00:00 2001 From: Jason E Date: Mon, 24 Aug 2026 19:02:18 -0500 Subject: [PATCH 001/194] Add request phase timing design spec and implementation plan (#1069) * Add request phase timing design spec (Server-Timing subtimings + access telemetry) * Address review round 1: freeze point, template-cache naming, snapshot semantics, KV scope, geo carry, route template, sink confirmation, sampling and query model, config rollback * Address review round 2: auction-wait placement modes, conservative private-only header emission, non-null sorting key with service identity, coarse publisher route template, telemetry snapshot and outage behavior, tinybird flag decoupling, adapter phase semantics * Add request phase timing implementation plan * Address engineer review: KV timing decorator, try_lock sampling, route metadata extension, adapter-derived env, typed template-cache state, adapter-owned emission context, per-mode delivery semantics, Axum outer wrapper --- .../plans/2026-08-24-request-phase-timing.md | 1038 +++++++++++++++++ .../2026-08-24-request-phase-timing-design.md | 553 +++++++++ 2 files changed, 1591 insertions(+) create mode 100644 docs/superpowers/plans/2026-08-24-request-phase-timing.md create mode 100644 docs/superpowers/specs/2026-08-24-request-phase-timing-design.md diff --git a/docs/superpowers/plans/2026-08-24-request-phase-timing.md b/docs/superpowers/plans/2026-08-24-request-phase-timing.md new file mode 100644 index 0000000000..9ff3905f42 --- /dev/null +++ b/docs/superpowers/plans/2026-08-24-request-phase-timing.md @@ -0,0 +1,1038 @@ +# Request Phase Timing Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Every application response attributes its own server time by phase via a +Server-Timing header and a sampled Tinybird access-telemetry row. + +**Architecture:** A core `RequestTimings` handle (Arc-shared, infallible recording) +collects phase spans always-on; the Fastly adapter freezes and emits at +`send_edgezero_response` immediately before `into_parts()`; a post-send emitter ships +one NDJSON row to the Tinybird Events API with a bounded, 2xx-validated await. + +**Tech Stack:** Rust 2024, `edgezero` HTTP types, Fastly Compute (wasm32-wasip1, +Viceroy tests), Axum (native tests), Tinybird Events API. + +**Spec:** `docs/superpowers/specs/2026-08-24-request-phase-timing-design.md`: the +plan argues from the spec; executors read both. Spec section numbers are cited per +task. + +## Global Constraints + +- Errors use `error-stack` (`Report`); errors defined with + `derive_more::Display`; never thiserror, never anyhow (except the Spin entry point). +- No `unwrap()` in production code; `expect("should ...")` only. Assertion messages + `"should ..."`. Tests use Arrange-Act-Assert. +- No inline comments; comments on their own line above the code. +- Functions never exceed 7 arguments; use a struct instead (this bit + `ec_finalize_response` in review; the timings handle travels inside existing state). +- No local imports inside functions; `use super::*` only in `#[cfg(test)]`. +- Only example/fictional data in tests and docs (`example.com` domains). +- Recording is infallible: saturating math, lock failure drops the sample, no panics + (spec 5, 13). +- Vendor identity never appears in emitted surfaces: the filter span is `ts-filter` + (spec 3). +- Test commands: `cargo test-axum` (native, fast inner loop), `cargo test-fastly` + (Viceroy) for adapter tasks. Before PR handoff: the full CI gate list in + `CLAUDE.md`. +- Commit style: sentence case, imperative, no prefixes, no trailers. + +## File Structure + +| File | Responsibility | +| ----------------------------------------------------------------- | -------------------------------------------------------------------------------------------------- | +| `crates/trusted-server-core/src/request_timing.rs` (new) | `Phase`, `AuctionWaitPlacement`, `RequestTimings`, `PhaseSpan`, `TimingSnapshot`, header rendering | +| `crates/trusted-server-core/src/access_telemetry.rs` (new) | `RouteClass`, `publisher_route_template`, `AccessTelemetrySnapshot`, `AccessEventRow` NDJSON | +| `crates/trusted-server-core/src/geo.rs` (modify) | `GeoLookupState` response-extension type | +| `crates/trusted-server-core/src/settings.rs` (modify) | `ObservabilitySettings`, tinybird flag decoupling, access validation | +| `crates/trusted-server-core/src/publisher.rs` (modify) | `ts-origin`, `ts-template-cache`, auction-wait spans | +| `crates/trusted-server-core/src/ec/kv.rs` (modify) | `ts-kv` at the graph abstraction | +| `crates/trusted-server-adapter-fastly/src/main.rs` (modify) | T0, appbuild span, freeze point, `DeliveryOutcome`, post-send emission ordering | +| `crates/trusted-server-adapter-fastly/src/app.rs` (modify) | filter span, geo span + `GeoLookupState` attach, route class assignment | +| `crates/trusted-server-adapter-fastly/src/middleware.rs` (modify) | finalize consumes `GeoLookupState` | +| `crates/trusted-server-adapter-fastly/src/tinybird.rs` (modify) | access sink with confirmed delivery | +| `crates/trusted-server-adapter-axum/src/` (modify) | terminal freeze layer, header emission | +| `tinybird/datasources/access_logs_raw.datasource` (modify) | phase-column schema, non-null sorting key | +| `trusted-server.example.toml` (modify) | `[observability]`, tinybird keys | + +Out of scope for this plan: the Grafana dashboard JSON (separate telemetry repo, +spec 11) and Cloudflare/Spin emission wiring (spec non-goal). + +--- + +### Task 1: Core `RequestTimings` + +**Files:** + +- Create: `crates/trusted-server-core/src/request_timing.rs` +- Modify: `crates/trusted-server-core/src/lib.rs` (add `pub mod request_timing;`) +- Test: same file, `#[cfg(test)]` + +**Interfaces:** + +- Consumes: nothing (leaf module; `std::time`, `std::sync`). +- Produces (later tasks rely on these exact names): + - `pub enum Phase { AppBuild, Filter, Geo, EcKv, Origin, TemplateCacheLookup, AuctionWait, Stream }` + - `pub enum AuctionWaitPlacement { PreHeader, InStream }` + - `#[derive(Clone)] pub struct RequestTimings` with: + - `pub fn new() -> Self` + - `pub fn record(&self, phase: Phase, dur: Duration)` (saturating accumulate) + - `pub fn record_auction_wait(&self, placement: AuctionWaitPlacement, dur: Duration)` + - `pub fn span(&self, phase: Phase) -> PhaseSpan` (records on drop) + - `pub fn mark_headers_ready(&self)` (first call wins) + - `pub fn mark_request_elapsed(&self)` (first call wins) + - `pub fn set_resp_bytes(&self, bytes: u64)` + - `pub fn server_timing_value(&self) -> Option` + - `pub fn snapshot(&self) -> TimingSnapshot` + - `pub struct TimingSnapshot { pub time_elapsed_ms: Option, pub request_elapsed_ms: Option, pub appbuild_ms: Option, pub filter_ms: Option, pub geo_ms: Option, pub kv_ms: Option, pub origin_ms: Option, pub template_cache_ms: Option, pub auction_wait_ms: Option, pub stream_ms: Option, pub auction_wait_placement: Option, pub resp_bytes: Option }` + +- [ ] **Step 1: Write the failing tests** + +```rust +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn render_omits_unrecorded_phases_and_orders_total_first() { + let timings = RequestTimings::new(); + timings.record(Phase::Filter, Duration::from_micros(9_100)); + timings.mark_headers_ready(); + let value = timings + .server_timing_value() + .expect("should render after mark_headers_ready"); + assert!( + value.starts_with("ts-total;dur="), + "should lead with ts-total: {value}" + ); + assert!(value.contains("ts-filter;dur=9.1"), "should render one decimal: {value}"); + assert!(!value.contains("ts-geo"), "should omit unrecorded phases: {value}"); + } + + #[test] + fn render_returns_none_before_headers_ready() { + let timings = RequestTimings::new(); + timings.record(Phase::Geo, Duration::from_millis(1)); + assert!(timings.server_timing_value().is_none(), "should require the snapshot"); + } + + #[test] + fn repeated_phases_accumulate_saturating() { + let timings = RequestTimings::new(); + timings.record(Phase::Geo, Duration::from_millis(2)); + timings.record(Phase::Geo, Duration::from_millis(3)); + timings.mark_headers_ready(); + let snapshot = timings.snapshot(); + assert_eq!(snapshot.geo_ms, Some(5), "should accumulate repeats"); + } + + #[test] + fn mark_headers_ready_is_first_call_wins() { + let timings = RequestTimings::new(); + timings.mark_headers_ready(); + let first = timings.snapshot().time_elapsed_ms; + std::thread::sleep(Duration::from_millis(5)); + timings.mark_headers_ready(); + assert_eq!(timings.snapshot().time_elapsed_ms, first, "should not restamp"); + } + + #[test] + fn span_guard_records_on_drop() { + let timings = RequestTimings::new(); + { + let _span = timings.span(Phase::Origin); + std::thread::sleep(Duration::from_millis(2)); + } + timings.mark_headers_ready(); + assert!( + timings.snapshot().origin_ms.expect("should record on drop") >= 1, + "should measure elapsed span time" + ); + } + + #[test] + fn auction_wait_records_placement() { + let timings = RequestTimings::new(); + timings.record_auction_wait(AuctionWaitPlacement::PreHeader, Duration::from_millis(40)); + let snapshot = timings.snapshot(); + assert_eq!(snapshot.auction_wait_ms, Some(40), "should record wait"); + assert_eq!( + snapshot.auction_wait_placement, + Some(AuctionWaitPlacement::PreHeader), + "should record placement" + ); + } + + #[test] + fn rendered_names_never_include_vendor_terms() { + let timings = RequestTimings::new(); + for phase in [Phase::AppBuild, Phase::Filter, Phase::Geo, Phase::EcKv, Phase::Origin, Phase::TemplateCacheLookup] { + timings.record(phase, Duration::from_millis(1)); + } + timings.mark_headers_ready(); + let value = timings.server_timing_value().expect("should render"); + assert!(!value.to_ascii_lowercase().contains("datadome"), "should mask vendors"); + } +} +``` + +- [ ] **Step 2: Run to verify failure** + +Run: `cargo test-axum -p trusted-server-core request_timing` +Expected: compile FAIL, module does not exist. + +- [ ] **Step 3: Implement** + +```rust +//! Per-request phase timing collection and Server-Timing rendering. +//! +//! Collection is always-on and infallible: saturating math, lock failure +//! drops the sample, no panics. See the design spec +//! `docs/superpowers/specs/2026-08-24-request-phase-timing-design.md`. + +use std::sync::{Arc, Mutex}; +use std::time::{Duration, Instant}; + +const PHASE_COUNT: usize = 8; + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum Phase { + AppBuild, + Filter, + Geo, + EcKv, + Origin, + TemplateCacheLookup, + AuctionWait, + Stream, +} + +impl Phase { + fn index(self) -> usize { /* match self -> 0..=7 */ } + + /// Header entry name; row-only phases return None. + fn header_name(self) -> Option<&'static str> { + match self { + Self::AppBuild => Some("ts-appbuild"), + Self::Filter => Some("ts-filter"), + Self::Geo => Some("ts-geo"), + Self::EcKv => Some("ts-kv"), + Self::Origin => Some("ts-origin"), + Self::TemplateCacheLookup => Some("ts-template-cache"), + Self::AuctionWait | Self::Stream => None, + } + } +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum AuctionWaitPlacement { + PreHeader, + InStream, +} + +struct Inner { + t0: Instant, + phases: [Option; PHASE_COUNT], + headers_ready_total: Option, + request_elapsed: Option, + auction_wait_placement: Option, + resp_bytes: Option, +} + +#[derive(Clone)] +pub struct RequestTimings(Arc>); +``` + +Implementation notes (all bodies in this task, none deferred): + +- Every method takes `if let Ok(mut inner) = self.0.try_lock()` and silently + returns otherwise: contention and poison both drop the sample instead of waiting, + per the infallibility constraint. +- `record` accumulates with `saturating_add` semantics + (`Some(existing.saturating_add(dur))`). +- `mark_headers_ready` and `mark_request_elapsed` write `t0.elapsed()` only when the + slot is `None`. +- `server_timing_value` returns `None` unless `headers_ready_total` is set; renders + `ts-total` first from the stored snapshot, then the six header phases in enum order + with `{:.1}` millisecond formatting (`dur.as_secs_f64() * 1000.0`). +- `PhaseSpan { timings: RequestTimings, phase: Phase, started: Instant }`; `Drop` + calls `record(self.phase, self.started.elapsed())`. +- `TimingSnapshot` converts each `Duration` with + `u32::try_from(dur.as_millis()).unwrap_or(u32::MAX)`. +- `impl Default for RequestTimings` delegates to `new()`. + +- [ ] **Step 4: Run to verify pass** + +Run: `cargo test-axum -p trusted-server-core request_timing` +Expected: all 7 tests PASS. + +- [ ] **Step 5: Commit** + +```bash +git add crates/trusted-server-core/src/request_timing.rs crates/trusted-server-core/src/lib.rs +git commit -m "Add RequestTimings phase collection and Server-Timing rendering" +``` + +--- + +### Task 2: Settings: `[observability]`, tinybird decoupling, access validation + +**Files:** + +- Modify: `crates/trusted-server-core/src/settings.rs` +- Modify: `trusted-server.example.toml` + +**Interfaces:** + +- Produces: + - `pub struct ObservabilitySettings { pub server_timing_enabled: bool }` as + `settings.observability`, `#[serde(default)]` on the field and + `#[serde(skip_serializing_if = "ObservabilitySettings::is_default")]`. + - `TinybirdSettings.auction_enabled: bool` (`#[serde(default = "default_true")]`). + - `prepare_runtime` validation: `access_enabled` requires `enabled`, non-empty + `api_host`, `secret_store`, `access_dataset`, `access_token_secret`, + `max_body_bytes > 0`, and `access_sample_rate > 0.0`. + +- [ ] **Step 1: Write the failing tests** (in `settings.rs` tests module) + +```rust +#[test] +fn observability_defaults_off_and_serializes_away() { + let settings = create_test_settings(); + assert!(!settings.observability.server_timing_enabled, "should default off"); + let toml = toml::to_string(&settings).expect("should serialize settings"); + assert!( + !toml.contains("[observability]"), + "should omit the default table so a prior binary can parse the config" + ); +} + +#[test] +fn access_enabled_requires_positive_sample_rate() { + // access_enabled = true with access_sample_rate = 0 is armed-but-silent: an error. + let err = settings_from_toml_with( + "[tinybird]\nenabled = true\napi_host = \"api.example.com\"\naccess_enabled = true\naccess_sample_rate = 0.0\n", + ) + .expect_err("should reject armed-but-silent access telemetry"); + assert!(format!("{err:?}").contains("access_sample_rate"), "should name the field"); +} + +#[test] +fn access_and_auction_emission_are_independent() { + let settings = settings_from_toml_with( + "[tinybird]\nenabled = true\napi_host = \"api.example.com\"\nauction_enabled = false\naccess_enabled = true\naccess_sample_rate = 1.0\n", + ) + .expect("should accept access without auction"); + assert!(!settings.tinybird.auction_enabled, "should disable auction emission"); + assert!(settings.tinybird.access_enabled, "should enable access emission"); +} + +#[test] +fn auction_enabled_defaults_true_for_existing_configs() { + let settings = settings_from_toml_with("[tinybird]\nenabled = true\napi_host = \"api.example.com\"\n") + .expect("should parse a pre-decoupling config"); + assert!(settings.tinybird.auction_enabled, "should preserve current behavior"); +} +``` + +Also REPLACE the existing rejection test +(`tinybird_access_enabled_is_rejected_until_emitter_is_wired`, `settings.rs:4123`) +with a wiring test asserting a fully-specified access config is accepted. + +- [ ] **Step 2: Run to verify failure** + +Run: `cargo test-axum -p trusted-server-core observability access_enabled auction_enabled` +Expected: compile FAIL (`observability` field missing). + +- [ ] **Step 3: Implement** + +- Add `ObservabilitySettings` (derive `Debug, Clone, Default, PartialEq, Deserialize, +Serialize`, `#[serde(deny_unknown_fields)]`), with + `fn is_default(&self) -> bool { *self == Self::default() }`. +- Add the `observability` field to `Settings` with the serde attributes above. +- Add `auction_enabled` to `TinybirdSettings` with `default_true()`; update + `Default for TinybirdSettings`. +- Extend `TinybirdSettings::prepare_runtime` with the access validation matrix; error + messages name the failing field (`"tinybird.access_sample_rate must be > 0 when +access_enabled"` and so on). +- `trusted-server.example.toml`: add a commented `[observability]` block with + `server_timing_enabled = false` present-but-false and the env-override note (the + overlay cannot create a missing leaf), plus `auction_enabled`/access keys in the + tinybird section comments. +- Gate the auction sink: in `crates/trusted-server-adapter-fastly/src/app.rs`, + `auction_sink_from_settings` condition becomes + `settings.tinybird.enabled && settings.tinybird.auction_enabled`. + +- [ ] **Step 4: Run to verify pass** + +Run: `cargo test-axum -p trusted-server-core` then `cargo test-fastly` (the sink gate +touches the Fastly adapter). +Expected: PASS, including the replaced wiring test. + +- [ ] **Step 5: Commit** + +```bash +git add crates/trusted-server-core/src/settings.rs trusted-server.example.toml crates/trusted-server-adapter-fastly/src/tinybird.rs crates/trusted-server-adapter-fastly/src/app.rs +git commit -m "Add observability settings and decouple tinybird access and auction emission" +``` + +--- + +### Task 3: Fastly freeze point, header emission, `DeliveryOutcome` + +**Files:** + +- Modify: `crates/trusted-server-adapter-fastly/src/main.rs` (entry T0, appbuild span, + `send_edgezero_response`) +- Test: `crates/trusted-server-adapter-fastly/src/app.rs` tests module (route-level + tests run under Viceroy) + +**Interfaces:** + +- Consumes: `RequestTimings`, `Phase` (Task 1); + `trusted_server_core::cache_policy::cache_control_headers_are_private_or_no_store`. +- Produces: + - `RequestTimings` inserted into request extensions at dispatch + (`core_req.extensions_mut().insert(timings.clone())`), alongside the existing + `config_store`/`device_signals`/`client_info` inserts. + - `send_edgezero_response(response, effects, timings) -> DeliveryOutcome` where + `pub(crate) struct DeliveryOutcome { pub bytes: u64, pub result: DeliveryResult }` + and `pub(crate) enum DeliveryResult { Complete, Error }` (streaming partial + detection lands in Task 6). + +- [ ] **Step 1: Write the failing tests** + +```rust +#[test] +fn server_timing_emitted_on_private_response_when_enabled() { + // Arrange: settings with observability.server_timing_enabled = true; publisher + // route fixture whose response is Cache-Control: private, no-store. + // Act: dispatch through the full adapter path. + // Assert: + let header = response_header(&response, "server-timing").expect("should emit header"); + assert!(header.contains("ts-total;dur="), "should carry the stored total"); + assert_eq!( + header.matches("ts-total").count(), 1, + "should emit exactly one TS-owned metric set" + ); +} + +#[test] +fn server_timing_absent_when_flag_off() { /* same fixture, flag false: no ts-total */ } + +#[test] +fn server_timing_absent_on_cacheable_responses() { + // tsjs route (public, max-age=31536000, immutable) and a bare max-age=60 response: + // both must carry no ts-total even with the flag on. +} + +#[test] +fn preexisting_server_timing_values_survive() { + // Fixture response already carrying Server-Timing: upstream;dur=1 stays present + // alongside the appended TS set. +} +``` + +- [ ] **Step 2: Run to verify failure** + +Run: `cargo test-fastly server_timing` +Expected: FAIL, no header emitted. + +- [ ] **Step 3: Implement** + +In `edgezero_main` (`main.rs`): + +```rust +let timings = RequestTimings::new(); +{ + let _appbuild = timings.span(Phase::AppBuild); + // existing: open_trusted_server_config_store() + build_app_with_state() +} +``` + +Move the config-store open inside the span scope. Insert `timings.clone()` into +request extensions before dispatch. Thread the handle into both send sites and the +error paths by value (it is a cheap clone). + +In `send_edgezero_response`, immediately before `response.into_parts()`: + +```rust +timings.mark_headers_ready(); +let conclusively_private = + cache_control_headers_are_private_or_no_store(response.headers()); +if settings_enabled_server_timing && conclusively_private { + if let Some(value) = timings.server_timing_value() { + match HeaderValue::from_str(&value) { + Ok(header_value) => { + response.headers_mut().append(header::SERVER_TIMING, header_value); + } + Err(error) => log::warn!("skipping server-timing header: {error}"), + } + } +} +``` + +`settings_enabled_server_timing` arrives inside a small +`SendContext { timings: RequestTimings, server_timing_enabled: bool }` so the +function stays at or under seven parameters. Return `DeliveryOutcome` with per-mode +semantics: buffered bodies capture the byte count from the body length before +`send_to_client()` (which returns no delivery result) and report complete-on-return; +the streaming branch gains a counting writer in Task 6. Existing callers ignore the +outcome in this task (Task 8 consumes it). + +- [ ] **Step 4: Run to verify pass** + +Run: `cargo test-fastly` and `cargo clippy-fastly` +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add crates/trusted-server-adapter-fastly/src/main.rs crates/trusted-server-adapter-fastly/src/app.rs +git commit -m "Emit Server-Timing at the send freeze point on conclusively private responses" +``` + +--- + +### Task 4: Filter span and geo span with `GeoLookupState` dedupe + +**Files:** + +- Modify: `crates/trusted-server-core/src/geo.rs` (add `GeoLookupState`) +- Modify: `crates/trusted-server-adapter-fastly/src/app.rs` + (`run_pre_route_filters` wrapper, `build_ec_request_state` geo span + state attach) +- Modify: `crates/trusted-server-adapter-fastly/src/middleware.rs` and `main.rs` + (`resolve_geo_for_response` consumes carried state) + +**Interfaces:** + +- Consumes: `RequestTimings` from request extensions (Task 3). +- Produces: + - `pub enum GeoLookupState { NotAttempted, Attempted, Resolved(GeoInfo) }` in + `trusted_server_core::geo`, attached as a response extension on every exit path + that attempted a lookup (including the asset fallback). + - `resolve_geo_for_response` gains the carried state as input: live lookup only on + `NotAttempted`; `Attempted` is never retried; fallback lookups are wrapped in + `timings.span(Phase::Geo)`. + +- [ ] **Step 1: Write the failing tests** + +```rust +#[test] +fn finalize_reuses_request_phase_geo_without_second_lookup() { + // Counting geo stub: dispatch a publisher route; assert lookup count == 1 and + // x-geo-country still set on the response. +} + +#[test] +fn failed_lookup_is_not_retried() { + // Stub returns None once; assert GeoLookupState::Attempted carried and the + // finalize path performs zero further lookups. +} + +#[test] +fn asset_fallback_carries_geo_state_without_ec_finalize_state() { + // Asset route: response extension holds GeoLookupState, EcFinalizeState absent. +} + +#[test] +fn filter_span_recorded_when_request_filter_runs() { + // Registry fixture with a test request filter; assert snapshot().filter_ms is Some. +} + +#[test] +fn geo_lookup_skipped_for_unauthorized_responses() { + // Existing 401 rule preserved: no lookup, state NotAttempted. +} +``` + +- [ ] **Step 2: Run to verify failure** + +Run: `cargo test-fastly geo_ filter_span` +Expected: FAIL (lookup count 2; no `GeoLookupState`). + +- [ ] **Step 3: Implement** + +- `GeoLookupState` derives `Debug, Clone`; store in response extensions from the + dispatch layer right after `build_ec_request_state` resolves (or fails) its lookup. +- Wrap the `build_ec_request_state` lookup and any finalize fallback lookup in + `timings.span(Phase::Geo)` (accumulating slot handles the repeat case). +- Wrap `run_pre_route_filters` (`app.rs:751`) in `timings.span(Phase::Filter)`, + recording only when at least one filter is registered (skip the span when the + registry has no request filters, so the header omits `ts-filter` on unconfigured + deployments). +- `resolve_geo_for_response(response, carried: &GeoLookupState, client_ip, lookup)` + keeps the 401 short-circuit first, then matches the carried state. + +- [ ] **Step 4: Run to verify pass** + +Run: `cargo test-fastly` and `cargo test-axum` +Expected: PASS including untouched existing geo header tests. + +- [ ] **Step 5: Commit** + +```bash +git add crates/trusted-server-core/src/geo.rs crates/trusted-server-adapter-fastly/src/app.rs crates/trusted-server-adapter-fastly/src/middleware.rs crates/trusted-server-adapter-fastly/src/main.rs +git commit -m "Record filter and geo spans and dedupe the per-request geo lookup" +``` + +--- + +### Task 5: Core spans: origin, template cache, KV abstraction + +**Files:** + +- Modify: `crates/trusted-server-core/src/publisher.rs` (origin send ~4496, template + cache lookup ~4391 on current main) +- Modify: `crates/trusted-server-core/src/ec/kv.rs` (graph-level `ts-kv`) +- Test: `publisher.rs` and `ec/kv.rs` test modules + +**Interfaces:** + +- Consumes: `RequestTimings` read from request extensions inside + `handle_publisher_request`; `KvIdentityGraph` gains + `pub fn with_timings(self, timings: RequestTimings) -> Self` (builder-style, + optional field), set where the graph is constructed in `main.rs`. +- Produces: `origin_ms`, `template_cache_ms`, `kv_ms` populated in snapshots. + +- [ ] **Step 1: Write the failing tests** + +```rust +#[test] +fn origin_span_covers_the_publisher_fetch() { + // Stubbed origin with a small injected delay; assert snapshot().origin_ms is Some. +} + +#[test] +fn template_cache_span_recorded_only_when_lookup_runs() { + // Inline mode fixture: template_cache_ms None. Shared-mode eligible fixture: + // template_cache_ms Some. +} + +#[test] +fn kv_span_accumulates_across_graph_operations() { + // Stub KV recording two operations through a TimedKvStore-wrapped graph; assert + // kv_ms Some and covers both (accumulated, not last-write). +} + +#[test] +fn consent_store_reads_are_timed_and_pull_sync_is_not() { + // Consent read through the decorated RuntimeServices store: kv_ms Some. + // Pull-sync graph built from the untimed store: records nothing. +} + +#[test] +fn ec_finalize_kv_lands_before_freeze() { + // Adapter-level (test-fastly): EC-enabled fixture with eids cookies; assert the + // emitted header contains ts-kv, proving the freeze point sits after finalize. +} +``` + +- [ ] **Step 2: Run to verify failure** + +Run: `cargo test-axum -p trusted-server-core origin_span template_cache_span kv_span` +Expected: FAIL. + +- [ ] **Step 3: Implement** + +- `handle_publisher_request` reads the handle: + `let timings = req.extensions().get::().cloned().unwrap_or_default();` + (a defaulted handle records into nothing that ever renders, keeping non-adapter + tests unchanged). +- Origin: `let origin_span = timings.span(Phase::Origin);` immediately before + `services.http_client().send(platform_request).await`; `drop(origin_span)` when the + response headers are available (directly after the `match` arm binds the response). +- Template cache: same guard pattern around + `services.template_cache().lookup_or_reserve(key).await`. +- KV: add `TimedKvStore` (new type in `crates/trusted-server-core/src/platform/`), + a decorator implementing `PlatformKvStore` that wraps `Arc` + plus a `RequestTimings` handle and records `Phase::EcKv` around every trait + method. Every request-path `KvIdentityGraph` construction site (request setup, + identify, admin lookup, batch sync, finalization) receives the timed store; + consent-store access through `RuntimeServices` uses the same decorator; pull-sync + constructs its graph from the untimed store explicitly (add a test asserting the + pull-sync store records nothing). `ec_finalize_response` keeps seven arguments: + the handle rides inside the store the graph already receives. + +- [ ] **Step 4: Run to verify pass** + +Run: `cargo test-axum` then `cargo test-fastly` +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add crates/trusted-server-core/src/publisher.rs crates/trusted-server-core/src/ec/kv.rs crates/trusted-server-adapter-fastly/src/main.rs +git commit -m "Record origin, template cache, and KV phase spans in core" +``` + +--- + +### Task 6: Body-phase capture: stream, auction wait placement, bytes + +**Files:** + +- Modify: `crates/trusted-server-core/src/publisher.rs` (seam wait + buffered wait) +- Modify: `crates/trusted-server-adapter-fastly/src/main.rs` (stream drive timing, + `DeliveryOutcome.bytes`) +- Test: `publisher.rs` tests + adapter tests + +**Interfaces:** + +- Consumes: `record_auction_wait` (Task 1), `DeliveryOutcome` (Task 3). +- Produces: `stream_ms`, `auction_wait_ms` + placement, `resp_bytes`, + `mark_request_elapsed()` called by the adapter immediately after the stream drive + returns. + +- [ ] **Step 1: Write the failing tests** + +```rust +#[test] +fn streaming_seam_wait_records_in_stream_placement() { + // Streaming fixture with a delayed auction: placement InStream, and + // stream_ms >= auction_wait_ms. +} + +#[test] +fn buffered_template_miss_records_pre_header_placement() { + // Shared-template authorized miss (buffered finalizer): placement PreHeader; the + // wait is recorded even though headers had not committed. +} + +#[test] +fn delivery_outcome_reports_bytes_and_request_elapsed_set() { + // Adapter: after send, snapshot has resp_bytes Some(body_len) and + // request_elapsed_ms Some; request_elapsed excludes post-send emitter time by + // construction (asserted by ordering test in Task 8). +} +``` + +- [ ] **Step 2: Run to verify failure** + +Run: `cargo test-fastly seam_wait buffered_template delivery_outcome` +Expected: FAIL. + +- [ ] **Step 3: Implement** + +- Streaming path: around the `collect_stream_auction(...)` await inside the body + stream, measure with `Instant::now()` and call + `timings.record_auction_wait(AuctionWaitPlacement::InStream, waited)`. The handle + reaches the stream closure through `OwnedProcessResponseParams`/assembly params (it + is `Clone`; add a field). +- Buffered path (`buffer_publisher_response_async` and the shared-template miss + finalizer): same measurement with `AuctionWaitPlacement::PreHeader`. +- Adapter stream drive: wrap the `block_on(stream_asset_body(...))` region with a + counting writer that tallies bytes and observes truncation/error, record + `Phase::Stream` with the elapsed drive time, populate `DeliveryOutcome` with + bytes and Complete/Partial/Error, call `timings.set_resp_bytes(bytes)` and + `timings.mark_request_elapsed()` immediately after the drive returns, before + anything else post-send. Buffered responses keep the Task 3 complete-on-return + semantics; `body_mode` distinguishes the regimes in the row. + +- [ ] **Step 4: Run to verify pass** + +Run: `cargo test-fastly` and `cargo test-axum` +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add crates/trusted-server-core/src/publisher.rs crates/trusted-server-adapter-fastly/src/main.rs +git commit -m "Capture stream duration, auction wait placement, and response bytes" +``` + +--- + +### Task 7: `AccessTelemetrySnapshot`, route class, route template + +**Files:** + +- Create: `crates/trusted-server-core/src/access_telemetry.rs` +- Modify: `crates/trusted-server-core/src/lib.rs` +- Modify: `crates/trusted-server-adapter-fastly/src/app.rs` (RouteMetadata attach at + handler wrappers), `main.rs` (snapshot build at freeze point), + `crates/trusted-server-core/src/publisher.rs` (typed template-cache state + extension) + +**Interfaces:** + +- Consumes: `TimingSnapshot` (Task 1), `GeoLookupState` (Task 4). +- Produces: + - `pub enum RouteClass { PublisherHtml, Tsjs, IntegrationProxy, Ec, AuctionApi, Other }` + with `pub fn as_str(&self) -> &'static str` (snake_case values from the spec). + - `pub fn publisher_route_template(path: &str) -> String`: `/` plus first segment + filtered to `[a-z0-9_-]`, truncated to 32 chars, plus `/*` when deeper; empty or + disallowed first segments render `/other/*`. + - `pub struct AccessTelemetrySnapshot { pub method: String, pub status: u16, pub route_class: RouteClass, pub route_template: String, pub publisher_domain: String, pub env: String, pub service_id: String, pub pop: String, pub ts_version: String, pub country: String, pub template_cache_state: String, pub body_mode: &'static str, pub sample_rate: f64 }` + - `pub fn access_event_row(snapshot: &AccessTelemetrySnapshot, timings: &TimingSnapshot, event_ts_epoch_ms: u64) -> String` (one NDJSON line). + +- [ ] **Step 1: Write the failing tests** (adversarial, per spec 9) + +```rust +#[test] +fn admin_ec_route_template_never_contains_the_identifier() { + // Named-route template comes from the route table: "/_ts/admin/ec/{id}". + // Assert a row built for that route never contains a 64-hex EC id fixture. +} + +#[test] +fn publisher_paths_normalize_to_coarse_templates() { + assert_eq!(publisher_route_template("/news/some-article-slug"), "/news/*"); + assert_eq!(publisher_route_template("/"), "/"); + assert_eq!( + publisher_route_template("/user@example.com/profile"), + "/other/*", + "should reject non-allowlisted characters" + ); + assert_eq!( + publisher_route_template(&format!("/{}", "a".repeat(500))), + format!("/{}", "a".repeat(32)), + "should bound segment length" + ); + assert_eq!(publisher_route_template("/search terms here"), "/other/*"); +} + +#[test] +fn row_serializes_nulls_for_missing_phases() { + // Sparse TimingSnapshot: absent phases serialize as JSON null, dimension fields + // never null (unknown sentinel). +} +``` + +- [ ] **Step 2: Run to verify failure** + +Run: `cargo test-axum -p trusted-server-core access_telemetry route_template` +Expected: compile FAIL. + +- [ ] **Step 3: Implement** + +Row serialization via `serde_json::json!` mapping spec section 9 column names exactly +(`time_elapsed_ms`, `appbuild_ms`, ..., `auction_wait_placement` as +`pre_header|in_stream|none`). `pop`/`service_id` read from Fastly env +(`FASTLY_SERVICE_ID`, `FASTLY_POP`), defaulting `"unknown"`; `env` derived by the +adapter from `FASTLY_IS_STAGING` (the `x-ts-env` input), never from `Settings`. +Route identity travels as a typed `RouteMetadata` response extension +(`pub struct RouteMetadata { pub route_class: RouteClass, pub route_template: String }` +in `access_telemetry.rs`): each named-route handler wrapper attaches its matched +route-table pattern verbatim, and the fallback and tsjs handlers attach their class +plus the coarse template; the freeze point consumes the extension (no `RouteClass` +column in `NAMED_ROUTES`, no reconstruction from a handler enum). Also in this task: +make `TemplateCacheResponseState` a typed response extension in `publisher.rs`, set +at every point that writes `x-ts-template-cache` so header and extension cannot +drift; the row reads the extension. The snapshot is built unconditionally in +`send_edgezero_response` right after `mark_headers_ready()` and returned inside +`DeliveryOutcome` (add field `pub snapshot: AccessTelemetrySnapshot`). + +- [ ] **Step 4: Run to verify pass** + +Run: `cargo test-axum` and `cargo test-fastly` +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add crates/trusted-server-core/src/access_telemetry.rs crates/trusted-server-core/src/lib.rs crates/trusted-server-adapter-fastly/src/app.rs crates/trusted-server-adapter-fastly/src/main.rs +git commit -m "Add access telemetry snapshot, route classes, and coarse route templates" +``` + +--- + +### Task 8: Access sink with confirmed delivery + post-send ordering + +**Files:** + +- Modify: `crates/trusted-server-adapter-fastly/src/tinybird.rs` (access sink) +- Modify: `crates/trusted-server-adapter-fastly/src/main.rs` (post-send ordering) + +**Interfaces:** + +- Consumes: `AccessTelemetrySnapshot` + `access_event_row` (Task 7), settings flags + (Task 2), `DeliveryOutcome` (Tasks 3/6). +- Produces: `pub(crate) async fn emit_access_event(client: &FastlyPlatformHttpClient, target: &TinybirdEventsTarget, row: String) -> Result<(), Report>`, + sending via the adapter's stateless platform client (the blocking variant, + post-delivery), checking `response.status().is_success()`, warning with status + otherwise. The transport context is adapter-owned and route-independent (target + derived from settings once at entry), so asset, admin, and error responses emit + without `RuntimeServices` or `EcFinalizeState`. + +- [ ] **Step 1: Write the failing tests** + +```rust +#[test] +fn access_emitter_posts_ndjson_and_validates_2xx() { + // RecordingHttpClient returning 202: assert URI is /v0/events?name=access_logs_raw, + // body is the row, Authorization bearer from the secret stub. +} + +#[test] +fn access_emitter_warns_and_drops_on_non_2xx() { + // RecordingHttpClient returning 422: emit returns Err naming the status; no retry + // request recorded (exactly one request seen). +} + +#[test] +fn sampled_out_requests_emit_nothing() { + // access_sample_rate stub decision false: RecordingHttpClient sees zero requests. +} + +#[test] +fn post_send_order_is_elapsed_then_pull_sync_then_telemetry() { + // Instrumented stubs record call order; assert request_elapsed snapshot precedes + // pull-sync dispatch which precedes the telemetry send. +} +``` + +- [ ] **Step 2: Run to verify failure** + +Run: `cargo test-fastly access_emitter post_send_order` +Expected: FAIL. + +- [ ] **Step 3: Implement** + +- Reuse `TinybirdEventsTarget` with a second constructor + `from_access_config(config: TinybirdSettings)` using `access_dataset` and + `access_token_secret`. +- Sampling decision: `fn sampled_in(rate: f64, entropy: u64) -> bool` where entropy is + derived from the event timestamp nanos XOR a per-request counter (no `rand` + dependency; document that uniformity is approximate and sufficient). +- `main.rs` post-send, in order: `timings.mark_request_elapsed()` (already placed in + Task 6), existing pull-sync dispatch unchanged, then when + `settings.tinybird.enabled && settings.tinybird.access_enabled` and sampled in: + build the row from `outcome.snapshot` + `timings.snapshot()`, call + `emit_access_event`, log one warning on `Err`. + +- [ ] **Step 4: Run to verify pass** + +Run: `cargo test-fastly` and `cargo clippy-fastly` +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add crates/trusted-server-adapter-fastly/src/tinybird.rs crates/trusted-server-adapter-fastly/src/main.rs +git commit -m "Emit confirmed access telemetry rows after pull-sync post-send" +``` + +--- + +### Task 9: Tinybird datasource schema + +**Files:** + +- Modify: `tinybird/datasources/access_logs_raw.datasource` + +**Interfaces:** + +- Consumes: column names exactly as serialized by `access_event_row` (Task 7). +- Produces: the deployed schema contract for the dashboard (separate repo). + +- [ ] **Step 1: Rewrite the schema** per spec section 9: keep + `event_ts DateTime64(3)`, `method`, `status UInt16`, `time_elapsed_ms UInt32`, + `sample_rate Float64`, `event_date` + 30-day TTL; add the columns from spec 9 with + dimension columns non-nullable `LowCardinality(String)` and phase columns + `Nullable(UInt32)`; drop `path` and `cache_state`; set + `ENGINE_SORTING_KEY "event_date, service_id, publisher_domain, env, route_class, pop, status"`. + +- [ ] **Step 2: Validate** with the tinybird toolchain if available locally + (`tb check` / project tests under `tinybird/tests`); otherwise assert the file + parses by review and rely on rollout step 4's remote verification. Add a fixture row + in `tinybird/fixtures` matching `access_event_row` output. + +- [ ] **Step 3: Commit** + +```bash +git add tinybird/datasources/access_logs_raw.datasource tinybird/fixtures +git commit -m "Extend access_logs_raw with phase columns and a non-null sorting key" +``` + +--- + +### Task 10: Axum adapter emission + +**Files:** + +- Modify: `crates/trusted-server-adapter-axum/src/` (terminal layer at the response + serialization boundary; locate the equivalent of the Fastly send path) +- Test: axum adapter tests (`cargo test-axum`) + +**Interfaces:** + +- Consumes: `RequestTimings`, header emission helper. Extract the emission block from + Task 3 into a shared core helper so both adapters call one function: + `pub fn append_server_timing_if_private(response: &mut Response, timings: &RequestTimings, enabled: bool)` + in `request_timing.rs` (move the Fastly inline logic here and re-point Task 3's call + site). +- Produces: Axum responses carry the header under the same conservative predicate; + `ts-appbuild` absent by construction (state built at startup); router-generated + 404/405 covered by the terminal layer; `/health` excluded by route match. + +- [ ] **Step 1: Write the failing tests** + +```rust +#[test] +fn axum_emits_header_on_private_response() { /* flag on, private response: ts-total present, ts-appbuild absent */ } + +#[test] +fn axum_404_carries_header_when_private() { /* router-generated 404 passes through the terminal layer */ } + +#[test] +fn axum_health_is_excluded() { /* /health: no ts-total */ } +``` + +- [ ] **Step 2: Run to verify failure** + +Run: `cargo test-axum axum_emits axum_404 axum_health` +Expected: FAIL. + +- [ ] **Step 3: Implement** an outer service wrapper around the `RouterService` + inside `AxumDevServer` (not router middleware, which router-generated 404/405 + responses bypass and which returns before body serialization): create + `RequestTimings::new()` per request in the wrapper, insert into request + extensions, and on the wrapper's response side call `mark_headers_ready()` + + `append_server_timing_if_private(...)`, skipping the `/health` path by match. + +- [ ] **Step 4: Run to verify pass** + +Run: `cargo test-axum` +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add crates/trusted-server-adapter-axum/src crates/trusted-server-core/src/request_timing.rs crates/trusted-server-adapter-fastly/src/main.rs +git commit -m "Emit Server-Timing from the Axum terminal layer with adapter-specific semantics" +``` + +--- + +### Task 11: Full gate, docs, and PR + +- [ ] **Step 1: Docs.** Add a short operator section to `docs/guide/configuration.md`: + the `[observability]` flag, the tinybird access keys, the deploy/rollback ordering + from spec section 12 (binary first, config second; config first on rollback), and + the conservative emission rule. Run `cd docs && npm run format`. + +- [ ] **Step 2: Full CI gate list** from `CLAUDE.md`: + `cargo fmt --all -- --check`; all six clippy aliases; `test-fastly`, `test-axum`, + `test-cloudflare`, `test-spin`; the integration-tests parity suite; JS build/test + and formats. Cloudflare/Spin compile the new core modules (collection only), which + is exactly what the non-goal requires. + +- [ ] **Step 3: Commit docs, push the branch, open the implementation PR** referencing + the spec PR #1069 and issue #1068, with the rollout section of the spec quoted as + the deployment checklist (staging pass-through + MISS/HIT replay before production + flag-on). + +--- + +## Self-Review + +- Spec coverage: sections 5 (Task 1), 12 (Task 2), 7 (Tasks 3, 10), 8/8a (Tasks 4, + 10), 6 (Tasks 4-6), 9 (Tasks 7, 9), 10 (Task 8), 13 (Tasks 1, 3, 8), 14 (test + steps throughout), 15 steps 1-4 (Task 11 + deployment checklist). Section 11 + (dashboard) is explicitly out of scope for this repo's plan. +- Type consistency: `RequestTimings`/`TimingSnapshot`/`RouteClass`/ + `AccessTelemetrySnapshot`/`DeliveryOutcome` names and signatures match across + Tasks 1, 3, 6, 7, 8, 10. +- Known intentional deferral: `DeliveryResult::Partial` detection is named in Task 3 + and wired when the stream drive reports bytes in Task 6; no other deferrals. diff --git a/docs/superpowers/specs/2026-08-24-request-phase-timing-design.md b/docs/superpowers/specs/2026-08-24-request-phase-timing-design.md new file mode 100644 index 0000000000..06f27c1c27 --- /dev/null +++ b/docs/superpowers/specs/2026-08-24-request-phase-timing-design.md @@ -0,0 +1,553 @@ +# Request phase timing: Server-Timing subtimings and access telemetry + +**Date:** 2026-08-24 +**Status:** Approved design, revised for review rounds 1 and 2, pending implementation +plan. +**Scope:** `trusted-server-core`, Fastly and Axum adapters, `tinybird/` schema, +performance dashboard (separate repo). + +--- + +## 1. Problem + +On 2026-08-21 a production deployment (publisher redacted, `prospect-a.example`) showed +an episodic stall: for a window of roughly 40 minutes, every request that reached the +application path carried a uniform extra ~600 ms of Fastly `time-elapsed`, and then +recovered to 20-50 ms with no deploy or config change we could observe. `/health` +(2-4 ms, short-circuits before app construction) and `/_ts/debug/ja4` (6-9 ms, settings +load only) stayed fast throughout, so the stall lived between app construction and +response send. + +Attributing that window required a live probing session: route-by-route bisection, +cookie-deletion experiments, and an eight-agent code trace. The trace found no +unconditional await on the path that could cost 570 ms, and exactly two +config-conditional candidates (the pre-route request filter's synchronous verification +POST, and EC identity KV writes before send), plus one dependency shared by every +application route (two geo hostcalls per request). We could not tell which one stalled, +because nothing in the response says where server time went. + +The Compute CPU budget is ~50 ms per request, so a large `time-elapsed` strongly +suggests wall-clock time outside active guest CPU: dependency awaits are the leading +explanation, with platform scheduling and hostcall queueing as the residual ones. The +comparison figure here is the fronting delivery layer's `time-elapsed` Server-Timing +entry, observed at its deliver phase. Either way, these are exactly the numbers a +response can carry about itself. + +## 2. Goals + +1. Every normal application response attributes its own server time by phase in a + standard header. Browsers expose the values to same-origin JavaScript via + `PerformanceResourceTiming.serverTiming`, so RUM tooling that reads that API can + surface the breakdown. Whether a given vendor or the publisher's own monitoring + extension actually collects it is verified separately in rollout; the publisher + extension needs a small change to render it. +2. The same numbers flow to Tinybird so we hold p50/p95/p99 per phase, per route class, + per PoP, per deployed version, and a future stall window self-diagnoses in one query. +3. No additional awaited I/O before first byte. The pre-send cost is a handful of + monotonic clock reads, one small allocation at entry, and rendering one header; + telemetry emission happens strictly after the last body byte. + +Scope note: phases cover the application lifecycle after T0. The `/health` and +`/_ts/debug/ja4` short-circuits, config-store open failures, and request-conversion +failures bypass the lifecycle and emit nothing. Requests served entirely by the +fronting cache never reach the guest and produce neither header entries nor rows. + +## 3. Non-goals + +- No trailer-based Server-Timing for body-phase spans (browsers do not expose trailer + values to JavaScript). +- No per-filter naming in any emitted surface. The request-filter span is `ts-filter` + regardless of which filter runs; vendor identity stays out of headers and telemetry. +- No Cloudflare or Spin emission wiring in v1. Core collection is adapter-neutral; those + adapters can wire emission later without core changes. +- No Tinybird endpoint pipe and no rollup materialized views in v1. Grafana queries the + datasource through the ClickHouse connector, matching the auction dashboards; rollups + only if panel latency demands them. +- No sampling of the header. The header is all-traffic when enabled; only Tinybird rows + sample. +- No cross-request circuit breaker for telemetry emission. Compute runs one isolate per + request; there is no shared mutable state to hold breaker state. The controls are the + bounded per-request cost and the `access_sample_rate` lever (section 10). + +## 4. Design overview + +``` +adapter entry (T0) + | RequestTimings::new() -> shared handle + v +app construction ................ ts-appbuild (adapter) +pre-route request filters ....... ts-filter (adapter wrapper) +geo lookup (single, deduped) .... ts-geo (adapter; result carried forward) +template cache lookup ........... ts-template-cache (core: publisher.rs) +origin fetch to resp headers .... ts-origin (core: publisher.rs) +EC identity KV, pre-send ........ ts-kv (core: KV abstraction) +auction wait, buffered mode ..... auction_wait_ms (row only; pre-header in this mode) + | +send_edgezero_response, immediately before into_parts(): + mark_headers_ready() snapshot (unconditional) + build AccessTelemetrySnapshot (unconditional) + append Server-Timing header (flag-gated, only on conclusively private responses) + | +headers committed; body streams + auction hold at seam .......... auction_wait_ms (row only; in-stream in this mode) + stream duration, bytes ........ stream_ms, resp_bytes (row only) + | +post-send (adapter main): + request_elapsed snapshot, then existing pull-sync, then: + sample gate -> one NDJSON row -> Tinybird Events API + bounded response await, 2xx validated +``` + +Collection is always-on and flag-free, including the `mark_headers_ready()` snapshot. +Two independent flags gate emission: the header (`observability.server_timing_enabled`) +and the telemetry row (`tinybird.access_enabled`). + +## 5. `RequestTimings` (core) + +New module `crates/trusted-server-core/src/request_timing.rs`. + +- `Phase`: a closed enum: `AppBuild`, `Filter`, `Geo`, `EcKv`, `Origin`, + `TemplateCacheLookup`, `AuctionWait`, `Stream`. Header rendering covers the first six + plus the stored total; the last two are row-only. +- Inner state: one fixed-size array of `Option` slots indexed by phase, + `t0: Instant`, `headers_ready_total: Option`, + `auction_wait_placement: Option` (`PreHeader` or `InStream`), + and `resp_bytes: Option`. Phases that repeat within a request (geo, KV) + accumulate by saturating addition into the same slot. +- `mark_headers_ready()`: stores `t0.elapsed()` once at the response-commit boundary, + unconditionally, before either emission flag is consulted. The header renders this + stored value as `ts-total`; the telemetry row reads the same stored value as + `time_elapsed_ms`. The two surfaces cannot disagree, and the row stays correct when + the header flag is off. Full request duration is captured separately as + `request_elapsed_ms`, snapshotted immediately after the body-stream drive returns and + before any other post-send work, so pull-sync and telemetry emission are never + included in it. +- Sharing: `RequestTimings` is a cheap-clone handle, `Arc>`. It crosses + three boundaries: adapter entry to core handlers, the streaming body closure (records + body-phase spans after the response object has been handed off), and the adapter's + post-send emission read. Access is exclusively `try_lock()`: a contended or + poisoned lock drops the sample immediately rather than waiting, so recording can + never delay a request. +- Recording API: `timings.record(Phase::Geo, dur)` and a scope guard + `timings.span(Phase::Origin)` that records on drop. Guards use saturating duration + math; a non-monotonic reading records zero rather than panicking. The auction-wait + recorder takes the placement explicitly so the two modes cannot be conflated. +- Rendering: `server_timing_value(&self) -> Option` produces + `ts-total;dur=41.2, ts-appbuild;dur=18.4, ts-filter;dur=9.1` with durations in + milliseconds at one decimal. Phases never recorded are omitted. Returns `None` when + `mark_headers_ready()` has not run. + +`Instant` is already used freely in the guest (`publisher.rs`, `auction/telemetry.rs`), +so no new clock abstraction is needed. + +## 6. Span taxonomy and recording sites + +| Entry | Measures | Site | +| ------------------- | ------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------- | +| `ts-total` | T0 to `mark_headers_ready()` at the response-commit boundary | stored snapshot | +| `ts-appbuild` | config-store open, Settings parse, orchestrator + registry + router build | Fastly `main.rs` around `open_trusted_server_config_store()` + `build_app_with_state()` | +| `ts-filter` | pre-route request filters, end to end (backend ensure, secret read, POST) | around `run_pre_route_filters` (`app.rs:751`) | +| `ts-geo` | geo hostcall (single after dedupe; accumulates if any path still repeats) | `build_ec_request_state` (`app.rs:410`) and timed finalizer fallback lookups | +| `ts-kv` | EC identity KV operations before response send (see enumeration below) | the shared KV abstraction | +| `ts-origin` | publisher backend send to response headers available (read-through cache hit or miss) | `publisher.rs` around the origin `send` | +| `ts-template-cache` | template cache `lookup_or_reserve`, including the hit-path full-body read | `publisher.rs` around the lookup | + +Naming follows the completed template-cache terminology migration (`x-ts-template-cache` +is the emitted header on `main`; `c2` naming is retired). + +`ts-kv` is instrumented by a timing decorator implementing `PlatformKvStore` that +wraps the store handed to request-scoped consumers, because no single existing +abstraction covers the taxonomy: EC graph operations go through `KvIdentityGraph` +while consent persistence uses `PlatformKvStore` directly, and graphs are constructed +independently in request setup, identify, admin lookup, batch sync, and finalization. +Every request-path graph construction receives the timed store; pull-sync explicitly +constructs its graph from an untimed store. Consent-store reads pass through the same +decorator and are timed like any other store call. Included pre-send operations: EC +generation `create_or_revive`, identify-path graph reads and evaluation, finalize-path +`ingest_eid_cookies`/`upsert_partner_ids` and withdrawal tombstones, consent-store +reads on consent routes, and batch-sync graph access when it runs before send. +Explicitly excluded: pull-sync work, which runs strictly after `send_to_client` and is +invisible to both surfaces. The decorator measures store-call latency only: no value +passing through it is read, parsed, or recorded, and the emitted surfaces carry no +consent or identity payloads. This feature therefore needs no consent gate; it is the +site measuring its own infrastructure, not processing user data. The timings handle +reaches `ec_finalize_response` inside the graph it already receives; that function +keeps the repository maximum of seven arguments and does not gain an eighth. + +Row-only fields: + +| Field | Measures | Site | +| -------------------- | --------------------------------------------------------- | ------------------------------------------------ | +| `auction_wait_ms` | wait on the dispatched auction (placement varies by mode) | seam hold (streaming) or buffered finalizer wait | +| `body_mode` | `streamed` or `buffered` response assembly | set where the response body is built | +| `stream_ms` | headers committed to last body byte | adapter around the body-stream drive | +| `resp_bytes` | bytes written to the client body | same | +| `request_elapsed_ms` | T0 to immediately after the body-stream drive returns | post-send snapshot, before pull-sync | + +Auction-wait placement is not universal. On the ordinary streaming path the wait +happens at the `` seam inside the body stream and nests inside `stream_ms`. On +buffered paths (the Fastly shared-template authorized miss, which buffers the full +transform and auction before returning a response, and every Axum response) the wait +completes before headers commit. The row therefore carries `body_mode` plus +`auction_wait_placement` (`pre_header` or `in_stream`), and derivations are +conditional: + +- `in_stream`: `stream_other_ms = greatest(coalesce(stream_ms, 0) - coalesce(auction_wait_ms, 0), 0)`. +- `pre_header`: `auction_wait_ms` joins the pre-header phase set, and `stream_other_ms = coalesce(stream_ms, 0)`. + +`unattributed_ms = greatest(coalesce(time_elapsed_ms, 0) - (coalesce(appbuild_ms, 0) + +coalesce(filter_ms, 0) + coalesce(geo_ms, 0) + coalesce(kv_ms, 0) + +coalesce(origin_ms, 0) + coalesce(template_cache_ms, 0) + pre-header auction wait), 0)`. +Every phase column is nullable, so every query-time formula wraps each term in +`coalesce(column, 0)` and every subtraction in `greatest(..., 0)`; query tests cover +sparse phase combinations. + +## 7. Freeze point and header emission + +The freeze-and-emit point is `send_edgezero_response` (Fastly `main.rs`), immediately +before `response.into_parts()`. This is the single choke point every send path shares, +and it runs after everything that can still mutate the response: the router middleware, +entry-point finalize (`apply_finalize_headers`, asset-policy reapplication), EC +finalization and its KV work, and terminal filter/privacy effects. +`apply_finalize_headers` itself does not emit; the `HEADER_X_TS_FINALIZED` sentinel +marks middleware finalization, not header commitment, and must not be treated as the +timing boundary. + +At the freeze point, in order: `mark_headers_ready()` (unconditional), the +`AccessTelemetrySnapshot` build (unconditional, section 10), then, gated on +`observability.server_timing_enabled`, append one `Server-Timing` header from +`server_timing_value()`. Append semantics, never insert: an origin-supplied +Server-Timing survives, and the fronting delivery layer's own entries (`time-elapsed`, +`hit-state`) are additive per the header's list semantics. + +Header emission is conservative: it happens only when the response is conclusively +non-storable by any shared cache, meaning `Cache-Control` contains `private` or +`no-store` (the existing `cache_control_headers_are_private_or_no_store` predicate). +Anything else, including bare `max-age`, `s-maxage` without `private`, +heuristically-cacheable responses with no cache header at all, and anything a fronting +cache override might store, emits no header, because a stored object would replay one +request's timings for its full lifetime. The long-lived immutable `tsjs` asset route +is the concrete excluded case. The snapshot and the telemetry row are unaffected by +this skip, so excluded routes still report through Tinybird. + +The Axum adapter applies the same emission rule at its terminal point before response +serialization, with adapter-specific phase semantics (section 8a). + +## 8. Geo lookup dedupe (rider) + +Today every dispatched request pays two geo hostcalls for one answer: request-phase in +`build_ec_request_state` (`app.rs:410`) and response-phase in +`FinalizeResponseMiddleware` (`middleware.rs:83`, alternate site `main.rs:285`). + +Plain request extensions cannot carry the result out: the middleware moves the request +context into `next.run(ctx)` and holds only the response afterward. The resolved geo +travels on a dedicated `GeoLookupState` response extension, attached on every exit +path that attempted a lookup, including the asset fallback, which runs +`build_ec_request_state` and then returns without `EcFinalizeState` (which is why +`EcFinalizeState` is not an acceptable carrier). States: `NotAttempted`, +`Attempted(None)` (lookup ran and failed, do not retry), and `Resolved(GeoInfo)`. The +finalize path consumes the carried value and performs a live lookup only in the +`NotAttempted` state; those legitimate fallback lookups (admin, batch, error paths) +are themselves timed into `ts-geo` so degraded geo cannot hide inside +`unattributed_ms`. The 401 rule (`resolve_geo_for_response` skips lookup for +unauthorized responses) is preserved. + +## 8a. Adapter phase semantics + +The Fastly adapter is the reference implementation of the taxonomy. Axum differs +structurally and its emissions are defined accordingly rather than pretending parity: + +- `ts-appbuild` is absent: Axum builds application state once at startup. +- `body_mode` is always `buffered`: the Axum HTTP client buffers upstream bodies, so + `stream_ms` measures buffered-body write-out and `auction_wait_placement` is always + `pre_header`. +- The freeze point is an outer service wrapper around the `RouterService` inside + `AxumDevServer`, not router middleware: router-generated 404/405 responses bypass + router middleware, and middleware returns before Axum serializes the body. The + wrapper sees every response including router-generated ones; `/health` is excluded + by path match inside the wrapper. +- Axum emits the header only; no Tinybird rows in v1 (unchanged). + +Cloudflare and Spin: collection compiles, no emission wiring in v1 (unchanged). + +## 9. Access telemetry row + +Extends the reserved `tinybird/datasources/access_logs_raw.datasource`. + +Kept columns: `event_ts`, `method`, `status`, `time_elapsed_ms` (defined as the +`mark_headers_ready()` snapshot), `sample_rate`, `event_date`, 30-day TTL. + +Removed: raw `path`. Route identifiers like `/_ts/admin/ec/{id}` would otherwise put +EC identifiers into a 30-day dataset, and publisher paths carry unbounded cardinality +and user-generated content (search terms, usernames, emails in slugs). Replaced by +`route_template`: + +- Named routes: the matched route-table pattern verbatim, parameters left as + placeholders. +- Publisher fallback: a coarse fixed template, `/` plus the first path segment + restricted to a bounded allowlisted charset, plus `/*` when deeper (for example + `/news/*`). The auction-telemetry normalizer is explicitly not sufficient here: it + redacts long tokens but preserves short identifiers and arbitrary slugs. +- Tests are adversarial, not just the happy path: a literal EC identifier on the admin + route, an email address in a path segment, search-term-shaped segments, and + overlong segments must all normalize to bounded, content-free templates. + +Added columns (all dimension columns non-nullable with an `unknown` sentinel, because +ClickHouse sorting keys cannot contain nullable columns): + +``` +`service_id` LowCardinality(String), -- FASTLY_SERVICE_ID; immutable deployment identity +`publisher_domain` LowCardinality(String), -- matches auction schema +`env` LowCardinality(String), -- adapter-derived: production | staging | unknown +`route_class` LowCardinality(String), -- publisher_html | tsjs | integration_proxy | ec | auction_api | other +`route_template` String, -- bounded, normalized; replaces path +`body_mode` LowCardinality(String), -- streamed | buffered +`auction_wait_placement` LowCardinality(String), -- pre_header | in_stream | none +`appbuild_ms` Nullable(UInt32), +`filter_ms` Nullable(UInt32), +`geo_ms` Nullable(UInt32), +`kv_ms` Nullable(UInt32), +`origin_ms` Nullable(UInt32), +`template_cache_ms` Nullable(UInt32), +`auction_wait_ms` Nullable(UInt32), +`stream_ms` Nullable(UInt32), +`request_elapsed_ms` Nullable(UInt32), +`resp_bytes` Nullable(UInt64), +`template_cache_state` LowCardinality(String), -- from the typed response extension, not the public header +`country` LowCardinality(String), +`ts_version` LowCardinality(String), +`pop` LowCardinality(String) -- FASTLY_POP, 'unknown' when absent +``` + +The matched route pattern does not survive dispatch today, so a typed +`RouteMetadata` response extension carries `route_class` and `route_template`: each +named-route handler wrapper attaches its route-table pattern verbatim (handlers +serving multiple patterns attach the one that matched), and the fallback and tsjs +handlers attach their class plus the coarse template. The freeze point consumes the +extension; nothing reconstructs routes from a handler enum or path regex. + +Typed sources only: `env` is adapter-owned, derived from the same Fastly +`FASTLY_IS_STAGING` input that drives `x-ts-env` (`Settings` has no environment +field and does not gain one). `template_cache_state` comes from a typed response +extension, not the `x-ts-template-cache` header (operator-configured response +headers can override managed headers): the currently private +`TemplateCacheResponseState` in `publisher.rs` becomes a typed response extension, +and every state transition sets the managed header and the extension together so +the two can never drift. `service_id` and `pop` come from the Fastly environment. `cache_state` +from the reserved schema is dropped: the guest cannot observe the fronting cache, and +guest-visible cache behavior is already carried by `template_cache_state` and +`origin_ms`. Rows exist only for guest-handled requests; fronting-cache hits are +invisible by construction and the dashboard documentation says so. + +Sorting key: `(event_date, service_id, publisher_domain, env, route_class, pop, +status)`. Grafana time filtering uses `$__timeFilter(event_ts)` and every panel query +also carries an `event_date` predicate so the primary index prunes; rollout validates +the panel queries with `EXPLAIN` before the dashboard is committed. This replaces the +reserved key `(event_date, path, status, method)`. Rollout step 4 verifies whether the +reserved datasource was ever deployed to the remote workspace; if it was, this schema +ships as a versioned replacement datasource with a cutover, not an in-place edit. + +## 10. Emission mechanics + +- `AccessTelemetrySnapshot`: built unconditionally at the freeze point, before + `into_parts()` consumes the response. It captures method, status, route metadata + (from the `RouteMetadata` extension), and typed dimension states (`env`, + `template_cache_state`, geo country). It exists because nothing else survives to + post-send on every path: the request is consumed by dispatch, the response by + `into_parts()`, and `EcFinalizeState` is absent on asset, admin, and error paths. +- The emitter's transport context is adapter-owned and route-independent: the + Events API target (backend spec, secret store name, dataset, token secret, sample + rate) derives from settings once at entry in `main.rs`, and the HTTP client is the + adapter's stateless platform client. Asset, admin, and error responses therefore + emit without `RuntimeServices` or `EcFinalizeState`. +- `send_edgezero_response` returns a delivery outcome instead of `()`, with + per-mode semantics because the two body paths observe different things. Streamed + bodies: a counting writer reports bytes written and distinguishes complete, + partial (truncated), and error outcomes. Buffered bodies: `send_to_client()` + returns no delivery result, so the byte count is captured from the body length + before the send and the outcome is complete-on-return with no partial detection; + `body_mode` in the row keeps the two regimes distinguishable in analysis. +- Ordering after the body-stream drive returns: snapshot `request_elapsed_ms` first, + run the existing pull-sync dispatch unchanged, then telemetry emission last, so + pull-sync is never delayed behind the ingest await and never included in + `request_elapsed_ms`. +- Sampling: uniform per-request decision against `tinybird.access_sample_rate`. No + client stickiness. Sampled-out requests are silent; every other drop (row build + failure, send failure, non-2xx) logs one warning naming the reason. There is no + cross-request warning suppression (per-request isolates hold no shared state); the + overload controls are the 2 s bounded await, the single-warning-per-request cap, and + `access_sample_rate` pushed down by config as the operational abort lever. Ingest + health is monitored from the Tinybird side via ingestion freshness on the + datasource, which catches quarantine and schema rejection that per-request warnings + cannot. +- Transport: one NDJSON row to the Tinybird Events API: same `api_host`, reserved + `access_dataset` and `access_token_secret`, 2 s first-byte and between-bytes + timeouts, `max_body_bytes` guard, no retry. +- Delivery confirmation: unlike the auction sink, which starts `send_async` and drops + the pending response (it runs before delivery completes and cannot afford to wait), + the access emitter runs after the client has the full response and therefore awaits + the bounded ingest response and validates 2xx. A non-2xx or timeout logs a warning + with the status. +- Budget: at `access_sample_rate = 1.0` this adds one backend request per request to + the service, after delivery; during a Tinybird outage each such request holds its + sandbox for up to the bounded timeout. The sample rate is the budget control; 1.0 is + a diagnosis setting, not a steady state, and rollout treats sustained emission + warnings as the signal to dial it down. +- Axum adapter: emits the header only; no Tinybird rows in v1. + +## 11. Dashboard and query model + +No endpoint pipe in v1. Grafana queries `access_logs_raw` directly through the +ClickHouse connector with `$__timeFilter(event_ts)` plus an `event_date` predicate, +matching the auction dashboards. + +Dashboard: a new standalone `grafana/dashboards/edge-performance.json` in the +telemetry repo (`trusted-server-tinybird`), performance only, no panels shared with +the revenue and auction dashboards. Panels: + +- Phase percentiles (p50/p95/p99) by `route_class`, per phase column. +- Stacked phase breakdown over time using the non-overlapping set: `appbuild_ms`, + `filter_ms`, `geo_ms`, `kv_ms`, `origin_ms`, `template_cache_ms`, pre-header + auction wait (where `auction_wait_placement = 'pre_header'`), and derived + `unattributed_ms`. In-stream auction wait and derived `stream_other_ms` chart in a + separate body-phase panel and never stack with pre-header phases. +- PoP split, `ts_version` overlay, template-cache state rates. +- Stall panel: rows with `request_elapsed_ms > 500` (post-body total, so body-only + stalls are caught) grouped by dominant phase, where `unattributed_ms` competes as a + phase so the panel cannot confidently blame a small measured span while most time is + uninstrumented. + +All derivations use the `coalesce`/`greatest` forms from section 6; query tests cover +sparse phase combinations and both `auction_wait_placement` modes. + +Sampling semantics for every aggregate: `sample_rate` must be operationally stable +within any queried window. Quantile panels filter strictly to a single `sample_rate` +value. Volume panels weight each row by `1.0 / sample_rate` (the inverse-probability +estimator is `sum(1.0 / sample_rate)` over emitted rows; `count() / rate` is valid +only when the query is already filtered to one rate). Pooled unweighted quantiles +across a rate change are documented as invalid. + +## 12. Config surface + +```toml +[observability] +# Append TS phase timings to the Server-Timing response header. +server_timing_enabled = false # example default +``` + +New `ObservabilitySettings` struct with the single boolean, default off, standard +environment override (`TRUSTED_SERVER__OBSERVABILITY__SERVER_TIMING_ENABLED`). +Collection has no flag: the flags gate the two emission surfaces independently. + +Tinybird flag structure: `tinybird.enabled` today arms the auction sink by itself, so +"enable Tinybird for access telemetry" would silently enable auction emission too. The +master flag is demoted to transport-only (host, store, credentials), and each emitter +gets its own switch: a new `tinybird.auction_enabled` defaulting to `true` (preserving +current behavior for existing configs) and the reserved `tinybird.access_enabled` +defaulting to `false`. A settings test locks the decoupling in both directions. + +Validation when `access_enabled = true`: `tinybird.enabled`, non-empty `api_host`, +non-empty `secret_store`, `access_dataset`, and `access_token_secret`, a positive +`max_body_bytes`, and `access_sample_rate > 0`. An armed-but-silent configuration +(`access_enabled = true`, `access_sample_rate = 0`) is a configuration error, not a +valid state; disabling is done with the flag, not the rate. + +Rollback and compatibility, because `Settings` is `deny_unknown_fields`: + +- Deployment order is binary first, config second. Rollback order is config first + (remove the `[observability]` table and any new tinybird keys), binary second. A + config containing the new fields must never be pushed while a pre-observability + binary can still run. +- Config serialization omits the table when it equals the default, so round-tripping a + config through tooling does not inject a field an older binary rejects. A + compatibility test asserts the serialized default config parses under the previous + schema. +- The environment-variable overlay cannot create a missing leaf, so the key ships + present-but-false in the base operator TOML (the same pattern the GPT integration + documents in `trusted-server.example.toml`) and is flipped by config push. + +## 13. Error handling + +- Recording is infallible: saturating math, lock-failure drops the sample, no panics. +- Header rendering failure (defensive `HeaderValue::from_str` error) logs and skips + the header. +- Row emission failure logs one warning naming the reason and drops the row. The + response has already been delivered; there is nothing to degrade. + +## 14. Testing + +- Core unit tests: phase accumulation, saturating math, `mark_headers_ready()` + idempotence and both-surface consistency, render format (one decimal, omission of + unrecorded phases), row serialization shape, auction-wait placement recording. +- Adapter tests (Fastly via Viceroy, Axum native): header present and well-formed on a + conclusively-private publisher route with the flag on; absent with the flag off; + absent on the shared-cacheable tsjs route and on a bare `max-age` response with the + flag on; exactly one TS-owned metric set (a single `ts-total`) with every + pre-existing Server-Timing value preserved, across all send paths; `ts-kv` captures + EC finalize work (proving the freeze point sits after it). +- Body-mode tests: ordinary streaming (in-stream wait nested in `stream_ms`), + Fastly shared-template authorized miss (buffered, pre-header wait), and Axum + (always buffered), each asserting placement and non-negative derivations. +- Geo dedupe: finalize consumes `Resolved`; no retry on `Attempted(None)`; live + lookup only on `NotAttempted`; fallback lookups timed into `ts-geo`; asset-fallback + path carries `GeoLookupState` without `EcFinalizeState`; 401 skip preserved. +- Route template: adversarial normalization tests (literal EC identifier on the admin + route, email address in a segment, search-term segments, overlong segments) all + producing bounded content-free templates. +- Settings: the access validation matrix including the armed-but-silent rejection; + auction/access flag decoupling in both directions; the former rejection test becomes + the wiring test; the serialized-default-config compatibility test against the + previous schema. +- Sink tests: `RecordingHttpClient` pattern; assert URI, NDJSON body shape, token + header, 2xx validation and warning on non-2xx, skip when sampled out, ordering after + pull-sync. +- Query tests: derivation formulas against sparse rows and both placements. + +## 15. Rollout and verification + +1. Land collection + freeze point + header emission behind the flag, off everywhere. + Full CI gate. +2. Staging deploy with the flag on. Delivery-layer verification is two-sided: a + pass-through request confirming the appended Server-Timing survives the fronting + VCL, and a MISS-then-HIT replay against a cacheable route confirming no stale + timing header is ever served from cache. Fallback if the VCL clobbers the header: a + one-line VCL change on the delivery service, or mirroring the value to + `x-ts-timing` while that lands. +3. Production flag on. Confirm + `performance.getEntriesByType('navigation')[0].serverTiming` shows `ts-*` entries + in a real browser session, and separately confirm what the publisher's RUM tooling + actually collects; the publisher monitoring extension renders it only after a small + change on their side. +4. Verify whether `access_logs_raw` exists in the remote Tinybird workspace. If yes, + ship the schema as a versioned replacement with cutover; if no, edit in place. + Validate the dashboard panel queries with `EXPLAIN` against the sorting key. Then + land the row schema, sink, and settings changes; sample at 1.0 during stall + diagnosis with ingestion-freshness monitoring on the datasource; then the + dashboard. +5. Success criterion: the next stall window is attributable from one response header + or one dashboard query, with no live probing session. + +## 16. Overhead + +Roughly ten monotonic clock reads, two stored snapshots, and one ~130-byte header per +request; one sampled HTTP POST with a bounded await after the response has fully +streamed. No allocation in the hot path beyond the one `Arc` at entry, the +`AccessTelemetrySnapshot` at the freeze point, and the rendered header string. + +## 17. Decisions and open questions + +- **Public exposure is a decision, not an open question.** The header is all-traffic + when enabled. Rationale: values are durations only; the delivery layer already + exposes `hit-state` and `time-elapsed` publicly on every response; filter vendor + identity is masked; emission is restricted to conclusively-private responses so no + cache can replay stale timings. Revisit (quantization or gating) only if a concrete + abuse surfaces. +- The fronting delivery layer's Server-Timing pass-through is unverified until the + first staging deploy (step 2). This is the only known external dependency. +- Body-phase capture threads the timings handle into the streaming closure in + `publisher.rs`; the exact seam is an implementation-plan detail, with the constraint + that a dropped handle (error paths, early client disconnect) must still yield a + valid row with null body-phase fields and a recorded delivery outcome. +- The stall window itself remains unattributed until this ships. If it recurs first, + the bisection runbook from 2026-08-21 (cookie-free curl UA request, static-asset + path versus HTML path) is the fallback. From 052eadad04af7964933178d4482881302de0c1b5 Mon Sep 17 00:00:00 2001 From: Jason Evans Date: Mon, 24 Aug 2026 19:31:18 -0500 Subject: [PATCH 002/194] Add RequestTimings phase collection and Server-Timing rendering --- crates/trusted-server-core/src/lib.rs | 1 + .../trusted-server-core/src/request_timing.rs | 437 ++++++++++++++++++ 2 files changed, 438 insertions(+) create mode 100644 crates/trusted-server-core/src/request_timing.rs diff --git a/crates/trusted-server-core/src/lib.rs b/crates/trusted-server-core/src/lib.rs index 48e92faed8..b8a1a5718b 100644 --- a/crates/trusted-server-core/src/lib.rs +++ b/crates/trusted-server-core/src/lib.rs @@ -61,6 +61,7 @@ pub mod proxy; pub mod publisher; pub mod redacted; pub mod request_signing; +pub mod request_timing; pub mod response_privacy; pub mod rsc_flight; pub(crate) mod s3_sigv4; diff --git a/crates/trusted-server-core/src/request_timing.rs b/crates/trusted-server-core/src/request_timing.rs new file mode 100644 index 0000000000..b0cb7b0ddd --- /dev/null +++ b/crates/trusted-server-core/src/request_timing.rs @@ -0,0 +1,437 @@ +//! Per-request phase timing collection and Server-Timing rendering. +//! +//! Collection is always-on and infallible: saturating math, lock failure +//! drops the sample, no panics. See the design spec +//! `docs/superpowers/specs/2026-08-24-request-phase-timing-design.md`. + +use std::sync::{Arc, Mutex}; +use std::time::{Duration, Instant}; + +/// Number of [`Phase`] variants; sizes the fixed-slot duration array in +/// [`Inner`]. +const PHASE_COUNT: usize = 8; + +/// A distinct stage of request handling that duration can be attributed to. +/// +/// Variants map to fixed slots in [`RequestTimings`], in declaration order. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum Phase { + /// Time spent constructing the app/handler before request processing + /// begins. + AppBuild, + /// Time spent in request/response filtering (e.g. HTML rewriting). + Filter, + /// Time spent resolving geographic signals for the request. + Geo, + /// Time spent reading or writing the Edge Cookie key-value store. + EcKv, + /// Time spent waiting on the origin fetch. + Origin, + /// Time spent looking up a cached template. + TemplateCacheLookup, + /// Time spent waiting on the auction. Row-only: never rendered as a + /// `Server-Timing` header entry. + AuctionWait, + /// Time spent streaming the response body. Row-only: never rendered as + /// a `Server-Timing` header entry. + Stream, +} + +impl Phase { + /// Maps this variant to its fixed slot in the [`Inner::phases`] array. + fn index(self) -> usize { + match self { + Self::AppBuild => 0, + Self::Filter => 1, + Self::Geo => 2, + Self::EcKv => 3, + Self::Origin => 4, + Self::TemplateCacheLookup => 5, + Self::AuctionWait => 6, + Self::Stream => 7, + } + } + + /// `Server-Timing` header entry name; row-only phases return `None`. + fn header_name(self) -> Option<&'static str> { + match self { + Self::AppBuild => Some("ts-appbuild"), + Self::Filter => Some("ts-filter"), + Self::Geo => Some("ts-geo"), + Self::EcKv => Some("ts-kv"), + Self::Origin => Some("ts-origin"), + Self::TemplateCacheLookup => Some("ts-template-cache"), + Self::AuctionWait | Self::Stream => None, + } + } +} + +/// Where in the response the auction wait occurred. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum AuctionWaitPlacement { + /// The auction was awaited before response headers were sent. + PreHeader, + /// The auction was awaited while streaming the response body. + InStream, +} + +/// Mutable state behind [`RequestTimings`], guarded by a [`Mutex`]. +struct Inner { + /// Instant the request started; the reference point for elapsed marks. + t0: Instant, + /// Accumulated duration per [`Phase`], indexed by [`Phase::index`]. + phases: [Option; PHASE_COUNT], + /// Elapsed time at the first [`RequestTimings::mark_headers_ready`] call. + headers_ready_total: Option, + /// Elapsed time at the first [`RequestTimings::mark_request_elapsed`] + /// call. + request_elapsed: Option, + /// Placement recorded by the most recent + /// [`RequestTimings::record_auction_wait`] call. + auction_wait_placement: Option, + /// Response body size in bytes, set via + /// [`RequestTimings::set_resp_bytes`]. + resp_bytes: Option, +} + +/// Per-request phase timing collector. +/// +/// Cheap to clone (an [`Arc`] handle) and safe to share across threads and +/// async tasks handling the same request. Every method is infallible: lock +/// contention or poisoning silently drops the sample rather than blocking or +/// panicking. +#[derive(Clone)] +pub struct RequestTimings(Arc>); + +impl RequestTimings { + /// Starts a new collector with its clock reference (`t0`) set to now. + #[must_use] + pub fn new() -> Self { + Self(Arc::new(Mutex::new(Inner { + t0: Instant::now(), + phases: [None; PHASE_COUNT], + headers_ready_total: None, + request_elapsed: None, + auction_wait_placement: None, + resp_bytes: None, + }))) + } + + /// Accumulates `dur` into `phase`'s running total. + /// + /// Repeated calls for the same phase saturate-add rather than overwrite. + /// Drops the sample silently on lock contention or poisoning. + pub fn record(&self, phase: Phase, dur: Duration) { + let Ok(mut inner) = self.0.try_lock() else { + return; + }; + let index = phase.index(); + let accumulated = inner.phases[index] + .unwrap_or(Duration::ZERO) + .saturating_add(dur); + inner.phases[index] = Some(accumulated); + } + + /// Records an auction wait duration under [`Phase::AuctionWait`] and + /// stores its placement. + /// + /// Drops the sample silently on lock contention or poisoning. + pub fn record_auction_wait(&self, placement: AuctionWaitPlacement, dur: Duration) { + let Ok(mut inner) = self.0.try_lock() else { + return; + }; + let index = Phase::AuctionWait.index(); + let accumulated = inner.phases[index] + .unwrap_or(Duration::ZERO) + .saturating_add(dur); + inner.phases[index] = Some(accumulated); + inner.auction_wait_placement = Some(placement); + } + + /// Starts a guard that records elapsed time into `phase` when dropped. + #[must_use] + pub fn span(&self, phase: Phase) -> PhaseSpan { + PhaseSpan { + timings: self.clone(), + phase, + started: Instant::now(), + } + } + + /// Stamps the elapsed time since `t0` as `headers_ready_total`, the + /// first time this is called. + /// + /// Subsequent calls are no-ops (first call wins). Drops the sample + /// silently on lock contention or poisoning. + pub fn mark_headers_ready(&self) { + let Ok(mut inner) = self.0.try_lock() else { + return; + }; + if inner.headers_ready_total.is_none() { + inner.headers_ready_total = Some(inner.t0.elapsed()); + } + } + + /// Stamps the elapsed time since `t0` as `request_elapsed`, the first + /// time this is called. + /// + /// Subsequent calls are no-ops (first call wins). Drops the sample + /// silently on lock contention or poisoning. + pub fn mark_request_elapsed(&self) { + let Ok(mut inner) = self.0.try_lock() else { + return; + }; + if inner.request_elapsed.is_none() { + inner.request_elapsed = Some(inner.t0.elapsed()); + } + } + + /// Records the response body size in bytes. + /// + /// Drops the sample silently on lock contention or poisoning. + pub fn set_resp_bytes(&self, bytes: u64) { + let Ok(mut inner) = self.0.try_lock() else { + return; + }; + inner.resp_bytes = Some(bytes); + } + + /// Renders a `Server-Timing` header value, or `None` before + /// [`RequestTimings::mark_headers_ready`] has been called. + /// + /// `ts-total` is rendered first from `headers_ready_total`, followed by + /// the recorded header-bearing phases (`ts-appbuild`, `ts-filter`, + /// `ts-geo`, `ts-kv`, `ts-origin`, `ts-template-cache`) in enum + /// declaration order. Unrecorded phases are omitted. Durations are + /// rendered as milliseconds with one decimal place. Drops the sample + /// silently (returning `None`) on lock contention or poisoning. + #[must_use] + pub fn server_timing_value(&self) -> Option { + let inner = self.0.try_lock().ok()?; + let total = inner.headers_ready_total?; + let mut entries = vec![format_entry("ts-total", total)]; + for phase in HEADER_PHASES { + let Some(name) = phase.header_name() else { + continue; + }; + if let Some(dur) = inner.phases[phase.index()] { + entries.push(format_entry(name, dur)); + } + } + Some(entries.join(", ")) + } + + /// Captures the current state as a [`TimingSnapshot`]. + /// + /// Returns an all-`None` snapshot on lock contention or poisoning, + /// consistent with the infallibility of every other method. + #[must_use] + pub fn snapshot(&self) -> TimingSnapshot { + let Ok(inner) = self.0.try_lock() else { + return TimingSnapshot::default(); + }; + TimingSnapshot { + time_elapsed_ms: duration_ms(inner.headers_ready_total), + request_elapsed_ms: duration_ms(inner.request_elapsed), + appbuild_ms: duration_ms(inner.phases[Phase::AppBuild.index()]), + filter_ms: duration_ms(inner.phases[Phase::Filter.index()]), + geo_ms: duration_ms(inner.phases[Phase::Geo.index()]), + kv_ms: duration_ms(inner.phases[Phase::EcKv.index()]), + origin_ms: duration_ms(inner.phases[Phase::Origin.index()]), + template_cache_ms: duration_ms(inner.phases[Phase::TemplateCacheLookup.index()]), + auction_wait_ms: duration_ms(inner.phases[Phase::AuctionWait.index()]), + stream_ms: duration_ms(inner.phases[Phase::Stream.index()]), + auction_wait_placement: inner.auction_wait_placement, + resp_bytes: inner.resp_bytes, + } + } +} + +impl Default for RequestTimings { + fn default() -> Self { + Self::new() + } +} + +/// The header-bearing phases (see [`Phase::header_name`]), in the enum +/// declaration order [`RequestTimings::server_timing_value`] renders them in. +const HEADER_PHASES: [Phase; 6] = [ + Phase::AppBuild, + Phase::Filter, + Phase::Geo, + Phase::EcKv, + Phase::Origin, + Phase::TemplateCacheLookup, +]; + +/// Formats one `Server-Timing` entry as `name;dur=`. +fn format_entry(name: &str, dur: Duration) -> String { + format!("{name};dur={:.1}", dur.as_secs_f64() * 1000.0) +} + +/// Converts a recorded [`Duration`] to whole milliseconds, saturating to +/// [`u32::MAX`] instead of overflowing. +fn duration_ms(dur: Option) -> Option { + dur.map(|dur| u32::try_from(dur.as_millis()).unwrap_or(u32::MAX)) +} + +/// RAII guard returned by [`RequestTimings::span`] that records its own +/// elapsed lifetime into the originating phase when dropped. +pub struct PhaseSpan { + /// The collector this span reports into on drop. + timings: RequestTimings, + /// The phase this span's elapsed time is recorded under. + phase: Phase, + /// The instant the span was created. + started: Instant, +} + +impl Drop for PhaseSpan { + fn drop(&mut self) { + self.timings.record(self.phase, self.started.elapsed()); + } +} + +/// A point-in-time, plain-data view of a [`RequestTimings`] collector. +/// +/// All durations are whole milliseconds; unrecorded phases are `None`. +#[derive(Debug, Clone, Default, PartialEq, Eq)] +pub struct TimingSnapshot { + /// Elapsed time from request start to + /// [`RequestTimings::mark_headers_ready`], in milliseconds. + pub time_elapsed_ms: Option, + /// Elapsed time from request start to + /// [`RequestTimings::mark_request_elapsed`], in milliseconds. + pub request_elapsed_ms: Option, + /// Accumulated [`Phase::AppBuild`] duration, in milliseconds. + pub appbuild_ms: Option, + /// Accumulated [`Phase::Filter`] duration, in milliseconds. + pub filter_ms: Option, + /// Accumulated [`Phase::Geo`] duration, in milliseconds. + pub geo_ms: Option, + /// Accumulated [`Phase::EcKv`] duration, in milliseconds. + pub kv_ms: Option, + /// Accumulated [`Phase::Origin`] duration, in milliseconds. + pub origin_ms: Option, + /// Accumulated [`Phase::TemplateCacheLookup`] duration, in milliseconds. + pub template_cache_ms: Option, + /// Accumulated [`Phase::AuctionWait`] duration, in milliseconds. + pub auction_wait_ms: Option, + /// Accumulated [`Phase::Stream`] duration, in milliseconds. + pub stream_ms: Option, + /// Placement recorded by the most recent + /// [`RequestTimings::record_auction_wait`] call. + pub auction_wait_placement: Option, + /// Response body size in bytes, set via + /// [`RequestTimings::set_resp_bytes`]. + pub resp_bytes: Option, +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn render_omits_unrecorded_phases_and_orders_total_first() { + let timings = RequestTimings::new(); + timings.record(Phase::Filter, Duration::from_micros(9_100)); + timings.mark_headers_ready(); + let value = timings + .server_timing_value() + .expect("should render after mark_headers_ready"); + assert!( + value.starts_with("ts-total;dur="), + "should lead with ts-total: {value}" + ); + assert!( + value.contains("ts-filter;dur=9.1"), + "should render one decimal: {value}" + ); + assert!( + !value.contains("ts-geo"), + "should omit unrecorded phases: {value}" + ); + } + + #[test] + fn render_returns_none_before_headers_ready() { + let timings = RequestTimings::new(); + timings.record(Phase::Geo, Duration::from_millis(1)); + assert!( + timings.server_timing_value().is_none(), + "should require the snapshot" + ); + } + + #[test] + fn repeated_phases_accumulate_saturating() { + let timings = RequestTimings::new(); + timings.record(Phase::Geo, Duration::from_millis(2)); + timings.record(Phase::Geo, Duration::from_millis(3)); + timings.mark_headers_ready(); + let snapshot = timings.snapshot(); + assert_eq!(snapshot.geo_ms, Some(5), "should accumulate repeats"); + } + + #[test] + fn mark_headers_ready_is_first_call_wins() { + let timings = RequestTimings::new(); + timings.mark_headers_ready(); + let first = timings.snapshot().time_elapsed_ms; + std::thread::sleep(Duration::from_millis(5)); + timings.mark_headers_ready(); + assert_eq!( + timings.snapshot().time_elapsed_ms, + first, + "should not restamp" + ); + } + + #[test] + fn span_guard_records_on_drop() { + let timings = RequestTimings::new(); + { + let _span = timings.span(Phase::Origin); + std::thread::sleep(Duration::from_millis(2)); + } + timings.mark_headers_ready(); + assert!( + timings.snapshot().origin_ms.expect("should record on drop") >= 1, + "should measure elapsed span time" + ); + } + + #[test] + fn auction_wait_records_placement() { + let timings = RequestTimings::new(); + timings.record_auction_wait(AuctionWaitPlacement::PreHeader, Duration::from_millis(40)); + let snapshot = timings.snapshot(); + assert_eq!(snapshot.auction_wait_ms, Some(40), "should record wait"); + assert_eq!( + snapshot.auction_wait_placement, + Some(AuctionWaitPlacement::PreHeader), + "should record placement" + ); + } + + #[test] + fn rendered_names_never_include_vendor_terms() { + let timings = RequestTimings::new(); + for phase in [ + Phase::AppBuild, + Phase::Filter, + Phase::Geo, + Phase::EcKv, + Phase::Origin, + Phase::TemplateCacheLookup, + ] { + timings.record(phase, Duration::from_millis(1)); + } + timings.mark_headers_ready(); + let value = timings.server_timing_value().expect("should render"); + assert!( + !value.to_ascii_lowercase().contains("datadome"), + "should mask vendors" + ); + } +} From 7505fb888e4bacf86504cf3a73a34c3502cf5d17 Mon Sep 17 00:00:00 2001 From: Jason Evans Date: Mon, 24 Aug 2026 20:27:41 -0500 Subject: [PATCH 003/194] Add observability settings and decouple tinybird access and auction emission --- .../src/tinybird.rs | 41 ++++- crates/trusted-server-core/src/settings.rs | 154 ++++++++++++++++-- trusted-server.example.toml | 29 ++++ 3 files changed, 205 insertions(+), 19 deletions(-) diff --git a/crates/trusted-server-adapter-fastly/src/tinybird.rs b/crates/trusted-server-adapter-fastly/src/tinybird.rs index f2df617446..08b5811fcf 100644 --- a/crates/trusted-server-adapter-fastly/src/tinybird.rs +++ b/crates/trusted-server-adapter-fastly/src/tinybird.rs @@ -22,9 +22,14 @@ const TINYBIRD_BETWEEN_BYTES_TIMEOUT: Duration = Duration::from_secs(2); const TINYBIRD_MAX_ROWS_PER_AUCTION_BATCH: usize = 512; /// Build the configured auction telemetry sink. +/// +/// Auction emission requires both the Tinybird master toggle +/// (`tinybird.enabled`) and the auction-specific toggle +/// (`tinybird.auction_enabled`), so access-log telemetry can be enabled +/// independently without also emitting auction events. #[must_use] pub(crate) fn auction_sink_from_settings(settings: &Settings) -> Arc { - if settings.tinybird.enabled { + if settings.tinybird.enabled && settings.tinybird.auction_enabled { Arc::new(FastlyTinybirdAuctionTelemetrySink::new( settings.tinybird.clone(), )) @@ -443,6 +448,7 @@ mod tests { fn enabled_config() -> TinybirdSettings { TinybirdSettings { enabled: true, + auction_enabled: true, api_host: "api.us-east.aws.tinybird.co".to_owned(), secret_store: "ts_secrets".to_owned(), auction_dataset: "auction_events_raw".to_owned(), @@ -455,6 +461,39 @@ mod tests { } } + #[test] + fn sink_from_settings_disables_when_auction_enabled_is_false() { + let settings = Settings { + tinybird: TinybirdSettings { + auction_enabled: false, + ..enabled_config() + }, + ..Settings::default() + }; + + let sink = auction_sink_from_settings(&settings); + + assert!( + !sink.is_enabled(), + "auction telemetry should stay off when auction_enabled is false, even if tinybird.enabled is true" + ); + } + + #[test] + fn sink_from_settings_enables_when_both_toggles_are_true() { + let settings = Settings { + tinybird: enabled_config(), + ..Settings::default() + }; + + let sink = auction_sink_from_settings(&settings); + + assert!( + sink.is_enabled(), + "auction telemetry should be on when both tinybird.enabled and tinybird.auction_enabled are true" + ); + } + #[test] fn events_uri_targets_dataset_on_region_host() { assert_eq!( diff --git a/crates/trusted-server-core/src/settings.rs b/crates/trusted-server-core/src/settings.rs index ddc8ac612e..bcb467fb13 100644 --- a/crates/trusted-server-core/src/settings.rs +++ b/crates/trusted-server-core/src/settings.rs @@ -1704,9 +1704,16 @@ impl Proxy { /// Direct Tinybird Events API telemetry configuration. #[derive(Debug, Clone, Deserialize, Serialize)] pub struct TinybirdSettings { - /// Master enablement for auction telemetry ingestion. + /// Master enablement for Tinybird telemetry. Required by both auction and + /// access-log emission; each is independently toggled below. #[serde(default)] pub enabled: bool, + /// Emit auction telemetry when `enabled`. Defaults to `true` so existing + /// configs preserve their current auction-emission behavior after + /// upgrading; set `false` to silence auction events while keeping + /// `enabled` on for other Tinybird telemetry (e.g. `access_enabled`). + #[serde(default = "default_true")] + pub auction_enabled: bool, /// Regional Tinybird API host, without scheme or path. #[serde(default)] pub api_host: String, @@ -1719,19 +1726,26 @@ pub struct TinybirdSettings { /// Secret key containing the auction datasource APPEND token. #[serde(default = "default_tinybird_auction_token_secret")] pub auction_token_secret: String, - /// Reserved for future access-log telemetry. + /// Emit access-log telemetry when `enabled`, independent of + /// `auction_enabled`. /// - /// `true` is rejected until an access-log emitter is wired, so operators - /// cannot enable a setting that silently emits nothing. + /// `true` requires `enabled`, non-empty `api_host`/`secret_store`/ + /// `access_dataset`/`access_token_secret`, `max_body_bytes > 0`, and + /// `access_sample_rate > 0.0`. This prevents an armed-but-silent sampler + /// that enables the flag but emits nothing. #[serde(default)] pub access_enabled: bool, - /// Future access-log Events API datasource name. + /// Access-log Events API datasource name. Required non-empty when + /// `access_enabled`. #[serde(default = "default_tinybird_access_dataset")] pub access_dataset: String, - /// Future Secret Store key containing the access-log datasource APPEND token. + /// Secret Store key containing the access-log datasource APPEND token. + /// Required non-empty when `access_enabled`. #[serde(default = "default_tinybird_access_token_secret")] pub access_token_secret: String, - /// Future fraction of requests to emit for optional access telemetry. + /// Fraction of requests to emit for access telemetry. Must be greater + /// than `0.0` when `access_enabled`, so an operator cannot enable access + /// telemetry while sampling it away entirely. #[serde(default)] pub access_sample_rate: f64, /// Defensive maximum NDJSON body size for one Events API request. @@ -1767,6 +1781,7 @@ impl Default for TinybirdSettings { fn default() -> Self { Self { enabled: false, + auction_enabled: default_true(), api_host: String::new(), secret_store: default_tinybird_secret_store(), auction_dataset: default_tinybird_auction_dataset(), @@ -1790,6 +1805,12 @@ impl TinybirdSettings { self.access_token_secret = self.access_token_secret.trim().to_owned(); } + /// Validate this settings block, including the access-telemetry matrix: + /// `access_enabled` requires `enabled`, a non-empty `api_host`, + /// `secret_store`, `access_dataset`, and `access_token_secret`, a + /// `max_body_bytes` above the defensive floor enforced below, and an + /// `access_sample_rate` greater than `0.0`. Auction emission is + /// independently gated by `auction_enabled` and validated the same way. fn prepare_runtime(&mut self) -> Result<(), Report> { self.normalize(); if !(0.0..=1.0).contains(&self.access_sample_rate) { @@ -1802,9 +1823,9 @@ impl TinybirdSettings { message: "tinybird.max_body_bytes must be at least 1024".to_owned(), })); } - if self.access_enabled { + if self.access_enabled && !self.enabled { return Err(Report::new(TrustedServerError::Configuration { - message: "tinybird.access_enabled is reserved for future access-log telemetry; no emitter is currently wired".to_owned(), + message: "tinybird.access_enabled requires tinybird.enabled".to_owned(), })); } if !self.enabled { @@ -1818,10 +1839,19 @@ impl TinybirdSettings { .to_owned(), })); } - if self.enabled { + if self.auction_enabled { validate_tinybird_dataset(&self.auction_dataset, "tinybird.auction_dataset")?; validate_tinybird_secret(&self.auction_token_secret, "tinybird.auction_token_secret")?; } + if self.access_enabled { + validate_tinybird_dataset(&self.access_dataset, "tinybird.access_dataset")?; + validate_tinybird_secret(&self.access_token_secret, "tinybird.access_token_secret")?; + if self.access_sample_rate <= 0.0 { + return Err(Report::new(TrustedServerError::Configuration { + message: "tinybird.access_sample_rate must be > 0 when tinybird.access_enabled is true".to_owned(), + })); + } + } Ok(()) } } @@ -2588,6 +2618,29 @@ pub enum AuctionDebugCommentFormat { Pretty, } +/// Request-observability toggles exposed to operators. +/// +/// The default table must stay omitted from serialized config blobs: this +/// struct denies unknown fields, so an older binary loading a config blob +/// carrying an `[observability]` table it does not know would reject it, +/// breaking rollback. See [`Settings::observability`]. +#[derive(Debug, Clone, Default, PartialEq, Deserialize, Serialize)] +#[serde(deny_unknown_fields)] +pub struct ObservabilitySettings { + /// Emit the `Server-Timing` response header with per-phase request + /// timing. Defaults to `false` (off). + #[serde(default)] + pub server_timing_enabled: bool, +} + +impl ObservabilitySettings { + /// True when every field is at its default, i.e. observability is fully + /// disabled and the table can be omitted from serialized output. + fn is_default(&self) -> bool { + *self == Self::default() + } +} + /// Tester-cookie endpoint configuration. #[derive(Debug, Default, Clone, Deserialize, Serialize)] pub struct TesterCookieConfig { @@ -2633,6 +2686,12 @@ pub struct Settings { pub tinybird: TinybirdSettings, #[serde(default)] pub debug: DebugConfig, + /// Request-observability toggles. The default table is omitted from + /// serialized config blobs so a config round-tripped without change + /// still parses under a prior binary's schema; see + /// [`ObservabilitySettings`]. + #[serde(default, skip_serializing_if = "ObservabilitySettings::is_default")] + pub observability: ObservabilitySettings, } impl Settings { @@ -3354,6 +3413,14 @@ mod tests { use crate::redacted::Redacted; use crate::test_support::tests::{crate_test_settings_str, create_test_settings}; + /// Parses `extra` appended to the shared test fixture TOML, mirroring the + /// `format!("{}\n...", crate_test_settings_str())` pattern used throughout + /// this module's other tests. + fn settings_from_toml_with(extra: &str) -> Result> { + let toml = format!("{}\n{extra}", crate_test_settings_str()); + Settings::from_toml(&toml) + } + #[test] fn auction_debug_comment_options_default_matches_serde_defaults() { let opts = AuctionDebugCommentOptions::default(); @@ -3565,17 +3632,68 @@ mod tests { } #[test] - fn tinybird_access_enabled_is_rejected_until_emitter_is_wired() { - let toml = format!( - "{}\n[tinybird]\naccess_enabled = true\n", - crate_test_settings_str() + fn tinybird_access_enabled_with_full_config_is_accepted() { + let settings = settings_from_toml_with( + "[tinybird]\nenabled = true\napi_host = \"api.example.com\"\naccess_enabled = true\naccess_sample_rate = 1.0\n", + ) + .expect("should accept a fully-specified access telemetry config"); + assert!( + settings.tinybird.access_enabled, + "should enable access emission" ); + } - let err = Settings::from_toml(&toml) - .expect_err("should reject access telemetry before emitter exists"); + #[test] + fn access_enabled_requires_positive_sample_rate() { + // access_enabled = true with access_sample_rate = 0 is armed-but-silent: an error. + let err = settings_from_toml_with( + "[tinybird]\nenabled = true\napi_host = \"api.example.com\"\naccess_enabled = true\naccess_sample_rate = 0.0\n", + ) + .expect_err("should reject armed-but-silent access telemetry"); + assert!( + format!("{err:?}").contains("access_sample_rate"), + "should name the field" + ); + } + + #[test] + fn access_and_auction_emission_are_independent() { + let settings = settings_from_toml_with( + "[tinybird]\nenabled = true\napi_host = \"api.example.com\"\nauction_enabled = false\naccess_enabled = true\naccess_sample_rate = 1.0\n", + ) + .expect("should accept access without auction"); + assert!( + !settings.tinybird.auction_enabled, + "should disable auction emission" + ); + assert!( + settings.tinybird.access_enabled, + "should enable access emission" + ); + } + + #[test] + fn auction_enabled_defaults_true_for_existing_configs() { + let settings = + settings_from_toml_with("[tinybird]\nenabled = true\napi_host = \"api.example.com\"\n") + .expect("should parse a pre-decoupling config"); + assert!( + settings.tinybird.auction_enabled, + "should preserve current behavior" + ); + } + + #[test] + fn observability_defaults_off_and_serializes_away() { + let settings = create_test_settings(); + assert!( + !settings.observability.server_timing_enabled, + "should default off" + ); + let toml = toml::to_string(&settings).expect("should serialize settings"); assert!( - format!("{err:?}").contains("tinybird.access_enabled"), - "should report unsupported tinybird.access_enabled setting: {err:?}" + !toml.contains("[observability]"), + "should omit the default table so a prior binary can parse the config" ); } diff --git a/trusted-server.example.toml b/trusted-server.example.toml index 71f0f8f785..c52a299b30 100644 --- a/trusted-server.example.toml +++ b/trusted-server.example.toml @@ -222,6 +222,35 @@ verbosity = "redacted" # JSON request/response bodies remain strings exactly as captured. format = "compact" +[tinybird] +enabled = false +# Regional Tinybird API host, no scheme/port/path, e.g. "api.us-east.aws.tinybird.co". +api_host = "" +secret_store = "ts_secrets" +auction_dataset = "auction_events_raw" +auction_token_secret = "tinybird_auction_append_token" +# Defaults to true: existing configs keep emitting auction telemetry once +# `enabled` is turned on. Set false to silence auction events while keeping +# `enabled` on for other Tinybird telemetry (e.g. `access_enabled`). +auction_enabled = true +# Access-log telemetry, decoupled from auction emission. `true` requires +# `enabled`, non-empty api_host/secret_store/access_dataset/access_token_secret, +# max_body_bytes > 0, and access_sample_rate > 0.0; an armed-but-silent sampler +# (access_enabled = true with access_sample_rate = 0) fails config load. +access_enabled = false +access_dataset = "access_logs_raw" +access_token_secret = "tinybird_access_append_token" +# Fraction (0.0-1.0) of requests to emit for access telemetry when access_enabled. +access_sample_rate = 0.0 +# Defensive maximum NDJSON body size, in bytes, for one Events API request. +max_body_bytes = 1048576 + +[observability] +# Keep this leaf present so the environment override can apply; the overlay +# cannot create a missing configuration leaf. The Server-Timing header stays +# off until enabled. +server_timing_enabled = false + [creative_opportunities] # Set to false to disable server-side ad templates while retaining slot definitions # and direct POST /auction callers. Structurally inactive templates use the From 4ce413ff87621429f731fc3a62a5267e43579c18 Mon Sep 17 00:00:00 2001 From: Jason Evans Date: Mon, 24 Aug 2026 21:01:00 -0500 Subject: [PATCH 004/194] Add test coverage for the access_enabled without tinybird.enabled rejection --- crates/trusted-server-core/src/settings.rs | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/crates/trusted-server-core/src/settings.rs b/crates/trusted-server-core/src/settings.rs index bcb467fb13..1aecf80a93 100644 --- a/crates/trusted-server-core/src/settings.rs +++ b/crates/trusted-server-core/src/settings.rs @@ -3643,6 +3643,21 @@ mod tests { ); } + #[test] + fn access_enabled_requires_tinybird_enabled() { + // access_enabled = true with tinybird.enabled omitted (defaults + // false) must be rejected: access telemetry cannot run without the + // master toggle on. + let err = settings_from_toml_with( + "[tinybird]\napi_host = \"api.example.com\"\naccess_enabled = true\naccess_sample_rate = 1.0\n", + ) + .expect_err("should reject access telemetry without tinybird.enabled"); + assert!( + format!("{err:?}").contains("tinybird.access_enabled"), + "should name the field: {err:?}" + ); + } + #[test] fn access_enabled_requires_positive_sample_rate() { // access_enabled = true with access_sample_rate = 0 is armed-but-silent: an error. From f83092ab2c7acf81ae3bdf0b2c81d97b3c5941e4 Mon Sep 17 00:00:00 2001 From: Jason Evans Date: Mon, 24 Aug 2026 21:20:34 -0500 Subject: [PATCH 005/194] Emit Server-Timing at the send freeze point on conclusively private responses --- .../trusted-server-adapter-fastly/src/app.rs | 109 +++++++++++- .../trusted-server-adapter-fastly/src/main.rs | 157 ++++++++++++++++-- 2 files changed, 248 insertions(+), 18 deletions(-) diff --git a/crates/trusted-server-adapter-fastly/src/app.rs b/crates/trusted-server-adapter-fastly/src/app.rs index 41e5e65ee0..8f21d89ac1 100644 --- a/crates/trusted-server-adapter-fastly/src/app.rs +++ b/crates/trusted-server-adapter-fastly/src/app.rs @@ -1311,7 +1311,9 @@ mod tests { use bytes::Bytes; use edgezero_core::body::Body; use edgezero_core::context::RequestContext; - use edgezero_core::http::{Method, Response, StatusCode, header, request_builder}; + use edgezero_core::http::{ + Method, Response, StatusCode, header, request_builder, response_builder, + }; use edgezero_core::key_value_store::NoopKvStore; use edgezero_core::params::PathParams; use edgezero_core::router::RouterService; @@ -1333,6 +1335,7 @@ mod tests { PlatformHttpRequest, PlatformKvStore, PlatformPendingRequest, PlatformResponse, PlatformSelectResult, RuntimeServices, }; + use trusted_server_core::request_timing::RequestTimings; use trusted_server_core::settings::Settings; fn settings_with_missing_consent_store() -> Settings { @@ -2750,4 +2753,108 @@ mod tests { "the filter's response-header effect must be threaded out" ); } + + /// Joins every instance of a response header into one comma-separated + /// string (mirroring how a client sees repeated header fields), or + /// `None` if the header is absent. + fn response_header(response: &Response, name: &str) -> Option { + let values: Vec<&str> = response + .headers() + .get_all(name) + .iter() + .filter_map(|value| value.to_str().ok()) + .collect(); + if values.is_empty() { + None + } else { + Some(values.join(", ")) + } + } + + #[test] + fn server_timing_emitted_on_private_response_when_enabled() { + let mut response = response_builder() + .header("cache-control", "private, no-store") + .body(Body::empty()) + .expect("should build a private response fixture"); + let timings = RequestTimings::new(); + + crate::apply_server_timing_header(&mut response, &timings, true); + + let header = response_header(&response, "server-timing").expect("should emit header"); + assert!( + header.contains("ts-total;dur="), + "should carry the stored total: {header}" + ); + assert_eq!( + header.matches("ts-total").count(), + 1, + "should emit exactly one TS-owned metric set" + ); + } + + #[test] + fn server_timing_absent_when_flag_off() { + let mut response = response_builder() + .header("cache-control", "private, no-store") + .body(Body::empty()) + .expect("should build a private response fixture"); + let timings = RequestTimings::new(); + + crate::apply_server_timing_header(&mut response, &timings, false); + + assert!( + response_header(&response, "server-timing").is_none(), + "should not emit server-timing when the flag is off" + ); + } + + #[test] + fn server_timing_absent_on_cacheable_responses() { + // tsjs route policy: public, long max-age, immutable. + let mut tsjs_response = response_builder() + .header("cache-control", "public, max-age=31536000, immutable") + .body(Body::empty()) + .expect("should build a tsjs-style response fixture"); + // A bare shared-cacheable response with no private/no-store directive. + let mut public_response = response_builder() + .header("cache-control", "max-age=60") + .body(Body::empty()) + .expect("should build a bare max-age response fixture"); + + crate::apply_server_timing_header(&mut tsjs_response, &RequestTimings::new(), true); + crate::apply_server_timing_header(&mut public_response, &RequestTimings::new(), true); + + assert!( + response_header(&tsjs_response, "server-timing").is_none(), + "should not emit on the public immutable tsjs cache policy" + ); + assert!( + response_header(&public_response, "server-timing").is_none(), + "should not emit on a bare shared-cacheable max-age response" + ); + } + + #[test] + fn preexisting_server_timing_values_survive() { + let mut response = response_builder() + .header("cache-control", "private, no-store") + .header("server-timing", "upstream;dur=1") + .body(Body::empty()) + .expect("should build a private response fixture carrying an upstream Server-Timing"); + let timings = RequestTimings::new(); + + crate::apply_server_timing_header(&mut response, &timings, true); + + let header = + response_header(&response, "server-timing").expect("should still carry a header"); + assert!( + header.contains("upstream;dur=1"), + "should preserve the pre-existing entry: {header}" + ); + assert!( + header.contains("ts-total"), + "should append the TS-owned set: {header}" + ); + } } diff --git a/crates/trusted-server-adapter-fastly/src/main.rs b/crates/trusted-server-adapter-fastly/src/main.rs index a19d0485d6..b04b84fe0f 100644 --- a/crates/trusted-server-adapter-fastly/src/main.rs +++ b/crates/trusted-server-adapter-fastly/src/main.rs @@ -5,13 +5,17 @@ use edgezero_adapter_fastly::request::into_core_request; use edgezero_core::body::Body as EdgeBody; use edgezero_core::config_store::ConfigStoreHandle; use edgezero_core::error::EdgeError; -use edgezero_core::http::{Request as HttpRequest, Response as HttpResponse}; +use edgezero_core::http::{ + HeaderName, HeaderValue, Request as HttpRequest, Response as HttpResponse, +}; use edgezero_core::response::IntoResponse; use error_stack::Report; use fastly::http::Method as FastlyMethod; use fastly::{Request as FastlyRequest, Response as FastlyResponse}; -use trusted_server_core::cache_policy::EdgeCacheHeader; +use trusted_server_core::cache_policy::{ + EdgeCacheHeader, cache_control_headers_are_private_or_no_store, +}; use trusted_server_core::ec::device::DeviceSignals; use trusted_server_core::ec::finalize::ec_finalize_response; use trusted_server_core::ec::kv::KvIdentityGraph; @@ -24,6 +28,7 @@ use trusted_server_core::integrations::RequestFilterEffects; use trusted_server_core::platform::PlatformGeo as _; use trusted_server_core::platform::RuntimeServices; use trusted_server_core::proxy::{AssetProxyCachePolicy, stream_asset_body}; +use trusted_server_core::request_timing::{Phase, RequestTimings}; use trusted_server_core::response_privacy::TerminalPrivateResponse; use trusted_server_core::settings::Settings; @@ -48,6 +53,12 @@ use crate::rate_limiter::{FastlyRateLimiter, RATE_COUNTER_NAME}; const TRUSTED_SERVER_CONFIG_STORE: &str = "trusted_server_config"; +/// `Server-Timing` header name. Not present in the `http` crate's `header` +/// module (unlike `CACHE_CONTROL` etc.), so declared locally following the +/// same `HeaderName::from_static` pattern used in +/// `trusted_server_core::constants`. +const HEADER_SERVER_TIMING: HeaderName = HeaderName::from_static("server-timing"); + /// Opens the Fastly Config Store used by the `EdgeZero` dispatcher. /// /// # Errors @@ -111,19 +122,27 @@ fn edgezero_main(mut req: FastlyRequest) { return; } - let config_store = match open_trusted_server_config_store() { - Ok(cs) => cs, - Err(e) => { - log::error!("failed to open config store: {e}"); - FastlyResponse::from_status(fastly::http::StatusCode::INTERNAL_SERVER_ERROR) - .with_body_text_plain("Internal Server Error") - .send_to_client(); - return; - } - }; + let timings = RequestTimings::new(); - let (app, app_state) = TrustedServerApp::build_app_with_state(); + let (config_store, app, app_state) = { + let _appbuild = timings.span(Phase::AppBuild); + let config_store = match open_trusted_server_config_store() { + Ok(cs) => cs, + Err(e) => { + log::error!("failed to open config store: {e}"); + FastlyResponse::from_status(fastly::http::StatusCode::INTERNAL_SERVER_ERROR) + .with_body_text_plain("Internal Server Error") + .send_to_client(); + return; + } + }; + let (app, app_state) = TrustedServerApp::build_app_with_state(); + (config_store, app, app_state) + }; let settings_snapshot = app_state.as_ref().map(|state| Arc::clone(&state.settings)); + let server_timing_enabled = settings_snapshot + .as_deref() + .is_some_and(|settings| settings.observability.server_timing_enabled); // Strip client-spoofable forwarded headers before dispatch. compat::sanitize_fastly_forwarded_headers(&mut req); @@ -171,6 +190,7 @@ fn edgezero_main(mut req: FastlyRequest) { core_req.extensions_mut().insert(config_store); core_req.extensions_mut().insert(device_signals); core_req.extensions_mut().insert(client_info); + core_req.extensions_mut().insert(timings.clone()); match futures::executor::block_on(app.router().oneshot(core_req)) { Ok(response) => response, Err(error) => edge_error_response(error), @@ -213,7 +233,14 @@ fn edgezero_main(mut req: FastlyRequest) { if let Some(settings) = settings_snapshot.as_deref() { match apply_edgezero_ec_finalize(settings, &ec_state, &mut response) { Ok(partner_registry) => { - send_edgezero_response(response, request_filter_effects.as_ref()); + send_edgezero_response( + response, + request_filter_effects.as_ref(), + &SendContext { + timings: timings.clone(), + server_timing_enabled, + }, + ); run_edgezero_pull_sync_after_send(settings, &partner_registry, &ec_state); return; } @@ -228,7 +255,14 @@ fn edgezero_main(mut req: FastlyRequest) { Ok(settings) => { match apply_edgezero_ec_finalize(&settings, &ec_state, &mut response) { Ok(partner_registry) => { - send_edgezero_response(response, request_filter_effects.as_ref()); + send_edgezero_response( + response, + request_filter_effects.as_ref(), + &SendContext { + timings: timings.clone(), + server_timing_enabled, + }, + ); run_edgezero_pull_sync_after_send( &settings, &partner_registry, @@ -250,7 +284,14 @@ fn edgezero_main(mut req: FastlyRequest) { } } - send_edgezero_response(response, request_filter_effects.as_ref()); + send_edgezero_response( + response, + request_filter_effects.as_ref(), + &SendContext { + timings, + server_timing_enabled, + }, + ); } fn edge_error_response(error: EdgeError) -> HttpResponse { @@ -323,6 +364,68 @@ fn run_edgezero_pull_sync_after_send( } } +/// Per-response context threaded into [`send_edgezero_response`] so the +/// function stays at or under seven parameters. +struct SendContext { + /// The request's phase-timing collector. + timings: RequestTimings, + /// Whether `observability.server_timing_enabled` is set. + server_timing_enabled: bool, +} + +/// Outcome of handing a finalized response to the client. +#[allow(dead_code)] +pub(crate) struct DeliveryOutcome { + /// Response body size in bytes. + pub bytes: u64, + /// Whether delivery completed or failed partway. + pub result: DeliveryResult, +} + +/// Whether [`send_edgezero_response`] completed delivery or failed partway. +pub(crate) enum DeliveryResult { + /// The response was handed to the client in full. + Complete, + /// Delivery failed partway through. + Error, +} + +/// Stamps [`RequestTimings::mark_headers_ready`] and, when observability is +/// enabled and the response is conclusively private, appends the rendered +/// `Server-Timing` header. +/// +/// Always stamps `mark_headers_ready` regardless of whether the header is +/// rendered, so the collector's `ts-total` reflects the moment headers +/// commit. Appends rather than overwrites so a pre-existing `Server-Timing` +/// value set upstream survives alongside the TS-owned set. A response is +/// never promoted to shared-cacheable just because the header would +/// otherwise be omitted: this only gates emission, it does not touch +/// `Cache-Control`. +pub(crate) fn apply_server_timing_header( + response: &mut HttpResponse, + timings: &RequestTimings, + server_timing_enabled: bool, +) { + timings.mark_headers_ready(); + + let conclusively_private = cache_control_headers_are_private_or_no_store(response.headers()); + if !server_timing_enabled || !conclusively_private { + return; + } + + let Some(value) = timings.server_timing_value() else { + return; + }; + match HeaderValue::from_str(&value) { + Ok(header_value) => { + response + .headers_mut() + .append(HEADER_SERVER_TIMING, header_value); + } + Err(error) => log::warn!("skipping server-timing header: {error}"), + } +} + /// Sends a finalized `EdgeZero` response to the client. /// /// Streaming `EdgeZero` bodies commit headers first, then pipe chunks to Fastly's @@ -331,8 +434,14 @@ fn run_edgezero_pull_sync_after_send( fn send_edgezero_response( mut response: HttpResponse, request_filter_effects: Option<&RequestFilterEffects>, -) { + context: &SendContext, +) -> DeliveryOutcome { apply_terminal_response_effects(&mut response, request_filter_effects); + apply_server_timing_header( + &mut response, + &context.timings, + context.server_timing_enabled, + ); let (parts, body) = response.into_parts(); @@ -348,15 +457,29 @@ fn send_edgezero_response( if let Err(e) = streaming_body.finish() { log::error!("failed to finish EdgeZero streaming body: {e}"); } + DeliveryOutcome { + bytes: 0, + result: DeliveryResult::Complete, + } } Err(e) => { log::error!("EdgeZero streaming failed: {e:?}"); drop(streaming_body); + DeliveryOutcome { + bytes: 0, + result: DeliveryResult::Error, + } } } } once => { + let bytes = + u64::try_from(once.as_bytes().map(<[u8]>::len).unwrap_or(0)).unwrap_or(u64::MAX); compat::to_fastly_response(HttpResponse::from_parts(parts, once)).send_to_client(); + DeliveryOutcome { + bytes, + result: DeliveryResult::Complete, + } } } } From 4b19c44a3cbb9abda692fd52054293fb0fcb8d56 Mon Sep 17 00:00:00 2001 From: Jason Evans Date: Mon, 24 Aug 2026 21:56:02 -0500 Subject: [PATCH 006/194] Record filter and geo spans and dedupe the per-request geo lookup --- .../trusted-server-adapter-fastly/src/app.rs | 342 ++++++++++++++++-- .../trusted-server-adapter-fastly/src/main.rs | 33 +- .../src/middleware.rs | 80 +++- crates/trusted-server-core/src/geo.rs | 18 + .../src/integrations/registry.rs | 11 + 5 files changed, 447 insertions(+), 37 deletions(-) diff --git a/crates/trusted-server-adapter-fastly/src/app.rs b/crates/trusted-server-adapter-fastly/src/app.rs index 8f21d89ac1..0cf80fb748 100644 --- a/crates/trusted-server-adapter-fastly/src/app.rs +++ b/crates/trusted-server-adapter-fastly/src/app.rs @@ -114,6 +114,7 @@ use trusted_server_core::ec::identify::{cors_preflight_identify, handle_identify use trusted_server_core::ec::kv::KvIdentityGraph; use trusted_server_core::ec::registry::PartnerRegistry; use trusted_server_core::error::{IntoHttpResponse as _, TrustedServerError}; +use trusted_server_core::geo::GeoLookupState; use trusted_server_core::http_util::is_navigation_request; use trusted_server_core::integrations::{ IntegrationRegistry, ProxyDispatchInput, RequestFilterEffects, RequestFilterRegistryInput, @@ -133,6 +134,7 @@ use trusted_server_core::request_signing::{ handle_deactivate_key, handle_rotate_key, handle_trusted_server_discovery, handle_verify_signature, }; +use trusted_server_core::request_timing::{Phase, RequestTimings}; use trusted_server_core::settings::{ProxyAssetRoute, Settings}; use trusted_server_core::settings_data::{ default_config_key, default_config_store_name, get_settings_from_config_store, @@ -355,6 +357,17 @@ impl EcRequestState { services: self.services, } } + + /// Derives the carried [`GeoLookupState`] from this request's geo lookup + /// outcome, so response-phase finalize can reuse it instead of repeating + /// the lookup. `build_ec_request_state` always attempts the lookup, so + /// `None` here means the lookup ran and failed, not that it was skipped. + fn geo_lookup_state(&self) -> GeoLookupState { + match &self.geo_info { + Some(info) => GeoLookupState::Resolved(info.clone()), + None => GeoLookupState::Attempted, + } + } } /// Derives device signals from the request's `User-Agent` header. @@ -410,13 +423,21 @@ fn build_ec_request_state( let eids_cookie = crate::extract_cookie_value(req, COOKIE_TS_EIDS); let sharedid_cookie = crate::extract_cookie_value(req, COOKIE_SHAREDID); - let geo_info = services - .geo() - .lookup(services.client_info().client_ip) - .unwrap_or_else(|e| { - log::warn!("geo lookup failed during EC setup: {e}"); - None - }); + let timings = req + .extensions() + .get::() + .cloned() + .unwrap_or_default(); + let geo_info = { + let _span = timings.span(Phase::Geo); + services + .geo() + .lookup(services.client_info().client_ip) + .unwrap_or_else(|e| { + log::warn!("geo lookup failed during EC setup: {e}"); + None + }) + }; let (ec_context, setup_error) = match EcContext::read_from_request_with_geo(settings, req, services, geo_info.as_ref()) { @@ -482,6 +503,18 @@ async fn run_pre_route_filters( req: &mut Request, geo_info: Option<&GeoInfo>, ) -> PreRoute { + // Only recorded when a filter is actually registered, so unconfigured + // deployments omit ts-filter from the Server-Timing header entirely. + let timings = req + .extensions() + .get::() + .cloned() + .unwrap_or_default(); + let _span = state + .registry + .has_request_filters() + .then(|| timings.span(Phase::Filter)); + match state .registry .filter_request(RequestFilterRegistryInput { @@ -515,6 +548,7 @@ fn attach_dispatch_extensions( ec: EcRequestState, effects: RequestFilterEffects, ) -> Response { + response.extensions_mut().insert(ec.geo_lookup_state()); response.extensions_mut().insert(ec.into_finalize_state()); if !effects.response_headers.is_empty() { response.extensions_mut().insert(effects); @@ -810,7 +844,15 @@ async fn dispatch_fallback( .then(|| state.settings.asset_route_for_path(&path)) .flatten(); if let Some(asset_route) = matched_asset_route { - return dispatch_asset_fallback(state, services, req, asset_route, &effects).await; + return dispatch_asset_fallback( + state, + services, + req, + asset_route, + &effects, + ec.geo_lookup_state(), + ) + .await; } // Generate an EC ID if needed — mirrors the legacy catch-all arm. @@ -900,7 +942,10 @@ fn asset_response_carries_body(method: &Method, status: StatusCode) -> bool { /// [`AssetProxyCachePolicy`] out via response extensions so `edgezero_main` /// can reapply protected cache directives after finalization. EC finalization /// is intentionally skipped: no [`EcFinalizeState`] is attached, matching the -/// legacy `should_finalize_ec = false` behavior for asset responses. +/// legacy `should_finalize_ec = false` behavior for asset responses. The +/// caller's [`GeoLookupState`] is still attached, since `build_ec_request_state` +/// already attempted the lookup before the asset route was matched — this is +/// the one exit path that carries geo state without an `EcFinalizeState`. /// /// Like legacy `route_request`, asset bodies are streamed straight to the client /// with no cap: the origin stream is attached to the response and `edgezero_main` @@ -915,6 +960,7 @@ async fn dispatch_asset_fallback( req: Request, asset_route: &ProxyAssetRoute, effects: &RequestFilterEffects, + geo_state: GeoLookupState, ) -> Response { log::info!("No explicit route matched; proxying via configured asset route"); @@ -936,6 +982,7 @@ async fn dispatch_asset_fallback( } response.extensions_mut().insert(cache_policy); + response.extensions_mut().insert(geo_state); attach_request_filter_effects(&mut response, effects); response } @@ -944,6 +991,7 @@ async fn dispatch_asset_fallback( response .extensions_mut() .insert(AssetProxyCachePolicy::NoStorePrivate); + response.extensions_mut().insert(geo_state); attach_request_filter_effects(&mut response, effects); response } @@ -1319,21 +1367,23 @@ mod tests { use edgezero_core::router::RouterService; use std::net::{IpAddr, Ipv4Addr}; use std::sync::Mutex; + use std::sync::atomic::{AtomicUsize, Ordering}; use error_stack::Report; use futures::executor::block_on; use serde_json::json; - use trusted_server_core::constants::HEADER_X_GEO_INFO_AVAILABLE; + use trusted_server_core::constants::{HEADER_X_GEO_COUNTRY, HEADER_X_GEO_INFO_AVAILABLE}; use trusted_server_core::ec::device::DeviceSignals; use trusted_server_core::error::TrustedServerError; + use trusted_server_core::geo::GeoLookupState; use trusted_server_core::integrations::{ HeaderMutation, IntegrationRegistry, IntegrationRequestFilter, RequestFilterDecision, RequestFilterEffects, RequestFilterInput, }; use trusted_server_core::platform::{ - ClientInfo, PlatformBackend, PlatformBackendSpec, PlatformError, PlatformHttpClient, - PlatformHttpRequest, PlatformKvStore, PlatformPendingRequest, PlatformResponse, - PlatformSelectResult, RuntimeServices, + ClientInfo, GeoInfo, PlatformBackend, PlatformBackendSpec, PlatformError, PlatformGeo, + PlatformHttpClient, PlatformHttpRequest, PlatformKvStore, PlatformPendingRequest, + PlatformResponse, PlatformSelectResult, RuntimeServices, }; use trusted_server_core::request_timing::RequestTimings; use trusted_server_core::settings::Settings; @@ -1481,19 +1531,19 @@ mod tests { ); } - /// Builds a router whose `AppState` uses a registry containing the given - /// request filters (and no routes), so dispatch-level request-filter - /// behavior can be exercised without a real integration. - fn router_with_request_filters( + /// Builds an `AppState` whose registry contains the given request + /// filters (and no routes), so dispatch-level request-filter behavior can + /// be exercised without a real integration. + fn state_with_request_filters( filters: Vec>, - ) -> RouterService { + ) -> Arc { let settings = test_settings(); let orchestrator = trusted_server_core::auction::build_orchestrator(&settings) .expect("should build orchestrator"); let registry = IntegrationRegistry::from_request_filters(filters); let default_kv_store = Arc::new(crate::platform::UnavailableKvStore) as Arc; - let state = Arc::new(super::AppState { + Arc::new(super::AppState { auction_telemetry_sink: Arc::new( trusted_server_core::auction::NoopAuctionTelemetrySink, ), @@ -1501,8 +1551,15 @@ mod tests { orchestrator: Arc::new(orchestrator), registry: Arc::new(registry), default_kv_store, - }); - TrustedServerApp::routes_for_state(&state) + }) + } + + /// Builds a router on top of [`state_with_request_filters`] so + /// dispatch-level request-filter behavior can be exercised end-to-end. + fn router_with_request_filters( + filters: Vec>, + ) -> RouterService { + TrustedServerApp::routes_for_state(&state_with_request_filters(filters)) } /// Continues routing while mutating the request and emitting a response @@ -2532,6 +2589,61 @@ mod tests { ); } + #[test] + fn asset_fallback_carries_geo_state_without_ec_finalize_state() { + // The asset-route fallback is the one exit path that skips + // EcFinalizeState but must still carry GeoLookupState, since + // build_ec_request_state (and its geo lookup) already ran before the + // asset route was matched. Without this, the finalize step would + // silently repeat the lookup for every asset request. + let settings = Settings::from_toml( + r#" + [[handlers]] + path = "^/_ts/admin" + username = "admin" + password = "admin-pass" + + [publisher] + domain = "test-publisher.com" + cookie_domain = ".test-publisher.com" + origin_url = "https://origin.test-publisher.com" + proxy_secret = "unit-test-proxy-secret" + + [ec] + passphrase = "test-secret-key-32-bytes-minimum" + + [request_signing] + enabled = false + config_store_id = "test-config-store-id" + secret_store_id = "test-secret-store-id" + + [proxy] + + [[proxy.asset_routes]] + prefix = "/.image/" + origin_url = "https://assets.example.com" + "#, + ) + .expect("should parse asset-route settings"); + let state = build_state_from_settings(settings).expect("should build state"); + let router = TrustedServerApp::routes_for_state(&state); + + let response = route(&router, empty_request(Method::GET, "/.image/banner.png")); + + assert!( + response.extensions().get::().is_some(), + "asset-route responses should still carry GeoLookupState even though \ + EC finalization is skipped" + ); + assert!( + response + .extensions() + .get::() + .is_none(), + "asset-route responses must skip EC finalization (no EcFinalizeState)" + ); + } + struct FixedBackend; impl PlatformBackend for FixedBackend { @@ -2652,6 +2764,7 @@ mod tests { req, asset_route, &effects, + trusted_server_core::geo::GeoLookupState::NotAttempted, )); assert_eq!( @@ -2695,6 +2808,193 @@ mod tests { ); } + /// A [`PlatformGeo`] stub that counts every `lookup` call and always + /// returns the same canned result, used to prove the request-phase geo + /// lookup is never repeated during finalize. + struct CountingGeo { + calls: Arc, + result: Option, + } + + impl PlatformGeo for CountingGeo { + fn lookup(&self, _: Option) -> Result, Report> { + self.calls.fetch_add(1, Ordering::SeqCst); + Ok(self.result.clone()) + } + } + + fn sample_geo_info() -> GeoInfo { + GeoInfo { + city: "Testville".to_string(), + country: "US".to_string(), + continent: "NorthAmerica".to_string(), + latitude: 0.0, + longitude: 0.0, + metro_code: 0, + region: None, + asn: None, + } + } + + fn runtime_services_with_geo(geo: Arc) -> RuntimeServices { + RuntimeServices::builder() + .config_store(Arc::new(crate::platform::FastlyPlatformConfigStore)) + .secret_store(Arc::new(crate::platform::FastlyPlatformSecretStore)) + .kv_store(Arc::new(NoopKvStore) as Arc) + .backend(Arc::new(FixedBackend)) + .http_client(Arc::new(StreamingHttpClient)) + .geo(geo) + .client_info(ClientInfo::default()) + .build() + } + + #[test] + fn finalize_reuses_request_phase_geo_without_second_lookup() { + // Dispatching a publisher route runs build_ec_request_state, which + // attempts the geo lookup once and carries the result via + // GeoLookupState. The finalize step (resolve_geo_for_response) must + // reuse that carried value instead of calling the geo backend again. + let calls = Arc::new(AtomicUsize::new(0)); + let geo = Arc::new(CountingGeo { + calls: Arc::clone(&calls), + result: Some(sample_geo_info()), + }); + let state = app_state_for_settings(test_settings()); + let services = runtime_services_with_geo(geo); + let req = empty_request(Method::GET, "/some-page"); + + let response = block_on(super::dispatch_fallback(&state, &services, req)); + + let carried = response + .extensions() + .get::() + .cloned() + .expect("dispatch should attach GeoLookupState"); + assert!( + matches!(carried, GeoLookupState::Resolved(_)), + "a successful lookup should carry Resolved" + ); + + let geo_info = + crate::middleware::resolve_geo_for_response(&response, &carried, None, |_| { + panic!("finalize must not repeat a resolved geo lookup"); + }); + + assert_eq!( + calls.load(Ordering::SeqCst), + 1, + "only the request-phase lookup should have run" + ); + + let mut response = response; + geo_info + .expect("geo info should have resolved") + .set_response_headers(&mut response); + assert!( + response.headers().get(HEADER_X_GEO_COUNTRY).is_some(), + "x-geo-country should still be set on the response after reusing the carried geo" + ); + } + + #[test] + fn failed_lookup_is_not_retried() { + // When the request-phase lookup fails (returns None), dispatch must + // carry GeoLookupState::Attempted rather than NotAttempted, and + // finalize must not retry it. + let calls = Arc::new(AtomicUsize::new(0)); + let geo = Arc::new(CountingGeo { + calls: Arc::clone(&calls), + result: None, + }); + let state = app_state_for_settings(test_settings()); + let services = runtime_services_with_geo(geo); + let req = empty_request(Method::GET, "/some-page"); + + let response = block_on(super::dispatch_fallback(&state, &services, req)); + + let carried = response + .extensions() + .get::() + .cloned() + .expect("dispatch should attach GeoLookupState even for a failed lookup"); + assert!( + matches!(carried, GeoLookupState::Attempted), + "a failed lookup should carry Attempted, not Resolved or NotAttempted" + ); + + let geo_info = + crate::middleware::resolve_geo_for_response(&response, &carried, None, |_| { + panic!("finalize must not retry a failed geo lookup"); + }); + + assert_eq!( + calls.load(Ordering::SeqCst), + 1, + "only the request-phase lookup should have run" + ); + assert!( + geo_info.is_none(), + "no geo info should be available after a failed lookup" + ); + } + + #[test] + fn filter_span_recorded_when_request_filter_runs() { + // The Filter phase span should only be recorded when the registry + // actually has a request filter registered, so unconfigured + // deployments omit ts-filter from the Server-Timing header entirely. + let state = state_with_request_filters(vec![Arc::new(RecordingRequestFilter)]); + let services = RuntimeServices::builder() + .config_store(Arc::new(crate::platform::FastlyPlatformConfigStore)) + .secret_store(Arc::new(crate::platform::FastlyPlatformSecretStore)) + .kv_store(Arc::new(NoopKvStore) as Arc) + .backend(Arc::new(FixedBackend)) + .http_client(Arc::new(StreamingHttpClient)) + .geo(Arc::new(crate::platform::FastlyPlatformGeo)) + .client_info(ClientInfo::default()) + .build(); + let mut req = empty_request(Method::GET, "/some-page"); + let timings = RequestTimings::new(); + req.extensions_mut().insert(timings.clone()); + + let _ = block_on(super::run_pre_route_filters( + &state, &services, &mut req, None, + )); + + assert!( + timings.snapshot().filter_ms.is_some(), + "should record the Filter phase span when a request filter is registered and runs" + ); + } + + #[test] + fn filter_span_not_recorded_when_no_request_filters_registered() { + // Mirror test: an empty registry must never record the Filter span, + // even though run_pre_route_filters still runs (as a no-op loop). + let state = state_with_request_filters(Vec::new()); + let services = RuntimeServices::builder() + .config_store(Arc::new(crate::platform::FastlyPlatformConfigStore)) + .secret_store(Arc::new(crate::platform::FastlyPlatformSecretStore)) + .kv_store(Arc::new(NoopKvStore) as Arc) + .backend(Arc::new(FixedBackend)) + .http_client(Arc::new(StreamingHttpClient)) + .geo(Arc::new(crate::platform::FastlyPlatformGeo)) + .client_info(ClientInfo::default()) + .build(); + let mut req = empty_request(Method::GET, "/some-page"); + let timings = RequestTimings::new(); + req.extensions_mut().insert(timings.clone()); + + let _ = block_on(super::run_pre_route_filters( + &state, &services, &mut req, None, + )); + + assert!( + timings.snapshot().filter_ms.is_none(), + "should omit the Filter phase span when no request filters are registered" + ); + } + #[test] fn dispatch_runs_request_filter_and_threads_response_effects() { // Regression guard for the EdgeZero request-filter bypass: the publisher diff --git a/crates/trusted-server-adapter-fastly/src/main.rs b/crates/trusted-server-adapter-fastly/src/main.rs index b04b84fe0f..ae9c600fb8 100644 --- a/crates/trusted-server-adapter-fastly/src/main.rs +++ b/crates/trusted-server-adapter-fastly/src/main.rs @@ -24,6 +24,7 @@ use trusted_server_core::ec::pull_sync::{ }; use trusted_server_core::ec::registry::PartnerRegistry; use trusted_server_core::error::TrustedServerError; +use trusted_server_core::geo::GeoLookupState; use trusted_server_core::integrations::RequestFilterEffects; use trusted_server_core::platform::PlatformGeo as _; use trusted_server_core::platform::RuntimeServices; @@ -209,14 +210,30 @@ fn edgezero_main(mut req: FastlyRequest) { let ec_state = response.extensions_mut().remove::(); let asset_cache_policy = response.extensions_mut().remove::(); let request_filter_effects = response.extensions_mut().remove::(); + let geo_lookup_state = response + .extensions_mut() + .remove::() + .unwrap_or(GeoLookupState::NotAttempted); if !take_finalize_sentinel(&mut response) { if let Some(settings) = settings_snapshot.as_deref() { - apply_entry_point_finalize_headers(settings, &mut response, client_ip); + apply_entry_point_finalize_headers( + settings, + &mut response, + client_ip, + &geo_lookup_state, + &timings, + ); } else { match load_settings_from_config_store() { Ok(settings) => { - apply_entry_point_finalize_headers(&settings, &mut response, client_ip); + apply_entry_point_finalize_headers( + &settings, + &mut response, + client_ip, + &geo_lookup_state, + &timings, + ); } Err(e) => { log::warn!("entry-point finalize skipped: failed to reload settings: {e:?}"); @@ -319,8 +336,11 @@ fn apply_entry_point_finalize_headers( settings: &Settings, response: &mut HttpResponse, client_ip: Option, + geo_state: &GeoLookupState, + timings: &RequestTimings, ) { - let geo_info = resolve_geo_for_response(response, client_ip, |client_ip| { + let geo_info = resolve_geo_for_response(response, geo_state, client_ip, |client_ip| { + let _span = timings.span(Phase::Geo); FastlyPlatformGeo.lookup(client_ip).unwrap_or_else(|e| { log::warn!("entry-point geo lookup failed: {e}"); None @@ -891,9 +911,10 @@ mod tests { .body(EdgeBody::empty()) .expect("should build response"); - let geo_info = resolve_geo_for_response(&response, None, |_| { - panic!("should skip entry-point geo lookup for 401 responses"); - }); + let geo_info = + resolve_geo_for_response(&response, &GeoLookupState::NotAttempted, None, |_| { + panic!("should skip entry-point geo lookup for 401 responses"); + }); apply_finalize_headers(&settings, geo_info.as_ref(), &mut response); assert_eq!( diff --git a/crates/trusted-server-adapter-fastly/src/middleware.rs b/crates/trusted-server-adapter-fastly/src/middleware.rs index 18f309c680..17ecc13a40 100644 --- a/crates/trusted-server-adapter-fastly/src/middleware.rs +++ b/crates/trusted-server-adapter-fastly/src/middleware.rs @@ -24,8 +24,9 @@ use trusted_server_core::constants::{ ENV_FASTLY_IS_STAGING, ENV_FASTLY_SERVICE_VERSION, HEADER_X_GEO_INFO_AVAILABLE, HEADER_X_TS_ENV, HEADER_X_TS_VERSION, }; -use trusted_server_core::geo::GeoInfo; +use trusted_server_core::geo::{GeoInfo, GeoLookupState}; use trusted_server_core::platform::PlatformGeo; +use trusted_server_core::request_timing::{Phase, RequestTimings}; use trusted_server_core::settings::Settings; pub(crate) const HEADER_X_TS_FINALIZED: &str = "x-ts-finalized"; @@ -68,6 +69,12 @@ impl FinalizeResponseMiddleware { impl Middleware for FinalizeResponseMiddleware { async fn handle(&self, ctx: RequestContext, next: Next<'_>) -> Result { let client_ip = FastlyRequestContext::get(ctx.request()).and_then(|c| c.client_ip); + let timings = ctx + .request() + .extensions() + .get::() + .cloned() + .unwrap_or_default(); let mut response = match next.run(ctx).await { Ok(r) => r, @@ -77,7 +84,13 @@ impl Middleware for FinalizeResponseMiddleware { } }; - let geo_info = resolve_geo_for_response(&response, client_ip, |ip| { + let carried = response + .extensions() + .get::() + .cloned() + .unwrap_or(GeoLookupState::NotAttempted); + let geo_info = resolve_geo_for_response(&response, &carried, client_ip, |ip| { + let _span = timings.span(Phase::Geo); self.geo.lookup(ip).unwrap_or_else(|e| { log::warn!("geo lookup failed: {e}"); None @@ -142,14 +155,20 @@ impl Middleware for AuthMiddleware { // Shared geo resolution helper // --------------------------------------------------------------------------- -/// Resolves geo for a response, skipping the lookup for 401 responses. +/// Resolves geo for a response, skipping the lookup for 401 responses and +/// reusing a request-phase lookup when one was already carried. /// -/// Returns `None` for authentication rejections (401) without calling `lookup_geo` -/// to avoid unnecessary work and exposing geo data to unauthenticated callers. -/// All other responses call `lookup_geo` and return its result. +/// Returns `None` for authentication rejections (401) without consulting +/// `carried` or calling `lookup_geo`, to avoid unnecessary work and exposing +/// geo data to unauthenticated callers. Otherwise dispatches on `carried`: +/// a [`GeoLookupState::Resolved`] value is reused as-is, a +/// [`GeoLookupState::Attempted`] value is treated as no geo info without +/// retrying the lookup, and [`GeoLookupState::NotAttempted`] falls back to +/// calling `lookup_geo`. /// /// Used by both [`FinalizeResponseMiddleware`] and the entry-point finalization -/// in `main.rs` so the 401-skip rule is defined in one place. +/// in `main.rs` so the 401-skip rule and the dedupe rule are each defined in +/// one place. /// /// # Parity note /// @@ -161,6 +180,7 @@ impl Middleware for AuthMiddleware { /// server or the upstream origin. pub(crate) fn resolve_geo_for_response( response: &Response, + carried: &GeoLookupState, client_ip: Option, lookup_geo: F, ) -> Option @@ -168,9 +188,12 @@ where F: FnOnce(Option) -> Option, { if response.status() == StatusCode::UNAUTHORIZED { - None - } else { - lookup_geo(client_ip) + return None; + } + match carried { + GeoLookupState::Resolved(geo) => Some(geo.clone()), + GeoLookupState::Attempted => None, + GeoLookupState::NotAttempted => lookup_geo(client_ip), } } @@ -277,6 +300,19 @@ mod tests { RequestContext::new(req, PathParams::new(HashMap::new())) } + fn sample_geo_info() -> GeoInfo { + GeoInfo { + city: "Testville".to_string(), + country: "US".to_string(), + continent: "NorthAmerica".to_string(), + latitude: 0.0, + longitude: 0.0, + metro_code: 0, + region: None, + asn: None, + } + } + struct FixedGeo(Option); impl PlatformGeo for FixedGeo { @@ -639,6 +675,30 @@ mod tests { ); } + #[test] + #[allow(clippy::panic)] + fn geo_lookup_skipped_for_unauthorized_responses() { + // The 401 short-circuit in resolve_geo_for_response must win + // regardless of what state the request phase carried in, and must + // never invoke the fallback lookup closure. + let mut response = empty_response(); + *response.status_mut() = StatusCode::UNAUTHORIZED; + + for carried in [ + GeoLookupState::NotAttempted, + GeoLookupState::Attempted, + GeoLookupState::Resolved(sample_geo_info()), + ] { + let geo_info = resolve_geo_for_response(&response, &carried, None, |_| { + panic!("401 responses must never trigger a geo lookup"); + }); + assert!( + geo_info.is_none(), + "401 responses should never resolve geo info, regardless of carried state" + ); + } + } + // --------------------------------------------------------------------------- // AuthMiddleware::handle tests // --------------------------------------------------------------------------- diff --git a/crates/trusted-server-core/src/geo.rs b/crates/trusted-server-core/src/geo.rs index 63f7907f53..fe5785d260 100644 --- a/crates/trusted-server-core/src/geo.rs +++ b/crates/trusted-server-core/src/geo.rs @@ -48,6 +48,24 @@ impl GeoInfo { } } +/// Carries the outcome of a request-phase geo lookup across to +/// response-phase finalization, so a finalize consumer can reuse it instead +/// of performing a second lookup for the same request. +/// +/// Attached as a response extension on every exit path that attempted a +/// lookup, including the asset-route fallback (which does not carry an EC +/// finalize state). +#[derive(Debug, Clone)] +pub enum GeoLookupState { + /// No lookup has been attempted for this request. + NotAttempted, + /// A lookup ran and failed (or returned no result). This must not be + /// retried: finalize treats it the same as no geo info being available. + Attempted, + /// A lookup ran and resolved geo info. + Resolved(GeoInfo), +} + fn insert_geo_header(headers: &mut http::HeaderMap, name: http::header::HeaderName, value: &str) { match HeaderValue::from_str(value) { Ok(header_value) => { diff --git a/crates/trusted-server-core/src/integrations/registry.rs b/crates/trusted-server-core/src/integrations/registry.rs index 280eae8472..376f5e4927 100644 --- a/crates/trusted-server-core/src/integrations/registry.rs +++ b/crates/trusted-server-core/src/integrations/registry.rs @@ -893,6 +893,17 @@ impl IntegrationRegistry { self.find_route(method, path).is_some() } + /// Return true when at least one integration request filter is + /// registered. + /// + /// Adapters use this to decide whether to record a request-filter phase + /// timing span, so unconfigured deployments (no request filters) omit + /// that entry from observability output entirely. + #[must_use] + pub fn has_request_filters(&self) -> bool { + !self.inner.request_filters.is_empty() + } + /// Run pre-routing request filters. /// /// Request header mutations are applied immediately so later filters and From ce295f1405a0f5be2ee7363bd4f6ba7f3fc94587 Mon Sep 17 00:00:00 2001 From: Jason Evans Date: Mon, 24 Aug 2026 22:37:53 -0600 Subject: [PATCH 007/194] Record origin, template cache, and KV phase spans in core --- .../trusted-server-adapter-fastly/src/app.rs | 132 +++++++++++-- .../trusted-server-adapter-fastly/src/main.rs | 184 +++++++++++++++++- crates/trusted-server-core/src/ec/kv.rs | 22 +++ .../trusted-server-core/src/platform/mod.rs | 2 + .../src/platform/timed_kv.rs | 180 +++++++++++++++++ crates/trusted-server-core/src/publisher.rs | 99 +++++++++- 6 files changed, 595 insertions(+), 24 deletions(-) create mode 100644 crates/trusted-server-core/src/platform/timed_kv.rs diff --git a/crates/trusted-server-adapter-fastly/src/app.rs b/crates/trusted-server-adapter-fastly/src/app.rs index 0cf80fb748..5cf7f97097 100644 --- a/crates/trusted-server-adapter-fastly/src/app.rs +++ b/crates/trusted-server-adapter-fastly/src/app.rs @@ -120,7 +120,9 @@ use trusted_server_core::integrations::{ IntegrationRegistry, ProxyDispatchInput, RequestFilterEffects, RequestFilterRegistryInput, RequestFilterRegistryOutcome, }; -use trusted_server_core::platform::{ClientInfo, GeoInfo, PlatformKvStore, RuntimeServices}; +use trusted_server_core::platform::{ + ClientInfo, GeoInfo, PlatformKvStore, RuntimeServices, TimedKvStore, +}; use trusted_server_core::proxy::{ AssetProxyCachePolicy, handle_asset_proxy_request, handle_first_party_click, handle_first_party_proxy, handle_first_party_proxy_rebuild, handle_first_party_proxy_sign, @@ -222,13 +224,18 @@ fn warn_if_certificate_check_disabled(settings: &Settings) { pub(crate) fn runtime_services_for_consent_route( settings: &Settings, runtime_services: &RuntimeServices, + timings: &RequestTimings, ) -> Result> { let Some(store_name) = settings.consent.consent_store.as_deref() else { return Ok(runtime_services.clone()); }; open_kv_store(store_name) - .map(|store| runtime_services.clone().with_kv_store(store)) + .map(|store| { + let timed_store = + Arc::new(TimedKvStore::new(store, timings.clone())) as Arc; + runtime_services.clone().with_kv_store(timed_store) + }) .map_err(|e| { Report::new(TrustedServerError::KvStore { store_name: store_name.to_string(), @@ -451,7 +458,7 @@ fn build_ec_request_state( // Bot gate: suppress KV-backed EC writes for unrecognized clients, except // consent withdrawals. Revocations keep the write path so tombstones stay // authoritative even for privacy-extension-heavy clients. - let kv_graph = crate::maybe_identity_graph(settings); + let kv_graph = crate::identity_graph_with_timing(settings, &timings); let finalize_kv_graph = if setup_error.is_none() && (is_real_browser || ec_consent_withdrawn(ec_context.consent())) { @@ -585,7 +592,12 @@ async fn execute_named( // Deliberately do not use an EC request-state graph: that // copy is bot-gated, while operators use curl for this // authenticated diagnostic. - let kv = crate::maybe_identity_graph(&state.settings); + let timings = req + .extensions() + .get::() + .cloned() + .unwrap_or_default(); + let kv = crate::identity_graph_with_timing(&state.settings, &timings); handle_admin_ec_lookup(kv.as_ref(), ®istry, &req) } NamedRouteHandler::AdminEidsLookup => handle_admin_eids_lookup(®istry, &req), @@ -656,7 +668,12 @@ async fn run_named_route( if req.method() == Method::OPTIONS { cors_preflight_identify(&state.settings, &req) } else { - let kv = crate::require_identity_graph(&state.settings)?; + let timings = req + .extensions() + .get::() + .cloned() + .unwrap_or_default(); + let kv = crate::require_identity_graph_with_timing(&state.settings, &timings)?; let partner_registry = PartnerRegistry::from_config(&state.settings.ec.partners)?; handle_identify( &state.settings, @@ -673,7 +690,13 @@ async fn run_named_route( // The auction reads consent data, so the consent KV store must be // available — fail closed with 503 when it is configured but // cannot be opened, matching legacy behavior. - let consent_services = runtime_services_for_consent_route(&state.settings, services)?; + let timings = req + .extensions() + .get::() + .cloned() + .unwrap_or_default(); + let consent_services = + runtime_services_for_consent_route(&state.settings, services, &timings)?; let partner_registry = PartnerRegistry::from_config(&state.settings.ec.partners)?; let registry_ref = if partner_registry.is_empty() { None @@ -701,7 +724,13 @@ async fn run_named_route( // Like the auction, page-bids reads consent data, so the consent KV // store must be available — fail closed with 503 when configured but // unopenable, matching legacy. - let consent_services = runtime_services_for_consent_route(&state.settings, services)?; + let timings = req + .extensions() + .get::() + .cloned() + .unwrap_or_default(); + let consent_services = + runtime_services_for_consent_route(&state.settings, services, &timings)?; let partner_registry = PartnerRegistry::from_config(&state.settings.ec.partners)?; let registry_ref = if partner_registry.is_empty() { None @@ -746,12 +775,18 @@ fn run_batch_sync(state: &AppState, services: &RuntimeServices, req: Request) -> let is_real_browser = device_signals.looks_like_browser(); let eids_cookie = crate::extract_cookie_value(&req, COOKIE_TS_EIDS); let sharedid_cookie = crate::extract_cookie_value(&req, COOKIE_SHAREDID); + let timings = req + .extensions() + .get::() + .cloned() + .unwrap_or_default(); - let result = crate::require_identity_graph(&state.settings).and_then(|kv| { - let partner_registry = PartnerRegistry::from_config(&state.settings.ec.partners)?; - let limiter = FastlyRateLimiter::new(RATE_COUNTER_NAME); - handle_batch_sync(&kv, &partner_registry, &limiter, req) - }); + let result = + crate::require_identity_graph_with_timing(&state.settings, &timings).and_then(|kv| { + let partner_registry = PartnerRegistry::from_config(&state.settings.ec.partners)?; + let limiter = FastlyRateLimiter::new(RATE_COUNTER_NAME); + handle_batch_sync(&kv, &partner_registry, &limiter, req) + }); let mut response = result.unwrap_or_else(|e| http_error(&e)); // Legacy parity: batch-sync responses still pass through @@ -870,7 +905,12 @@ async fn dispatch_fallback( // Publisher pages read consent data, so the consent KV store must be // available — fail closed with 503 when it is configured but cannot // be opened, matching legacy behavior. - match runtime_services_for_consent_route(&state.settings, services) { + let timings = req + .extensions() + .get::() + .cloned() + .unwrap_or_default(); + match runtime_services_for_consent_route(&state.settings, services, &timings) { Ok(publisher_services) => { // Run the server-side auction with the configured creative- // opportunity slots and collect dispatched bids from the lazy @@ -2995,6 +3035,72 @@ mod tests { ); } + fn settings_with_consent_and_ec_store() -> Settings { + Settings::from_toml( + r#" + [[handlers]] + path = "^/_ts/admin" + username = "admin" + password = "admin-pass" + + [publisher] + domain = "test-publisher.com" + cookie_domain = ".test-publisher.com" + origin_url = "https://origin.test-publisher.com" + proxy_secret = "unit-test-proxy-secret" + + [ec] + passphrase = "test-secret-key-32-bytes-minimum" + ec_store = "ec_identity_store" + + [consent] + consent_store = "consent_store" + + [request_signing] + enabled = false + config_store_id = "test-config-store-id" + secret_store_id = "test-secret-store-id" + "#, + ) + .expect("should parse settings with consent and EC KV stores configured") + } + + #[test] + fn consent_store_reads_are_timed_and_pull_sync_is_not() { + // Consent-store access threaded through RuntimeServices uses the same + // TimedKvStore decorator as request-path KvIdentityGraph + // construction, so a read through it records Phase::EcKv. + let settings = settings_with_consent_and_ec_store(); + let services = streaming_runtime_services(); + let timings = RequestTimings::new(); + + let consent_services = + super::runtime_services_for_consent_route(&settings, &services, &timings) + .expect("should open the configured consent store"); + let _ = block_on(consent_services.kv_store().get_bytes("consent-read-key")); + + timings.mark_headers_ready(); + assert!( + timings.snapshot().kv_ms.is_some(), + "a consent-store read through the decorated RuntimeServices store should record Phase::EcKv" + ); + + // Pull-sync's identity graph is built by `require_identity_graph`, + // which takes no `timings` parameter at all — the untimed store it + // constructs cannot record into any handle, including a fresh one. + let graph = crate::require_identity_graph(&settings) + .expect("should construct the pull-sync identity graph"); + let ec_id = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.test01"; + let _ = graph.get(ec_id); + + let pull_sync_timings = RequestTimings::new(); + pull_sync_timings.mark_headers_ready(); + assert!( + pull_sync_timings.snapshot().kv_ms.is_none(), + "pull-sync's untimed graph construction has no timings handle to record into" + ); + } + #[test] fn dispatch_runs_request_filter_and_threads_response_effects() { // Regression guard for the EdgeZero request-filter bypass: the publisher diff --git a/crates/trusted-server-adapter-fastly/src/main.rs b/crates/trusted-server-adapter-fastly/src/main.rs index ae9c600fb8..ca97f64750 100644 --- a/crates/trusted-server-adapter-fastly/src/main.rs +++ b/crates/trusted-server-adapter-fastly/src/main.rs @@ -27,7 +27,7 @@ use trusted_server_core::error::TrustedServerError; use trusted_server_core::geo::GeoLookupState; use trusted_server_core::integrations::RequestFilterEffects; use trusted_server_core::platform::PlatformGeo as _; -use trusted_server_core::platform::RuntimeServices; +use trusted_server_core::platform::{RuntimeServices, TimedKvStore}; use trusted_server_core::proxy::{AssetProxyCachePolicy, stream_asset_body}; use trusted_server_core::request_timing::{Phase, RequestTimings}; use trusted_server_core::response_privacy::TerminalPrivateResponse; @@ -248,7 +248,7 @@ fn edgezero_main(mut req: FastlyRequest) { if let Some(ec_state) = ec_state { if let Some(settings) = settings_snapshot.as_deref() { - match apply_edgezero_ec_finalize(settings, &ec_state, &mut response) { + match apply_edgezero_ec_finalize(settings, &ec_state, &mut response, &timings) { Ok(partner_registry) => { send_edgezero_response( response, @@ -270,7 +270,8 @@ fn edgezero_main(mut req: FastlyRequest) { } else { match load_settings_from_config_store() { Ok(settings) => { - match apply_edgezero_ec_finalize(&settings, &ec_state, &mut response) { + match apply_edgezero_ec_finalize(&settings, &ec_state, &mut response, &timings) + { Ok(partner_registry) => { send_edgezero_response( response, @@ -353,10 +354,11 @@ fn apply_edgezero_ec_finalize( settings: &Settings, ec_state: &EcFinalizeState, response: &mut HttpResponse, + timings: &RequestTimings, ) -> Result> { let partner_registry = PartnerRegistry::from_config(&settings.ec.partners)?; let finalize_kv_graph = if ec_state.use_finalize_kv { - maybe_identity_graph(settings) + identity_graph_with_timing(settings, timings) } else { None }; @@ -575,12 +577,21 @@ fn build_ja4_debug_response(req: &FastlyRequest) -> FastlyResponse { .with_body(body) } -pub(crate) fn maybe_identity_graph(settings: &Settings) -> Option { - settings - .ec - .ec_store - .as_ref() - .map(|store_name| KvIdentityGraph::new(FastlyEcKvStore::new(store_name))) +/// Constructs a `KvIdentityGraph` wrapped in the [`Phase::EcKv`] timing +/// decorator, for request-path callers with a `RequestTimings` handle. +/// +/// Returns `None` when `ec.ec_store` is not configured, matching +/// [`require_identity_graph_with_timing`]'s contract on every other axis. +pub(crate) fn identity_graph_with_timing( + settings: &Settings, + timings: &RequestTimings, +) -> Option { + settings.ec.ec_store.as_ref().map(|store_name| { + KvIdentityGraph::new(TimedKvStore::new( + FastlyEcKvStore::new(store_name), + timings.clone(), + )) + }) } fn run_pull_sync_after_send( @@ -603,6 +614,12 @@ fn run_pull_sync_after_send( /// Constructs a `KvIdentityGraph` from settings, or returns an error if the /// `ec_store` config is not set. +/// +/// Deliberately untimed: pull-sync (this function's only caller) runs after +/// `send_edgezero_response`'s Server-Timing freeze point, so a decorated +/// store here would record into a handle nothing ever renders. +/// Request-path callers with a `RequestTimings` handle use +/// [`require_identity_graph_with_timing`] instead. pub(crate) fn require_identity_graph( settings: &Settings, ) -> Result> { @@ -615,6 +632,27 @@ pub(crate) fn require_identity_graph( Ok(KvIdentityGraph::new(FastlyEcKvStore::new(store_name))) } +/// Constructs a `KvIdentityGraph` wrapped in the [`Phase::EcKv`] timing +/// decorator, or returns an error if the `ec_store` config is not set. +/// +/// Request-path sibling of [`require_identity_graph`], which pull-sync uses +/// unwrapped because pull-sync runs after the Server-Timing freeze point. +pub(crate) fn require_identity_graph_with_timing( + settings: &Settings, + timings: &RequestTimings, +) -> Result> { + let store_name = settings.ec.ec_store.as_deref().ok_or_else(|| { + Report::new(TrustedServerError::KvStore { + store_name: "ec.ec_store".to_owned(), + message: "ec.ec_store is not configured".to_owned(), + }) + })?; + Ok(KvIdentityGraph::new(TimedKvStore::new( + FastlyEcKvStore::new(store_name), + timings.clone(), + ))) +} + /// Extracts a named cookie value from the request's `Cookie` header. pub(crate) fn extract_cookie_value(req: &HttpRequest, name: &str) -> Option { let cookie_header = req.headers().get("cookie").and_then(|v| v.to_str().ok())?; @@ -644,6 +682,7 @@ pub(crate) fn derive_device_signals(req: &FastlyRequest) -> DeviceSignals { #[cfg(test)] mod tests { use super::*; + use base64::Engine as _; use edgezero_core::body::Body as EdgeBody; use edgezero_core::http::HeaderValue; use edgezero_core::http::response_builder; @@ -980,4 +1019,129 @@ mod tests { "should include sec-ch-ua-platform fallback" ); } + + fn ec_finalize_settings() -> Settings { + Settings::from_toml( + r#" + [[handlers]] + path = "^/_ts/admin" + username = "admin" + password = "admin-pass" + + [publisher] + domain = "test-publisher.com" + cookie_domain = ".test-publisher.com" + origin_url = "https://origin.test-publisher.com" + proxy_secret = "unit-test-proxy-secret" + + [ec] + passphrase = "test-secret-key-32-bytes-minimum" + ec_store = "ec_identity_store" + + [[ec.partners]] + name = "Example Partner" + source_domain = "example.com" + api_token = "test-vendor-token-32-bytes-minimum" + + [request_signing] + enabled = false + config_store_id = "test-config-store-id" + secret_store_id = "test-secret-store-id" + "#, + ) + .expect("should parse EC finalize test settings") + } + + /// Minimal `RuntimeServices` for `EcFinalizeState.services`. Real + /// `FastlyPlatform*` handles are used as inert placeholders: EC + /// finalization never calls through them, it only satisfies the field. + fn inert_runtime_services() -> RuntimeServices { + RuntimeServices::builder() + .config_store(Arc::new(crate::platform::FastlyPlatformConfigStore)) + .secret_store(Arc::new(crate::platform::FastlyPlatformSecretStore)) + .kv_store(Arc::new(edgezero_core::key_value_store::NoopKvStore) + as Arc) + .backend(Arc::new(crate::platform::FastlyPlatformBackend)) + .http_client(Arc::new(crate::platform::FastlyPlatformHttpClient)) + .geo(Arc::new(crate::platform::FastlyPlatformGeo)) + .client_info(trusted_server_core::platform::ClientInfo::default()) + .build() + } + + #[test] + fn ec_finalize_kv_lands_before_freeze() { + // A pre-seeded EC entry (see fastly.toml's ec_identity_store fixture) + // for a returning user carrying an eids cookie that matches the + // configured partner. This drives ec_finalize_response into + // ingest_eid_cookies, which reads and writes the KV identity graph. + let settings = ec_finalize_settings(); + let ec_id = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.test01"; + let eids = serde_json::json!([{ + "source": "example.com", + "uids": [{ "id": "example-uid", "atype": 1 }] + }]); + let eids_cookie = base64::engine::general_purpose::STANDARD.encode(eids.to_string()); + let request = edgezero_core::http::request_builder() + .method(fastly::http::Method::GET) + .uri("https://test-publisher.com/article") + .header("cookie", format!("ts-ec={ec_id}; ts-eids={eids_cookie}")) + .body(EdgeBody::empty()) + .expect("should build EC finalize test request"); + + let services = inert_runtime_services(); + let geo_info = trusted_server_core::platform::GeoInfo { + city: String::new(), + country: "US".to_owned(), + continent: "NorthAmerica".to_owned(), + latitude: 0.0, + longitude: 0.0, + metro_code: 0, + region: None, + asn: None, + }; + let ec_context = trusted_server_core::ec::EcContext::read_from_request_with_geo( + &settings, + &request, + &services, + Some(&geo_info), + ) + .expect("should read EC context from a non-regulated request"); + assert!( + ec_context.ec_was_present(), + "the pre-seeded ts-ec cookie should be recognized" + ); + + let ec_state = EcFinalizeState { + ec_context, + use_finalize_kv: true, + eids_cookie: Some(eids_cookie), + sharedid_cookie: None, + is_real_browser: true, + services, + }; + let mut response = response_builder() + .header("cache-control", "private, no-store") + .body(EdgeBody::empty()) + .expect("should build EC finalize response fixture"); + let timings = RequestTimings::new(); + + // Mirrors edgezero_main's ordering: EC finalize runs, then the freeze + // point (apply_server_timing_header, called just before + // response.into_parts() inside send_edgezero_response) renders the + // header. Calling both directly exercises exactly this order without + // requiring a live Fastly client connection. + apply_edgezero_ec_finalize(&settings, &ec_state, &mut response, &timings) + .expect("should finalize EC response"); + apply_server_timing_header(&mut response, &timings, true); + + let header = response + .headers() + .get("server-timing") + .and_then(|v| v.to_str().ok()) + .expect("should emit a Server-Timing header"); + assert!( + header.contains("ts-kv"), + "the freeze point must run after EC finalization recorded KV time: {header}" + ); + } } diff --git a/crates/trusted-server-core/src/ec/kv.rs b/crates/trusted-server-core/src/ec/kv.rs index 3572581ce8..dc9885fdec 100644 --- a/crates/trusted-server-core/src/ec/kv.rs +++ b/crates/trusted-server-core/src/ec/kv.rs @@ -810,6 +810,28 @@ mod tests { assert!(ts > 0, "should return a nonzero timestamp"); } + #[test] + fn kv_span_accumulates_across_graph_operations() { + let timings = crate::request_timing::RequestTimings::new(); + let graph = KvIdentityGraph::new(crate::platform::TimedKvStore::new( + crate::ec::kv_backend::test_support::InMemoryEcKv::new("test-store"), + timings.clone(), + )); + + graph + .create("ec-1", &live_entry()) + .expect("should create entry through the timed store"); + graph + .get("ec-1") + .expect("should read the entry back through the timed store"); + + timings.mark_headers_ready(); + assert!( + timings.snapshot().kv_ms.is_some(), + "should accumulate Phase::EcKv across both graph operations, not just the last write" + ); + } + #[test] fn serialize_entry_produces_valid_json() { let entry = KvEntry::tombstone(1000); diff --git a/crates/trusted-server-core/src/platform/mod.rs b/crates/trusted-server-core/src/platform/mod.rs index 1c5bf4c2a2..e6ed3ef903 100644 --- a/crates/trusted-server-core/src/platform/mod.rs +++ b/crates/trusted-server-core/src/platform/mod.rs @@ -42,6 +42,7 @@ mod template_assembly; mod template_cache; #[cfg(test)] pub(crate) mod test_support; +mod timed_kv; mod traits; mod types; @@ -67,6 +68,7 @@ pub use template_cache::{ TemplateEntry, TemplateMetadata, TemplateMetadataEncodeError, UnavailableTemplateCache, VaryHeaderValues, VarySpec, }; +pub use timed_kv::TimedKvStore; pub use traits::{PlatformBackend, PlatformConfigStore, PlatformGeo, PlatformSecretStore}; pub use types::{ ClientInfo, GeoInfo, PlatformBackendSpec, RuntimeServices, RuntimeServicesBuilder, StoreId, diff --git a/crates/trusted-server-core/src/platform/timed_kv.rs b/crates/trusted-server-core/src/platform/timed_kv.rs new file mode 100644 index 0000000000..0594b7cef1 --- /dev/null +++ b/crates/trusted-server-core/src/platform/timed_kv.rs @@ -0,0 +1,180 @@ +//! Latency-only timing decorator for KV store handles. +//! +//! [`TimedKvStore`] wraps an inner store plus a [`RequestTimings`] handle and +//! records [`Phase::EcKv`] around every call. It implements both +//! [`PlatformKvStore`] (for consent-store access obtained through +//! [`RuntimeServices`](super::RuntimeServices)) and [`EcKvStore`] (for +//! [`KvIdentityGraph`](crate::ec::kv::KvIdentityGraph) construction sites), +//! because no single existing abstraction covers the whole `ts-kv` taxonomy: +//! EC graph operations go through [`EcKvStore`] while consent persistence +//! uses [`PlatformKvStore`] directly. +//! +//! The decorator measures store-call latency only: it never reads, parses, +//! or logs any value passing through it. + +use std::sync::Arc; +use std::time::Duration; + +use async_trait::async_trait; +use bytes::Bytes; +use edgezero_core::key_value_store::{KvError, KvPage, KvStore as PlatformKvStore}; +use error_stack::Report; + +use crate::ec::kv_backend::{EcKvLookup, EcKvStore, EcKvWrite, EcKvWriteOutcome}; +use crate::error::TrustedServerError; +use crate::request_timing::{Phase, RequestTimings}; + +/// Wraps `inner` plus a [`RequestTimings`] handle, recording [`Phase::EcKv`] +/// around every store call made through it. +pub struct TimedKvStore { + /// The wrapped store handle. + inner: S, + /// The request's phase-timing collector. + timings: RequestTimings, +} + +impl TimedKvStore { + /// Creates a decorator around `inner` that records into `timings`. + #[must_use] + pub fn new(inner: S, timings: RequestTimings) -> Self { + Self { inner, timings } + } +} + +#[async_trait(?Send)] +impl PlatformKvStore for TimedKvStore> { + async fn get_bytes(&self, key: &str) -> Result, KvError> { + let _span = self.timings.span(Phase::EcKv); + self.inner.get_bytes(key).await + } + + async fn put_bytes(&self, key: &str, value: Bytes) -> Result<(), KvError> { + let _span = self.timings.span(Phase::EcKv); + self.inner.put_bytes(key, value).await + } + + async fn put_bytes_with_ttl( + &self, + key: &str, + value: Bytes, + ttl: Duration, + ) -> Result<(), KvError> { + let _span = self.timings.span(Phase::EcKv); + self.inner.put_bytes_with_ttl(key, value, ttl).await + } + + async fn delete(&self, key: &str) -> Result<(), KvError> { + let _span = self.timings.span(Phase::EcKv); + self.inner.delete(key).await + } + + async fn list_keys_page( + &self, + prefix: &str, + cursor: Option<&str>, + limit: usize, + ) -> Result { + let _span = self.timings.span(Phase::EcKv); + self.inner.list_keys_page(prefix, cursor, limit).await + } +} + +impl EcKvStore for TimedKvStore { + fn store_name(&self) -> &str { + self.inner.store_name() + } + + fn lookup(&self, key: &str) -> Result, Report> { + let _span = self.timings.span(Phase::EcKv); + self.inner.lookup(key) + } + + fn insert( + &self, + key: &str, + write: EcKvWrite<'_>, + ) -> Result> { + let _span = self.timings.span(Phase::EcKv); + self.inner.insert(key, write) + } + + fn count_keys_with_prefix( + &self, + prefix: &str, + limit: u32, + ) -> Result> { + let _span = self.timings.span(Phase::EcKv); + self.inner.count_keys_with_prefix(prefix, limit) + } + + fn delete(&self, key: &str) -> Result<(), Report> { + let _span = self.timings.span(Phase::EcKv); + self.inner.delete(key) + } +} + +#[cfg(test)] +mod tests { + use std::time::Duration as StdDuration; + + use super::*; + use crate::ec::kv_backend::test_support::InMemoryEcKv; + + #[test] + fn ec_kv_store_operations_accumulate_into_ec_kv_phase() { + let timings = RequestTimings::new(); + let store = TimedKvStore::new(InMemoryEcKv::new("test-store"), timings.clone()); + + store + .insert( + "key-a", + EcKvWrite { + body: "{}", + metadata: "{}", + ttl: StdDuration::from_secs(60), + mode: crate::ec::kv_backend::EcKvWriteMode::Add, + }, + ) + .expect("should insert into the in-memory store"); + store.lookup("key-a").expect("should read back the entry"); + + timings.mark_headers_ready(); + assert!( + timings.snapshot().kv_ms.is_some(), + "should record Phase::EcKv across both store calls" + ); + } + + #[test] + fn store_name_is_not_timed() { + let timings = RequestTimings::new(); + let store = TimedKvStore::new(InMemoryEcKv::new("test-store"), timings.clone()); + + assert_eq!(store.store_name(), "test-store"); + timings.mark_headers_ready(); + assert!( + timings.snapshot().kv_ms.is_none(), + "store_name is a metadata accessor, not a store operation" + ); + } + + #[test] + fn platform_kv_store_operations_accumulate_into_ec_kv_phase() { + let timings = RequestTimings::new(); + let inner: Arc = Arc::new(crate::platform::UnavailableKvStore); + let store = TimedKvStore::new(inner, timings.clone()); + + // UnavailableKvStore errors on every call; the decorator still times + // the attempt regardless of outcome. + futures::executor::block_on(async { + let _ = store.get_bytes("key").await; + let _ = store.put_bytes("key", Bytes::from_static(b"value")).await; + }); + + timings.mark_headers_ready(); + assert!( + timings.snapshot().kv_ms.is_some(), + "should record Phase::EcKv even when the inner store errors" + ); + } +} diff --git a/crates/trusted-server-core/src/publisher.rs b/crates/trusted-server-core/src/publisher.rs index b0d63b82ae..07d6704a7c 100644 --- a/crates/trusted-server-core/src/publisher.rs +++ b/crates/trusted-server-core/src/publisher.rs @@ -70,6 +70,7 @@ use crate::platform::{ contains_publisher_esi_directive, }; use crate::price_bucket::{PriceGranularity, price_bucket}; +use crate::request_timing::{Phase, RequestTimings}; use crate::response_privacy::{ apply_inactive_ad_stack_browser_cache_policy, cache_control_forbids_shared_storage, enforce_synthesized_html_cache_privacy, enforce_terminal_private_cache_privacy, @@ -4044,6 +4045,14 @@ pub async fn handle_publisher_request( ) -> Result> { log::debug!("Proxying request to publisher_origin"); + // A defaulted handle records into nothing that ever renders, so tests + // that don't populate the request extension are unaffected. + let timings = req + .extensions() + .get::() + .cloned() + .unwrap_or_default(); + // Adapter fallbacks prepare this before EC/cookie handling. Keep this // idempotent call as a direct-handler safety net and for focused tests. let gpt_diagnostics = @@ -4476,7 +4485,10 @@ pub async fn handle_publisher_request( // not be served a shared template even if that template is perfectly cacheable. let mut template_cache_reservation = None; if let Some(key) = template_cache_key.as_ref() { - match services.template_cache().lookup_or_reserve(key).await { + let template_cache_span = timings.span(Phase::TemplateCacheLookup); + let template_cache_lookup = services.template_cache().lookup_or_reserve(key).await; + drop(template_cache_span); + match template_cache_lookup { Ok(crate::platform::TemplateCacheLookup::Hit(entry)) => { log::debug!("template_cache hit: {} bytes", entry.body.len()); @@ -4577,6 +4589,7 @@ pub async fn handle_publisher_request( platform_request = platform_request.with_cache_bypass(); } + let origin_span = timings.span(Phase::Origin); let mut response = match services.http_client().send(platform_request).await { Ok(platform_response) => platform_response.response, Err(err) => { @@ -4594,6 +4607,7 @@ pub async fn handle_publisher_request( })); } }; + drop(origin_span); log::debug!( "Publisher origin response received: status={}, header_count={}", @@ -7737,6 +7751,35 @@ mod tests { .expect("should proxy publisher request") } + #[tokio::test] + async fn origin_span_covers_the_publisher_fetch() { + let settings = create_test_settings(); + let stub = Arc::new(StubHttpClient::new()); + stub.push_response_with_headers( + 200, + b"origin".to_vec(), + vec![(header::CONTENT_TYPE.as_str(), "text/html; charset=utf-8")], + ); + let services = + build_services_with_http_client(stub as Arc); + let mut request = HttpRequest::builder() + .method(Method::GET) + .uri("https://publisher.example/some-page") + .header(header::HOST, "publisher.example") + .body(EdgeBody::empty()) + .expect("should build request"); + let timings = RequestTimings::new(); + request.extensions_mut().insert(timings.clone()); + + let _response = run_publisher_proxy(&settings, &services, request).await; + + timings.mark_headers_ready(); + assert!( + timings.snapshot().origin_ms.is_some(), + "should record the Origin phase span around the publisher fetch" + ); + } + mod rendered_template_identity_tests { //! The gate the plan's Task 3 Step 2 actually asks for. //! @@ -8973,6 +9016,60 @@ mod tests { ); } + #[tokio::test] + async fn template_cache_span_recorded_only_when_lookup_runs() { + // Inline mode: no shared-cache key is ever computed, so the lookup + // never runs and the span is never recorded. + let inline_settings = create_test_settings(); + let inline_stub = Arc::new(StubHttpClient::new()); + inline_stub.push_response_with_headers( + 200, + b"origin".to_vec(), + vec![(header::CONTENT_TYPE.as_str(), "text/html; charset=utf-8")], + ); + let inline_services = build_services_with_http_client( + inline_stub as Arc, + ); + let mut inline_request = HttpRequest::builder() + .method(Method::GET) + .uri("https://publisher.example/some-page") + .header(header::HOST, "publisher.example") + .body(EdgeBody::empty()) + .expect("should build request"); + let inline_timings = RequestTimings::new(); + inline_request + .extensions_mut() + .insert(inline_timings.clone()); + + let _inline_response = + run_publisher_proxy(&inline_settings, &inline_services, inline_request).await; + + inline_timings.mark_headers_ready(); + assert!( + inline_timings.snapshot().template_cache_ms.is_none(), + "inline mode should never run the template-cache lookup" + ); + + // Shared-mode eligible: a matched slot plus a template-cache-eligible + // assembly mode compute a cache key, so the lookup runs and is timed. + let stub = Arc::new(StubHttpClient::new()); + let cache = Arc::new(MemoryTemplateCache::default()); + let settings = Arc::new(settings_with_mode("esi")); + let services = services(Arc::clone(&stub), Arc::clone(&cache)); + queue_shareable_html(&stub); + let mut request = navigation_request(); + let timings = RequestTimings::new(); + request.extensions_mut().insert(timings.clone()); + + let _response = run(&settings, &services, request).await; + + timings.mark_headers_ready(); + assert!( + timings.snapshot().template_cache_ms.is_some(), + "a shared-cache-eligible request should record the TemplateCacheLookup span" + ); + } + #[tokio::test] async fn only_the_cold_miss_uses_the_platform_assembler() { let stub = Arc::new(StubHttpClient::new()); From c222ae280931999fdd822473581ac22a632cab27 Mon Sep 17 00:00:00 2001 From: Jason Evans Date: Tue, 25 Aug 2026 00:57:06 -0700 Subject: [PATCH 008/194] Capture stream duration, auction wait placement, and response bytes --- .../trusted-server-adapter-fastly/src/main.rs | 269 ++++++++++++++++-- crates/trusted-server-core/src/publisher.rs | 210 +++++++++++++- 2 files changed, 462 insertions(+), 17 deletions(-) diff --git a/crates/trusted-server-adapter-fastly/src/main.rs b/crates/trusted-server-adapter-fastly/src/main.rs index ca97f64750..5747dc8fde 100644 --- a/crates/trusted-server-adapter-fastly/src/main.rs +++ b/crates/trusted-server-adapter-fastly/src/main.rs @@ -1,4 +1,5 @@ use std::sync::Arc; +use std::time::Instant; use edgezero_adapter_fastly::config_store::FastlyConfigStore as EdgeZeroFastlyConfigStore; use edgezero_adapter_fastly::request::into_core_request; @@ -405,10 +406,15 @@ pub(crate) struct DeliveryOutcome { } /// Whether [`send_edgezero_response`] completed delivery or failed partway. +#[derive(Debug, PartialEq, Eq)] pub(crate) enum DeliveryResult { /// The response was handed to the client in full. Complete, - /// Delivery failed partway through. + /// Delivery started but did not finish cleanly: some bytes reached the + /// client's transport before a stream error, or the transport could not + /// be closed cleanly after every byte was written. + Partial, + /// Delivery failed before any bytes reached the client. Error, } @@ -448,6 +454,93 @@ pub(crate) fn apply_server_timing_header( } } +/// A [`Write`](std::io::Write) wrapper that tallies bytes successfully written +/// to the inner writer. +/// +/// Wraps the client transport during a streaming drive so a truncated or +/// failed drive still reports how many bytes actually reached it, instead of +/// the placeholder `0` a failed/aborted drive would otherwise report. +struct CountingWriter { + inner: W, + bytes: u64, +} + +impl CountingWriter { + fn new(inner: W) -> Self { + Self { inner, bytes: 0 } + } + + /// Bytes successfully written to the inner writer so far. + fn bytes(&self) -> u64 { + self.bytes + } + + fn into_inner(self) -> W { + self.inner + } +} + +impl std::io::Write for CountingWriter { + fn write(&mut self, buf: &[u8]) -> std::io::Result { + let written = self.inner.write(buf)?; + self.bytes = self.bytes.saturating_add(written as u64); + Ok(written) + } + + fn flush(&mut self) -> std::io::Result<()> { + self.inner.flush() + } +} + +/// Drives a streaming `EdgeZero` body through `output`, tallying bytes written +/// and timing the drive into `timings`. +/// +/// Stamps `resp_bytes` and `request_elapsed` immediately once the drive +/// returns — before the caller does anything transport-specific (finishing +/// the streaming body, logging) — so `request_elapsed` never includes that +/// work. Returns the counting writer (so the caller can recover both the +/// tallied byte count and the wrapped transport) alongside the drive's +/// result. +fn drive_streaming_body( + body: EdgeBody, + output: W, + timings: &RequestTimings, +) -> (CountingWriter, Result<(), Report>) { + let mut counting = CountingWriter::new(output); + let drive_started = Instant::now(); + let result = futures::executor::block_on(stream_asset_body(body, &mut counting)); + timings.record(Phase::Stream, drive_started.elapsed()); + timings.set_resp_bytes(counting.bytes()); + timings.mark_request_elapsed(); + (counting, result) +} + +/// Classifies a completed streaming drive into a [`DeliveryResult`]. +/// +/// A drive that failed after writing at least one byte delivered a truncated +/// response rather than nothing at all, so it is [`DeliveryResult::Partial`], +/// not [`DeliveryResult::Error`]. +fn classify_stream_delivery( + drive_result: &Result<(), Report>, + bytes: u64, +) -> DeliveryResult { + match drive_result { + Ok(()) => DeliveryResult::Complete, + Err(_) if bytes > 0 => DeliveryResult::Partial, + Err(_) => DeliveryResult::Error, + } +} + +/// Stamps `resp_bytes`/`request_elapsed` for an already-materialized body, +/// immediately before it is handed to the Fastly client transport, and +/// returns its byte length. +fn record_buffered_delivery(body: &EdgeBody, timings: &RequestTimings) -> u64 { + let bytes = u64::try_from(body.as_bytes().map(<[u8]>::len).unwrap_or(0)).unwrap_or(u64::MAX); + timings.set_resp_bytes(bytes); + timings.mark_request_elapsed(); + bytes +} + /// Sends a finalized `EdgeZero` response to the client. /// /// Streaming `EdgeZero` bodies commit headers first, then pipe chunks to Fastly's @@ -473,30 +566,40 @@ fn send_edgezero_response( parts, EdgeBody::empty(), )); - let mut streaming_body = skeleton.stream_to_client(); - match futures::executor::block_on(stream_asset_body(body, &mut streaming_body)) { - Ok(()) => { - if let Err(e) = streaming_body.finish() { - log::error!("failed to finish EdgeZero streaming body: {e}"); - } - DeliveryOutcome { - bytes: 0, + let (counting, drive_result) = + drive_streaming_body(body, skeleton.stream_to_client(), &context.timings); + let bytes = counting.bytes(); + let streaming_body = counting.into_inner(); + // Computed before `drive_result` is matched by value below, since + // the `Err` arm there moves its `Report` out. + let result = classify_stream_delivery(&drive_result, bytes); + match drive_result { + Ok(()) => match streaming_body.finish() { + Ok(()) => DeliveryOutcome { + bytes, result: DeliveryResult::Complete, + }, + Err(e) => { + // Every byte was handed to the transport (the drive + // above returned Ok), but the transport itself could + // not close cleanly — the client may still see a + // truncated response. + log::error!("failed to finish EdgeZero streaming body: {e}"); + DeliveryOutcome { + bytes, + result: DeliveryResult::Partial, + } } - } + }, Err(e) => { log::error!("EdgeZero streaming failed: {e:?}"); drop(streaming_body); - DeliveryOutcome { - bytes: 0, - result: DeliveryResult::Error, - } + DeliveryOutcome { bytes, result } } } } once => { - let bytes = - u64::try_from(once.as_bytes().map(<[u8]>::len).unwrap_or(0)).unwrap_or(u64::MAX); + let bytes = record_buffered_delivery(&once, &context.timings); compat::to_fastly_response(HttpResponse::from_parts(parts, once)).send_to_client(); DeliveryOutcome { bytes, @@ -687,7 +790,9 @@ mod tests { use edgezero_core::http::HeaderValue; use edgezero_core::http::response_builder; use fastly::mime; + use std::time::Duration; use trusted_server_core::integrations::HeaderMutation; + use trusted_server_core::request_timing::AuctionWaitPlacement; fn test_settings() -> Settings { Settings::from_toml( @@ -1144,4 +1249,136 @@ mod tests { "the freeze point must run after EC finalization recorded KV time: {header}" ); } + + #[test] + fn delivery_outcome_reports_bytes_and_request_elapsed_set() { + let timings = RequestTimings::new(); + let body = EdgeBody::stream(futures::stream::iter(vec![ + bytes::Bytes::from_static(b"hello "), + bytes::Bytes::from_static(b"world"), + ])); + + let (counting, drive_result) = drive_streaming_body(body, Vec::new(), &timings); + drive_result.expect("streaming a well-formed body should not fail"); + let bytes = counting.bytes(); + let outcome = DeliveryOutcome { + bytes, + result: DeliveryResult::Complete, + }; + + assert_eq!( + counting.into_inner(), + b"hello world", + "should write every byte to the underlying transport" + ); + assert_eq!( + outcome.bytes, + "hello world".len() as u64, + "DeliveryOutcome.bytes should equal the streamed body length" + ); + + let snapshot = timings.snapshot(); + assert_eq!( + snapshot.resp_bytes, + Some("hello world".len() as u64), + "should stamp resp_bytes to the tallied byte count" + ); + assert!( + snapshot.request_elapsed_ms.is_some(), + "should stamp request_elapsed once the drive returns" + ); + } + + #[test] + fn buffered_delivery_stamps_bytes_and_request_elapsed() { + let timings = RequestTimings::new(); + let body = EdgeBody::from(b"a buffered body".to_vec()); + + let bytes = record_buffered_delivery(&body, &timings); + + assert_eq!( + bytes, + "a buffered body".len() as u64, + "should report the buffered body length" + ); + let snapshot = timings.snapshot(); + assert_eq!( + snapshot.resp_bytes, + Some("a buffered body".len() as u64), + "should stamp resp_bytes for the buffered path too" + ); + assert!( + snapshot.request_elapsed_ms.is_some(), + "should stamp request_elapsed for the buffered path too" + ); + } + + #[test] + fn stream_drive_records_stream_ms_covering_the_in_stream_auction_wait() { + // A streaming seam wait (Task 6, publisher.rs) records into the same + // `RequestTimings` handle the adapter drives with. `Phase::Stream` + // wraps the entire drive, so it must cover — and therefore be at + // least as large as — any `AuctionWait` recorded while the body was + // being polled. + let timings = RequestTimings::new(); + let wait_timings = timings.clone(); + let stream = futures::stream::once(async move { + let waited = Duration::from_millis(5); + std::thread::sleep(waited); + wait_timings.record_auction_wait(AuctionWaitPlacement::InStream, waited); + bytes::Bytes::from_static(b"") + }); + let body = EdgeBody::stream(stream); + + let (_counting, drive_result) = drive_streaming_body(body, Vec::new(), &timings); + drive_result.expect("streaming a well-formed body should not fail"); + + let snapshot = timings.snapshot(); + assert_eq!( + snapshot.auction_wait_placement, + Some(AuctionWaitPlacement::InStream), + "should preserve the placement recorded from inside the polled body" + ); + let auction_wait_ms = snapshot + .auction_wait_ms + .expect("should record the auction wait"); + let stream_ms = snapshot.stream_ms.expect("should record the stream drive"); + assert!( + stream_ms >= auction_wait_ms, + "the drive's Phase::Stream span must cover the in-stream auction wait: \ + stream_ms={stream_ms} auction_wait_ms={auction_wait_ms}" + ); + } + + #[test] + fn classify_stream_delivery_treats_bytes_written_before_an_error_as_partial() { + let err = Report::new(TrustedServerError::Proxy { + message: "boom".to_string(), + }); + assert_eq!( + classify_stream_delivery(&Err(err), 42), + DeliveryResult::Partial, + "bytes already on the wire before a stream error is a truncated delivery" + ); + } + + #[test] + fn classify_stream_delivery_treats_an_error_with_no_bytes_as_error() { + let err = Report::new(TrustedServerError::Proxy { + message: "boom".to_string(), + }); + assert_eq!( + classify_stream_delivery(&Err(err), 0), + DeliveryResult::Error, + "a failure before any byte reached the client is a clean failure, not a truncation" + ); + } + + #[test] + fn classify_stream_delivery_treats_ok_as_complete() { + assert_eq!( + classify_stream_delivery(&Ok(()), 123), + DeliveryResult::Complete + ); + } } diff --git a/crates/trusted-server-core/src/publisher.rs b/crates/trusted-server-core/src/publisher.rs index 07d6704a7c..1943d9b3cf 100644 --- a/crates/trusted-server-core/src/publisher.rs +++ b/crates/trusted-server-core/src/publisher.rs @@ -70,7 +70,7 @@ use crate::platform::{ contains_publisher_esi_directive, }; use crate::price_bucket::{PriceGranularity, price_bucket}; -use crate::request_timing::{Phase, RequestTimings}; +use crate::request_timing::{AuctionWaitPlacement, Phase, RequestTimings}; use crate::response_privacy::{ apply_inactive_ad_stack_browser_cache_policy, cache_control_forbids_shared_storage, enforce_synthesized_html_cache_privacy, enforce_terminal_private_cache_privacy, @@ -1617,6 +1617,12 @@ pub struct OwnedProcessResponseParams { /// rescanned from the output, which cannot tell a `nonce` attribute from the same /// word inside a script. pub(crate) csp_nonce_observed: Option>, + /// Per-request phase-timing handle, carried into the streaming/buffered + /// finalizers so the `` seam wait can be recorded with the right + /// [`AuctionWaitPlacement`]. Cheap to clone (an `Arc` handle); a request that + /// never attached one to its extensions gets a fresh, unattached collector + /// that nothing ever renders. + pub(crate) timings: RequestTimings, } /// Response-authorized template cache insert inputs. The key is built before origin lookup; the @@ -1860,6 +1866,8 @@ pub async fn buffer_publisher_response_async( ¶ms.request_scheme, ¶ms.request_host, ), + timings: params.timings.clone(), + placement: AuctionWaitPlacement::PreHeader, }, ) .await; @@ -2019,6 +2027,7 @@ fn build_template_assembly_params( request_scheme: &str, price_granularity: PriceGranularity, ad_bids_state: AdBidsState, + timings: RequestTimings, ) -> OwnedProcessResponseParams { OwnedProcessResponseParams { csp_nonce_observed: None, @@ -2041,6 +2050,7 @@ fn build_template_assembly_params( price_granularity, gpt_diagnostics: None, suppress_datadome_client_side_tag: false, + timings, } } @@ -2383,6 +2393,8 @@ pub async fn publisher_response_into_streaming_response( ¶ms.request_scheme, ¶ms.request_host, ), + timings: params.timings.clone(), + placement: AuctionWaitPlacement::InStream, }, ) .await; @@ -2501,6 +2513,7 @@ pub async fn publisher_response_into_streaming_response( &orchestrator, &services, &settings, + AuctionWaitPlacement::InStream, ) .await; // Collection reached a terminal result; disarm only now @@ -2522,6 +2535,8 @@ pub async fn publisher_response_into_streaming_response( ¶ms.request_scheme, ¶ms.request_host, ), + timings: params.timings.clone(), + placement: AuctionWaitPlacement::InStream, }; while let Some(step) = hold_step_next_chunk( @@ -2857,6 +2872,7 @@ pub async fn stream_publisher_body_async( orchestrator, services, settings, + AuctionWaitPlacement::PreHeader, ) .await; if body.is_stream() { @@ -2923,6 +2939,8 @@ pub async fn stream_publisher_body_async( services, settings, request_origin: request_origin(¶ms.request_scheme, ¶ms.request_host), + timings: params.timings.clone(), + placement: AuctionWaitPlacement::PreHeader, }, }, ) @@ -3490,6 +3508,12 @@ struct AuctionCollectDeps<'a> { settings: &'a Settings, /// Trusted request origin (`scheme://host`) for absolute inline creative URLs. request_origin: String, + /// Phase-timing handle the collect step records the auction wait into. + timings: RequestTimings, + /// Where this collect call sits relative to response headers: streaming + /// callers await inside the body already handed to the client, buffered + /// callers await before anything has been sent. + placement: AuctionWaitPlacement, } /// Run the close-body hold loop for HTML bodies, collecting the auction before @@ -3870,6 +3894,11 @@ async fn emit_abandoned_auction( /// Collect a dispatched auction before a non-HTML body streams: there is no /// `` to inject into, so bids are written to state up front and the /// auction telemetry completes immediately. +/// +/// `placement` records where this wait sits relative to response headers — the +/// caller decides, since this collector runs from both the buffered finalizer +/// (headers not yet committed) and the true streaming path (headers already +/// sent, this body only just started being polled). async fn collect_non_html_auction( dispatched: DispatchedAuction, telemetry: AuctionTelemetryCarry, @@ -3877,12 +3906,14 @@ async fn collect_non_html_auction( orchestrator: &AuctionOrchestrator, services: &RuntimeServices, settings: &Settings, + placement: AuctionWaitPlacement, ) { let auction_id = telemetry .auction_request .as_ref() .and_then(|_| diagnostics_auction_id(settings)); let placeholder = mediator_placeholder_request(); + let wait_started = Instant::now(); let result = orchestrator .collect_dispatched_auction( dispatched, @@ -3890,6 +3921,9 @@ async fn collect_non_html_auction( &make_collect_context(settings, services, &placeholder), ) .await; + params + .timings + .record_auction_wait(placement, wait_started.elapsed()); let delivered_winner_slots = write_bids_to_state( &result.winning_bids, params.price_granularity, @@ -3931,6 +3965,8 @@ async fn collect_stream_auction( services, settings, request_origin, + timings, + placement, } = deps; let auction_id = telemetry .auction_request @@ -3939,9 +3975,11 @@ async fn collect_stream_auction( log::info!("body_close_hold_loop: collecting dispatched auction before held body tail"); let placeholder = mediator_placeholder_request(); let collect_ctx = make_collect_context(settings, services, &placeholder); + let wait_started = Instant::now(); let result = orchestrator .collect_dispatched_auction(dispatched, services, &collect_ctx) .await; + timings.record_auction_wait(*placement, wait_started.elapsed()); log::info!( "body_close_hold_loop: collect complete - {} winning bid(s)", result.winning_bids.len() @@ -4538,6 +4576,7 @@ pub async fn handle_publisher_request( request_scheme, price_granularity, ad_bids_state.clone(), + timings.clone(), ); params.seam_ad_slots = seam_ad_slots.clone(); params.dispatched_auction = dispatched_auction.take(); @@ -4910,6 +4949,7 @@ pub async fn handle_publisher_request( dispatched_auction, price_granularity, gpt_diagnostics: Some(gpt_diagnostics), + timings: timings.clone(), }), }) } @@ -7546,6 +7586,7 @@ mod tests { price_granularity: Default::default(), gpt_diagnostics: None, suppress_datadome_client_side_tag: false, + timings: RequestTimings::new(), } } @@ -14592,6 +14633,8 @@ mod tests { services: &services, settings: &settings, request_origin: String::new(), + timings: RequestTimings::new(), + placement: AuctionWaitPlacement::PreHeader, }, }; let mut output = Vec::new(); @@ -14641,6 +14684,8 @@ mod tests { services: &services, settings: &settings, request_origin: String::new(), + timings: RequestTimings::new(), + placement: AuctionWaitPlacement::PreHeader, }; // Passthrough processor: the ordering contract is about collection, not // HTML rewriting, so keep the emitted bytes verbatim. @@ -15517,6 +15562,7 @@ mod tests { price_granularity: crate::price_bucket::PriceGranularity::default(), gpt_diagnostics: None, suppress_datadome_client_side_tag: false, + timings: RequestTimings::new(), }; let mut output = Vec::new(); @@ -15570,6 +15616,7 @@ mod tests { price_granularity: crate::price_bucket::PriceGranularity::default(), gpt_diagnostics: None, suppress_datadome_client_side_tag: false, + timings: RequestTimings::new(), }; let mut output = Vec::new(); @@ -15612,6 +15659,7 @@ mod tests { price_granularity: crate::price_bucket::PriceGranularity::default(), gpt_diagnostics: None, suppress_datadome_client_side_tag: false, + timings: RequestTimings::new(), }; let body = EdgeBody::from_stream(futures::stream::iter(vec![Ok::<_, io::Error>( bytes::Bytes::from_static(b"live"), @@ -15732,6 +15780,7 @@ mod tests { price_granularity: crate::price_bucket::PriceGranularity::default(), gpt_diagnostics: None, suppress_datadome_client_side_tag: false, + timings: RequestTimings::new(), }; let body = EdgeBody::stream(futures::stream::iter(vec![ bytes::Bytes::from_static(b"body{background:url('https://origin.example.com/"), @@ -15790,6 +15839,7 @@ mod tests { price_granularity: crate::price_bucket::PriceGranularity::default(), gpt_diagnostics: None, suppress_datadome_client_side_tag: false, + timings: RequestTimings::new(), }; let compressed = gzip_encode(b"body{background:url('https://origin.example.com/asset.png')}"); @@ -15851,6 +15901,7 @@ mod tests { price_granularity: crate::price_bucket::PriceGranularity::default(), gpt_diagnostics: None, suppress_datadome_client_side_tag: false, + timings: RequestTimings::new(), }; let compressed = deflate_encode(b"body{background:url('https://origin.example.com/asset.png')}"); @@ -15912,6 +15963,7 @@ mod tests { price_granularity: crate::price_bucket::PriceGranularity::default(), gpt_diagnostics: None, suppress_datadome_client_side_tag: false, + timings: RequestTimings::new(), }; let compressed = brotli_encode(b"body{background:url('https://origin.example.com/asset.png')}"); @@ -15973,6 +16025,7 @@ mod tests { price_granularity: crate::price_bucket::PriceGranularity::default(), gpt_diagnostics: None, suppress_datadome_client_side_tag: false, + timings: RequestTimings::new(), }; let compressed = brotli_encode(b"body{background:url('https://origin.example.com/asset.png')}"); @@ -16022,6 +16075,7 @@ mod tests { price_granularity: crate::price_bucket::PriceGranularity::default(), gpt_diagnostics: None, suppress_datadome_client_side_tag: false, + timings: RequestTimings::new(), } } @@ -16221,6 +16275,7 @@ mod tests { price_granularity: crate::price_bucket::PriceGranularity::default(), gpt_diagnostics: None, suppress_datadome_client_side_tag: false, + timings: RequestTimings::new(), }; let body = EdgeBody::stream(futures::stream::iter(vec![ bytes::Bytes::from_static(b"hello"), @@ -16290,6 +16345,7 @@ mod tests { price_granularity: crate::price_bucket::PriceGranularity::default(), gpt_diagnostics: None, suppress_datadome_client_side_tag: false, + timings: RequestTimings::new(), }; // The `` that triggers bid injection lives in the SECOND gzip // member. `flate2::read::GzDecoder` decodes only the first member, so @@ -16358,6 +16414,7 @@ mod tests { price_granularity: crate::price_bucket::PriceGranularity::default(), gpt_diagnostics: None, suppress_datadome_client_side_tag: false, + timings: RequestTimings::new(), }; let body = EdgeBody::stream(futures::stream::iter(vec![bytes::Bytes::from_static( b"body{background:url('https://origin.example.com/asset.png')}", @@ -16419,6 +16476,7 @@ mod tests { price_granularity: crate::price_bucket::PriceGranularity::default(), gpt_diagnostics: None, suppress_datadome_client_side_tag: false, + timings: RequestTimings::new(), }; let publisher_response = PublisherResponse::Stream { response, @@ -16568,6 +16626,7 @@ mod tests { price_granularity: crate::price_bucket::PriceGranularity::default(), gpt_diagnostics: None, suppress_datadome_client_side_tag: false, + timings: RequestTimings::new(), } } @@ -16958,6 +17017,7 @@ mod tests { price_granularity: PriceGranularity::default(), gpt_diagnostics: None, suppress_datadome_client_side_tag: false, + timings: RequestTimings::new(), } }; let make_stream_response = || PublisherResponse::Stream { @@ -17020,6 +17080,149 @@ mod tests { assert_bodiless_abandoned(&buffered_sink); } + #[test] + fn streaming_seam_wait_records_in_stream_placement() { + // The true Fastly streaming path: `publisher_response_into_streaming_response` + // hands back a lazy body after headers have already been committed by + // `stream_to_client()`. The `` seam wait polled from inside that body + // must therefore be attributed `InStream`, never `PreHeader`. + let settings = Arc::new(create_test_settings()); + let registry = + IntegrationRegistry::new(&settings).expect("should create integration registry"); + let orchestrator = Arc::new(AuctionOrchestrator::new(settings.auction.clone())); + let timings = RequestTimings::new(); + + let mut params = make_stream_params(&settings, ""); + params.content_type = "text/html; charset=utf-8".to_string(); + params.dispatched_auction = Some(DispatchedAuction::empty_for_test( + test_auction_request(), + 500, + )); + params.auction_request = Some(test_auction_request()); + params.timings = timings.clone(); + + let response = Response::builder() + .status(StatusCode::OK) + .header(header::CONTENT_TYPE, "text/html; charset=utf-8") + .body(EdgeBody::empty()) + .expect("should build response"); + let body = EdgeBody::stream(futures::stream::iter(vec![ + bytes::Bytes::from_static(b"hello"), + bytes::Bytes::from_static(b""), + ])); + + let response = futures::executor::block_on(publisher_response_into_streaming_response( + PublisherResponse::Stream { + response, + body, + params: Box::new(params), + }, + &Method::GET, + Arc::clone(&settings), + ®istry, + Arc::clone(&orchestrator), + noop_services(), + )) + .expect("streaming finalize should succeed"); + + // The wait is only recorded once the lazy body is actually polled — the + // finalizer call above only constructs it. + let drained = futures::executor::block_on( + response + .into_body() + .into_bytes_bounded(settings.publisher.max_buffered_body_bytes), + ) + .expect("body should drain"); + assert!( + String::from_utf8_lossy(&drained).contains("hello"), + "should still stream the document" + ); + + let snapshot = timings.snapshot(); + assert_eq!( + snapshot.auction_wait_placement, + Some(AuctionWaitPlacement::InStream), + "the streaming seam wait must be attributed InStream" + ); + assert!( + snapshot.auction_wait_ms.is_some(), + "should record an auction wait duration" + ); + } + + #[test] + fn buffered_template_miss_records_pre_header_placement() { + // The buffered finalizer materializes the entire response — headers and + // body — before any of it reaches the client. Even though the wait runs + // through the same `` seam code path as the streaming finalizer + // above, headers have not committed here, so it must be attributed + // `PreHeader`. (This exercises the same collect step a shared-template + // authorized miss rides through: `template_cache_key`'s presence only + // changes what happens *after* collection — whether the transformed bytes + // are stored — not where the wait itself is measured.) + let settings = create_test_settings(); + let registry = + IntegrationRegistry::new(&settings).expect("should create integration registry"); + let orchestrator = AuctionOrchestrator::new(settings.auction.clone()); + let services = noop_services(); + let timings = RequestTimings::new(); + + let mut params = make_stream_params(&settings, ""); + params.content_type = "text/html; charset=utf-8".to_string(); + params.dispatched_auction = Some(DispatchedAuction::empty_for_test( + test_auction_request(), + 500, + )); + params.auction_request = Some(test_auction_request()); + params.timings = timings.clone(); + + let response = Response::builder() + .status(StatusCode::OK) + .header(header::CONTENT_TYPE, "text/html; charset=utf-8") + .body(EdgeBody::empty()) + .expect("should build response"); + let body = EdgeBody::stream(futures::stream::iter(vec![ + bytes::Bytes::from_static(b"hello"), + bytes::Bytes::from_static(b""), + ])); + + let response = futures::executor::block_on(buffer_publisher_response_async( + PublisherResponse::Stream { + response, + body, + params: Box::new(params), + }, + &Method::GET, + &settings, + ®istry, + &orchestrator, + &services, + )) + .expect("buffered finalize should succeed"); + + let html = String::from_utf8( + response + .into_body() + .into_bytes() + .unwrap_or_default() + .to_vec(), + ) + .expect("should be valid UTF-8"); + assert!(html.contains("hello"), "should still assemble the document"); + + let snapshot = timings.snapshot(); + assert_eq!( + snapshot.auction_wait_placement, + Some(AuctionWaitPlacement::PreHeader), + "the buffered finalizer's wait must be attributed PreHeader even though \ + it shares the seam code path with the streaming finalizer" + ); + assert!( + snapshot.auction_wait_ms.is_some(), + "should record an auction wait duration" + ); + } + #[test] fn publisher_response_streaming_finalize_processes_gzip_stream() { let compressed = @@ -17142,6 +17345,7 @@ mod tests { price_granularity: crate::price_bucket::PriceGranularity::default(), gpt_diagnostics: None, suppress_datadome_client_side_tag: false, + timings: RequestTimings::new(), }; let publisher_response = PublisherResponse::Stream { response, @@ -17214,6 +17418,7 @@ mod tests { price_granularity: crate::price_bucket::PriceGranularity::default(), gpt_diagnostics: None, suppress_datadome_client_side_tag: false, + timings: RequestTimings::new(), }; let mut output = Vec::new(); @@ -17269,6 +17474,7 @@ mod tests { price_granularity: crate::price_bucket::PriceGranularity::default(), gpt_diagnostics: None, suppress_datadome_client_side_tag: false, + timings: RequestTimings::new(), }; let bogus_body = EdgeBody::from(b"not gzip".to_vec()); @@ -17382,6 +17588,7 @@ mod tests { price_granularity: crate::price_bucket::PriceGranularity::default(), gpt_diagnostics: None, suppress_datadome_client_side_tag: false, + timings: RequestTimings::new(), }; let mut output = Vec::new(); stream_publisher_body(body, &mut output, ¶ms, &settings, ®istry) @@ -17444,6 +17651,7 @@ mod tests { price_granularity: crate::price_bucket::PriceGranularity::default(), gpt_diagnostics: None, suppress_datadome_client_side_tag: false, + timings: RequestTimings::new(), }; let mut output = Vec::new(); From 8b028bb90aca365eddd4f106e3bc2f5ec818735b Mon Sep 17 00:00:00 2001 From: Jason Evans Date: Tue, 25 Aug 2026 09:27:44 -0700 Subject: [PATCH 009/194] Add access telemetry snapshot, route classes, and coarse route templates --- .../trusted-server-adapter-fastly/src/app.rs | 196 +++++++- .../trusted-server-adapter-fastly/src/main.rs | 283 +++++++++++- .../src/access_telemetry.rs | 418 ++++++++++++++++++ crates/trusted-server-core/src/constants.rs | 2 + crates/trusted-server-core/src/lib.rs | 1 + crates/trusted-server-core/src/publisher.rs | 78 +++- 6 files changed, 967 insertions(+), 11 deletions(-) create mode 100644 crates/trusted-server-core/src/access_telemetry.rs diff --git a/crates/trusted-server-adapter-fastly/src/app.rs b/crates/trusted-server-adapter-fastly/src/app.rs index 5cf7f97097..0a0cc8c5ae 100644 --- a/crates/trusted-server-adapter-fastly/src/app.rs +++ b/crates/trusted-server-adapter-fastly/src/app.rs @@ -98,6 +98,7 @@ use edgezero_core::http::{ }; use edgezero_core::router::RouterService; use error_stack::Report; +use trusted_server_core::access_telemetry::{RouteClass, RouteMetadata, publisher_route_template}; use trusted_server_core::auction::AuctionTelemetrySink; use trusted_server_core::auction::endpoints::handle_auction; use trusted_server_core::auction::{AuctionOrchestrator, build_orchestrator}; @@ -303,6 +304,12 @@ fn uses_dynamic_tsjs_fallback(method: &Method, path: &str) -> bool { *method == Method::GET && path.starts_with("/static/tsjs=") } +/// Coarse route template for every `tsjs` bundle request, used as the +/// `route_template` in the [`RouteMetadata`] attached by the tsjs branch of +/// [`dispatch_fallback`]. Actual filenames vary by module/hash; the prefix +/// alone is the route identity that matters for access telemetry. +const TSJS_ROUTE_TEMPLATE: &str = "/static/tsjs=*"; + // --------------------------------------------------------------------------- // EC request state // --------------------------------------------------------------------------- @@ -846,12 +853,28 @@ async fn dispatch_fallback( PreRoute::Continue { effects } => effects, }; + // Assigned exactly once, per branch below, alongside the routing + // decision itself, so the access-telemetry route identity always + // reflects which branch actually dispatched the request — including + // when that branch's handler errors. The asset-route sub-branch is an + // early return handled separately by `dispatch_asset_fallback`, so it + // never reaches (or needs to assign) this binding. + let route_metadata: Option; + let result = if uses_dynamic_tsjs_fallback(&method, &path) { + route_metadata = Some(RouteMetadata { + route_class: RouteClass::Tsjs, + route_template: TSJS_ROUTE_TEMPLATE.to_owned(), + }); handle_tsjs_dynamic(&req, &state.registry, EdgeCacheHeader::SurrogateControl) } else if state.registry.has_route(&method, &path) { // Integration-proxy responses are not bounded by // publisher.max_buffered_body_bytes. Publisher fallback below uses the // publisher-specific streaming finalizer instead. + route_metadata = Some(RouteMetadata { + route_class: RouteClass::IntegrationProxy, + route_template: publisher_route_template(&path), + }); state .registry .handle_proxy(ProxyDispatchInput { @@ -890,6 +913,11 @@ async fn dispatch_fallback( .await; } + route_metadata = Some(RouteMetadata { + route_class: RouteClass::PublisherHtml, + route_template: publisher_route_template(&path), + }); + // Generate an EC ID if needed — mirrors the legacy catch-all arm. // Only for document navigations by recognised browsers; subresource // requests may lack consent signals such as Sec-GPC. @@ -958,7 +986,10 @@ async fn dispatch_fallback( } }; - let response = result.unwrap_or_else(|e| http_error(&e)); + let mut response = result.unwrap_or_else(|e| http_error(&e)); + if let Some(metadata) = route_metadata { + response.extensions_mut().insert(metadata); + } attach_dispatch_extensions(response, ec, effects) } @@ -1154,6 +1185,10 @@ struct NamedRoute { path: &'static str, primary_methods: &'static [Method], handler: NamedRouteHandler, + /// Access-telemetry traffic category for this row. Attached verbatim + /// alongside `path` (the route-table pattern) to every response this + /// route produces — see [`named_route_handler`]. + route_class: RouteClass, } const LEGACY_ADMIN_DENY_METHODS: &[Method] = &[ @@ -1171,21 +1206,25 @@ const NAMED_ROUTES: &[NamedRoute] = &[ path: "/.well-known/trusted-server.json", primary_methods: &[Method::GET], handler: NamedRouteHandler::TrustedServerDiscovery, + route_class: RouteClass::Other, }, NamedRoute { path: "/verify-signature", primary_methods: &[Method::POST], handler: NamedRouteHandler::VerifySignature, + route_class: RouteClass::Ec, }, NamedRoute { path: "/_ts/admin/keys/rotate", primary_methods: &[Method::POST], handler: NamedRouteHandler::RotateKey, + route_class: RouteClass::Ec, }, NamedRoute { path: "/_ts/admin/keys/deactivate", primary_methods: &[Method::POST], handler: NamedRouteHandler::DeactivateKey, + route_class: RouteClass::Ec, }, // Admin EC lookup: the bare route reads the EC ID from the caller's // `ts-ec` cookie; the parameterized route takes an explicit EC ID. @@ -1193,11 +1232,13 @@ const NAMED_ROUTES: &[NamedRoute] = &[ path: "/_ts/admin/ec", primary_methods: &[Method::GET], handler: NamedRouteHandler::AdminEcLookup, + route_class: RouteClass::Ec, }, NamedRoute { path: "/_ts/admin/ec/{id}", primary_methods: &[Method::GET], handler: NamedRouteHandler::AdminEcLookup, + route_class: RouteClass::Ec, }, // Admin EIDs echo: decodes the request's ts-eids/sharedId cookies with // an ingestion preview. Pure request inspection — no KV access. @@ -1205,6 +1246,7 @@ const NAMED_ROUTES: &[NamedRoute] = &[ path: "/_ts/admin/eids", primary_methods: &[Method::GET], handler: NamedRouteHandler::AdminEidsLookup, + route_class: RouteClass::Ec, }, // The legacy non-`/_ts` aliases (`/admin/keys/*`) are denied locally with a // 404 instead of executing key operations: the production basic-auth handler @@ -1216,36 +1258,43 @@ const NAMED_ROUTES: &[NamedRoute] = &[ path: "/admin/keys/rotate", primary_methods: LEGACY_ADMIN_DENY_METHODS, handler: NamedRouteHandler::LegacyAdminDenied, + route_class: RouteClass::Other, }, NamedRoute { path: "/admin/keys/deactivate", primary_methods: LEGACY_ADMIN_DENY_METHODS, handler: NamedRouteHandler::LegacyAdminDenied, + route_class: RouteClass::Other, }, NamedRoute { path: "/_ts/api/v1/batch-sync", primary_methods: &[Method::POST], handler: NamedRouteHandler::BatchSync, + route_class: RouteClass::Ec, }, NamedRoute { path: "/_ts/api/v1/identify", primary_methods: &[Method::GET, Method::OPTIONS], handler: NamedRouteHandler::Identify, + route_class: RouteClass::Ec, }, NamedRoute { path: "/_ts/set-tester", primary_methods: &[Method::GET], handler: NamedRouteHandler::SetTester, + route_class: RouteClass::Other, }, NamedRoute { path: "/_ts/clear-tester", primary_methods: &[Method::GET], handler: NamedRouteHandler::ClearTester, + route_class: RouteClass::Other, }, NamedRoute { path: "/auction", primary_methods: &[Method::POST], handler: NamedRouteHandler::Auction, + route_class: RouteClass::AuctionApi, }, // GET runs the SPA re-auction; OPTIONS is denied in-handler as a CORS // preflight guard for this side-effecting endpoint. @@ -1253,6 +1302,7 @@ const NAMED_ROUTES: &[NamedRoute] = &[ path: PAGE_BIDS_PATH, primary_methods: &[Method::GET, Method::OPTIONS], handler: NamedRouteHandler::PageBids, + route_class: RouteClass::AuctionApi, }, // Deprecated double-underscore alias. tsjs bundles served before the // `/_ts/page-bids` rename keep requesting this path from already-loaded @@ -1263,21 +1313,29 @@ const NAMED_ROUTES: &[NamedRoute] = &[ path: PAGE_BIDS_LEGACY_PATH, primary_methods: &[Method::GET, Method::OPTIONS], handler: NamedRouteHandler::PageBids, + route_class: RouteClass::AuctionApi, }, + // Classified `Other` rather than `IntegrationProxy`: that class is + // reserved for `state.registry.handle_proxy` (the js-integration proxy + // dispatch in `dispatch_fallback`), which these first-party proxy routes + // do not go through. NamedRoute { path: "/first-party/proxy", primary_methods: &[Method::GET], handler: NamedRouteHandler::FirstPartyProxy, + route_class: RouteClass::Other, }, NamedRoute { path: "/first-party/click", primary_methods: &[Method::GET], handler: NamedRouteHandler::FirstPartyClick, + route_class: RouteClass::Other, }, NamedRoute { path: "/first-party/sign", primary_methods: &[Method::GET, Method::POST], handler: NamedRouteHandler::FirstPartySign, + route_class: RouteClass::Other, }, NamedRoute { path: "/first-party/proxy-rebuild", @@ -1286,16 +1344,35 @@ const NAMED_ROUTES: &[NamedRoute] = &[ // POST is blocked by CORS and the guard navigates here for a 302 instead. primary_methods: &[Method::GET, Method::POST], handler: NamedRouteHandler::FirstPartyProxyRebuild, + route_class: RouteClass::Other, }, ]; +/// Wraps [`execute_named`], attaching a [`RouteMetadata`] extension carrying +/// `route_class` and the route-table pattern (`route_template`, verbatim, +/// with parameters left as placeholders) to every response the handler +/// produces — including its early-return diagnostic and setup-error arms, +/// since the attachment happens once around the whole future rather than in +/// each branch. fn named_route_handler( state: Arc, handler: NamedRouteHandler, + route_class: RouteClass, + route_template: &'static str, ) -> impl Fn(RequestContext) -> HandlerFuture + Clone + Send + Sync + 'static { move |ctx: RequestContext| { let state = Arc::clone(&state); - Box::pin(execute_named(state, ctx, handler)) + Box::pin(async move { + execute_named(state, ctx, handler) + .await + .map(|mut response| { + response.extensions_mut().insert(RouteMetadata { + route_class, + route_template: route_template.to_owned(), + }); + response + }) + }) } } @@ -1354,7 +1431,12 @@ impl TrustedServerApp { router = router.route( route.path, method.clone(), - named_route_handler(Arc::clone(state), route.handler), + named_route_handler( + Arc::clone(state), + route.handler, + route.route_class, + route.path, + ), ); } @@ -1392,7 +1474,8 @@ mod tests { use super::{ AppState, NAMED_ROUTES, NamedRouteHandler, PAGE_BIDS_LEGACY_PATH, PAGE_BIDS_PATH, - TrustedServerApp, build_per_request_services, build_state_from_settings, + RouteClass, RouteMetadata, TSJS_ROUTE_TEMPLATE, TrustedServerApp, + build_per_request_services, build_state_from_settings, publisher_route_template, startup_error_router, }; use base64::Engine as _; @@ -2223,6 +2306,111 @@ mod tests { ); } + /// `Authorization: Basic` header value for `test_settings()`'s + /// `^/_ts/admin` handler (`admin` / `admin-pass`). + fn admin_basic_auth_header() -> edgezero_core::http::HeaderValue { + let credentials = base64::engine::general_purpose::STANDARD.encode("admin:admin-pass"); + format!("Basic {credentials}") + .parse() + .expect("should parse basic-auth header value") + } + + #[test] + fn named_route_attaches_the_table_pattern_verbatim_even_with_a_real_id_in_the_path() { + // A named-route response must carry the route-TABLE pattern + // (`{id}` left as a placeholder), never the caller's actual matched + // path segment — this is what keeps a real EC identifier out of + // access telemetry, independent of anything the row-serialization + // layer does. + let router = test_router(); + let ec_id = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.test01"; + let mut req = empty_request(Method::GET, &format!("/_ts/admin/ec/{ec_id}")); + req.headers_mut() + .insert(header::AUTHORIZATION, admin_basic_auth_header()); + let response = route(&router, req); + + let metadata = response + .extensions() + .get::() + .expect("named-route responses should carry RouteMetadata"); + assert_eq!(metadata.route_class, RouteClass::Ec); + assert_eq!(metadata.route_template, "/_ts/admin/ec/{id}"); + assert!( + !metadata.route_template.contains(ec_id), + "the attached template must never contain the matched id" + ); + } + + #[test] + fn named_route_attaches_metadata_even_on_a_read_only_diagnostic_early_return() { + // AdminEidsLookup is handled by an early-return arm inside + // execute_named, before the normal EC lifecycle runs (see the + // "read-only diagnostics" comment there). named_route_handler wraps + // the whole future, so the attachment must still happen here too. + let router = test_router(); + let mut req = empty_request(Method::GET, "/_ts/admin/eids"); + req.headers_mut() + .insert(header::AUTHORIZATION, admin_basic_auth_header()); + let response = route(&router, req); + + let metadata = response + .extensions() + .get::() + .expect("even a read-only diagnostic early-return response should carry RouteMetadata"); + assert_eq!(metadata.route_class, RouteClass::Ec); + assert_eq!(metadata.route_template, "/_ts/admin/eids"); + } + + #[test] + fn tsjs_fallback_attaches_tsjs_route_metadata() { + let router = test_router(); + let response = route( + &router, + empty_request(Method::GET, "/static/tsjs=tsjs-unified.min.js"), + ); + + let metadata = response + .extensions() + .get::() + .expect("tsjs fallback responses should carry RouteMetadata"); + assert_eq!(metadata.route_class, RouteClass::Tsjs); + assert_eq!(metadata.route_template, TSJS_ROUTE_TEMPLATE); + } + + #[test] + fn integration_proxy_fallback_attaches_integration_proxy_route_metadata() { + // test_settings() enables the prebid integration, which registers a + // proxy route at /integrations/prebid/bundle.js. + let router = test_router(); + let response = route( + &router, + empty_request(Method::GET, "/integrations/prebid/bundle.js"), + ); + + let metadata = response + .extensions() + .get::() + .expect("integration-proxy fallback responses should carry RouteMetadata"); + assert_eq!(metadata.route_class, RouteClass::IntegrationProxy); + assert_eq!( + metadata.route_template, + publisher_route_template("/integrations/prebid/bundle.js") + ); + } + + #[test] + fn publisher_fallback_attaches_publisher_html_route_metadata() { + let router = test_router(); + let response = route(&router, empty_request(Method::GET, "/news/some-article")); + + let metadata = response + .extensions() + .get::() + .expect("publisher fallback responses should carry RouteMetadata"); + assert_eq!(metadata.route_class, RouteClass::PublisherHtml); + assert_eq!(metadata.route_template, "/news/*"); + } + #[test] fn browser_device_signals_from_extension_reach_ec_finalize_state() { // Regression guard for the EdgeZero JA4/H2 signal loss: `edgezero_main` diff --git a/crates/trusted-server-adapter-fastly/src/main.rs b/crates/trusted-server-adapter-fastly/src/main.rs index 5747dc8fde..c09663d135 100644 --- a/crates/trusted-server-adapter-fastly/src/main.rs +++ b/crates/trusted-server-adapter-fastly/src/main.rs @@ -14,9 +14,13 @@ use error_stack::Report; use fastly::http::Method as FastlyMethod; use fastly::{Request as FastlyRequest, Response as FastlyResponse}; +use trusted_server_core::access_telemetry::{AccessTelemetrySnapshot, RouteClass, RouteMetadata}; use trusted_server_core::cache_policy::{ EdgeCacheHeader, cache_control_headers_are_private_or_no_store, }; +use trusted_server_core::constants::{ + ENV_FASTLY_IS_STAGING, ENV_FASTLY_POP, ENV_FASTLY_SERVICE_ID, ENV_FASTLY_SERVICE_VERSION, +}; use trusted_server_core::ec::device::DeviceSignals; use trusted_server_core::ec::finalize::ec_finalize_response; use trusted_server_core::ec::kv::KvIdentityGraph; @@ -30,6 +34,7 @@ use trusted_server_core::integrations::RequestFilterEffects; use trusted_server_core::platform::PlatformGeo as _; use trusted_server_core::platform::{RuntimeServices, TimedKvStore}; use trusted_server_core::proxy::{AssetProxyCachePolicy, stream_asset_body}; +use trusted_server_core::publisher::TemplateCacheResponseState; use trusted_server_core::request_timing::{Phase, RequestTimings}; use trusted_server_core::response_privacy::TerminalPrivateResponse; use trusted_server_core::settings::Settings; @@ -145,6 +150,18 @@ fn edgezero_main(mut req: FastlyRequest) { let server_timing_enabled = settings_snapshot .as_deref() .is_some_and(|settings| settings.observability.server_timing_enabled); + // Both read once here rather than at each `send_edgezero_response` call + // site: if `app_state` failed to build, there is no settings snapshot to + // read them from at all, so every call site would need the same + // degraded-mode fallback. `access_sample_rate` defaults to `0.0` (never + // sampled in) and `publisher_domain` to `"unknown"` in that case. + let access_sample_rate = settings_snapshot + .as_deref() + .map_or(0.0, |settings| settings.tinybird.access_sample_rate); + let publisher_domain = settings_snapshot.as_deref().map_or_else( + || "unknown".to_owned(), + |settings| settings.publisher.domain.clone(), + ); // Strip client-spoofable forwarded headers before dispatch. compat::sanitize_fastly_forwarded_headers(&mut req); @@ -159,8 +176,11 @@ fn edgezero_main(mut req: FastlyRequest) { req.set_header("fastly-ssl", "1"); } - // Capture client IP before the request is consumed by dispatch. + // Capture client IP and method before the request is consumed by + // dispatch: nothing else survives to the freeze point in + // `send_edgezero_response`, which only receives the response. let client_ip = req.get_client_ip_addr(); + let request_method = req.get_method_str().to_owned(); // Strip any client-supplied x-ts-tls-* headers before injecting the trusted // values from the Fastly SDK. Must run after sanitize_fastly_forwarded_headers. @@ -211,9 +231,13 @@ fn edgezero_main(mut req: FastlyRequest) { let ec_state = response.extensions_mut().remove::(); let asset_cache_policy = response.extensions_mut().remove::(); let request_filter_effects = response.extensions_mut().remove::(); + // Read rather than pop: the access-telemetry snapshot built later in + // `send_edgezero_response` reads this same extension, so it must still + // be attached to `response` at that point. let geo_lookup_state = response - .extensions_mut() - .remove::() + .extensions() + .get::() + .cloned() .unwrap_or(GeoLookupState::NotAttempted); if !take_finalize_sentinel(&mut response) { @@ -257,6 +281,9 @@ fn edgezero_main(mut req: FastlyRequest) { &SendContext { timings: timings.clone(), server_timing_enabled, + method: request_method.clone(), + publisher_domain: publisher_domain.clone(), + access_sample_rate, }, ); run_edgezero_pull_sync_after_send(settings, &partner_registry, &ec_state); @@ -280,6 +307,9 @@ fn edgezero_main(mut req: FastlyRequest) { &SendContext { timings: timings.clone(), server_timing_enabled, + method: request_method.clone(), + publisher_domain: publisher_domain.clone(), + access_sample_rate, }, ); run_edgezero_pull_sync_after_send( @@ -309,6 +339,9 @@ fn edgezero_main(mut req: FastlyRequest) { &SendContext { timings, server_timing_enabled, + method: request_method, + publisher_domain, + access_sample_rate, }, ); } @@ -349,6 +382,18 @@ fn apply_entry_point_finalize_headers( }) }); apply_finalize_headers(settings, geo_info.as_ref(), response); + + // This path runs only when the middleware chain was bypassed (e.g. a + // router-level 404/405 for an unregistered method), so `geo_state` may + // still be `NotAttempted` even after a fresh lookup just ran above. + // Write the resolved outcome back so the access-telemetry snapshot built + // later in `send_edgezero_response` sees what was actually looked up, + // not the stale carried-in state. + let resolved_state = match &geo_info { + Some(info) => GeoLookupState::Resolved(info.clone()), + None => GeoLookupState::Attempted, + }; + response.extensions_mut().insert(resolved_state); } fn apply_edgezero_ec_finalize( @@ -394,6 +439,13 @@ struct SendContext { timings: RequestTimings, /// Whether `observability.server_timing_enabled` is set. server_timing_enabled: bool, + /// The request's HTTP method, captured before the request was consumed + /// by dispatch. + method: String, + /// The configured publisher domain. + publisher_domain: String, + /// The configured access-telemetry sample rate. + access_sample_rate: f64, } /// Outcome of handing a finalized response to the client. @@ -403,6 +455,9 @@ pub(crate) struct DeliveryOutcome { pub bytes: u64, /// Whether delivery completed or failed partway. pub result: DeliveryResult, + /// Access-telemetry dimensions captured for this response at the + /// freeze point. + pub snapshot: AccessTelemetrySnapshot, } /// Whether [`send_edgezero_response`] completed delivery or failed partway. @@ -558,6 +613,12 @@ fn send_edgezero_response( context.server_timing_enabled, ); + // Built unconditionally, right after the freeze point and before + // `into_parts()` consumes `response`: nothing else survives to + // post-send on every path (the request was consumed by dispatch, and + // `EcFinalizeState` is absent on asset, admin, and error paths). + let snapshot = build_access_telemetry_snapshot(&response, context); + let (parts, body) = response.into_parts(); match body { @@ -578,6 +639,7 @@ fn send_edgezero_response( Ok(()) => DeliveryOutcome { bytes, result: DeliveryResult::Complete, + snapshot, }, Err(e) => { // Every byte was handed to the transport (the drive @@ -588,13 +650,18 @@ fn send_edgezero_response( DeliveryOutcome { bytes, result: DeliveryResult::Partial, + snapshot, } } }, Err(e) => { log::error!("EdgeZero streaming failed: {e:?}"); drop(streaming_body); - DeliveryOutcome { bytes, result } + DeliveryOutcome { + bytes, + result, + snapshot, + } } } } @@ -604,11 +671,89 @@ fn send_edgezero_response( DeliveryOutcome { bytes, result: DeliveryResult::Complete, + snapshot, } } } } +/// Builds the [`AccessTelemetrySnapshot`] for `response` at the +/// `Server-Timing` freeze point. +/// +/// Reads route identity, geo country, and template-cache state from typed +/// response extensions rather than the headers those extensions back — +/// operator-configured response headers can override a managed header, so +/// reading a header here could silently drift from what actually happened. +/// Falls back to `"unknown"`/[`RouteClass::Other`] sentinels when an +/// extension was never attached (router-generated, asset, and other +/// responses that never passed through a `RouteMetadata`-attaching +/// wrapper). +fn build_access_telemetry_snapshot( + response: &HttpResponse, + context: &SendContext, +) -> AccessTelemetrySnapshot { + let (route_class, route_template) = match response.extensions().get::() { + Some(metadata) => (metadata.route_class, metadata.route_template.clone()), + None => (RouteClass::Other, "unknown".to_owned()), + }; + + let country = match response.extensions().get::() { + Some(GeoLookupState::Resolved(info)) => info.country.clone(), + Some(GeoLookupState::Attempted | GeoLookupState::NotAttempted) | None => { + "unknown".to_owned() + } + }; + + let template_cache_state = response + .extensions() + .get::() + .map_or_else(|| "unknown".to_owned(), |state| state.as_str().to_owned()); + + let body_mode = if matches!(response.body(), EdgeBody::Stream(_)) { + "streamed" + } else { + "buffered" + }; + + AccessTelemetrySnapshot { + method: context.method.clone(), + status: response.status().as_u16(), + route_class, + route_template, + publisher_domain: context.publisher_domain.clone(), + env: resolve_env_dimension(), + service_id: env_var_or_unknown(ENV_FASTLY_SERVICE_ID), + pop: env_var_or_unknown(ENV_FASTLY_POP), + ts_version: env_var_or_unknown(ENV_FASTLY_SERVICE_VERSION), + country, + template_cache_state, + body_mode, + sample_rate: context.access_sample_rate, + } +} + +/// Derives the `env` access-telemetry dimension from the same +/// `FASTLY_IS_STAGING` input that drives the `x-ts-env` response header +/// (see [`apply_finalize_headers`]), never from [`Settings`] — `Settings` +/// has no environment field and does not gain one for this. +/// +/// `"unknown"` covers contexts where the variable is entirely absent (for +/// example native unit tests run outside Fastly Compute); on the Fastly +/// platform the variable is always present, as either `"1"` or not. +fn resolve_env_dimension() -> String { + match std::env::var(ENV_FASTLY_IS_STAGING) { + Ok(value) if value == "1" => "staging".to_owned(), + Ok(_) => "production".to_owned(), + Err(_) => "unknown".to_owned(), + } +} + +/// Reads a Fastly-provided environment variable, defaulting to `"unknown"` +/// when unset. +fn env_var_or_unknown(name: &str) -> String { + std::env::var(name).unwrap_or_else(|_| "unknown".to_owned()) +} + /// Apply every late response mutation, then restore privacy invariants before headers commit. fn apply_terminal_response_effects( response: &mut HttpResponse, @@ -820,6 +965,26 @@ mod tests { .expect("should parse test settings") } + /// A minimal [`AccessTelemetrySnapshot`] fixture for tests that only + /// need a `DeliveryOutcome` to exist, not its telemetry content. + fn sample_access_snapshot() -> AccessTelemetrySnapshot { + AccessTelemetrySnapshot { + method: "GET".to_owned(), + status: 200, + route_class: RouteClass::Other, + route_template: "/other/*".to_owned(), + publisher_domain: "unknown".to_owned(), + env: "unknown".to_owned(), + service_id: "unknown".to_owned(), + pop: "unknown".to_owned(), + ts_version: "unknown".to_owned(), + country: "unknown".to_owned(), + template_cache_state: "unknown".to_owned(), + body_mode: "buffered", + sample_rate: 0.0, + } + } + #[test] fn health_response_short_circuits_get_health() { let req = FastlyRequest::get("https://example.com/health"); @@ -1264,6 +1429,7 @@ mod tests { let outcome = DeliveryOutcome { bytes, result: DeliveryResult::Complete, + snapshot: sample_access_snapshot(), }; assert_eq!( @@ -1313,6 +1479,115 @@ mod tests { ); } + fn send_context_fixture() -> SendContext { + SendContext { + timings: RequestTimings::new(), + server_timing_enabled: false, + method: "GET".to_owned(), + publisher_domain: "test-publisher.com".to_owned(), + access_sample_rate: 0.25, + } + } + + #[test] + fn access_snapshot_defaults_when_no_extensions_are_attached() { + // Router-generated 404/405 responses and other paths that never pass + // through a RouteMetadata-attaching wrapper must still produce a + // usable snapshot: RouteClass::Other and "unknown" sentinels, never + // a missing/panicking build. + let response = response_builder() + .status(404) + .body(EdgeBody::empty()) + .expect("should build response"); + let context = send_context_fixture(); + + let snapshot = build_access_telemetry_snapshot(&response, &context); + + assert_eq!(snapshot.status, 404); + assert_eq!(snapshot.method, "GET"); + assert!(matches!(snapshot.route_class, RouteClass::Other)); + assert_eq!(snapshot.route_template, "unknown"); + assert_eq!(snapshot.country, "unknown"); + assert_eq!(snapshot.template_cache_state, "unknown"); + assert_eq!(snapshot.body_mode, "buffered"); + assert_eq!(snapshot.publisher_domain, "test-publisher.com"); + assert_eq!(snapshot.sample_rate, 0.25); + } + + #[test] + fn access_snapshot_reads_route_geo_and_template_cache_extensions() { + let mut response = response_builder() + .status(200) + .body(EdgeBody::empty()) + .expect("should build response"); + response.extensions_mut().insert(RouteMetadata { + route_class: RouteClass::AuctionApi, + route_template: "/auction".to_owned(), + }); + response.extensions_mut().insert(GeoLookupState::Resolved( + trusted_server_core::platform::GeoInfo { + city: String::new(), + country: "US".to_owned(), + continent: "NorthAmerica".to_owned(), + latitude: 0.0, + longitude: 0.0, + metro_code: 0, + region: None, + asn: None, + }, + )); + response + .extensions_mut() + .insert(TemplateCacheResponseState::Hit); + let context = send_context_fixture(); + + let snapshot = build_access_telemetry_snapshot(&response, &context); + + assert!(matches!(snapshot.route_class, RouteClass::AuctionApi)); + assert_eq!(snapshot.route_template, "/auction"); + assert_eq!(snapshot.country, "US"); + assert_eq!(snapshot.template_cache_state, "hit"); + } + + #[test] + fn access_snapshot_treats_attempted_geo_lookup_as_unknown_country() { + let mut response = response_builder() + .status(200) + .body(EdgeBody::empty()) + .expect("should build response"); + response.extensions_mut().insert(GeoLookupState::Attempted); + let context = send_context_fixture(); + + let snapshot = build_access_telemetry_snapshot(&response, &context); + + assert_eq!( + snapshot.country, "unknown", + "an attempted-but-unresolved lookup must not surface a stale country" + ); + } + + #[test] + fn access_snapshot_body_mode_reflects_the_response_body_variant() { + let streamed = response_builder() + .status(200) + .body(EdgeBody::stream(futures::stream::empty())) + .expect("should build streaming response"); + let buffered = response_builder() + .status(200) + .body(EdgeBody::from(b"hi".to_vec())) + .expect("should build buffered response"); + let context = send_context_fixture(); + + assert_eq!( + build_access_telemetry_snapshot(&streamed, &context).body_mode, + "streamed" + ); + assert_eq!( + build_access_telemetry_snapshot(&buffered, &context).body_mode, + "buffered" + ); + } + #[test] fn stream_drive_records_stream_ms_covering_the_in_stream_auction_wait() { // A streaming seam wait (Task 6, publisher.rs) records into the same diff --git a/crates/trusted-server-core/src/access_telemetry.rs b/crates/trusted-server-core/src/access_telemetry.rs new file mode 100644 index 0000000000..9daf77abc2 --- /dev/null +++ b/crates/trusted-server-core/src/access_telemetry.rs @@ -0,0 +1,418 @@ +//! Access telemetry: route classification and the per-request access log row. +//! +//! Extends the reserved `access_logs_raw` Tinybird datasource with bounded, +//! content-free route identity (see [`RouteClass`] and +//! [`publisher_route_template`]) instead of the raw request path, which would +//! otherwise carry identifiers, search terms, and other user-generated +//! content into a 30-day dataset. See the design spec +//! `docs/superpowers/specs/2026-08-24-request-phase-timing-design.md` +//! section 9. + +use serde_json::json; + +use crate::request_timing::{AuctionWaitPlacement, TimingSnapshot}; + +/// Maximum number of characters kept from a publisher path's first segment +/// by [`publisher_route_template`]. +const MAX_SEGMENT_LEN: usize = 32; + +/// Coarse traffic category for one response, used as a `LowCardinality` +/// dimension in the access telemetry row. +/// +/// Assigned per route by the adapter at dispatch time (see +/// [`RouteMetadata`]) rather than reconstructed from a handler enum or a +/// path regex at emission time, so the mapping lives in exactly one place. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum RouteClass { + /// A publisher-origin page served through the assembly/auction pipeline. + PublisherHtml, + /// The unified `tsjs` static bundle. + Tsjs, + /// A request served by a registered [`crate::integrations::IntegrationProxy`]. + IntegrationProxy, + /// An Edge Cookie identity endpoint (verify, rotate, identify, admin + /// lookups, batch sync). + Ec, + /// The server-side auction or SPA re-auction (`page-bids`) endpoint. + AuctionApi, + /// Everything else: discovery, tester-cookie toggles, denied legacy + /// aliases, and any response with no attached [`RouteMetadata`]. + Other, +} + +impl RouteClass { + /// Renders this variant as the `snake_case` string stored in the + /// `route_class` column. + #[must_use] + pub const fn as_str(self) -> &'static str { + match self { + Self::PublisherHtml => "publisher_html", + Self::Tsjs => "tsjs", + Self::IntegrationProxy => "integration_proxy", + Self::Ec => "ec", + Self::AuctionApi => "auction_api", + Self::Other => "other", + } + } +} + +/// Route identity for one response, carried from dispatch to the freeze +/// point as a response extension. +/// +/// The matched route pattern does not otherwise survive dispatch: the +/// request is consumed by the router, and nothing else records which +/// route-table row (or coarse fallback bucket) produced the response. Each +/// named-route handler wrapper attaches its matched route-table pattern +/// verbatim; the publisher fallback and `tsjs` handlers attach their class +/// plus a coarse template ([`publisher_route_template`] for the former). +#[derive(Debug, Clone)] +pub struct RouteMetadata { + /// Coarse traffic category for this response. + pub route_class: RouteClass, + /// Bounded, content-free route identifier. For named routes this is the + /// route-table pattern verbatim (e.g. `/_ts/admin/ec/{id}`); for + /// publisher-fallback traffic it is the output of + /// [`publisher_route_template`]. + pub route_template: String, +} + +/// Normalizes a publisher-fallback request path into a bounded, +/// content-free route template. +/// +/// Returns `/` plus the first path segment, lowercased and restricted to +/// `[a-z0-9_-]`, truncated to [`MAX_SEGMENT_LEN`] characters, with a +/// trailing `/*` appended when the path has additional segments beyond the +/// first. The root path `/` maps to itself. An empty first segment, or one +/// containing any character outside the allowlist (after lowercasing), +/// maps to `/other/*` — the segment is rejected outright rather than +/// filtered, so no fragment of a disallowed segment (an email address, a +/// search phrase) ever reaches the row. +/// +/// This is deliberately coarser than the auction-telemetry path +/// normalizer, which redacts long tokens but preserves short identifiers +/// and arbitrary slugs; that normalizer is not sufficient for a dataset +/// this broad. +/// +/// # Examples +/// +/// ``` +/// use trusted_server_core::access_telemetry::publisher_route_template; +/// +/// assert_eq!(publisher_route_template("/news/some-article-slug"), "/news/*"); +/// assert_eq!(publisher_route_template("/"), "/"); +/// assert_eq!(publisher_route_template("/user@example.com/profile"), "/other/*"); +/// ``` +#[must_use] +pub fn publisher_route_template(path: &str) -> String { + if path == "/" { + return "/".to_owned(); + } + + let trimmed = path.strip_prefix('/').unwrap_or(path); + let (first_segment, rest) = match trimmed.split_once('/') { + Some((first, rest)) => (first, rest), + None => (trimmed, ""), + }; + let has_more_depth = !rest.is_empty(); + + let lowered = first_segment.to_ascii_lowercase(); + let is_allowlisted = !lowered.is_empty() + && lowered + .chars() + .all(|c| c.is_ascii_lowercase() || c.is_ascii_digit() || c == '_' || c == '-'); + + if !is_allowlisted { + return "/other/*".to_owned(); + } + + let truncated: String = lowered.chars().take(MAX_SEGMENT_LEN).collect(); + if has_more_depth { + format!("/{truncated}/*") + } else { + format!("/{truncated}") + } +} + +/// A point-in-time view of the access-log dimensions for one response, +/// captured unconditionally at the `Server-Timing` freeze point. +/// +/// Built from typed response extensions ([`RouteMetadata`], the geo +/// lookup state, and the template-cache response state) plus adapter-owned +/// environment values, never from the public response headers those +/// extensions back — operator-configured response headers can override a +/// managed header, so reading the header instead of the extension would let +/// the row silently drift from the extension of record. +#[derive(Debug, Clone)] +pub struct AccessTelemetrySnapshot { + /// The request's HTTP method (e.g. `GET`). + pub method: String, + /// The response's HTTP status code. + pub status: u16, + /// Coarse traffic category (see [`RouteClass`]). + pub route_class: RouteClass, + /// Bounded, content-free route identifier (see + /// [`publisher_route_template`]). + pub route_template: String, + /// The configured publisher domain. + pub publisher_domain: String, + /// Adapter-derived deployment environment: `production`, `staging`, or + /// `unknown`. + pub env: String, + /// Fastly service ID, or `unknown` when unavailable. + pub service_id: String, + /// Fastly POP code, or `unknown` when unavailable. + pub pop: String, + /// Trusted Server build/version identifier, or `unknown` when + /// unavailable. + pub ts_version: String, + /// Two-letter geo country code, or `unknown` when no geo lookup + /// resolved one. + pub country: String, + /// Template-cache outcome for this response, or `unknown` when the + /// response never passed through the assembly pipeline. + pub template_cache_state: String, + /// Whether the response body was streamed or buffered to the client. + pub body_mode: &'static str, + /// The configured access-telemetry sample rate at the time this + /// response was handled. + pub sample_rate: f64, +} + +/// Renders one NDJSON access-log row for the Tinybird Events API. +/// +/// Column names match spec section 9 exactly. Phase columns come from +/// `timings` and serialize as JSON `null` for phases that were never +/// recorded; every dimension column comes from `snapshot` and is a +/// non-nullable string (callers are expected to substitute an `unknown` +/// sentinel rather than leave a dimension empty). `event_date` is omitted: +/// the datasource derives it from `event_ts` by default. +#[must_use] +pub fn access_event_row( + snapshot: &AccessTelemetrySnapshot, + timings: &TimingSnapshot, + event_ts_epoch_ms: u64, +) -> String { + let auction_wait_placement = match timings.auction_wait_placement { + Some(AuctionWaitPlacement::PreHeader) => "pre_header", + Some(AuctionWaitPlacement::InStream) => "in_stream", + None => "none", + }; + + let row = json!({ + "event_ts": format_event_timestamp(event_ts_epoch_ms), + "method": snapshot.method, + "status": snapshot.status, + "time_elapsed_ms": timings.time_elapsed_ms, + "sample_rate": snapshot.sample_rate, + "service_id": snapshot.service_id, + "publisher_domain": snapshot.publisher_domain, + "env": snapshot.env, + "route_class": snapshot.route_class.as_str(), + "route_template": snapshot.route_template, + "body_mode": snapshot.body_mode, + "auction_wait_placement": auction_wait_placement, + "appbuild_ms": timings.appbuild_ms, + "filter_ms": timings.filter_ms, + "geo_ms": timings.geo_ms, + "kv_ms": timings.kv_ms, + "origin_ms": timings.origin_ms, + "template_cache_ms": timings.template_cache_ms, + "auction_wait_ms": timings.auction_wait_ms, + "stream_ms": timings.stream_ms, + "request_elapsed_ms": timings.request_elapsed_ms, + "resp_bytes": timings.resp_bytes, + "template_cache_state": snapshot.template_cache_state, + "country": snapshot.country, + "ts_version": snapshot.ts_version, + "pop": snapshot.pop, + }); + row.to_string() +} + +/// Formats `epoch_ms` as a `ClickHouse` `DateTime64(3)`-compatible string +/// (`%Y-%m-%d %H:%M:%S%.3f`), matching the format the auction telemetry +/// sink already uses for `event_ts`. +/// +/// Falls back to the Unix epoch when `epoch_ms` cannot be represented as a +/// valid timestamp, which never happens for a real wall-clock reading. +fn format_event_timestamp(epoch_ms: u64) -> String { + let epoch_ms = i64::try_from(epoch_ms).unwrap_or(i64::MAX); + let timestamp = chrono::DateTime::::from_timestamp_millis(epoch_ms) + .unwrap_or(chrono::DateTime::::UNIX_EPOCH); + timestamp.format("%Y-%m-%d %H:%M:%S%.3f").to_string() +} + +#[cfg(test)] +mod tests { + use super::*; + + /// A 64-hex-plus-suffix EC identifier, matching the format the admin + /// EC lookup route (`/_ts/admin/ec/{id}`) accepts as a path parameter. + const SYNTHETIC_EC_ID: &str = + "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.test01"; + + fn unknown_snapshot(route_class: RouteClass, route_template: &str) -> AccessTelemetrySnapshot { + AccessTelemetrySnapshot { + method: "GET".to_owned(), + status: 200, + route_class, + route_template: route_template.to_owned(), + publisher_domain: "unknown".to_owned(), + env: "unknown".to_owned(), + service_id: "unknown".to_owned(), + pop: "unknown".to_owned(), + ts_version: "unknown".to_owned(), + country: "unknown".to_owned(), + template_cache_state: "unknown".to_owned(), + body_mode: "buffered", + sample_rate: 0.0, + } + } + + #[test] + fn admin_ec_route_template_never_contains_the_identifier() { + // Named-route templates come from the route table verbatim, never + // from the matched request path, so a row built for this route can + // never carry the caller's actual EC id — even when a caller + // supplies one shaped exactly like a real one. + let snapshot = unknown_snapshot(RouteClass::Ec, "/_ts/admin/ec/{id}"); + let row = access_event_row(&snapshot, &TimingSnapshot::default(), 0); + + assert!( + !row.contains(SYNTHETIC_EC_ID), + "row must never contain a literal EC identifier: {row}" + ); + assert!( + row.contains("/_ts/admin/ec/{id}"), + "row should still carry the route-table pattern: {row}" + ); + } + + #[test] + fn publisher_paths_normalize_to_coarse_templates() { + assert_eq!( + publisher_route_template("/news/some-article-slug"), + "/news/*" + ); + assert_eq!(publisher_route_template("/"), "/"); + assert_eq!( + publisher_route_template("/user@example.com/profile"), + "/other/*", + "should reject non-allowlisted characters" + ); + assert_eq!( + publisher_route_template(&format!("/{}", "a".repeat(500))), + format!("/{}", "a".repeat(32)), + "should bound segment length" + ); + assert_eq!(publisher_route_template("/search terms here"), "/other/*"); + } + + #[test] + fn publisher_route_template_rejects_empty_first_segment() { + assert_eq!( + publisher_route_template("//double-slash"), + "/other/*", + "an empty first segment should not be treated as allowlisted" + ); + } + + #[test] + fn publisher_route_template_uppercases_lowercase_before_allowlisting() { + assert_eq!( + publisher_route_template("/News/Article"), + "/news/*", + "should lowercase before validating and truncating" + ); + } + + #[test] + fn row_serializes_nulls_for_missing_phases() { + let snapshot = unknown_snapshot(RouteClass::Other, "/other/*"); + let row = access_event_row(&snapshot, &TimingSnapshot::default(), 0); + let parsed: serde_json::Value = + serde_json::from_str(&row).expect("should serialize valid JSON"); + + for field in [ + "time_elapsed_ms", + "appbuild_ms", + "filter_ms", + "geo_ms", + "kv_ms", + "origin_ms", + "template_cache_ms", + "auction_wait_ms", + "stream_ms", + "request_elapsed_ms", + "resp_bytes", + ] { + assert!( + parsed[field].is_null(), + "unrecorded phase `{field}` should serialize as null: {row}" + ); + } + + for field in [ + "service_id", + "publisher_domain", + "env", + "route_class", + "route_template", + "body_mode", + "template_cache_state", + "country", + "ts_version", + "pop", + ] { + assert!( + parsed[field].is_string(), + "dimension `{field}` must never be null: {row}" + ); + } + assert_eq!(parsed["auction_wait_placement"], "none"); + } + + #[test] + fn row_serializes_recorded_phases_as_numbers() { + let snapshot = unknown_snapshot(RouteClass::AuctionApi, "/auction"); + let timings = TimingSnapshot { + time_elapsed_ms: Some(12), + request_elapsed_ms: Some(15), + appbuild_ms: Some(1), + filter_ms: Some(2), + geo_ms: Some(3), + kv_ms: Some(4), + origin_ms: Some(5), + template_cache_ms: Some(6), + auction_wait_ms: Some(7), + stream_ms: Some(8), + auction_wait_placement: Some(AuctionWaitPlacement::InStream), + resp_bytes: Some(1024), + }; + let row = access_event_row(&snapshot, &timings, 1_700_000_000_000); + let parsed: serde_json::Value = + serde_json::from_str(&row).expect("should serialize valid JSON"); + + assert_eq!(parsed["appbuild_ms"], 1); + assert_eq!(parsed["stream_ms"], 8); + assert_eq!(parsed["resp_bytes"], 1024); + assert_eq!(parsed["auction_wait_placement"], "in_stream"); + } + + #[test] + fn route_class_renders_snake_case() { + assert_eq!(RouteClass::PublisherHtml.as_str(), "publisher_html"); + assert_eq!(RouteClass::Tsjs.as_str(), "tsjs"); + assert_eq!(RouteClass::IntegrationProxy.as_str(), "integration_proxy"); + assert_eq!(RouteClass::Ec.as_str(), "ec"); + assert_eq!(RouteClass::AuctionApi.as_str(), "auction_api"); + assert_eq!(RouteClass::Other.as_str(), "other"); + } + + #[test] + fn format_event_timestamp_matches_clickhouse_datetime64_shape() { + // 2023-11-14T22:13:20.000Z + let rendered = format_event_timestamp(1_700_000_000_000); + assert_eq!(rendered, "2023-11-14 22:13:20.000"); + } +} diff --git a/crates/trusted-server-core/src/constants.rs b/crates/trusted-server-core/src/constants.rs index e1152b1e71..1f85facb89 100644 --- a/crates/trusted-server-core/src/constants.rs +++ b/crates/trusted-server-core/src/constants.rs @@ -34,6 +34,8 @@ pub const HEADER_X_TS_ENV: HeaderName = HeaderName::from_static("x-ts-env"); // Fastly environment variables pub const ENV_FASTLY_SERVICE_VERSION: &str = "FASTLY_SERVICE_VERSION"; pub const ENV_FASTLY_IS_STAGING: &str = "FASTLY_IS_STAGING"; +pub const ENV_FASTLY_SERVICE_ID: &str = "FASTLY_SERVICE_ID"; +pub const ENV_FASTLY_POP: &str = "FASTLY_POP"; // Common standard header names used across modules pub const HEADER_USER_AGENT: HeaderName = HeaderName::from_static("user-agent"); diff --git a/crates/trusted-server-core/src/lib.rs b/crates/trusted-server-core/src/lib.rs index b8a1a5718b..de1bbad21e 100644 --- a/crates/trusted-server-core/src/lib.rs +++ b/crates/trusted-server-core/src/lib.rs @@ -31,6 +31,7 @@ ) )] +pub mod access_telemetry; pub(crate) mod asset_image_optimizer; pub mod auction; pub mod auction_config_types; diff --git a/crates/trusted-server-core/src/publisher.rs b/crates/trusted-server-core/src/publisher.rs index 1943d9b3cf..6a30611c8b 100644 --- a/crates/trusted-server-core/src/publisher.rs +++ b/crates/trusted-server-core/src/publisher.rs @@ -91,21 +91,44 @@ const DEFAULT_PUBLISHER_FIRST_BYTE_TIMEOUT: Duration = Duration::from_secs(15); const HEADER_X_TS_TEMPLATE_CACHE: &str = "x-ts-template-cache"; const HEADER_X_TS_ASSEMBLY: &str = "x-ts-assembly"; -#[derive(Clone, Copy, PartialEq, Eq)] -enum TemplateCacheResponseState { +/// Outcome of a template-cache lookup/store attempt for one response. +/// +/// Set on every response that passes through the assembly pipeline via +/// [`set_template_cache_response_state`], which writes both the +/// `x-ts-template-cache` response header and this same value as a typed +/// response extension, so the two can never drift. Access telemetry reads +/// the extension rather than the header, since operator-configured response +/// headers can override a managed header but cannot touch extensions. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum TemplateCacheResponseState { + /// The cached template was found and reused. Hit, + /// No cached template existed; the cache store is reserved for this + /// content type. MissReserved, + /// No cached template existed; one was stored after assembly. MissStored, + /// No cached template existed; storing the freshly assembled template + /// failed. MissStoreError, + /// The request bypassed the cache lookup. BypassRequest, + /// The response bypassed the cache store. BypassResponse, + /// The response's content type is not supported by the template cache. Unsupported, + /// The cached template entry was invalid and could not be reused. Invalid, + /// A backend error prevented the cache lookup or store. BackendError, } impl TemplateCacheResponseState { - const fn as_str(self) -> &'static str { + /// Renders this variant as the string written to the + /// `x-ts-template-cache` header and the `template_cache_state` access + /// telemetry column. + #[must_use] + pub const fn as_str(self) -> &'static str { match self { Self::Hit => "hit", Self::MissReserved => "miss-reserved", @@ -128,6 +151,7 @@ fn set_template_cache_response_state( HEADER_X_TS_TEMPLATE_CACHE, HeaderValue::from_static(state.as_str()), ); + response.extensions_mut().insert(state); } #[derive(Clone, Copy, PartialEq, Eq)] @@ -9057,6 +9081,54 @@ mod tests { ); } + #[tokio::test] + async fn template_cache_response_extension_matches_the_header_on_every_transition() { + // The typed extension and the `x-ts-template-cache` header are written + // together by a single setter, so they must always agree — access + // telemetry reads the extension precisely because it cannot drift from + // what an operator-configured header override might otherwise show. + let stub = Arc::new(StubHttpClient::new()); + let cache = Arc::new(MemoryTemplateCache::default()); + let settings = Arc::new(settings_with_mode("esi")); + let services = services(Arc::clone(&stub), Arc::clone(&cache)); + + queue_shareable_html(&stub); + + let cold = run(&settings, &services, navigation_request()).await; + let cold_header = cold + .headers() + .get(HEADER_X_TS_TEMPLATE_CACHE) + .and_then(|value| value.to_str().ok()) + .map(str::to_owned); + let cold_extension = cold + .extensions() + .get::() + .map(|state| state.as_str()); + assert_eq!( + cold_extension, + cold_header.as_deref(), + "the cold-fill extension must match the header" + ); + assert_eq!(cold_extension, Some("miss-stored")); + + let warm = run(&settings, &services, navigation_request()).await; + let warm_header = warm + .headers() + .get(HEADER_X_TS_TEMPLATE_CACHE) + .and_then(|value| value.to_str().ok()) + .map(str::to_owned); + let warm_extension = warm + .extensions() + .get::() + .map(|state| state.as_str()); + assert_eq!( + warm_extension, + warm_header.as_deref(), + "the warm-hit extension must match the header" + ); + assert_eq!(warm_extension, Some("hit")); + } + #[tokio::test] async fn template_cache_span_recorded_only_when_lookup_runs() { // Inline mode: no shared-cache key is ever computed, so the lookup From 48acd816d7044c5a6d77d6780e0afdf1473bd214 Mon Sep 17 00:00:00 2001 From: Jason Evans Date: Tue, 25 Aug 2026 10:00:30 -0700 Subject: [PATCH 010/194] Emit confirmed access telemetry rows after pull-sync post-send --- .../trusted-server-adapter-fastly/src/main.rs | 207 ++++++++++- .../src/tinybird.rs | 336 +++++++++++++++++- 2 files changed, 524 insertions(+), 19 deletions(-) diff --git a/crates/trusted-server-adapter-fastly/src/main.rs b/crates/trusted-server-adapter-fastly/src/main.rs index c09663d135..65e89cd1c8 100644 --- a/crates/trusted-server-adapter-fastly/src/main.rs +++ b/crates/trusted-server-adapter-fastly/src/main.rs @@ -1,5 +1,5 @@ use std::sync::Arc; -use std::time::Instant; +use std::time::{Instant, SystemTime, UNIX_EPOCH}; use edgezero_adapter_fastly::config_store::FastlyConfigStore as EdgeZeroFastlyConfigStore; use edgezero_adapter_fastly::request::into_core_request; @@ -14,7 +14,9 @@ use error_stack::Report; use fastly::http::Method as FastlyMethod; use fastly::{Request as FastlyRequest, Response as FastlyResponse}; -use trusted_server_core::access_telemetry::{AccessTelemetrySnapshot, RouteClass, RouteMetadata}; +use trusted_server_core::access_telemetry::{ + AccessTelemetrySnapshot, RouteClass, RouteMetadata, access_event_row, +}; use trusted_server_core::cache_policy::{ EdgeCacheHeader, cache_control_headers_are_private_or_no_store, }; @@ -275,7 +277,7 @@ fn edgezero_main(mut req: FastlyRequest) { if let Some(settings) = settings_snapshot.as_deref() { match apply_edgezero_ec_finalize(settings, &ec_state, &mut response, &timings) { Ok(partner_registry) => { - send_edgezero_response( + let outcome = send_edgezero_response( response, request_filter_effects.as_ref(), &SendContext { @@ -287,6 +289,7 @@ fn edgezero_main(mut req: FastlyRequest) { }, ); run_edgezero_pull_sync_after_send(settings, &partner_registry, &ec_state); + emit_access_telemetry_after_send(settings, &outcome, &timings); return; } Err(e) => { @@ -301,7 +304,7 @@ fn edgezero_main(mut req: FastlyRequest) { match apply_edgezero_ec_finalize(&settings, &ec_state, &mut response, &timings) { Ok(partner_registry) => { - send_edgezero_response( + let outcome = send_edgezero_response( response, request_filter_effects.as_ref(), &SendContext { @@ -317,6 +320,7 @@ fn edgezero_main(mut req: FastlyRequest) { &partner_registry, &ec_state, ); + emit_access_telemetry_after_send(&settings, &outcome, &timings); return; } Err(e) => { @@ -333,17 +337,31 @@ fn edgezero_main(mut req: FastlyRequest) { } } - send_edgezero_response( + let outcome = send_edgezero_response( response, request_filter_effects.as_ref(), &SendContext { - timings, + timings: timings.clone(), server_timing_enabled, method: request_method, publisher_domain, access_sample_rate, }, ); + // The asset/admin/error fallback path: no `EcFinalizeState` (or the ec + // finalize branch above failed), so there is no pull-sync dispatch here + // at all — telemetry is the only post-send step. Reload settings when + // `app_state` never built, matching the fallback used earlier in this + // function for entry-point finalize headers. + match settings_snapshot.as_deref() { + Some(settings) => emit_access_telemetry_after_send(settings, &outcome, &timings), + None => match load_settings_from_config_store() { + Ok(settings) => emit_access_telemetry_after_send(&settings, &outcome, &timings), + Err(e) => { + log::warn!("access telemetry emission skipped: failed to reload settings: {e:?}"); + } + }, + } } fn edge_error_response(error: EdgeError) -> HttpResponse { @@ -432,6 +450,59 @@ fn run_edgezero_pull_sync_after_send( } } +/// Builds and emits the access-telemetry row for one delivered response, +/// when access telemetry is enabled and this request is sampled in. +/// +/// Called last at every `send_edgezero_response` call site in +/// [`edgezero_main`] — after `run_edgezero_pull_sync_after_send` on the two +/// EC-finalized paths, and directly after send on the asset/admin/error +/// fallback path, which never builds an [`EcFinalizeState`] or route-scoped +/// `RuntimeServices` at all. The Tinybird transport context is therefore +/// constructed fresh from `settings` here rather than threaded through +/// either of those per-route types, so every response class can emit. +/// +/// Sampled-out requests return silently — that is the expected, high-volume +/// case and not worth a log line. Every other drop (row build, token load, +/// send, or non-2xx status — all folded into `emit_access_event`'s `Result`) +/// logs exactly one warning naming the reason. +fn emit_access_telemetry_after_send( + settings: &Settings, + outcome: &DeliveryOutcome, + timings: &RequestTimings, +) { + if !settings.tinybird.enabled || !settings.tinybird.access_enabled { + return; + } + + let since_epoch = SystemTime::now() + .duration_since(UNIX_EPOCH) + .unwrap_or_default(); + let epoch_ms = u64::try_from(since_epoch.as_millis()).unwrap_or(u64::MAX); + // Entropy for the sampling decision: the timestamp's nanosecond + // resolution XORed with a per-request value already on hand + // (`outcome.bytes`), so two requests handled in the same instance never + // collide on sampling decisions purely because they read the same + // millisecond. There is no `rand` crate dependency here — see + // `tinybird::sampled_in`. + let entropy_nanos = u64::try_from(since_epoch.as_nanos()).unwrap_or(u64::MAX); + let entropy = entropy_nanos ^ outcome.bytes; + + if !tinybird::sampled_in(settings.tinybird.access_sample_rate, entropy) { + return; + } + + let row = access_event_row(&outcome.snapshot, &timings.snapshot(), epoch_ms); + let target = tinybird::TinybirdEventsTarget::from_access_config(settings.tinybird.clone()); + let result = futures::executor::block_on(tinybird::emit_access_event( + &platform::FastlyPlatformHttpClient, + &target, + row, + )); + if let Err(error) = result { + log::warn!("access telemetry emission dropped: {error:?}"); + } +} + /// Per-response context threaded into [`send_edgezero_response`] so the /// function stays at or under seven parameters. struct SendContext { @@ -935,6 +1006,7 @@ mod tests { use edgezero_core::http::HeaderValue; use edgezero_core::http::response_builder; use fastly::mime; + use std::sync::Mutex; use std::time::Duration; use trusted_server_core::integrations::HeaderMutation; use trusted_server_core::request_timing::AuctionWaitPlacement; @@ -1656,4 +1728,127 @@ mod tests { DeliveryResult::Complete ); } + + /// Records `"telemetry"` into a shared order log instead of sending a + /// real request, standing in for the adapter's platform HTTP client in + /// [`post_send_order_is_elapsed_then_pull_sync_then_telemetry`]. + struct OrderingHttpClient { + log: Arc>>, + } + + #[async_trait::async_trait(?Send)] + impl trusted_server_core::platform::PlatformHttpClient for OrderingHttpClient { + async fn send( + &self, + _request: trusted_server_core::platform::PlatformHttpRequest, + ) -> Result< + trusted_server_core::platform::PlatformResponse, + Report, + > { + self.log + .lock() + .expect("should lock order log") + .push("telemetry"); + let response = response_builder() + .status(edgezero_core::http::StatusCode::ACCEPTED) + .body(EdgeBody::empty()) + .expect("should build ordering test response"); + Ok(trusted_server_core::platform::PlatformResponse::new( + response, + )) + } + + async fn send_async( + &self, + _request: trusted_server_core::platform::PlatformHttpRequest, + ) -> Result< + trusted_server_core::platform::PlatformPendingRequest, + Report, + > { + Err(Report::new( + trusted_server_core::platform::PlatformError::Unsupported, + )) + } + + async fn select( + &self, + _pending_requests: Vec, + ) -> Result< + trusted_server_core::platform::PlatformSelectResult, + Report, + > { + Err(Report::new( + trusted_server_core::platform::PlatformError::Unsupported, + )) + } + } + + #[test] + fn post_send_order_is_elapsed_then_pull_sync_then_telemetry() { + // `edgezero_main` cannot be driven directly in a unit test (it + // consumes a live `fastly::Request::from_client()`), and + // `run_edgezero_pull_sync_after_send` has no injectable seam of its + // own — it dispatches through the real identity-graph pull-sync + // path, which needs a configured EC KV store, partner registry, and + // rate limiter wired together. This test instead exercises the two + // REAL functions `edgezero_main` calls that DO have a testable seam + // — `send_edgezero_response` (which stamps `request_elapsed` before + // returning, per Task 6/7) and `tinybird::emit_access_event` (the + // telemetry send added by this task) — around an instrumented + // stand-in for the pull-sync dispatch call, in the exact order + // `edgezero_main` places them. + // + // This proves the elapsed-before-telemetry leg from real production + // code (the assertion below reads the real `timings` snapshot + // between the two calls). The pull-sync-before-telemetry leg is a + // source-order invariant in `edgezero_main`'s three call sites + // (verified by code review, not by this test) because + // `run_edgezero_pull_sync_after_send` itself has no seam to + // instrument — see task-8-report.md for this residual. + let log: Arc>> = Arc::new(Mutex::new(Vec::new())); + let timings = RequestTimings::new(); + let response = response_builder() + .body(EdgeBody::from("ok")) + .expect("should build response"); + + let outcome = send_edgezero_response( + response, + None, + &SendContext { + timings: timings.clone(), + server_timing_enabled: false, + method: "GET".to_owned(), + publisher_domain: "test-publisher.com".to_owned(), + access_sample_rate: 1.0, + }, + ); + assert!( + timings.snapshot().request_elapsed_ms.is_some(), + "request_elapsed should already be stamped before pull-sync/telemetry run" + ); + + // Stand-in for `run_edgezero_pull_sync_after_send`, which has no + // injectable seam (see the test doc comment above). + log.lock().expect("should lock order log").push("pull_sync"); + + let http_client = OrderingHttpClient { + log: Arc::clone(&log), + }; + let target = tinybird::TinybirdEventsTarget::from_access_config( + trusted_server_core::settings::TinybirdSettings { + api_host: "api.us-east.aws.tinybird.co".to_owned(), + ..trusted_server_core::settings::TinybirdSettings::default() + }, + ); + let row = access_event_row(&outcome.snapshot, &timings.snapshot(), 0); + + futures::executor::block_on(tinybird::emit_access_event(&http_client, &target, row)) + .expect("should send access telemetry"); + + assert_eq!( + *log.lock().expect("should lock order log"), + vec!["pull_sync", "telemetry"], + "pull-sync must dispatch before telemetry emits" + ); + } } diff --git a/crates/trusted-server-adapter-fastly/src/tinybird.rs b/crates/trusted-server-adapter-fastly/src/tinybird.rs index 08b5811fcf..5025f9a73c 100644 --- a/crates/trusted-server-adapter-fastly/src/tinybird.rs +++ b/crates/trusted-server-adapter-fastly/src/tinybird.rs @@ -11,10 +11,13 @@ use trusted_server_core::auction::telemetry::{ }; use trusted_server_core::error::TrustedServerError; use trusted_server_core::platform::{ - PlatformBackendSpec, PlatformHttpRequest, RuntimeServices, StoreName, + PlatformBackend as _, PlatformBackendSpec, PlatformHttpClient, PlatformHttpRequest, + PlatformSecretStore as _, RuntimeServices, StoreName, }; use trusted_server_core::settings::{Settings, TinybirdSettings}; +use crate::platform::{FastlyPlatformBackend, FastlyPlatformSecretStore}; + const TINYBIRD_EVENTS_PATH: &str = "/v0/events"; const TINYBIRD_NDJSON_CONTENT_TYPE: &str = "application/x-ndjson"; const TINYBIRD_FIRST_BYTE_TIMEOUT: Duration = Duration::from_secs(2); @@ -45,7 +48,7 @@ struct FastlyTinybirdAuctionTelemetrySink { } #[derive(Debug, Clone)] -struct TinybirdEventsTarget { +pub(crate) struct TinybirdEventsTarget { api_host: String, dataset: String, secret_store: StoreName, @@ -69,6 +72,27 @@ impl TinybirdEventsTarget { max_body_bytes: config.max_body_bytes, } } + + /// Builds the Events API target for the access-log datasource. + /// + /// Shares [`from_config`](Self::from_config)'s host/secret-store/ + /// body-size-limit derivation, but points at `access_dataset` and + /// `access_token_secret` instead of the auction pair, so access-log + /// emission never shares a datasource or token with auction telemetry + /// even though both configs come from the same [`TinybirdSettings`]. + pub(crate) fn from_access_config(config: TinybirdSettings) -> Self { + let uri = tinybird_events_uri(&config.api_host, &config.access_dataset); + let backend_spec = tinybird_backend_spec(&config.api_host); + Self { + api_host: config.api_host, + dataset: config.access_dataset, + secret_store: StoreName::from(config.secret_store), + token_secret: config.access_token_secret, + uri, + backend_spec, + max_body_bytes: config.max_body_bytes, + } + } } impl FastlyTinybirdAuctionTelemetrySink { @@ -208,6 +232,149 @@ impl AuctionTelemetrySink for FastlyTinybirdAuctionTelemetrySink { } } +// --------------------------------------------------------------------------- +// Access telemetry: confirmed-delivery emitter +// --------------------------------------------------------------------------- + +/// Bucket count [`sampled_in`] maps `entropy` into. +/// +/// Large enough that `rate` values with several significant digits (e.g. +/// `0.015`) still land in a distinct bucket instead of rounding away, while +/// staying well inside `u64` range once multiplied by `rate`. +const ACCESS_SAMPLE_BUCKETS: u64 = 1_000_000; + +/// Decides whether one request's access-telemetry row should be emitted. +/// +/// `entropy` should vary from request to request — callers derive it from +/// the wall-clock event timestamp `XORed` with a cheap per-request value (see +/// the call site in `main.rs`). There is no `rand` crate dependency here: +/// the wasm32-wasip1 guest has no equivalent to `Math.random()`. Mapping +/// `entropy % ACCESS_SAMPLE_BUCKETS` into `[0, 1)` and comparing against +/// `rate` is not cryptographically uniform (the low bits of a timestamp are +/// not perfectly evenly distributed), but access-telemetry sampling only +/// needs an approximately even sample, not a provably unbiased one. +/// +/// `rate <= 0.0` always returns `false` and `rate >= 1.0` always returns +/// `true`, independent of `entropy`, so both boundary configurations behave +/// predictably. `0.0` cannot actually occur while `access_enabled` is `true` +/// (`Settings` validation requires `access_sample_rate > 0.0` in that case), +/// but this function stays total rather than leaning on that invariant. +#[must_use] +pub(crate) fn sampled_in(rate: f64, entropy: u64) -> bool { + if rate >= 1.0 { + return true; + } + if rate <= 0.0 { + return false; + } + let threshold = (rate * ACCESS_SAMPLE_BUCKETS as f64) as u64; + entropy % ACCESS_SAMPLE_BUCKETS < threshold +} + +/// Loads and validates the access-log APPEND token from the Fastly secret store. +/// +/// Constructs [`FastlyPlatformSecretStore`] directly instead of routing +/// through [`RuntimeServices`]: access-telemetry emission runs post-delivery +/// for every response class — including asset, admin, and error responses +/// that never build a route-scoped `RuntimeServices` — so the transport +/// context here must be adapter-owned and route-independent rather than +/// threaded from wherever the route happened to construct one. +fn load_access_token(target: &TinybirdEventsTarget) -> Result> { + let token = FastlyPlatformSecretStore + .get_string(&target.secret_store, &target.token_secret) + .change_context(TrustedServerError::Proxy { + message: "Tinybird access append token unavailable".to_owned(), + })?; + let token = token.trim().to_owned(); + if token.is_empty() { + return Err(Report::new(TrustedServerError::Proxy { + message: "Tinybird access append token is empty".to_owned(), + })); + } + Ok(token) +} + +/// Builds the Events API POST request for one access-log row. +fn build_access_events_request( + target: &TinybirdEventsTarget, + body: String, + auth_header: HeaderValue, +) -> Result> { + request_builder() + .method(Method::POST) + .uri(target.uri.as_str()) + .header(header::AUTHORIZATION, auth_header) + .header(header::CONTENT_TYPE, TINYBIRD_NDJSON_CONTENT_TYPE) + .body(Body::from(body)) + .change_context(TrustedServerError::Proxy { + message: "failed to build Tinybird Events API request".to_owned(), + }) +} + +/// Sends one confirmed access-log row to the Tinybird Events API and waits +/// for the response. +/// +/// Unlike [`FastlyTinybirdAuctionTelemetrySink::emit_auction_events`] (fire- +/// and-forget, dispatched mid-request so it never adds latency to the +/// response), this runs post-delivery: the response has already reached the +/// client, so there is no latency budget left to protect, and the send can +/// afford to wait for — and validate — the reply. `client` is the adapter's +/// stateless platform HTTP client in production +/// ([`crate::platform::FastlyPlatformHttpClient`]); accepting it as `&dyn +/// PlatformHttpClient` here (rather than that concrete type) is what lets +/// tests substitute a recording double instead of performing a real network +/// send, matching how [`RuntimeServices::http_client`] is consumed +/// elsewhere. `target` is derived from settings once at the post-send call +/// site rather than threaded through any per-route state. +/// +/// A non-2xx status is reported as `Err` naming the status; there is no +/// retry — the caller logs exactly one warning and moves on. +/// +/// # Errors +/// +/// Returns `Err` when the access-log APPEND token cannot be loaded, the +/// backend cannot be registered, the request cannot be built or sent, or the +/// Tinybird Events API responds with a non-2xx status. +pub(crate) async fn emit_access_event( + client: &dyn PlatformHttpClient, + target: &TinybirdEventsTarget, + row: String, +) -> Result<(), Report> { + let token = load_access_token(target)?; + let auth_header = FastlyTinybirdAuctionTelemetrySink::authorization_header(&token)?; + let backend_name = FastlyPlatformBackend + .ensure(&target.backend_spec) + .change_context(TrustedServerError::Proxy { + message: "Tinybird backend registration failed".to_owned(), + })?; + let request = build_access_events_request(target, row, auth_header)?; + + log::info!( + "sending access telemetry to Tinybird dataset={} host={} backend={}", + target.dataset, + target.api_host, + backend_name + ); + + let response = client + .send(PlatformHttpRequest::new(request, backend_name)) + .await + .change_context(TrustedServerError::Proxy { + message: "failed to send Tinybird access telemetry request".to_owned(), + })?; + + if response.response.status().is_success() { + Ok(()) + } else { + Err(Report::new(TrustedServerError::Proxy { + message: format!( + "Tinybird access telemetry request failed with status {}", + response.response.status() + ), + })) + } +} + fn tinybird_backend_spec(api_host: &str) -> PlatformBackendSpec { PlatformBackendSpec { scheme: "https".to_owned(), @@ -327,25 +494,28 @@ mod tests { body: Vec, } + /// Records outbound requests and, for [`PlatformHttpClient::send`] (the + /// blocking variant `emit_access_event` uses), returns a synthetic + /// response carrying `respond_status` instead of performing a real + /// network send. #[derive(Default)] struct RecordingHttpClient { requests: Mutex>, select_calls: Mutex, + respond_status: Mutex, } - #[async_trait::async_trait(?Send)] - impl PlatformHttpClient for RecordingHttpClient { - async fn send( - &self, - _request: PlatformHttpRequest, - ) -> Result> { - Err(Report::new(PlatformError::Unsupported)) + impl RecordingHttpClient { + /// Status [`PlatformHttpClient::send`] should reply with. Irrelevant + /// to auction-sink tests, which only exercise `send_async`. + fn respond_with(status: u16) -> Self { + Self { + respond_status: Mutex::new(status), + ..Self::default() + } } - async fn send_async( - &self, - request: PlatformHttpRequest, - ) -> Result> { + fn record(&self, request: PlatformHttpRequest) { let backend_name = request.backend_name; let (parts, body) = request.request.into_parts(); let headers = parts @@ -369,6 +539,35 @@ mod tests { .lock() .expect("should lock recorded requests") .push(recorded); + } + } + + #[async_trait::async_trait(?Send)] + impl PlatformHttpClient for RecordingHttpClient { + async fn send( + &self, + request: PlatformHttpRequest, + ) -> Result> { + self.record(request); + let status = *self + .respond_status + .lock() + .expect("should lock configured response status"); + let response = edgezero_core::http::response_builder() + .status( + edgezero_core::http::StatusCode::from_u16(status) + .expect("should build a valid test status code"), + ) + .body(edgezero_core::body::Body::empty()) + .expect("should build test response"); + Ok(PlatformResponse::new(response)) + } + + async fn send_async( + &self, + request: PlatformHttpRequest, + ) -> Result> { + self.record(request); Ok(PlatformPendingRequest::new(()).with_backend_name("tinybird-backend")) } @@ -698,6 +897,117 @@ mod tests { ); } + #[test] + fn access_emitter_posts_ndjson_and_validates_2xx() { + // `ts_secrets`/`tinybird_access_append_token` is seeded in + // fastly.toml's `[local_server.secret_stores]` fixture (value + // "test-tinybird-access-append-token"), so `emit_access_event` can + // load a real token through Viceroy without a secret-store test + // double — the same fixture backs the auction-token secret used + // above. + let target = TinybirdEventsTarget::from_access_config(enabled_config()); + let http_client = RecordingHttpClient::respond_with(202); + let row = r#"{"status":200}"#.to_owned(); + + futures::executor::block_on(emit_access_event(&http_client, &target, row.clone())) + .expect("should accept a 202 response"); + + let requests = http_client + .requests + .lock() + .expect("should lock recorded requests"); + assert_eq!(requests.len(), 1, "should send exactly one request"); + assert_eq!( + requests[0].uri, + "https://api.us-east.aws.tinybird.co/v0/events?name=access_logs_raw" + ); + assert_eq!(requests[0].method, Method::POST.to_string()); + assert_eq!( + header_value(&requests[0].headers, header::AUTHORIZATION.as_str()), + Some("Bearer test-tinybird-access-append-token") + ); + assert_eq!( + std::str::from_utf8(&requests[0].body).expect("should record utf8 body"), + row, + "should send the row verbatim as the request body" + ); + } + + #[test] + fn access_emitter_warns_and_drops_on_non_2xx() { + let target = TinybirdEventsTarget::from_access_config(enabled_config()); + let http_client = RecordingHttpClient::respond_with(422); + + let result = futures::executor::block_on(emit_access_event( + &http_client, + &target, + r#"{"status":422}"#.to_owned(), + )); + + let error = result.expect_err("a 422 response should be reported as an error"); + assert!( + error.to_string().contains("422"), + "error should name the failing status: {error}" + ); + assert_eq!( + http_client + .requests + .lock() + .expect("should lock recorded requests") + .len(), + 1, + "should not retry after a non-2xx response" + ); + } + + #[test] + fn sampled_out_requests_emit_nothing() { + // Mirrors main.rs's post-send gate exactly (`if sampled_in(rate, + // entropy) { emit_access_event(...) }`): `emit_access_event` is only + // reached when `sampled_in` returns `true`. With a `0.0` rate it + // never does, for any entropy, so the http client should never see + // a request. + let http_client = RecordingHttpClient::respond_with(202); + let target = TinybirdEventsTarget::from_access_config(enabled_config()); + let rate = 0.0; + let entropy = 123_456_789_u64; + + if sampled_in(rate, entropy) { + futures::executor::block_on(emit_access_event(&http_client, &target, "{}".to_owned())) + .expect("should send when sampled in"); + } + + assert_eq!( + http_client + .requests + .lock() + .expect("should lock recorded requests") + .len(), + 0, + "sampled-out requests must never reach emit_access_event" + ); + } + + #[test] + fn sampled_in_boundary_rates_are_unconditional() { + assert!( + sampled_in(1.0, 0), + "a 1.0 sample rate should always sample in" + ); + assert!( + sampled_in(1.0, u64::MAX), + "a 1.0 sample rate should always sample in regardless of entropy" + ); + assert!( + !sampled_in(0.0, 0), + "a 0.0 sample rate should never sample in" + ); + assert!( + !sampled_in(0.0, u64::MAX), + "a 0.0 sample rate should never sample in regardless of entropy" + ); + } + fn header_value<'a>(headers: &'a [(String, String)], name: &str) -> Option<&'a str> { headers .iter() From 72d57557827453d18409494e4301cc7fd6fe3734 Mon Sep 17 00:00:00 2001 From: Jason Evans Date: Tue, 25 Aug 2026 10:10:07 -0700 Subject: [PATCH 011/194] Extend access_logs_raw with phase columns and a non-null sorting key --- .../datasources/access_logs_raw.datasource | 28 +++++++++++++++---- tinybird/fixtures/access_logs_raw.ndjson | 1 + 2 files changed, 24 insertions(+), 5 deletions(-) create mode 100644 tinybird/fixtures/access_logs_raw.ndjson diff --git a/tinybird/datasources/access_logs_raw.datasource b/tinybird/datasources/access_logs_raw.datasource index 42f214e078..a484964dbb 100644 --- a/tinybird/datasources/access_logs_raw.datasource +++ b/tinybird/datasources/access_logs_raw.datasource @@ -1,19 +1,37 @@ DESCRIPTION > - Optional sampled Trusted Server access telemetry rows. Disabled by default in Fastly config. + Per-request phase-timing telemetry rows, sampled and emitted post-send by the edge service. SCHEMA > `event_ts` DateTime64(3), `method` LowCardinality(String), - `path` String, `status` UInt16, `time_elapsed_ms` UInt32, - `cache_state` LowCardinality(Nullable(String)), - `country` LowCardinality(String), `sample_rate` Float64, + `service_id` LowCardinality(String), + `publisher_domain` LowCardinality(String), + `env` LowCardinality(String), + `route_class` LowCardinality(String), + `route_template` String, + `body_mode` LowCardinality(String), + `auction_wait_placement` LowCardinality(String), + `appbuild_ms` Nullable(UInt32), + `filter_ms` Nullable(UInt32), + `geo_ms` Nullable(UInt32), + `kv_ms` Nullable(UInt32), + `origin_ms` Nullable(UInt32), + `template_cache_ms` Nullable(UInt32), + `auction_wait_ms` Nullable(UInt32), + `stream_ms` Nullable(UInt32), + `request_elapsed_ms` Nullable(UInt32), + `resp_bytes` Nullable(UInt64), + `template_cache_state` LowCardinality(String), + `country` LowCardinality(String), + `ts_version` LowCardinality(String), + `pop` LowCardinality(String), `event_date` Date DEFAULT toDate(event_ts) ENGINE "MergeTree" -ENGINE_SORTING_KEY "event_date, path, status, method" +ENGINE_SORTING_KEY "event_date, service_id, publisher_domain, env, route_class, pop, status" TTL "event_date + INTERVAL 30 DAY" TOKEN ts_access_ingest APPEND diff --git a/tinybird/fixtures/access_logs_raw.ndjson b/tinybird/fixtures/access_logs_raw.ndjson new file mode 100644 index 0000000000..3c82c5ca6e --- /dev/null +++ b/tinybird/fixtures/access_logs_raw.ndjson @@ -0,0 +1 @@ +{"event_ts":"2026-06-23 12:00:00.000","method":"GET","status":200,"time_elapsed_ms":145,"sample_rate":0.1,"service_id":"abc123","publisher_domain":"test-publisher.com","env":"production","route_class":"publisher_html","route_template":"/news/*","body_mode":"streamed","auction_wait_placement":"in_stream","appbuild_ms":12,"filter_ms":5,"geo_ms":3,"kv_ms":8,"origin_ms":25,"template_cache_ms":10,"auction_wait_ms":45,"stream_ms":18,"request_elapsed_ms":145,"resp_bytes":8192,"template_cache_state":"hit","country":"US","ts_version":"v1.2.3","pop":"SFO"} From c50be0323d258cae2b449dd4cb7ea98e232e115d Mon Sep 17 00:00:00 2001 From: Jason Evans Date: Tue, 25 Aug 2026 10:17:02 -0700 Subject: [PATCH 012/194] Widen time_elapsed_ms to nullable so dropped snapshots cannot quarantine rows --- .../specs/2026-08-24-request-phase-timing-design.md | 3 ++- tinybird/datasources/access_logs_raw.datasource | 2 +- 2 files changed, 3 insertions(+), 2 deletions(-) diff --git a/docs/superpowers/specs/2026-08-24-request-phase-timing-design.md b/docs/superpowers/specs/2026-08-24-request-phase-timing-design.md index 06f27c1c27..d9bbd4bdd7 100644 --- a/docs/superpowers/specs/2026-08-24-request-phase-timing-design.md +++ b/docs/superpowers/specs/2026-08-24-request-phase-timing-design.md @@ -275,7 +275,8 @@ Cloudflare and Spin: collection compiles, no emission wiring in v1 (unchanged). Extends the reserved `tinybird/datasources/access_logs_raw.datasource`. Kept columns: `event_ts`, `method`, `status`, `time_elapsed_ms` (defined as the -`mark_headers_ready()` snapshot), `sample_rate`, `event_date`, 30-day TTL. +`mark_headers_ready()` snapshot; nullable because a contended lock drop can lose the +snapshot), `sample_rate`, `event_date`, 30-day TTL. Removed: raw `path`. Route identifiers like `/_ts/admin/ec/{id}` would otherwise put EC identifiers into a 30-day dataset, and publisher paths carry unbounded cardinality diff --git a/tinybird/datasources/access_logs_raw.datasource b/tinybird/datasources/access_logs_raw.datasource index a484964dbb..918f3d5fd4 100644 --- a/tinybird/datasources/access_logs_raw.datasource +++ b/tinybird/datasources/access_logs_raw.datasource @@ -5,7 +5,7 @@ SCHEMA > `event_ts` DateTime64(3), `method` LowCardinality(String), `status` UInt16, - `time_elapsed_ms` UInt32, + `time_elapsed_ms` Nullable(UInt32), `sample_rate` Float64, `service_id` LowCardinality(String), `publisher_domain` LowCardinality(String), From 0aead79727d81cde132f03be94a022db9ef18cb8 Mon Sep 17 00:00:00 2001 From: Jason Evans Date: Tue, 25 Aug 2026 12:24:22 -0700 Subject: [PATCH 013/194] Emit Server-Timing from the Axum terminal layer with adapter-specific semantics --- crates/trusted-server-adapter-axum/Cargo.toml | 6 +- crates/trusted-server-adapter-axum/src/app.rs | 32 ++- crates/trusted-server-adapter-axum/src/lib.rs | 3 + .../trusted-server-adapter-axum/src/main.rs | 71 ++++- .../trusted-server-adapter-axum/src/timing.rs | 242 ++++++++++++++++++ .../trusted-server-adapter-fastly/src/main.rs | 51 +--- .../trusted-server-core/src/request_timing.rs | 91 +++++++ 7 files changed, 438 insertions(+), 58 deletions(-) create mode 100644 crates/trusted-server-adapter-axum/src/timing.rs diff --git a/crates/trusted-server-adapter-axum/Cargo.toml b/crates/trusted-server-adapter-axum/Cargo.toml index 15b6ee59d6..09e8c77d26 100644 --- a/crates/trusted-server-adapter-axum/Cargo.toml +++ b/crates/trusted-server-adapter-axum/Cargo.toml @@ -20,6 +20,7 @@ path = "src/main.rs" [dependencies] async-trait = { workspace = true } +axum = { workspace = true } edgezero-adapter-axum = { workspace = true, features = ["axum"] } edgezero-core = { workspace = true } error-stack = { workspace = true } @@ -27,12 +28,11 @@ futures = { workspace = true } log = { workspace = true } reqwest = { workspace = true } simple_logger = { workspace = true } -tokio = { workspace = true, features = ["rt-multi-thread", "macros", "sync", "time"] } +tokio = { workspace = true, features = ["rt-multi-thread", "macros", "net", "signal", "sync", "time"] } +tower = { workspace = true, features = ["util"] } trusted-server-core = { workspace = true } [dev-dependencies] -axum = { workspace = true } base64 = { workspace = true } temp-env = { workspace = true } tokio = { workspace = true, features = ["rt-multi-thread", "macros"] } -tower = { workspace = true, features = ["util"] } diff --git a/crates/trusted-server-adapter-axum/src/app.rs b/crates/trusted-server-adapter-axum/src/app.rs index 4b71d07ce0..4bdf27d011 100644 --- a/crates/trusted-server-adapter-axum/src/app.rs +++ b/crates/trusted-server-adapter-axum/src/app.rs @@ -565,15 +565,7 @@ impl Hooks for TrustedServerApp { } fn routes() -> RouterService { - let state = match build_state() { - Ok(s) => s, - Err(ref e) => { - log::error!("failed to build application state: {:?}", e); - return startup_error_router(e); - } - }; - - build_router(&state) + Self::routes_with_server_timing_flag().0 } } @@ -594,6 +586,28 @@ impl TrustedServerApp { let state = build_state_with_settings(settings)?; Ok(build_router(&state)) } + + /// Build the router alongside whether `Server-Timing` emission is + /// enabled, read from the same settings snapshot used to build the + /// router. + /// + /// The Axum dev server's terminal timing layer ([`crate::timing`]) needs + /// this flag once at startup: unlike the Fastly adapter, which rebuilds + /// `Settings` per request, the Axum dev server builds its application + /// state once and reuses the same [`RouterService`] for every request. + #[must_use] + pub fn routes_with_server_timing_flag() -> (RouterService, bool) { + let state = match build_state() { + Ok(s) => s, + Err(ref e) => { + log::error!("failed to build application state: {:?}", e); + return (startup_error_router(e), false); + } + }; + + let server_timing_enabled = state.settings.observability.server_timing_enabled; + (build_router(&state), server_timing_enabled) + } } fn build_router(state: &Arc) -> RouterService { diff --git a/crates/trusted-server-adapter-axum/src/lib.rs b/crates/trusted-server-adapter-axum/src/lib.rs index 2f15e566d2..b1d4c3dd8f 100644 --- a/crates/trusted-server-adapter-axum/src/lib.rs +++ b/crates/trusted-server-adapter-axum/src/lib.rs @@ -10,3 +10,6 @@ pub mod app; pub mod middleware; /// Platform-trait implementations backed by env vars and `reqwest`. pub mod platform; +/// Terminal timing layer wrapping the Axum dev server's tower `Service` +/// boundary with the request-phase `Server-Timing` freeze point. +pub mod timing; diff --git a/crates/trusted-server-adapter-axum/src/main.rs b/crates/trusted-server-adapter-axum/src/main.rs index 960982176d..b8bc28ae21 100644 --- a/crates/trusted-server-adapter-axum/src/main.rs +++ b/crates/trusted-server-adapter-axum/src/main.rs @@ -1,6 +1,16 @@ -use edgezero_adapter_axum::dev_server::{AxumDevServer, AxumDevServerConfig}; -use edgezero_core::app::Hooks as _; +use std::net::SocketAddr; + +use axum::Router; +use edgezero_adapter_axum::dev_server::AxumDevServerConfig; +use edgezero_adapter_axum::service::EdgeZeroAxumService; +use edgezero_core::router::RouterService; +use tokio::net::TcpListener; +use tokio::runtime::Builder as RuntimeBuilder; +use tokio::signal; +use tower::Service as _; +use tower::service_fn; use trusted_server_adapter_axum::app::TrustedServerApp; +use trusted_server_adapter_axum::timing::TimingService; #[allow(clippy::print_stderr)] fn main() { @@ -20,13 +30,66 @@ fn main() { }; log::info!("Listening on http://{}", config.addr); - let router = TrustedServerApp::routes(); - if let Err(err) = AxumDevServer::with_config(router, config).run() { + let (router, server_timing_enabled) = TrustedServerApp::routes_with_server_timing_flag(); + if let Err(err) = run(router, server_timing_enabled, config) { log::error!("trusted-server-adapter-axum failed: {err}"); std::process::exit(1); } } +/// Runs the Axum dev server with the request-phase timing terminal layer +/// ([`trusted_server_adapter_axum::timing::TimingService`]) wrapped around +/// `EdgeZeroAxumService`, ahead of `axum::serve`. +/// +/// This does not use `edgezero_adapter_axum::dev_server::AxumDevServer::run`: +/// that helper only accepts a bare [`RouterService`] and builds its own +/// `EdgeZeroAxumService` and `axum::Router` internally, with no seam for an +/// outer service wrapper. Router-generated 404/405 responses bypass +/// `RouterBuilder::middleware` (see `trusted_server_adapter_axum::timing`), +/// so the freeze point has to wrap the tower `Service` boundary itself. +/// Driving `axum::serve` directly here mirrors that helper's own internal +/// bind/wrap/serve/shutdown sequence closely enough to keep behavior +/// identical for callers (`PORT` env var, ctrl-c graceful shutdown). +/// +/// # Errors +/// +/// Returns an error if the Tokio runtime fails to start, the listener fails +/// to bind, or the underlying serve loop errors. +fn run( + router: RouterService, + server_timing_enabled: bool, + config: AxumDevServerConfig, +) -> std::io::Result<()> { + let runtime = RuntimeBuilder::new_multi_thread().enable_all().build()?; + runtime.block_on(serve(router, server_timing_enabled, config)) +} + +async fn serve( + router: RouterService, + server_timing_enabled: bool, + config: AxumDevServerConfig, +) -> std::io::Result<()> { + let listener = TcpListener::bind(config.addr).await?; + + let service = TimingService::new(EdgeZeroAxumService::new(router), server_timing_enabled); + let axum_router = Router::new().fallback_service(service_fn(move |req| { + let mut svc = service.clone(); + async move { svc.call(req).await } + })); + let make_service = axum_router.into_make_service_with_connect_info::(); + + let server = axum::serve(listener, make_service); + if config.enable_ctrl_c { + server + .with_graceful_shutdown(async { + let _ctrl_c = signal::ctrl_c().await; + }) + .await + } else { + server.await + } +} + /// Read a port number from the `PORT` environment variable. /// /// Returns `None` when the variable is unset. Exits non-zero if the value diff --git a/crates/trusted-server-adapter-axum/src/timing.rs b/crates/trusted-server-adapter-axum/src/timing.rs new file mode 100644 index 0000000000..832823c153 --- /dev/null +++ b/crates/trusted-server-adapter-axum/src/timing.rs @@ -0,0 +1,242 @@ +//! Terminal timing layer for the Axum dev server. +//! +//! [`TimingService`](crate::timing::TimingService) wraps the tower `Service` +//! boundary the Axum dev server's router sits behind: it creates a +//! [`RequestTimings`](trusted_server_core::request_timing::RequestTimings) +//! collector per request, threads it through request extensions so +//! downstream core handlers can record into it, and on the way back stamps +//! `mark_headers_ready` and appends the `Server-Timing` header via +//! [`append_server_timing_if_private`](trusted_server_core::request_timing::append_server_timing_if_private). +//! +//! This wraps *outside* `RouterService` rather than registering as +//! `RouterBuilder::middleware`. A router-generated 404/405 short-circuits +//! `RouterInner::dispatch` before its middleware chain ever runs, so +//! middleware never sees those responses. By the time a response reaches +//! this layer -- after `RouterService::oneshot` inside +//! `EdgeZeroAxumService::call` has already converted any dispatch error into +//! a plain response -- every response is covered uniformly, router-generated +//! or not. +//! +//! `/health` is excluded by path match before a +//! [`RequestTimings`](trusted_server_core::request_timing::RequestTimings) +//! collector is even created: health checks never carry timing data on any +//! adapter. +//! +//! Unlike the Fastly adapter (state built per request, adding +//! `Phase::AppBuild` to the rendered header), the Axum dev server builds its +//! application state once at startup. There is no per-request app-build +//! interval to measure, so `ts-appbuild` never appears in the header here. + +use std::convert::Infallible; +use std::future::Future; +use std::pin::Pin; +use std::task::{Context, Poll}; + +use axum::body::Body as AxumBody; +use axum::http::{Request, Response}; +use tower::Service; +use trusted_server_core::request_timing::{RequestTimings, append_server_timing_if_private}; + +/// Path excluded from timing collection and `Server-Timing` emission: health +/// checks never carry timing data on any adapter. +const HEALTH_PATH: &str = "/health"; + +/// Wraps an inner Axum tower service with the request-phase timing freeze +/// point described in the module docs. +#[derive(Clone)] +pub struct TimingService { + inner: S, + server_timing_enabled: bool, +} + +impl TimingService { + /// Wraps `inner`, appending `Server-Timing` when `server_timing_enabled` + /// is set and the response is conclusively private. + #[must_use] + pub fn new(inner: S, server_timing_enabled: bool) -> Self { + Self { + inner, + server_timing_enabled, + } + } +} + +impl Service> for TimingService +where + S: Service, Response = Response, Error = Infallible> + + Clone + + Send + + 'static, + S::Future: Send + 'static, +{ + type Error = Infallible; + type Future = Pin> + Send>>; + type Response = Response; + + fn call(&mut self, mut req: Request) -> Self::Future { + let mut inner = self.inner.clone(); + + // Excluded before a collector is even created: `/health` never + // carries timing data, on any adapter. + if req.uri().path() == HEALTH_PATH { + return Box::pin(async move { inner.call(req).await }); + } + + let server_timing_enabled = self.server_timing_enabled; + let timings = RequestTimings::new(); + req.extensions_mut().insert(timings.clone()); + + Box::pin(async move { + let mut response = inner.call(req).await?; + append_server_timing_if_private(&mut response, &timings, server_timing_enabled); + Ok(response) + }) + } + + fn poll_ready(&mut self, cx: &mut Context<'_>) -> Poll> { + self.inner.poll_ready(cx) + } +} + +#[cfg(test)] +mod tests { + use super::*; + use axum::http::header::CACHE_CONTROL; + use axum::http::{HeaderValue, StatusCode}; + use edgezero_adapter_axum::service::EdgeZeroAxumService; + use edgezero_core::body::Body as EdgeBody; + use edgezero_core::context::RequestContext; + use edgezero_core::error::EdgeError; + use edgezero_core::http::response_builder; + use edgezero_core::router::RouterService; + use tower::{ServiceExt as _, service_fn}; + + /// Builds a private (`cache-control: private, no-store`) response for a + /// handler under test. + fn private_ok_response() -> Result { + Ok(response_builder() + .status(StatusCode::OK) + .header("cache-control", "private, no-store") + .body(EdgeBody::from("ok")) + .expect("should build a private response fixture")) + } + + /// Reads a response header as a UTF-8 string, or `None` if absent. + fn header(response: &Response, name: &str) -> Option { + response + .headers() + .get(name) + .and_then(|value| value.to_str().ok()) + .map(ToOwned::to_owned) + } + + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] + async fn axum_emits_header_on_private_response() { + let router = RouterService::builder() + .get("/private", |_ctx: RequestContext| async { + private_ok_response() + }) + .build(); + let mut service = TimingService::new(EdgeZeroAxumService::new(router), true); + + let request = Request::builder() + .uri("/private") + .body(AxumBody::empty()) + .expect("should build request"); + let response = service + .ready() + .await + .expect("should be ready") + .call(request) + .await + .expect("should not fail"); + + let server_timing = header(&response, "server-timing").expect("should emit header"); + assert!( + server_timing.contains("ts-total;dur="), + "should carry the collected total: {server_timing}" + ); + assert!( + !server_timing.contains("ts-appbuild"), + "the Axum dev server builds state once at startup, so there is no \ + per-request app-build interval to render: {server_timing}" + ); + } + + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] + async fn axum_404_carries_header_when_private() { + // An empty router has no routes at all, so any path dispatches + // through `RouterInner::dispatch`'s `NotFound` branch -- exactly the + // path that bypasses `RouterBuilder::middleware`. The router's own + // `EdgeError::into_response` does not attach `Cache-Control`, so a + // small wrapping service forces the response private here, standing + // in for whatever upstream layer would normally mark a genuinely + // private 404. This proves the freeze point still runs for a + // router-generated response without weakening + // `append_server_timing_if_private`'s real gating logic. + let empty_router = RouterService::builder().build(); + let inner = EdgeZeroAxumService::new(empty_router); + let force_private = service_fn(move |req: Request| { + let mut svc = inner.clone(); + async move { + let mut response = svc.call(req).await?; + response + .headers_mut() + .insert(CACHE_CONTROL, HeaderValue::from_static("private, no-store")); + Ok::<_, Infallible>(response) + } + }); + let mut service = TimingService::new(force_private, true); + + let request = Request::builder() + .uri("/does-not-exist") + .body(AxumBody::empty()) + .expect("should build request"); + let response = service + .ready() + .await + .expect("should be ready") + .call(request) + .await + .expect("should not fail"); + + assert_eq!( + response.status(), + StatusCode::NOT_FOUND, + "should still be the router's own not-found response" + ); + let server_timing = header(&response, "server-timing") + .expect("a router-generated 404 must still carry the header when private"); + assert!( + server_timing.contains("ts-total;dur="), + "should carry the collected total: {server_timing}" + ); + } + + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] + async fn axum_health_is_excluded() { + let router = RouterService::builder() + .get("/health", |_ctx: RequestContext| async { + private_ok_response() + }) + .build(); + let mut service = TimingService::new(EdgeZeroAxumService::new(router), true); + + let request = Request::builder() + .uri("/health") + .body(AxumBody::empty()) + .expect("should build request"); + let response = service + .ready() + .await + .expect("should be ready") + .call(request) + .await + .expect("should not fail"); + + assert!( + header(&response, "server-timing").is_none(), + "/health must never carry a server-timing header" + ); + } +} diff --git a/crates/trusted-server-adapter-fastly/src/main.rs b/crates/trusted-server-adapter-fastly/src/main.rs index 65e89cd1c8..07c15b8b80 100644 --- a/crates/trusted-server-adapter-fastly/src/main.rs +++ b/crates/trusted-server-adapter-fastly/src/main.rs @@ -6,9 +6,7 @@ use edgezero_adapter_fastly::request::into_core_request; use edgezero_core::body::Body as EdgeBody; use edgezero_core::config_store::ConfigStoreHandle; use edgezero_core::error::EdgeError; -use edgezero_core::http::{ - HeaderName, HeaderValue, Request as HttpRequest, Response as HttpResponse, -}; +use edgezero_core::http::{Request as HttpRequest, Response as HttpResponse}; use edgezero_core::response::IntoResponse; use error_stack::Report; use fastly::http::Method as FastlyMethod; @@ -17,9 +15,7 @@ use fastly::{Request as FastlyRequest, Response as FastlyResponse}; use trusted_server_core::access_telemetry::{ AccessTelemetrySnapshot, RouteClass, RouteMetadata, access_event_row, }; -use trusted_server_core::cache_policy::{ - EdgeCacheHeader, cache_control_headers_are_private_or_no_store, -}; +use trusted_server_core::cache_policy::EdgeCacheHeader; use trusted_server_core::constants::{ ENV_FASTLY_IS_STAGING, ENV_FASTLY_POP, ENV_FASTLY_SERVICE_ID, ENV_FASTLY_SERVICE_VERSION, }; @@ -37,7 +33,7 @@ use trusted_server_core::platform::PlatformGeo as _; use trusted_server_core::platform::{RuntimeServices, TimedKvStore}; use trusted_server_core::proxy::{AssetProxyCachePolicy, stream_asset_body}; use trusted_server_core::publisher::TemplateCacheResponseState; -use trusted_server_core::request_timing::{Phase, RequestTimings}; +use trusted_server_core::request_timing::{Phase, RequestTimings, append_server_timing_if_private}; use trusted_server_core::response_privacy::TerminalPrivateResponse; use trusted_server_core::settings::Settings; @@ -62,12 +58,6 @@ use crate::rate_limiter::{FastlyRateLimiter, RATE_COUNTER_NAME}; const TRUSTED_SERVER_CONFIG_STORE: &str = "trusted_server_config"; -/// `Server-Timing` header name. Not present in the `http` crate's `header` -/// module (unlike `CACHE_CONTROL` etc.), so declared locally following the -/// same `HeaderName::from_static` pattern used in -/// `trusted_server_core::constants`. -const HEADER_SERVER_TIMING: HeaderName = HeaderName::from_static("server-timing"); - /// Opens the Fastly Config Store used by the `EdgeZero` dispatcher. /// /// # Errors @@ -544,40 +534,17 @@ pub(crate) enum DeliveryResult { Error, } -/// Stamps [`RequestTimings::mark_headers_ready`] and, when observability is -/// enabled and the response is conclusively private, appends the rendered -/// `Server-Timing` header. -/// -/// Always stamps `mark_headers_ready` regardless of whether the header is -/// rendered, so the collector's `ts-total` reflects the moment headers -/// commit. Appends rather than overwrites so a pre-existing `Server-Timing` -/// value set upstream survives alongside the TS-owned set. A response is -/// never promoted to shared-cacheable just because the header would -/// otherwise be omitted: this only gates emission, it does not touch -/// `Cache-Control`. +/// Thin Fastly-adapter wrapper around +/// [`append_server_timing_if_private`], the freeze point shared with the +/// Axum adapter's terminal timing layer. See that function's doc for the +/// emission rules (always stamps `mark_headers_ready`; appends rather than +/// overwrites; never promotes a response to shared-cacheable). pub(crate) fn apply_server_timing_header( response: &mut HttpResponse, timings: &RequestTimings, server_timing_enabled: bool, ) { - timings.mark_headers_ready(); - - let conclusively_private = cache_control_headers_are_private_or_no_store(response.headers()); - if !server_timing_enabled || !conclusively_private { - return; - } - - let Some(value) = timings.server_timing_value() else { - return; - }; - match HeaderValue::from_str(&value) { - Ok(header_value) => { - response - .headers_mut() - .append(HEADER_SERVER_TIMING, header_value); - } - Err(error) => log::warn!("skipping server-timing header: {error}"), - } + append_server_timing_if_private(response, timings, server_timing_enabled); } /// A [`Write`](std::io::Write) wrapper that tallies bytes successfully written diff --git a/crates/trusted-server-core/src/request_timing.rs b/crates/trusted-server-core/src/request_timing.rs index b0cb7b0ddd..ccd3935551 100644 --- a/crates/trusted-server-core/src/request_timing.rs +++ b/crates/trusted-server-core/src/request_timing.rs @@ -7,6 +7,16 @@ use std::sync::{Arc, Mutex}; use std::time::{Duration, Instant}; +use http::{HeaderName, HeaderValue, Response}; + +use crate::cache_policy::cache_control_headers_are_private_or_no_store; + +/// `Server-Timing` header name. Not present in the `http` crate's `header` +/// module (unlike `CACHE_CONTROL` etc.), so declared locally following the +/// same `HeaderName::from_static` pattern used in +/// `trusted_server_core::constants`. +const HEADER_SERVER_TIMING: HeaderName = HeaderName::from_static("server-timing"); + /// Number of [`Phase`] variants; sizes the fixed-slot duration array in /// [`Inner`]. const PHASE_COUNT: usize = 8; @@ -253,6 +263,46 @@ impl Default for RequestTimings { } } +/// Stamps [`RequestTimings::mark_headers_ready`] and, when `enabled` and the +/// response is conclusively private, appends the rendered `Server-Timing` +/// header. +/// +/// Always stamps `mark_headers_ready` regardless of whether the header is +/// rendered, so the collector's `ts-total` reflects the moment headers +/// commit. Appends rather than overwrites so a pre-existing `Server-Timing` +/// value set upstream survives alongside the TS-owned set. A response is +/// never promoted to shared-cacheable just because the header would +/// otherwise be omitted: this only gates emission, it does not touch +/// `Cache-Control`. +/// +/// Generic over the response body type so every adapter's terminal layer can +/// call the same emission logic regardless of which body type its HTTP stack +/// uses. +pub fn append_server_timing_if_private( + response: &mut Response, + timings: &RequestTimings, + enabled: bool, +) { + timings.mark_headers_ready(); + + let conclusively_private = cache_control_headers_are_private_or_no_store(response.headers()); + if !enabled || !conclusively_private { + return; + } + + let Some(value) = timings.server_timing_value() else { + return; + }; + match HeaderValue::from_str(&value) { + Ok(header_value) => { + response + .headers_mut() + .append(HEADER_SERVER_TIMING, header_value); + } + Err(error) => log::warn!("skipping server-timing header: {error}"), + } +} + /// The header-bearing phases (see [`Phase::header_name`]), in the enum /// declaration order [`RequestTimings::server_timing_value`] renders them in. const HEADER_PHASES: [Phase; 6] = [ @@ -434,4 +484,45 @@ mod tests { "should mask vendors" ); } + + #[test] + fn append_server_timing_emits_on_private_response_when_enabled() { + let mut response = Response::builder() + .header("cache-control", "private, no-store") + .body(()) + .expect("should build a private response fixture"); + let timings = RequestTimings::new(); + + append_server_timing_if_private(&mut response, &timings, true); + + let header = response + .headers() + .get("server-timing") + .and_then(|value| value.to_str().ok()) + .expect("should emit a Server-Timing header"); + assert!( + header.starts_with("ts-total;dur="), + "should lead with the stored total: {header}" + ); + } + + #[test] + fn append_server_timing_marks_headers_ready_even_when_not_emitted() { + let mut response = Response::builder() + .header("cache-control", "max-age=60") + .body(()) + .expect("should build a shared-cacheable response fixture"); + let timings = RequestTimings::new(); + + append_server_timing_if_private(&mut response, &timings, true); + + assert!( + response.headers().get("server-timing").is_none(), + "should not emit on a shared-cacheable response" + ); + assert!( + timings.server_timing_value().is_some(), + "should still stamp mark_headers_ready so ts-total reflects the freeze point" + ); + } } From e9cf5b97a3677e734184b2604abc7df325e37ec5 Mon Sep 17 00:00:00 2001 From: Jason Evans Date: Tue, 25 Aug 2026 15:50:12 -0700 Subject: [PATCH 014/194] Document the observability and access telemetry configuration surface --- docs/guide/configuration.md | 126 ++++++++++++++++++++++++++++++++++++ 1 file changed, 126 insertions(+) diff --git a/docs/guide/configuration.md b/docs/guide/configuration.md index a1f172429f..675aaf40ac 100644 --- a/docs/guide/configuration.md +++ b/docs/guide/configuration.md @@ -77,6 +77,8 @@ fail and the service will return its startup-error response. | `[request_signing]` | Ed25519 request signing | | `[auction]` | Auction orchestration | | `[integrations.*]` | Partner integrations (Prebid, Next.js, etc.) | +| `[observability]` | Server-Timing header emission | +| `[tinybird]` | Auction and access telemetry transport | ## Example: Production Setup @@ -1805,6 +1807,130 @@ Rollback to the legacy entry point is no longer controlled by runtime config keys. Use the normal deployment rollback path to restore a pre-cleanup service version if that is required. +## Observability and Access Telemetry Configuration + +Settings for the `Server-Timing` response header and the sampled +access-telemetry sink. Both are off by default and are independent switches: +enabling one does not enable the other. + +### `[observability]` + +| Field | Type | Required | Default | Description | +| ----------------------- | ------- | -------- | ------- | ------------------------------------------------------------------- | +| `server_timing_enabled` | Boolean | No | `false` | Append request-phase timings to the `Server-Timing` response header | + +**Purpose**: Surfaces per-phase request timing (`ts-total` plus recorded +phases such as `ts-appbuild`, `ts-filter`, `ts-geo`, `ts-kv`, `ts-origin`, and +`ts-template-cache`) as a standard `Server-Timing` header, in milliseconds +with one decimal place. An unrecorded phase is omitted from the header +rather than rendered as zero. + +**Emission is conservative**: the header is appended only on responses that +are conclusively private, meaning `Cache-Control` contains `private` or +`no-store`. A response that is heuristically cacheable, carries a bare +`max-age`, or has no cache header at all never receives the header, because a +shared-cache object would otherwise replay one request's timings for its +entire stored lifetime. The long-lived, shared-cacheable `tsjs` asset route is +the concrete case this excludes. The header is appended, never inserted, so +an origin-supplied `Server-Timing` value and any entries the fronting +delivery layer adds are preserved alongside the TS entries. + +The Axum adapter applies the same private-response rule at its own terminal +point before serializing the response, and emits the header only; it does not +send access-telemetry rows. + +**Example**: + +```toml +[observability] +server_timing_enabled = true +``` + +**Environment Override**: + +```bash +TRUSTED_SERVER__OBSERVABILITY__SERVER_TIMING_ENABLED=true +``` + +::: tip Present-but-false by default +`server_timing_enabled` ships as `false` in the base operator config rather +than being left out, even though `false` is also its default. The +environment-variable overlay can only override a leaf that already exists in +the parsed TOML; it cannot create a missing one. Keeping the leaf present lets +`TRUSTED_SERVER__OBSERVABILITY__SERVER_TIMING_ENABLED` take effect without an +extra edit to add the table first. +::: + +### `[tinybird]` access telemetry keys + +`[tinybird]` configures a shared Events API transport (`enabled`, `api_host`, +`secret_store`, and per-sink dataset and token fields) used by two +independent emitters: auction telemetry (`auction_dataset`, +`auction_token_secret`) and access telemetry. The keys below cover the +access-telemetry sink and the shared enable flags. + +| Field | Type | Required | Default | Description | +| -------------------- | ------- | ------------------------------------ | ------- | ------------------------------------------------------------------------------- | +| `enabled` | Boolean | Yes, when `access_enabled` | `false` | Master switch for the shared Tinybird transport (host, store, credentials) | +| `auction_enabled` | Boolean | No | `true` | Independently gates auction telemetry emission, decoupled from access telemetry | +| `access_enabled` | Boolean | No | `false` | Enables the sampled access-telemetry row sent after each response is delivered | +| `access_sample_rate` | Float | Yes (`> 0.0`), when `access_enabled` | `0.0` | Fraction (`0.0`-`1.0`) of requests to emit an access-telemetry row for | + +**Purpose**: `access_enabled` and `auction_enabled` gate the two Tinybird +sinks separately so that turning on one does not silently turn on (or leave +off) the other; a settings test locks this decoupling in both directions. +Setting `access_enabled = true` with `access_sample_rate = 0.0` is rejected at +config load as an armed-but-silent configuration; use `access_enabled` itself +to turn the sink off, not the sample rate. Enabling `access_enabled` also +requires the shared transport fields (`enabled`, non-empty `api_host`, +`secret_store`, `access_dataset`, `access_token_secret`, and a positive +`max_body_bytes`) to already be set. + +**Example**: + +```toml +[tinybird] +enabled = true +api_host = "api.tinybird.example.com" +secret_store = "ts_secrets" +auction_enabled = true + +# Access-log telemetry, decoupled from auction emission. +access_enabled = true +access_dataset = "access_logs_raw" +access_token_secret = "tinybird_access_append_token" +access_sample_rate = 0.05 +max_body_bytes = 1048576 +``` + +**Environment Override**: + +```bash +TRUSTED_SERVER__TINYBIRD__ACCESS_ENABLED=true +TRUSTED_SERVER__TINYBIRD__ACCESS_SAMPLE_RATE=0.05 +TRUSTED_SERVER__TINYBIRD__AUCTION_ENABLED=true +``` + +A sampled request emits one access-telemetry row to `access_dataset` after +the response has already been delivered to the client, so ingest never delays +the response the reader sees. + +### Deploy and rollback ordering + +::: warning `Settings` rejects unknown fields; order matters +Both `[observability].server_timing_enabled` and the new `[tinybird]` access +keys are new fields on a config schema that uses `deny_unknown_fields`, so an +older binary fails to load a config that carries them. + +**Deploying**: upgrade the binary first, then push a config containing the +new fields second. Never push a config with these fields while a +pre-observability binary can still receive it. + +**Rolling back**: reverse the order. Remove the `[observability]` table and +any new `[tinybird]` access keys from the config and push that first, then +roll back the binary second. +::: + ## Validation ### Automatic Validation From 7942c74aa6c2a298fed80dfade8ee3059c34cc7c Mon Sep 17 00:00:00 2001 From: Jason Evans Date: Tue, 25 Aug 2026 18:45:28 -0700 Subject: [PATCH 015/194] Normalize telemetry method, guard zero sample rate, and mirror geo write-back in middleware Three final-review fixes for access telemetry correctness: - Normalize the HTTP method to an allowlist (GET/HEAD/POST/PUT/DELETE/ PATCH/OPTIONS, else "other") inside access_event_row, so a client- controlled extension-method token can never inflate the LowCardinality method column, regardless of which adapter builds the row. - Guard emit_access_telemetry_after_send against snapshots carrying a degraded sample_rate of 0.0 (captured on the app-state-build-failure fallback path), which could otherwise be sampled in by freshly reloaded settings and corrupt the sum(1.0/sample_rate) volume estimator. - Mirror the geo lookup write-back from apply_entry_point_finalize_headers into FinalizeResponseMiddleware::handle, so a middleware-finalized response that resolved geo via fallback carries the resolved GeoLookupState for the access-telemetry snapshot instead of showing country "unknown". --- .../trusted-server-adapter-fastly/src/main.rs | 88 ++++++++++++++++++- .../src/middleware.rs | 73 +++++++++++++++ .../src/access_telemetry.rs | 79 ++++++++++++++++- 3 files changed, 235 insertions(+), 5 deletions(-) diff --git a/crates/trusted-server-adapter-fastly/src/main.rs b/crates/trusted-server-adapter-fastly/src/main.rs index 07c15b8b80..ec2370272d 100644 --- a/crates/trusted-server-adapter-fastly/src/main.rs +++ b/crates/trusted-server-adapter-fastly/src/main.rs @@ -452,9 +452,12 @@ fn run_edgezero_pull_sync_after_send( /// either of those per-route types, so every response class can emit. /// /// Sampled-out requests return silently — that is the expected, high-volume -/// case and not worth a log line. Every other drop (row build, token load, -/// send, or non-2xx status — all folded into `emit_access_event`'s `Result`) -/// logs exactly one warning naming the reason. +/// case and not worth a log line. A snapshot carrying a degraded +/// `sample_rate` (see [`should_sample_access_row`]) also returns silently, +/// since it only occurs on an already-degraded path. Every other drop (row +/// build, token load, send, or non-2xx status — all folded into +/// `emit_access_event`'s `Result`) logs exactly one warning naming the +/// reason. fn emit_access_telemetry_after_send( settings: &Settings, outcome: &DeliveryOutcome, @@ -477,7 +480,11 @@ fn emit_access_telemetry_after_send( let entropy_nanos = u64::try_from(since_epoch.as_nanos()).unwrap_or(u64::MAX); let entropy = entropy_nanos ^ outcome.bytes; - if !tinybird::sampled_in(settings.tinybird.access_sample_rate, entropy) { + if !should_sample_access_row( + outcome.snapshot.sample_rate, + settings.tinybird.access_sample_rate, + entropy, + ) { return; } @@ -493,6 +500,39 @@ fn emit_access_telemetry_after_send( } } +/// Whether one response's access-telemetry row should be emitted, combining +/// the degraded-snapshot guard with the sampling roll. +/// +/// `snapshot_sample_rate` is the rate recorded on the [`AccessTelemetrySnapshot`] +/// itself (the value serialized into the row's `sample_rate` column, which +/// the documented volume estimator divides by as `1.0 / sample_rate`). +/// `settings_sample_rate` is the rate used for the sampling decision at +/// call time. The two can diverge: when `app_state` fails to build, +/// [`edgezero_main`] captures a snapshot with `sample_rate` defaulted to +/// `0.0` before any settings ever load, but the two settings-reload +/// emission sites still gate and sample using the *reloaded* settings' +/// (nonzero) rate. Without this guard, such a row could be sampled in and +/// emitted while carrying `sample_rate: 0.0`, corrupting the volume +/// estimator. Dropping these rows is acceptable: they only occur on an +/// already-degraded path, consistent with this pipeline's fail-quiet +/// telemetry policy. Split out of [`emit_access_telemetry_after_send`] so +/// the guard is unit-testable without a network seam. +/// +/// Callers must already have applied the coarse +/// `tinybird.enabled`/`access_enabled` gate. +#[must_use] +fn should_sample_access_row( + snapshot_sample_rate: f64, + settings_sample_rate: f64, + entropy: u64, +) -> bool { + if snapshot_sample_rate <= 0.0 { + return false; + } + + tinybird::sampled_in(settings_sample_rate, entropy) +} + /// Per-response context threaded into [`send_edgezero_response`] so the /// function stays at or under seven parameters. struct SendContext { @@ -1818,4 +1858,44 @@ mod tests { "pull-sync must dispatch before telemetry emits" ); } + + #[test] + fn should_sample_access_row_rejects_a_degraded_zero_sample_rate() { + // A snapshot captured on the app-state-build-failure fallback path + // carries `sample_rate: 0.0`. Even when the reloaded settings' rate + // would sample every request in (1.0), the row must not emit — + // otherwise it would claim `sample_rate: 0.0` and corrupt the + // `sum(1.0 / sample_rate)` volume estimator. + assert!( + !should_sample_access_row(0.0, 1.0, 0), + "a snapshot with sample_rate 0.0 must never emit, regardless of entropy or settings' rate" + ); + assert!( + !should_sample_access_row(0.0, 1.0, u64::MAX), + "the degraded-rate guard must not depend on the entropy value" + ); + } + + #[test] + fn should_sample_access_row_rejects_a_negative_sample_rate() { + assert!( + !should_sample_access_row(-1.0, 1.0, 0), + "a negative snapshot sample_rate is equally degraded and must not emit" + ); + } + + #[test] + fn should_sample_access_row_defers_to_the_settings_sampling_roll_when_not_degraded() { + // With a healthy (nonzero) snapshot sample_rate, the outcome should + // match `tinybird::sampled_in` exactly, since that is the only + // remaining decision. + assert!( + should_sample_access_row(0.25, 1.0, 0), + "a settings rate of 1.0 always samples in, independent of entropy" + ); + assert!( + !should_sample_access_row(0.25, 0.0, 0), + "a settings rate of 0.0 always samples out, independent of the snapshot's rate" + ); + } } diff --git a/crates/trusted-server-adapter-fastly/src/middleware.rs b/crates/trusted-server-adapter-fastly/src/middleware.rs index 17ecc13a40..ae1efe7c3c 100644 --- a/crates/trusted-server-adapter-fastly/src/middleware.rs +++ b/crates/trusted-server-adapter-fastly/src/middleware.rs @@ -97,6 +97,18 @@ impl Middleware for FinalizeResponseMiddleware { }) }); + // Write the resolved outcome back so a downstream access-telemetry + // snapshot (built from response extensions after finalize) sees + // what was actually looked up here rather than the stale carried-in + // state — mirrors the entry-point finalize site in `main.rs` + // (`apply_entry_point_finalize_headers`), which writes back for the + // same reason. + let resolved_state = match &geo_info { + Some(geo) => GeoLookupState::Resolved(geo.clone()), + None => GeoLookupState::Attempted, + }; + response.extensions_mut().insert(resolved_state); + apply_finalize_headers(&self.settings, geo_info.as_ref(), &mut response); response .headers_mut() @@ -594,6 +606,67 @@ mod tests { ); } + #[test] + fn finalize_handle_writes_back_resolved_geo_state_after_fallback_lookup() { + // The request phase never attempted a geo lookup (no GeoLookupState + // extension on the handler's response), so the middleware resolves + // one via the fallback closure. That resolved outcome must be + // written back into response extensions -- mirroring + // apply_entry_point_finalize_headers in main.rs -- so a downstream + // access-telemetry snapshot sees the freshly resolved country + // instead of a stale/missing GeoLookupState. + let settings = settings_with_response_headers(vec![]); + let middleware = FinalizeResponseMiddleware::new( + Arc::new(settings), + Arc::new(FixedGeo(Some(sample_geo_info()))), + ); + let handler = + Arc::new( + |_ctx: RequestContext| async move { Ok::(empty_response()) }, + ); + + let response = block_on(middleware.handle(empty_ctx(), Next::new(&[], &*handler))) + .expect("should succeed"); + + match response.extensions().get::() { + Some(GeoLookupState::Resolved(info)) => { + assert_eq!( + info.country, "US", + "should carry the fallback-resolved geo info" + ); + } + other => { + panic!("expected GeoLookupState::Resolved after a fallback lookup, got {other:?}") + } + } + } + + #[test] + fn finalize_handle_writes_back_attempted_geo_state_when_fallback_finds_nothing() { + // The fallback lookup ran but resolved no geo info. The middleware + // must still record that the lookup was attempted, so a later + // consumer of the extension does not mistake this for + // GeoLookupState::NotAttempted and retry the lookup. + let settings = settings_with_response_headers(vec![]); + let middleware = + FinalizeResponseMiddleware::new(Arc::new(settings), Arc::new(FixedGeo(None))); + let handler = + Arc::new( + |_ctx: RequestContext| async move { Ok::(empty_response()) }, + ); + + let response = block_on(middleware.handle(empty_ctx(), Next::new(&[], &*handler))) + .expect("should succeed"); + + assert!( + matches!( + response.extensions().get::(), + Some(GeoLookupState::Attempted) + ), + "should write back Attempted when the fallback lookup finds no geo info" + ); + } + #[test] fn finalize_handle_marks_response_as_finalized() { let settings = settings_with_response_headers(vec![]); diff --git a/crates/trusted-server-core/src/access_telemetry.rs b/crates/trusted-server-core/src/access_telemetry.rs index 9daf77abc2..1642a1334c 100644 --- a/crates/trusted-server-core/src/access_telemetry.rs +++ b/crates/trusted-server-core/src/access_telemetry.rs @@ -16,6 +16,42 @@ use crate::request_timing::{AuctionWaitPlacement, TimingSnapshot}; /// by [`publisher_route_template`]. const MAX_SEGMENT_LEN: usize = 32; +/// Normalizes an HTTP method token into the bounded set of values stored in +/// the `method` `LowCardinality` column. +/// +/// HTTP permits arbitrary extension-method tokens (`PROPFIND`, `MKCOL`, or +/// any client-supplied garbage), and the token on an inbound request is +/// entirely client controlled. Capturing one verbatim into a 30-day +/// `LowCardinality(String)` column would let a single caller inflate that +/// column's cardinality without bound and would violate this dataset's +/// bounded-dimension privacy rule (see the module doc). Every standard +/// method maps to its uppercase form; anything else maps to `"other"`. Runs +/// inside [`access_event_row`] rather than at each capture site, so every +/// row-building path is covered regardless of how `method` was populated. +/// +/// # Examples +/// +/// ``` +/// use trusted_server_core::access_telemetry::normalize_method; +/// +/// assert_eq!(normalize_method("get"), "GET"); +/// assert_eq!(normalize_method("PROPFIND"), "other"); +/// assert_eq!(normalize_method(""), + "other", + "an unbounded client-controlled token must not reach the row verbatim" + ); + } + + #[test] + fn row_normalizes_method_even_when_snapshot_carries_a_raw_token() { + // The normalizer runs inside `access_event_row` so every row-building + // path is covered, regardless of what the snapshot's `method` field + // holds — a caller-controlled extension method must never leak into + // the row unnormalized. + let mut snapshot = unknown_snapshot(RouteClass::Other, "/other/*"); + snapshot.method = "PROPFIND".to_owned(); + let row = access_event_row(&snapshot, &TimingSnapshot::default(), 0); + let parsed: serde_json::Value = + serde_json::from_str(&row).expect("should serialize valid JSON"); + + assert_eq!(parsed["method"], "other"); + } } From d8fcb5e933e0cc8f0df15e98613dfce43636bd0f Mon Sep 17 00:00:00 2001 From: Jason Evans Date: Tue, 25 Aug 2026 19:51:06 -0700 Subject: [PATCH 016/194] Add a local dev config envelope generator example --- .../examples/local_dev_config.rs | 126 ++++++++++++++++++ 1 file changed, 126 insertions(+) create mode 100644 crates/trusted-server-core/examples/local_dev_config.rs diff --git a/crates/trusted-server-core/examples/local_dev_config.rs b/crates/trusted-server-core/examples/local_dev_config.rs new file mode 100644 index 0000000000..19826231dc --- /dev/null +++ b/crates/trusted-server-core/examples/local_dev_config.rs @@ -0,0 +1,126 @@ +//! Generate a ready-to-use local dev config envelope for the Axum adapter. +//! +//! Reads `trusted-server.example.toml`, replaces the placeholder secrets with +//! random values, flips the flags a local smoke test needs, validates the +//! result through [`trusted_server_core::settings::Settings::from_toml`], and +//! prints the blob envelope JSON that +//! `TRUSTED_SERVER_CONFIG_TRUSTED_SERVER_CONFIG_TRUSTED_SERVER_CONFIG` expects. +//! +//! The random values are time-and-pid seeded, not cryptographic. This tool +//! exists for throwaway local test instances only; never use its output for a +//! deployed service. +//! +//! Usage: +//! +//! ```text +//! cargo run -p trusted-server-core --example local_dev_config \ +//! --target -- [origin-url] [--realistic] +//! ``` +//! +//! `origin-url` defaults to `https://www.example.com`. By default every +//! response is forced `Cache-Control: private, no-store` so the Server-Timing +//! header is visible on all routes; pass `--realistic` to keep the origin's +//! own cache policy instead. + +use std::time::{SystemTime, UNIX_EPOCH}; + +/// Deliberately non-cryptographic generator for local placeholder secrets. +struct WeakRandom(u64); + +impl WeakRandom { + fn from_environment() -> Self { + let nanos = SystemTime::now() + .duration_since(UNIX_EPOCH) + .expect("should compute epoch time") + .subsec_nanos() as u64; + let secs = SystemTime::now() + .duration_since(UNIX_EPOCH) + .expect("should compute epoch time") + .as_secs(); + let pid = std::process::id() as u64; + Self(nanos ^ (secs << 20) ^ (pid << 40) ^ 0x9e37_79b9_7f4a_7c15) + } + + fn next(&mut self) -> u64 { + let mut x = self.0; + x ^= x << 13; + x ^= x >> 7; + x ^= x << 17; + self.0 = x; + x + } + + fn hex(&mut self, chars: usize) -> String { + let mut out = String::with_capacity(chars); + while out.len() < chars { + out.push_str(&format!("{:016x}", self.next())); + } + out.truncate(chars); + out + } +} + +#[allow(clippy::print_stdout, clippy::print_stderr)] +fn main() { + let args: Vec = std::env::args().skip(1).collect(); + let realistic = args.iter().any(|a| a == "--realistic"); + let origin = args + .iter() + .find(|a| !a.starts_with("--")) + .cloned() + .unwrap_or_else(|| "https://www.example.com".to_string()); + + let template = std::fs::read_to_string("trusted-server.example.toml") + .expect("should read trusted-server.example.toml from the repo root"); + + let mut random = WeakRandom::from_environment(); + let mut config = template + .replace( + "password = \"replace-with-admin-password-32-bytes\"", + &format!("password = \"{}\"", random.hex(48)), + ) + .replace( + "proxy_secret = \"change-me-proxy-secret\"", + &format!("proxy_secret = \"{}\"", random.hex(48)), + ) + .replace( + "passphrase = \"trusted-server-placeholder-secret\"", + &format!("passphrase = \"{}\"", random.hex(48)), + ) + .replace( + "server_timing_enabled = false", + "server_timing_enabled = true", + ); + + let origin_line = config + .lines() + .find(|line| line.starts_with("origin_url = ")) + .expect("should find the origin_url line in the template") + .to_string(); + config = config.replace(&origin_line, &format!("origin_url = \"{origin}\"")); + + if !realistic { + config = config.replace( + "# [response_headers]", + "[response_headers]\n\"Cache-Control\" = \"private, no-store\"", + ); + } + + let settings = trusted_server_core::settings::Settings::from_toml(&config) + .expect("should validate the generated local config"); + let data = serde_json::to_value(&settings).expect("should serialize settings"); + let generated_at = SystemTime::now() + .duration_since(UNIX_EPOCH) + .expect("should compute epoch time") + .as_secs() + .to_string(); + let envelope = edgezero_core::blob_envelope::BlobEnvelope::new(data, generated_at); + println!( + "{}", + serde_json::to_string(&envelope).expect("should serialize the envelope") + ); + eprintln!( + "local dev envelope generated: origin={origin} force_private={}", + !realistic + ); +} From 7cf7d86712c996e80ce1cd4ae436df7974a69732 Mon Sep 17 00:00:00 2001 From: Jason Evans Date: Tue, 25 Aug 2026 20:41:15 -0700 Subject: [PATCH 017/194] Add JSONPaths and expression sorting key to the access datasource --- .../2026-08-24-request-phase-timing-design.md | 7 ++- .../datasources/access_logs_raw.datasource | 60 ++++++++++--------- 2 files changed, 36 insertions(+), 31 deletions(-) diff --git a/docs/superpowers/specs/2026-08-24-request-phase-timing-design.md b/docs/superpowers/specs/2026-08-24-request-phase-timing-design.md index d9bbd4bdd7..d8791a799e 100644 --- a/docs/superpowers/specs/2026-08-24-request-phase-timing-design.md +++ b/docs/superpowers/specs/2026-08-24-request-phase-timing-design.md @@ -340,8 +340,11 @@ guest-visible cache behavior is already carried by `template_cache_state` and `origin_ms`. Rows exist only for guest-handled requests; fronting-cache hits are invisible by construction and the dashboard documentation says so. -Sorting key: `(event_date, service_id, publisher_domain, env, route_class, pop, -status)`. Grafana time filtering uses `$__timeFilter(event_ts)` and every panel query +Sorting key: `(toDate(event_ts), service_id, publisher_domain, env, route_class, +pop, status)`. Every column carries a `json:$.` path (the Events API rejects +NDJSON into a datasource without JSONPaths, discovered live); `event_date` was +dropped in favor of the sorting-key expression because a DEFAULT column cannot +carry a JSONPath the producer never sends. Grafana time filtering uses `$__timeFilter(event_ts)` and every panel query also carries an `event_date` predicate so the primary index prunes; rollout validates the panel queries with `EXPLAIN` before the dashboard is committed. This replaces the reserved key `(event_date, path, status, method)`. Rollout step 4 verifies whether the diff --git a/tinybird/datasources/access_logs_raw.datasource b/tinybird/datasources/access_logs_raw.datasource index 918f3d5fd4..062b884bac 100644 --- a/tinybird/datasources/access_logs_raw.datasource +++ b/tinybird/datasources/access_logs_raw.datasource @@ -2,36 +2,38 @@ DESCRIPTION > Per-request phase-timing telemetry rows, sampled and emitted post-send by the edge service. SCHEMA > - `event_ts` DateTime64(3), - `method` LowCardinality(String), - `status` UInt16, - `time_elapsed_ms` Nullable(UInt32), - `sample_rate` Float64, - `service_id` LowCardinality(String), - `publisher_domain` LowCardinality(String), - `env` LowCardinality(String), - `route_class` LowCardinality(String), - `route_template` String, - `body_mode` LowCardinality(String), - `auction_wait_placement` LowCardinality(String), - `appbuild_ms` Nullable(UInt32), - `filter_ms` Nullable(UInt32), - `geo_ms` Nullable(UInt32), - `kv_ms` Nullable(UInt32), - `origin_ms` Nullable(UInt32), - `template_cache_ms` Nullable(UInt32), - `auction_wait_ms` Nullable(UInt32), - `stream_ms` Nullable(UInt32), - `request_elapsed_ms` Nullable(UInt32), - `resp_bytes` Nullable(UInt64), - `template_cache_state` LowCardinality(String), - `country` LowCardinality(String), - `ts_version` LowCardinality(String), - `pop` LowCardinality(String), - `event_date` Date DEFAULT toDate(event_ts) + `event_ts` DateTime64(3) `json:$.event_ts`, + `method` LowCardinality(String) `json:$.method`, + `status` UInt16 `json:$.status`, + `time_elapsed_ms` Nullable(UInt32) `json:$.time_elapsed_ms`, + `sample_rate` Float64 `json:$.sample_rate`, + `service_id` LowCardinality(String) `json:$.service_id`, + `publisher_domain` LowCardinality(String) `json:$.publisher_domain`, + `env` LowCardinality(String) `json:$.env`, + `route_class` LowCardinality(String) `json:$.route_class`, + `route_template` String `json:$.route_template`, + `body_mode` LowCardinality(String) `json:$.body_mode`, + `auction_wait_placement` LowCardinality(String) `json:$.auction_wait_placement`, + `appbuild_ms` Nullable(UInt32) `json:$.appbuild_ms`, + `filter_ms` Nullable(UInt32) `json:$.filter_ms`, + `geo_ms` Nullable(UInt32) `json:$.geo_ms`, + `kv_ms` Nullable(UInt32) `json:$.kv_ms`, + `origin_ms` Nullable(UInt32) `json:$.origin_ms`, + `template_cache_ms` Nullable(UInt32) `json:$.template_cache_ms`, + `auction_wait_ms` Nullable(UInt32) `json:$.auction_wait_ms`, + `stream_ms` Nullable(UInt32) `json:$.stream_ms`, + `request_elapsed_ms` Nullable(UInt32) `json:$.request_elapsed_ms`, + `resp_bytes` Nullable(UInt64) `json:$.resp_bytes`, + `template_cache_state` LowCardinality(String) `json:$.template_cache_state`, + `country` LowCardinality(String) `json:$.country`, + `ts_version` LowCardinality(String) `json:$.ts_version`, + `pop` LowCardinality(String) `json:$.pop` ENGINE "MergeTree" -ENGINE_SORTING_KEY "event_date, service_id, publisher_domain, env, route_class, pop, status" -TTL "event_date + INTERVAL 30 DAY" +ENGINE_SORTING_KEY "toDate(event_ts), service_id, publisher_domain, env, route_class, pop, status" +TTL "toDate(event_ts) + INTERVAL 30 DAY" + +FORWARD_QUERY > + SELECT event_ts, method, status, time_elapsed_ms, sample_rate, service_id, publisher_domain, env, route_class, route_template, body_mode, auction_wait_placement, appbuild_ms, filter_ms, geo_ms, kv_ms, origin_ms, template_cache_ms, auction_wait_ms, stream_ms, request_elapsed_ms, resp_bytes, template_cache_state, country, ts_version, pop TOKEN ts_access_ingest APPEND From 600746f957f2a92c49fca93cea4d7d1a9c10de7c Mon Sep 17 00:00:00 2001 From: Jason Evans Date: Wed, 26 Aug 2026 07:08:16 -0700 Subject: [PATCH 018/194] Use web_time Instant on request timing paths std::time::Instant::now() panics on wasm32-unknown-unknown, so every publisher request on the Cloudflare adapter trapped when the timing collector was constructed, and the two auction-wait sites would trap once an auction dispatched. web_time re-exports std's Instant on every other target, so Fastly, Axum, and Spin behavior is unchanged. The publisher.rs sites are qualified locally because that module's std Instant import still serves the pre-existing template-cache sites, which are out of scope here. --- crates/trusted-server-core/src/publisher.rs | 10 ++++++++-- crates/trusted-server-core/src/request_timing.rs | 6 +++++- 2 files changed, 13 insertions(+), 3 deletions(-) diff --git a/crates/trusted-server-core/src/publisher.rs b/crates/trusted-server-core/src/publisher.rs index 6a30611c8b..53d09bfc61 100644 --- a/crates/trusted-server-core/src/publisher.rs +++ b/crates/trusted-server-core/src/publisher.rs @@ -3937,7 +3937,10 @@ async fn collect_non_html_auction( .as_ref() .and_then(|_| diagnostics_auction_id(settings)); let placeholder = mediator_placeholder_request(); - let wait_started = Instant::now(); + // Qualified: `std::time::Instant::now()` panics on Cloudflare's + // `wasm32-unknown-unknown` target; this module's `Instant` import stays + // std for the pre-existing template-cache sites. + let wait_started = web_time::Instant::now(); let result = orchestrator .collect_dispatched_auction( dispatched, @@ -3999,7 +4002,10 @@ async fn collect_stream_auction( log::info!("body_close_hold_loop: collecting dispatched auction before held body tail"); let placeholder = mediator_placeholder_request(); let collect_ctx = make_collect_context(settings, services, &placeholder); - let wait_started = Instant::now(); + // Qualified: `std::time::Instant::now()` panics on Cloudflare's + // `wasm32-unknown-unknown` target; this module's `Instant` import stays + // std for the pre-existing template-cache sites. + let wait_started = web_time::Instant::now(); let result = orchestrator .collect_dispatched_auction(dispatched, services, &collect_ctx) .await; diff --git a/crates/trusted-server-core/src/request_timing.rs b/crates/trusted-server-core/src/request_timing.rs index ccd3935551..6ac9a9692b 100644 --- a/crates/trusted-server-core/src/request_timing.rs +++ b/crates/trusted-server-core/src/request_timing.rs @@ -5,9 +5,13 @@ //! `docs/superpowers/specs/2026-08-24-request-phase-timing-design.md`. use std::sync::{Arc, Mutex}; -use std::time::{Duration, Instant}; +use std::time::Duration; use http::{HeaderName, HeaderValue, Response}; +// `std::time::Instant::now()` panics on `wasm32-unknown-unknown` (the +// Cloudflare adapter's target); `web_time` re-exports std's `Instant` on +// every other target. +use web_time::Instant; use crate::cache_policy::cache_control_headers_are_private_or_no_store; From 3d7e697cad8d1fb6b3206249d66a54e117d40527 Mon Sep 17 00:00:00 2001 From: Jason Evans Date: Wed, 26 Aug 2026 07:08:33 -0700 Subject: [PATCH 019/194] Reject opaque identifier segments in publisher route templates The character allowlist alone does not bound identity: [a-z0-9_-] is exactly the alphabet UUIDs, hex ids, reset tokens, and article slugs are built from, and truncating to 32 characters still leaves a globally unique prefix. A first segment now rejects whole to /other/* when it exceeds 32 characters or carries more than 7 ASCII digits, alongside the existing charset rejection. Year archives and hyphenated section names still pass. Extends the adversarial tests to the publisher-fallback path with UUID, hex-id, token, and slug shapes, and fixes the stale event_date reference in the row-builder doc. --- .../src/access_telemetry.rs | 114 +++++++++++++++--- .../2026-08-24-request-phase-timing-design.md | 11 +- 2 files changed, 105 insertions(+), 20 deletions(-) diff --git a/crates/trusted-server-core/src/access_telemetry.rs b/crates/trusted-server-core/src/access_telemetry.rs index 1642a1334c..28e391f7e7 100644 --- a/crates/trusted-server-core/src/access_telemetry.rs +++ b/crates/trusted-server-core/src/access_telemetry.rs @@ -12,10 +12,20 @@ use serde_json::json; use crate::request_timing::{AuctionWaitPlacement, TimingSnapshot}; -/// Maximum number of characters kept from a publisher path's first segment -/// by [`publisher_route_template`]. +/// Maximum length of a publisher path's first segment before +/// [`publisher_route_template`] rejects it to `/other/*`. Longer segments +/// are opaque-identifier or slug shaped (a UUID is 36 characters), and a +/// truncated prefix of either would still be identifying, so the segment +/// is rejected whole rather than truncated. const MAX_SEGMENT_LEN: usize = 32; +/// Maximum number of ASCII digits in a publisher path's first segment +/// before [`publisher_route_template`] rejects it to `/other/*`. Hex ids, +/// base36 ids, and reset tokens are digit-heavy; real section names carry +/// at most a year (`2026`) or a small version number, so a segment with +/// more digits than this is treated as an identifier, not a name. +const MAX_SEGMENT_DIGITS: usize = 7; + /// Normalizes an HTTP method token into the bounded set of values stored in /// the `method` `LowCardinality` column. /// @@ -116,18 +126,27 @@ pub struct RouteMetadata { /// content-free route template. /// /// Returns `/` plus the first path segment, lowercased and restricted to -/// `[a-z0-9_-]`, truncated to [`MAX_SEGMENT_LEN`] characters, with a -/// trailing `/*` appended when the path has additional segments beyond the -/// first. The root path `/` maps to itself. An empty first segment, or one -/// containing any character outside the allowlist (after lowercasing), -/// maps to `/other/*` — the segment is rejected outright rather than -/// filtered, so no fragment of a disallowed segment (an email address, a -/// search phrase) ever reaches the row. +/// `[a-z0-9_-]`, with a trailing `/*` appended when the path has +/// additional segments beyond the first. The root path `/` maps to itself. +/// A first segment is rejected to `/other/*` — outright, never filtered or +/// truncated, so no fragment of it ever reaches the row — when it: +/// +/// - is empty, or contains any character outside the allowlist after +/// lowercasing (an email address, a search phrase); +/// - is longer than [`MAX_SEGMENT_LEN`] characters (UUIDs, long hex +/// tokens, and full article slugs all exceed it — a truncated prefix of +/// any of these would still be identifying); or +/// - contains more than [`MAX_SEGMENT_DIGITS`] ASCII digits. Opaque +/// identifiers (hex ids, base36 ids, reset tokens) are digit-heavy; +/// publisher section names are words, at most a year or a version +/// number. /// /// This is deliberately coarser than the auction-telemetry path /// normalizer, which redacts long tokens but preserves short identifiers /// and arbitrary slugs; that normalizer is not sufficient for a dataset -/// this broad. +/// this broad. Short all-alpha slugs on single-segment paths are +/// indistinguishable from section names and still pass; the bound here is +/// shape-based, not semantic. /// /// # Examples /// @@ -137,6 +156,10 @@ pub struct RouteMetadata { /// assert_eq!(publisher_route_template("/news/some-article-slug"), "/news/*"); /// assert_eq!(publisher_route_template("/"), "/"); /// assert_eq!(publisher_route_template("/user@example.com/profile"), "/other/*"); +/// assert_eq!( +/// publisher_route_template("/550e8400-e29b-41d4-a716-446655440000"), +/// "/other/*" +/// ); /// ``` #[must_use] pub fn publisher_route_template(path: &str) -> String { @@ -156,16 +179,17 @@ pub fn publisher_route_template(path: &str) -> String { && lowered .chars() .all(|c| c.is_ascii_lowercase() || c.is_ascii_digit() || c == '_' || c == '-'); + let within_length = lowered.chars().count() <= MAX_SEGMENT_LEN; + let digit_count = lowered.chars().filter(char::is_ascii_digit).count(); - if !is_allowlisted { + if !is_allowlisted || !within_length || digit_count > MAX_SEGMENT_DIGITS { return "/other/*".to_owned(); } - let truncated: String = lowered.chars().take(MAX_SEGMENT_LEN).collect(); if has_more_depth { - format!("/{truncated}/*") + format!("/{lowered}/*") } else { - format!("/{truncated}") + format!("/{lowered}") } } @@ -220,8 +244,9 @@ pub struct AccessTelemetrySnapshot { /// `timings` and serialize as JSON `null` for phases that were never /// recorded; every dimension column comes from `snapshot` and is a /// non-nullable string (callers are expected to substitute an `unknown` -/// sentinel rather than leave a dimension empty). `event_date` is omitted: -/// the datasource derives it from `event_ts` by default. +/// sentinel rather than leave a dimension empty). There is no `event_date` +/// column: the datasource's sorting key derives the date via +/// `toDate(event_ts)`. #[must_use] pub fn access_event_row( snapshot: &AccessTelemetrySnapshot, @@ -338,12 +363,65 @@ mod tests { ); assert_eq!( publisher_route_template(&format!("/{}", "a".repeat(500))), - format!("/{}", "a".repeat(32)), - "should bound segment length" + "/other/*", + "should reject overlong segments whole rather than truncate" ); assert_eq!(publisher_route_template("/search terms here"), "/other/*"); } + #[test] + fn publisher_route_template_rejects_opaque_identifier_segments() { + // Every row here passes the character allowlist (`[a-z0-9_-]` is + // exactly what UUIDs, hex ids, and tokens are built from) and must + // be caught by the length and digit-count bounds instead. A + // truncated prefix of any of these would still be identifying, so + // rejection must be whole-segment. + assert_eq!( + publisher_route_template("/550e8400-e29b-41d4-a716-446655440000"), + "/other/*", + "should reject a UUID (36 chars) by length" + ); + assert_eq!( + publisher_route_template("/550e8400-e29b-41d4-a716-446655440000/profile"), + "/other/*", + "should reject a UUID first segment on deeper paths too" + ); + assert_eq!( + publisher_route_template("/8f3a9c2b1d4e5f6a7b8c9d0e1f2a3b4c"), + "/other/*", + "should reject a 32-char hex id by digit count" + ); + assert_eq!( + publisher_route_template(&format!("/{}", "a1".repeat(32))), + "/other/*", + "should reject a 64-char token by length" + ); + assert_eq!( + publisher_route_template("/reset-password-token-9f2b1c7d4e8a"), + "/other/*", + "should reject a reset token by length" + ); + assert_eq!( + publisher_route_template("/how-to-treat-my-recent-hiv-diagnosis"), + "/other/*", + "should reject a full article slug by length" + ); + } + + #[test] + fn publisher_route_template_keeps_digit_light_section_names() { + assert_eq!( + publisher_route_template("/2026/08/some-article"), + "/2026/*", + "a year archive segment should pass the digit bound" + ); + assert_eq!( + publisher_route_template("/wp-content/themes/site/app.css"), + "/wp-content/*", + "a hyphenated section name should pass" + ); + } + #[test] fn publisher_route_template_rejects_empty_first_segment() { assert_eq!( diff --git a/docs/superpowers/specs/2026-08-24-request-phase-timing-design.md b/docs/superpowers/specs/2026-08-24-request-phase-timing-design.md index d8791a799e..214d146a1a 100644 --- a/docs/superpowers/specs/2026-08-24-request-phase-timing-design.md +++ b/docs/superpowers/specs/2026-08-24-request-phase-timing-design.md @@ -289,9 +289,16 @@ and user-generated content (search terms, usernames, emails in slugs). Replaced restricted to a bounded allowlisted charset, plus `/*` when deeper (for example `/news/*`). The auction-telemetry normalizer is explicitly not sufficient here: it redacts long tokens but preserves short identifiers and arbitrary slugs. +- Rejection is whole-segment, never truncation: a segment is dropped to `/other/*` + when it fails the charset allowlist, exceeds 32 characters, or carries more than 7 + ASCII digits. The character allowlist alone does not bound identity (`[a-z0-9_-]` + is exactly the alphabet of UUIDs, hex ids, and reset tokens), and a truncated + prefix of any of those is still identifying, so the length and digit bounds reject + the segment outright. - Tests are adversarial, not just the happy path: a literal EC identifier on the admin - route, an email address in a path segment, search-term-shaped segments, and - overlong segments must all normalize to bounded, content-free templates. + route, an email address in a path segment, search-term-shaped segments, overlong + segments, UUIDs, hex ids, reset tokens, and full article slugs must all normalize + to bounded, content-free templates. Added columns (all dimension columns non-nullable with an `unknown` sentinel, because ClickHouse sorting keys cannot contain nullable columns): From 38043d7464362d44519153a09fe850bacc256b58 Mon Sep 17 00:00:00 2001 From: Jason Evans Date: Wed, 26 Aug 2026 07:08:41 -0700 Subject: [PATCH 020/194] Address access telemetry review feedback - Gate building the access snapshot on tinybird.enabled and access_enabled, threaded through SendContext: a disabled deployment (the default) no longer pays env reads and String allocations on the pre-send path. DeliveryOutcome.snapshot becomes Option and the emitter treats None as nothing to send. - Classify asset-fallback responses as route_class asset with the operator-configured route prefix as the template, instead of landing in the other/unknown bucket alongside 404s. - Pin Phase::index() to PHASE_COUNT with a uniqueness-and-bounds test so a future variant fails the suite instead of panicking at runtime. - Drop the tautological sampled-out emission test; the 0.0-rate behavior is covered by sampled_in_boundary_rates_are_unconditional. - Clarify that the local dev config env var name genuinely triples trusted_server_config (prefix, store, key) rather than reading as a find/replace mistake. --- .../trusted-server-adapter-fastly/src/app.rs | 11 +++- .../trusted-server-adapter-fastly/src/main.rs | 51 +++++++++++++++---- .../src/tinybird.rs | 28 ---------- .../examples/local_dev_config.rs | 7 ++- .../src/access_telemetry.rs | 4 ++ .../trusted-server-core/src/request_timing.rs | 34 +++++++++++++ 6 files changed, 93 insertions(+), 42 deletions(-) diff --git a/crates/trusted-server-adapter-fastly/src/app.rs b/crates/trusted-server-adapter-fastly/src/app.rs index 0a0cc8c5ae..3a4ceda8f8 100644 --- a/crates/trusted-server-adapter-fastly/src/app.rs +++ b/crates/trusted-server-adapter-fastly/src/app.rs @@ -902,7 +902,14 @@ async fn dispatch_fallback( .then(|| state.settings.asset_route_for_path(&path)) .flatten(); if let Some(asset_route) = matched_asset_route { - return dispatch_asset_fallback( + // The template is the operator-configured route prefix, so it + // is bounded and content-free by construction (unlike request + // paths, which need `publisher_route_template`). + let asset_metadata = RouteMetadata { + route_class: RouteClass::Asset, + route_template: format!("{}/*", asset_route.prefix.trim_end_matches('/')), + }; + let mut response = dispatch_asset_fallback( state, services, req, @@ -911,6 +918,8 @@ async fn dispatch_fallback( ec.geo_lookup_state(), ) .await; + response.extensions_mut().insert(asset_metadata); + return response; } route_metadata = Some(RouteMetadata { diff --git a/crates/trusted-server-adapter-fastly/src/main.rs b/crates/trusted-server-adapter-fastly/src/main.rs index ec2370272d..934f984027 100644 --- a/crates/trusted-server-adapter-fastly/src/main.rs +++ b/crates/trusted-server-adapter-fastly/src/main.rs @@ -150,6 +150,9 @@ fn edgezero_main(mut req: FastlyRequest) { let access_sample_rate = settings_snapshot .as_deref() .map_or(0.0, |settings| settings.tinybird.access_sample_rate); + let access_telemetry_enabled = settings_snapshot + .as_deref() + .is_some_and(|settings| settings.tinybird.enabled && settings.tinybird.access_enabled); let publisher_domain = settings_snapshot.as_deref().map_or_else( || "unknown".to_owned(), |settings| settings.publisher.domain.clone(), @@ -276,6 +279,7 @@ fn edgezero_main(mut req: FastlyRequest) { method: request_method.clone(), publisher_domain: publisher_domain.clone(), access_sample_rate, + access_telemetry_enabled, }, ); run_edgezero_pull_sync_after_send(settings, &partner_registry, &ec_state); @@ -303,6 +307,7 @@ fn edgezero_main(mut req: FastlyRequest) { method: request_method.clone(), publisher_domain: publisher_domain.clone(), access_sample_rate, + access_telemetry_enabled, }, ); run_edgezero_pull_sync_after_send( @@ -336,6 +341,7 @@ fn edgezero_main(mut req: FastlyRequest) { method: request_method, publisher_domain, access_sample_rate, + access_telemetry_enabled, }, ); // The asset/admin/error fallback path: no `EcFinalizeState` (or the ec @@ -467,6 +473,12 @@ fn emit_access_telemetry_after_send( return; } + // No snapshot means access telemetry was disabled when the response + // was sent (the flag is read once, before dispatch); nothing to emit. + let Some(snapshot) = &outcome.snapshot else { + return; + }; + let since_epoch = SystemTime::now() .duration_since(UNIX_EPOCH) .unwrap_or_default(); @@ -481,14 +493,14 @@ fn emit_access_telemetry_after_send( let entropy = entropy_nanos ^ outcome.bytes; if !should_sample_access_row( - outcome.snapshot.sample_rate, + snapshot.sample_rate, settings.tinybird.access_sample_rate, entropy, ) { return; } - let row = access_event_row(&outcome.snapshot, &timings.snapshot(), epoch_ms); + let row = access_event_row(snapshot, &timings.snapshot(), epoch_ms); let target = tinybird::TinybirdEventsTarget::from_access_config(settings.tinybird.clone()); let result = futures::executor::block_on(tinybird::emit_access_event( &platform::FastlyPlatformHttpClient, @@ -547,6 +559,12 @@ struct SendContext { publisher_domain: String, /// The configured access-telemetry sample rate. access_sample_rate: f64, + /// Whether `tinybird.enabled` and `tinybird.access_enabled` were both + /// set when settings were first read. Gates building the + /// [`AccessTelemetrySnapshot`] at all: the snapshot costs env reads and + /// `String` allocations on the pre-send path, which a disabled + /// deployment (the default) should not pay. + access_telemetry_enabled: bool, } /// Outcome of handing a finalized response to the client. @@ -557,8 +575,9 @@ pub(crate) struct DeliveryOutcome { /// Whether delivery completed or failed partway. pub result: DeliveryResult, /// Access-telemetry dimensions captured for this response at the - /// freeze point. - pub snapshot: AccessTelemetrySnapshot, + /// freeze point. `None` when access telemetry was disabled at snapshot + /// time; the emitter treats that as nothing to send. + pub snapshot: Option, } /// Whether [`send_edgezero_response`] completed delivery or failed partway. @@ -691,11 +710,15 @@ fn send_edgezero_response( context.server_timing_enabled, ); - // Built unconditionally, right after the freeze point and before - // `into_parts()` consumes `response`: nothing else survives to - // post-send on every path (the request was consumed by dispatch, and - // `EcFinalizeState` is absent on asset, admin, and error paths). - let snapshot = build_access_telemetry_snapshot(&response, context); + // Built right after the freeze point and before `into_parts()` + // consumes `response`: nothing else survives to post-send on every + // path (the request was consumed by dispatch, and `EcFinalizeState` + // is absent on asset, admin, and error paths). Skipped entirely when + // access telemetry is disabled, so the default configuration pays no + // env reads or allocations here. + let snapshot = context + .access_telemetry_enabled + .then(|| build_access_telemetry_snapshot(&response, context)); let (parts, body) = response.into_parts(); @@ -1508,7 +1531,7 @@ mod tests { let outcome = DeliveryOutcome { bytes, result: DeliveryResult::Complete, - snapshot: sample_access_snapshot(), + snapshot: Some(sample_access_snapshot()), }; assert_eq!( @@ -1565,6 +1588,7 @@ mod tests { method: "GET".to_owned(), publisher_domain: "test-publisher.com".to_owned(), access_sample_rate: 0.25, + access_telemetry_enabled: true, } } @@ -1827,6 +1851,7 @@ mod tests { method: "GET".to_owned(), publisher_domain: "test-publisher.com".to_owned(), access_sample_rate: 1.0, + access_telemetry_enabled: true, }, ); assert!( @@ -1847,7 +1872,11 @@ mod tests { ..trusted_server_core::settings::TinybirdSettings::default() }, ); - let row = access_event_row(&outcome.snapshot, &timings.snapshot(), 0); + let snapshot = outcome + .snapshot + .as_ref() + .expect("should build a snapshot when access telemetry is enabled"); + let row = access_event_row(snapshot, &timings.snapshot(), 0); futures::executor::block_on(tinybird::emit_access_event(&http_client, &target, row)) .expect("should send access telemetry"); diff --git a/crates/trusted-server-adapter-fastly/src/tinybird.rs b/crates/trusted-server-adapter-fastly/src/tinybird.rs index 5025f9a73c..1158ada68c 100644 --- a/crates/trusted-server-adapter-fastly/src/tinybird.rs +++ b/crates/trusted-server-adapter-fastly/src/tinybird.rs @@ -960,34 +960,6 @@ mod tests { ); } - #[test] - fn sampled_out_requests_emit_nothing() { - // Mirrors main.rs's post-send gate exactly (`if sampled_in(rate, - // entropy) { emit_access_event(...) }`): `emit_access_event` is only - // reached when `sampled_in` returns `true`. With a `0.0` rate it - // never does, for any entropy, so the http client should never see - // a request. - let http_client = RecordingHttpClient::respond_with(202); - let target = TinybirdEventsTarget::from_access_config(enabled_config()); - let rate = 0.0; - let entropy = 123_456_789_u64; - - if sampled_in(rate, entropy) { - futures::executor::block_on(emit_access_event(&http_client, &target, "{}".to_owned())) - .expect("should send when sampled in"); - } - - assert_eq!( - http_client - .requests - .lock() - .expect("should lock recorded requests") - .len(), - 0, - "sampled-out requests must never reach emit_access_event" - ); - } - #[test] fn sampled_in_boundary_rates_are_unconditional() { assert!( diff --git a/crates/trusted-server-core/examples/local_dev_config.rs b/crates/trusted-server-core/examples/local_dev_config.rs index 19826231dc..acbb61b6cb 100644 --- a/crates/trusted-server-core/examples/local_dev_config.rs +++ b/crates/trusted-server-core/examples/local_dev_config.rs @@ -3,8 +3,11 @@ //! Reads `trusted-server.example.toml`, replaces the placeholder secrets with //! random values, flips the flags a local smoke test needs, validates the //! result through [`trusted_server_core::settings::Settings::from_toml`], and -//! prints the blob envelope JSON that -//! `TRUSTED_SERVER_CONFIG_TRUSTED_SERVER_CONFIG_TRUSTED_SERVER_CONFIG` expects. +//! prints the blob envelope JSON that the Axum adapter's +//! `TRUSTED_SERVER_CONFIG_{STORE}_{KEY}` environment variable expects. With +//! the default store and key both named `trusted_server_config`, the +//! concrete variable resolves (not a typo) to +//! `TRUSTED_SERVER_CONFIG_TRUSTED_SERVER_CONFIG_TRUSTED_SERVER_CONFIG`. //! //! The random values are time-and-pid seeded, not cryptographic. This tool //! exists for throwaway local test instances only; never use its output for a diff --git a/crates/trusted-server-core/src/access_telemetry.rs b/crates/trusted-server-core/src/access_telemetry.rs index 28e391f7e7..2aac6d7d36 100644 --- a/crates/trusted-server-core/src/access_telemetry.rs +++ b/crates/trusted-server-core/src/access_telemetry.rs @@ -81,6 +81,9 @@ pub enum RouteClass { Ec, /// The server-side auction or SPA re-auction (`page-bids`) endpoint. AuctionApi, + /// A response proxied through a configured asset route (the + /// non-document fallback for scripts, styles, images, and fonts). + Asset, /// Everything else: discovery, tester-cookie toggles, denied legacy /// aliases, and any response with no attached [`RouteMetadata`]. Other, @@ -97,6 +100,7 @@ impl RouteClass { Self::IntegrationProxy => "integration_proxy", Self::Ec => "ec", Self::AuctionApi => "auction_api", + Self::Asset => "asset", Self::Other => "other", } } diff --git a/crates/trusted-server-core/src/request_timing.rs b/crates/trusted-server-core/src/request_timing.rs index 6ac9a9692b..8d1b92cc1c 100644 --- a/crates/trusted-server-core/src/request_timing.rs +++ b/crates/trusted-server-core/src/request_timing.rs @@ -385,6 +385,40 @@ pub struct TimingSnapshot { mod tests { use super::*; + #[test] + fn every_phase_index_is_unique_and_in_bounds() { + // `PHASE_COUNT` and `Phase::index()` are hand-synced; nothing at + // compile time ties them together. A new variant whose `index()` + // returns `PHASE_COUNT` would panic at runtime on first `record`, + // contradicting the module's no-panics claim, so this test fails + // first instead. (A variant missing from this list is a compile + // error via the exhaustive `match` in `index()` once added there.) + let phases = [ + Phase::AppBuild, + Phase::Filter, + Phase::Geo, + Phase::EcKv, + Phase::Origin, + Phase::TemplateCacheLookup, + Phase::AuctionWait, + Phase::Stream, + ]; + let mut seen = [false; PHASE_COUNT]; + for phase in phases { + let index = phase.index(); + assert!( + index < PHASE_COUNT, + "should be in bounds: {phase:?} -> {index}" + ); + assert!(!seen[index], "should be unique: {phase:?} -> {index}"); + seen[index] = true; + } + assert!( + seen.iter().all(|slot| *slot), + "should cover every phases-array slot" + ); + } + #[test] fn render_omits_unrecorded_phases_and_orders_total_first() { let timings = RequestTimings::new(); From 274241b9ebb0f9bcabff420bc56a3a82ab1920a0 Mon Sep 17 00:00:00 2001 From: Jason Evans Date: Wed, 26 Aug 2026 11:00:29 -0700 Subject: [PATCH 021/194] Add auction timeline offsets spec amendment Adds section 18 to the request phase timing spec: three first-call-wins T0 offsets (auction dispatched, resolved, committed) on RequestTimings, emitted as additive nullable columns on access_logs_raw with auction_id as the join key to the per-bidder auction dataset. Answers the overlap-proof questions the two existing clocks cannot: when the auction started relative to request entry, when the final bid landed, and when targeting was committed toward GAM. --- .../2026-08-24-request-phase-timing-design.md | 105 ++++++++++++++++++ 1 file changed, 105 insertions(+) diff --git a/docs/superpowers/specs/2026-08-24-request-phase-timing-design.md b/docs/superpowers/specs/2026-08-24-request-phase-timing-design.md index 214d146a1a..56c2685b3a 100644 --- a/docs/superpowers/specs/2026-08-24-request-phase-timing-design.md +++ b/docs/superpowers/specs/2026-08-24-request-phase-timing-design.md @@ -562,3 +562,108 @@ streamed. No allocation in the hot path beyond the one `Arc` at entry, the - The stall window itself remains unattributed until this ships. If it recurs first, the bisection runbook from 2026-08-21 (cookie-free curl UA request, static-asset path versus HTML path) is the fallback. + +## 18. Auction timeline offsets (follow-up increment) + +Status: spec amendment for a follow-up PR; not part of the initial implementation +(#1074). Builds only on machinery that spec sections 5, 9, and 10 already define. + +### Problem + +The pipeline has two clocks that never meet. The auction dataset +(`auction_events_raw`, PR #813) measures the auction internally: `total_time_ms` +from auction start to terminal, `provider_response_time_ms` per bidder call. Its +clock starts when the auction observation is created, so nothing places those +numbers on the request timeline. The access row is T0-anchored but records only +`auction_wait_ms`: time the handler was blocked at collect, deliberately not the +auction's own timeline. + +That leaves three questions unanswerable today: + +1. At what request-relative time did the auction start (dispatch leave the edge)? +2. At what request-relative time did the auction resolve (final bid or timeout)? +3. At what request-relative time were the results committed toward GAM? + +These are the overlap-proof questions. A client-side wrapper cannot dispatch until +the browser boots (t≈3000ms on measured prospect pages); the server-side auction +dispatches while the origin fetch is in flight. Proving that requires all +milestones on one clock. + +### Design + +Three first-call-wins marks on `RequestTimings`, in the style of +`mark_headers_ready()`, each storing `Option` since T0: + +| Mark | Recorded at | Meaning | +| --------------------------- | --------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------ | +| `mark_auction_dispatched()` | immediately before `orchestrator.dispatch_auction` returns control to the caller (`publisher.rs` dispatch site) | bid requests have left the edge | +| `mark_auction_resolved()` | immediately after `collect_dispatched_auction` returns, both collect sites | final bid returned or auction timed out; terminal either way | +| `mark_auction_committed()` | immediately after `write_bids_to_state` returns, both call sites | winning bids are in page state, available to the response pipeline | + +Notes on the definitions: + +- "Committed toward GAM" is defined as `write_bids_to_state` returning: the common + point in buffered and streaming modes where targeting becomes part of the + response. TS never calls GAM server-side; the browser's GPT call carries the + targeting, and that half of the timeline belongs to client-side measurement. + The edge proves when targeting was available; the client proves when GAM saw it. +- First-call-wins on all three marks. A request produces at most one publisher-path + auction today; if a second auction ever occurs in one request, the row describes + the first and the auction dataset still carries both in full. +- Same locking and failure model as every other `RequestTimings` write: `try_lock`, + drop on contention, saturating conversion at serialization. + +### Row changes + +Four additive columns on `access_logs_raw`, all populated from the +`TimingSnapshot` at the existing freeze/emission points (no new emission path): + +``` +`auction_dispatched_ms` Nullable(UInt32), `json:$.auction_dispatched_ms` +`auction_resolved_ms` Nullable(UInt32), `json:$.auction_resolved_ms` +`auction_committed_ms` Nullable(UInt32), `json:$.auction_committed_ms` +`auction_id` String, `json:$.auction_id` +``` + +- The three offsets are null when no auction ran (the common case: assets, EC + endpoints, auction-disabled deployments). Null means "no auction", never "zero". +- `auction_id` is the telemetry auction UUID already present on every + `auction_events_raw` row, carried onto the access row as the join key between + the T0 timeline and per-bidder detail. Sentinel `none` when no auction ran, + matching the non-nullable-dimension convention of section 9. It is a random + UUID, not identity-bearing; unbounded cardinality is accepted for the same + reason it is accepted in the auction dataset. +- Schema evolution is additive with JSONPaths on every new column and + `FORWARD_QUERY` carrying the existing columns, per the deployed datasource's + established evolution path. Verified with `tb --cloud deploy --check` before + deploy. + +### Interpretation model + +Combined with existing columns, one access row now reads as a timeline: + +``` +t=0 ......... request entry +t=D ......... auction_dispatched_ms (bids out; origin fetch typically in flight) +t=R ......... auction_resolved_ms (R - D ~ auction duration; join auction_id + for the per-bidder long pole) +t=C ......... auction_committed_ms (targeting in page state) +t=H ......... time_elapsed_ms (headers committed) +``` + +Derivations the dashboard can add without schema help: auction duration on the +request clock (`R - D`), commit latency (`C - R`), and overlap ratio (share of +`R - D` that ran concurrently with `ts-origin`). `auction_wait_ms` keeps its +existing meaning (blocked time only) and is now interpretable next to the +timeline: `R - D` minus `auction_wait_ms` approximates how much of the auction +was absorbed by work the request needed anyway. + +### Scope + +- Fastly emits; Axum, Cloudflare, and Spin collect the marks but do not emit, + matching section 8a adapter semantics. +- No header emission for any of these values: they are post-hoc analysis fields, + and two of the three are typically unknown at the header freeze point in + streaming mode. +- No config surface: the marks are always-on collection like every other phase, + gated at emission by the existing `tinybird.access_enabled`. From 29d45e8c8087173ef8daa221898892b58a399989 Mon Sep 17 00:00:00 2001 From: Jason Evans Date: Wed, 26 Aug 2026 11:31:09 -0700 Subject: [PATCH 022/194] Add auction timeline offsets implementation plan --- .../2026-08-26-auction-timeline-offsets.md | 59 +++++++++++++++++++ 1 file changed, 59 insertions(+) create mode 100644 docs/superpowers/plans/2026-08-26-auction-timeline-offsets.md diff --git a/docs/superpowers/plans/2026-08-26-auction-timeline-offsets.md b/docs/superpowers/plans/2026-08-26-auction-timeline-offsets.md new file mode 100644 index 0000000000..08e14dabc2 --- /dev/null +++ b/docs/superpowers/plans/2026-08-26-auction-timeline-offsets.md @@ -0,0 +1,59 @@ +# Auction Timeline Offsets Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Record three T0-anchored auction milestones (dispatched, resolved, committed) plus the auction id on `RequestTimings`, and emit them as four additive columns on the `access_logs_raw` row. + +**Architecture:** Follows spec section 18 exactly. All state lives in the existing `RequestTimings` inner (same `try_lock`/first-call-wins/saturating model as `mark_headers_ready`); the row builder reads the values from `TimingSnapshot`, so no new emission path and no adapter changes. + +**Tech Stack:** Rust (core crate only), Tinybird datasource file. + +**Spec:** `docs/superpowers/specs/2026-08-24-request-phase-timing-design.md` section 18. + +## Global Constraints + +- Marks are first-call-wins; `try_lock` only; a contended lock drops the sample. +- Null offsets mean "no auction ran", never zero. `auction_id` sentinel is `none`. +- Column names: `auction_dispatched_ms`, `auction_resolved_ms`, `auction_committed_ms`, `auction_id`; JSONPaths `json:$.`; FORWARD_QUERY extended in the same order. +- Dispatch mark records only on `DispatchAuctionOutcome::Dispatched`; a failed dispatch leaves all three offsets null (the auction dataset still records the failure). +- No header emission, no config surface, no changes outside `trusted-server-core` and `tinybird/`. + +--- + +### Task 1: RequestTimings marks and snapshot fields + +**Files:** +- Modify: `crates/trusted-server-core/src/request_timing.rs` + +**Interfaces:** +- Produces: `mark_auction_dispatched(&self, auction_id: String)`, `mark_auction_resolved(&self)`, `mark_auction_committed(&self)`; `TimingSnapshot { auction_dispatched_ms, auction_resolved_ms, auction_committed_ms: Option, auction_id: Option, .. }` + +- [ ] Add `auction_dispatched`, `auction_resolved`, `auction_committed: Option` and `auction_id: Option` to `Inner`; initialize `None`. +- [ ] Add the three mark methods, first-call-wins on their own field, storing `inner.t0.elapsed()`; dispatched also stores the id first-call-wins. +- [ ] Map all four into `TimingSnapshot` via `duration_ms` / clone. +- [ ] Tests: first-call-wins per mark; snapshot maps offsets and id; unmarked snapshot yields all `None`. +- [ ] `cargo test-fastly request_timing`, commit. + +### Task 2: Publisher call sites + +**Files:** +- Modify: `crates/trusted-server-core/src/publisher.rs` + +**Interfaces:** +- Consumes: Task 1 methods; `observation.auction_id` (`AuctionObservationContext`), in scope at the dispatch site. + +- [ ] In the `DispatchAuctionOutcome::Dispatched` arm (~line 4341): `timings.mark_auction_dispatched(observation.auction_id.to_string());` +- [ ] After both `record_auction_wait` calls (collect sites ~3952 and ~4012): `mark_auction_resolved()`. +- [ ] After both `write_bids_to_state` calls (~3954 and ~4017): `mark_auction_committed()`. +- [ ] `cargo test-fastly`, commit. + +### Task 3: Row columns and datasource + +**Files:** +- Modify: `crates/trusted-server-core/src/access_telemetry.rs` +- Modify: `tinybird/datasources/access_logs_raw.datasource` + +- [ ] `access_event_row`: add the three offset keys (nullable) and `auction_id` with `none` sentinel, after the existing phase keys. +- [ ] Extend `row_serializes_nulls_for_missing_phases` and `row_serializes_recorded_phases_as_numbers` for the new keys. +- [ ] Datasource: four schema columns with JSONPaths (`Nullable(UInt32)` ×3, `String`), appended at the end of SCHEMA and FORWARD_QUERY so existing column order stays stable. +- [ ] Full gates: fmt, clippy (all six), test-fastly/axum/cloudflare/spin, parity. Commit. From 46911a647347adc13791171995839e43a92c56fb Mon Sep 17 00:00:00 2001 From: Jason Evans Date: Wed, 26 Aug 2026 11:31:09 -0700 Subject: [PATCH 023/194] Record T0-anchored auction timeline offsets in the access row Implements spec section 18: three first-call-wins marks on RequestTimings (dispatched at the DispatchAuctionOutcome::Dispatched arm, resolved after collect at both sites, committed after write_bids_to_state at both sites), carried through TimingSnapshot into four additive access_logs_raw columns: auction_dispatched_ms, auction_resolved_ms, auction_committed_ms, and auction_id as the join key to the per-bidder auction dataset. Null offsets mean no auction ran; a failed dispatch records nothing. FORWARD_QUERY fills the new columns with typed defaults for pre-existing rows. No header emission, no config surface, no adapter changes: the values ride the existing snapshot and the tinybird.access_enabled gate. --- .../wrangler.integration.generated.toml | 16 +++ .../src/access_telemetry.rs | 19 +++ crates/trusted-server-core/src/publisher.rs | 14 ++ .../trusted-server-core/src/request_timing.rs | 134 ++++++++++++++++++ .../datasources/access_logs_raw.datasource | 8 +- 5 files changed, 189 insertions(+), 2 deletions(-) create mode 100644 crates/trusted-server-adapter-cloudflare/wrangler.integration.generated.toml diff --git a/crates/trusted-server-adapter-cloudflare/wrangler.integration.generated.toml b/crates/trusted-server-adapter-cloudflare/wrangler.integration.generated.toml new file mode 100644 index 0000000000..263403193a --- /dev/null +++ b/crates/trusted-server-adapter-cloudflare/wrangler.integration.generated.toml @@ -0,0 +1,16 @@ +name = "trusted-server" +main = "build/index.js" +compatibility_date = "2024-09-23" +# Keep in sync with wrangler.toml. `cache_option_enabled` is required for the +# outbound `CacheMode::NoStore` cache bypass under this compatibility date. +compatibility_flags = ["nodejs_compat", "cache_option_enabled"] +# No [build] section — bundle is pre-built in CI; wrangler dev must not rebuild. + +[[kv_namespaces]] +binding = "TRUSTED_SERVER_KV" +id = "ci-local-kv" + +[vars] +# Placeholder replaced by the integration test harness with a JSON object that +# contains the runtime Trusted Server app-config blob envelope. +TRUSTED_SERVER_CONFIG = '''{"app_config":"{\"data\":{\"auction\":{\"allowed_context_keys\":[],\"creative_store\":\"creative_store\",\"enabled\":false,\"mediator\":null,\"providers\":[],\"timeout_ms\":2000},\"cache\":{\"asset_rules\":[]},\"consent\":{\"check_expiration\":true,\"conflict_resolution\":{\"freshness_threshold_days\":30,\"mode\":\"restrictive\"},\"gdpr\":{\"applies_in\":[\"AT\",\"BE\",\"BG\",\"HR\",\"CY\",\"CZ\",\"DK\",\"EE\",\"FI\",\"FR\",\"DE\",\"GR\",\"HU\",\"IE\",\"IT\",\"LV\",\"LT\",\"LU\",\"MT\",\"NL\",\"PL\",\"PT\",\"RO\",\"SK\",\"SI\",\"ES\",\"SE\",\"IS\",\"LI\",\"NO\",\"GB\"]},\"max_consent_age_days\":395,\"mode\":\"interpreter\",\"us_privacy_defaults\":{\"gpc_implies_optout\":true,\"lspa_covered\":false,\"notice_given\":true},\"us_states\":{\"privacy_states\":[\"CA\",\"VA\",\"CO\",\"CT\",\"UT\",\"MT\",\"OR\",\"TX\",\"FL\",\"DE\",\"IA\",\"NE\",\"NH\",\"NJ\",\"TN\",\"MN\",\"MD\",\"IN\",\"KY\",\"RI\"]}},\"creative_opportunities\":null,\"debug\":{\"auction_html_comment\":false,\"inject_adm_for_testing\":false,\"ja4_endpoint_enabled\":false},\"ec\":{\"cluster_recheck_secs\":3600,\"cluster_trust_threshold\":10,\"ec_store\":\"ec_identity_store\",\"partners\":[{\"api_token\":\"integration-test-token-alpha-32-bytes-ok\",\"batch_rate_limit\":60,\"bidstream_enabled\":true,\"name\":\"Integration Test Partner\",\"openrtb_atype\":3,\"pull_sync_allowed_domains\":[],\"pull_sync_enabled\":false,\"pull_sync_rate_limit\":10,\"pull_sync_ttl_sec\":86400,\"pull_sync_url\":null,\"source_domain\":\"inttest.example.com\",\"ts_pull_token\":null},{\"api_token\":\"integration-test-token-bravo-32-bytes-ok\",\"batch_rate_limit\":60,\"bidstream_enabled\":true,\"name\":\"Integration Test Partner 2\",\"openrtb_atype\":3,\"pull_sync_allowed_domains\":[],\"pull_sync_enabled\":false,\"pull_sync_rate_limit\":10,\"pull_sync_ttl_sec\":86400,\"pull_sync_url\":null,\"source_domain\":\"inttest2.example.com\",\"ts_pull_token\":null}],\"passphrase\":\"integration-test-ec-secret-padded-32\",\"pull_sync_concurrency\":3},\"handlers\":[{\"password\":\"integration-admin-password-32-bytes-ok\",\"path\":\"^/_ts/admin\",\"username\":\"admin\"}],\"image_optimizer\":{\"profile_sets\":{}},\"integrations\":{\"adserver_mock\":{\"context_query_params\":{\"example_segments\":\"segments\"},\"enabled\":false,\"endpoint\":\"https://adserver.example.com/mediate\",\"timeout_ms\":1000},\"aps\":{\"account_id\":\"example-aps-account-id\",\"allow_script_creatives\":false,\"enabled\":true,\"endpoint\":\"https://aps.example.com/e/pb/bid\",\"timeout_ms\":1000},\"datadome\":{\"api_origin\":\"https://api.example.com\",\"cache_ttl_seconds\":3600,\"enabled\":false,\"rewrite_sdk\":true,\"sdk_origin\":\"https://sdk.example.com\"},\"didomi\":{\"api_origin\":\"https://api.example.com\",\"enabled\":false,\"sdk_origin\":\"https://sdk.example.com\"},\"google_tag_manager\":{\"container_id\":\"GTM-EXAMPLE\",\"enabled\":false,\"upstream_url\":\"https://tags.example.com\"},\"gpt\":{\"cache_ttl_seconds\":3600,\"enabled\":false,\"gam_attribution_enabled\":false,\"rewrite_script\":true,\"script_url\":\"https://ads.example.com/gpt.js\"},\"gpt_diagnostics\":{\"enabled\":true},\"lockr\":{\"api_endpoint\":\"https://identity.example.com\",\"app_id\":\"\",\"cache_ttl_seconds\":3600,\"enabled\":false,\"rewrite_sdk\":true,\"sdk_url\":\"https://identity.example.com/trusted-server.js\"},\"nextjs\":{\"enabled\":false,\"max_combined_payload_bytes\":10485760,\"rewrite_attributes\":[\"href\",\"link\",\"siteBaseUrl\",\"siteProductionDomain\",\"url\"]},\"permutive\":{\"api_endpoint\":\"https://api.example.com\",\"enabled\":false,\"organization_id\":\"\",\"project_id\":\"\",\"secure_signals_endpoint\":\"https://secure-signals.example.com\",\"workspace_id\":\"\"},\"prebid\":{\"bidders\":[],\"client_side_bidders\":[],\"debug\":false,\"enabled\":false,\"server_url\":\"https://prebid.example.com/openrtb2/auction\",\"timeout_ms\":1000},\"sourcepoint\":{\"cache_ttl_seconds\":3600,\"cdn_origin\":\"https://cdn.example.com\",\"enabled\":false,\"rewrite_sdk\":true},\"testlight\":{\"enabled\":false,\"endpoint\":\"https://testlight.example.com/openrtb2/auction\",\"rewrite_scripts\":true,\"timeout_ms\":1200}},\"proxy\":{\"allowed_domains\":[],\"asset_routes\":[],\"certificate_check\":false},\"publisher\":{\"cookie_domain\":\"localhost\",\"domain\":\"localhost\",\"max_buffered_body_bytes\":16777216,\"origin_host_header_override\":null,\"origin_url\":\"http://127.0.0.1:8888\",\"proxy_secret\":\"integration-test-proxy-secret\"},\"request_signing\":{\"config_store_id\":\"app_config\",\"enabled\":false,\"secret_store_id\":\"secrets\"},\"response_headers\":{},\"rewrite\":{\"exclude_domains\":[]},\"tester_cookie\":{\"enabled\":false},\"tinybird\":{\"access_dataset\":\"access_logs_raw\",\"access_enabled\":false,\"access_sample_rate\":0.0,\"access_token_secret\":\"tinybird_access_append_token\",\"api_host\":\"\",\"auction_dataset\":\"auction_events_raw\",\"auction_enabled\":true,\"auction_token_secret\":\"tinybird_auction_append_token\",\"enabled\":false,\"max_body_bytes\":1048576,\"secret_store\":\"ts_secrets\"}},\"generated_at\":\"2026-06-23T00:00:00Z\",\"sha256\":\"895f7fad0ce924476d1c04c68b0bf95f463d1fb631a4f639dcc7cf0c510383b4\",\"version\":1}"}''' diff --git a/crates/trusted-server-core/src/access_telemetry.rs b/crates/trusted-server-core/src/access_telemetry.rs index 2aac6d7d36..083adbe8ec 100644 --- a/crates/trusted-server-core/src/access_telemetry.rs +++ b/crates/trusted-server-core/src/access_telemetry.rs @@ -286,6 +286,10 @@ pub fn access_event_row( "stream_ms": timings.stream_ms, "request_elapsed_ms": timings.request_elapsed_ms, "resp_bytes": timings.resp_bytes, + "auction_dispatched_ms": timings.auction_dispatched_ms, + "auction_resolved_ms": timings.auction_resolved_ms, + "auction_committed_ms": timings.auction_committed_ms, + "auction_id": timings.auction_id.as_deref().unwrap_or("none"), "template_cache_state": snapshot.template_cache_state, "country": snapshot.country, "ts_version": snapshot.ts_version, @@ -463,6 +467,9 @@ mod tests { "stream_ms", "request_elapsed_ms", "resp_bytes", + "auction_dispatched_ms", + "auction_resolved_ms", + "auction_committed_ms", ] { assert!( parsed[field].is_null(), @@ -488,6 +495,10 @@ mod tests { ); } assert_eq!(parsed["auction_wait_placement"], "none"); + assert_eq!( + parsed["auction_id"], "none", + "auction_id should carry the none sentinel when no auction ran" + ); } #[test] @@ -506,6 +517,10 @@ mod tests { stream_ms: Some(8), auction_wait_placement: Some(AuctionWaitPlacement::InStream), resp_bytes: Some(1024), + auction_dispatched_ms: Some(9), + auction_resolved_ms: Some(10), + auction_committed_ms: Some(11), + auction_id: Some("33333333-3333-3333-3333-333333333333".to_owned()), }; let row = access_event_row(&snapshot, &timings, 1_700_000_000_000); let parsed: serde_json::Value = @@ -515,6 +530,10 @@ mod tests { assert_eq!(parsed["stream_ms"], 8); assert_eq!(parsed["resp_bytes"], 1024); assert_eq!(parsed["auction_wait_placement"], "in_stream"); + assert_eq!(parsed["auction_dispatched_ms"], 9); + assert_eq!(parsed["auction_resolved_ms"], 10); + assert_eq!(parsed["auction_committed_ms"], 11); + assert_eq!(parsed["auction_id"], "33333333-3333-3333-3333-333333333333"); } #[test] diff --git a/crates/trusted-server-core/src/publisher.rs b/crates/trusted-server-core/src/publisher.rs index 53d09bfc61..3981c386be 100644 --- a/crates/trusted-server-core/src/publisher.rs +++ b/crates/trusted-server-core/src/publisher.rs @@ -3951,6 +3951,8 @@ async fn collect_non_html_auction( params .timings .record_auction_wait(placement, wait_started.elapsed()); + // T0-anchored timeline mark (spec section 18): final bid or timeout. + params.timings.mark_auction_resolved(); let delivered_winner_slots = write_bids_to_state( &result.winning_bids, params.price_granularity, @@ -3960,6 +3962,9 @@ async fn collect_non_html_auction( settings.debug.inject_adm_for_testing, auction_id.as_deref(), ); + // T0-anchored timeline mark (spec section 18): winning bids are in page + // state, available to the response pipeline. + params.timings.mark_auction_committed(); if let (Some(observation), Some(auction_request)) = (telemetry.observation, telemetry.auction_request.as_ref()) { @@ -4010,6 +4015,8 @@ async fn collect_stream_auction( .collect_dispatched_auction(dispatched, services, &collect_ctx) .await; timings.record_auction_wait(*placement, wait_started.elapsed()); + // T0-anchored timeline mark (spec section 18): final bid or timeout. + timings.mark_auction_resolved(); log::info!( "body_close_hold_loop: collect complete - {} winning bid(s)", result.winning_bids.len() @@ -4023,6 +4030,9 @@ async fn collect_stream_auction( settings.debug.inject_adm_for_testing, auction_id.as_deref(), ); + // T0-anchored timeline mark (spec section 18): winning bids are in page + // state, available to the response pipeline. + timings.mark_auction_committed(); if let (Some(observation), Some(auction_request)) = (telemetry.observation, telemetry.auction_request.as_ref()) { @@ -4340,6 +4350,10 @@ pub async fn handle_publisher_request( .await { DispatchAuctionOutcome::Dispatched(dispatched) => { + // T0-anchored timeline mark (spec section 18): bid + // requests have left the edge. A failed dispatch never + // marks, so all three auction offsets stay null for it. + timings.mark_auction_dispatched(observation.auction_id.to_string()); auction_request_for_telemetry = Some(auction_request); auction_observation = Some(observation); Some(dispatched) diff --git a/crates/trusted-server-core/src/request_timing.rs b/crates/trusted-server-core/src/request_timing.rs index 8d1b92cc1c..edcd9bfc2b 100644 --- a/crates/trusted-server-core/src/request_timing.rs +++ b/crates/trusted-server-core/src/request_timing.rs @@ -106,6 +106,19 @@ struct Inner { /// Response body size in bytes, set via /// [`RequestTimings::set_resp_bytes`]. resp_bytes: Option, + /// Elapsed time at the first + /// [`RequestTimings::mark_auction_dispatched`] call. + auction_dispatched: Option, + /// Elapsed time at the first + /// [`RequestTimings::mark_auction_resolved`] call. + auction_resolved: Option, + /// Elapsed time at the first + /// [`RequestTimings::mark_auction_committed`] call. + auction_committed: Option, + /// Telemetry auction UUID recorded by the first + /// [`RequestTimings::mark_auction_dispatched`] call; joins the access + /// row to the per-bidder auction dataset. + auction_id: Option, } /// Per-request phase timing collector. @@ -128,6 +141,10 @@ impl RequestTimings { request_elapsed: None, auction_wait_placement: None, resp_bytes: None, + auction_dispatched: None, + auction_resolved: None, + auction_committed: None, + auction_id: None, }))) } @@ -200,6 +217,53 @@ impl RequestTimings { } } + /// Stamps the elapsed time since `t0` as the auction dispatch offset and + /// records the telemetry auction id, the first time this is called. + /// + /// Called when `dispatch_auction` reports the bid requests dispatched; + /// a failed dispatch never records, so all three auction offsets stay + /// `None` for it. Subsequent calls are no-ops (first call wins). Drops + /// the sample silently on lock contention or poisoning. + pub fn mark_auction_dispatched(&self, auction_id: String) { + let Ok(mut inner) = self.0.try_lock() else { + return; + }; + if inner.auction_dispatched.is_none() { + inner.auction_dispatched = Some(inner.t0.elapsed()); + inner.auction_id = Some(auction_id); + } + } + + /// Stamps the elapsed time since `t0` as the auction resolve offset (the + /// final bid returned or the auction timed out), the first time this is + /// called. + /// + /// Subsequent calls are no-ops (first call wins). Drops the sample + /// silently on lock contention or poisoning. + pub fn mark_auction_resolved(&self) { + let Ok(mut inner) = self.0.try_lock() else { + return; + }; + if inner.auction_resolved.is_none() { + inner.auction_resolved = Some(inner.t0.elapsed()); + } + } + + /// Stamps the elapsed time since `t0` as the auction commit offset + /// (winning bids written into page state), the first time this is + /// called. + /// + /// Subsequent calls are no-ops (first call wins). Drops the sample + /// silently on lock contention or poisoning. + pub fn mark_auction_committed(&self) { + let Ok(mut inner) = self.0.try_lock() else { + return; + }; + if inner.auction_committed.is_none() { + inner.auction_committed = Some(inner.t0.elapsed()); + } + } + /// Records the response body size in bytes. /// /// Drops the sample silently on lock contention or poisoning. @@ -257,6 +321,10 @@ impl RequestTimings { stream_ms: duration_ms(inner.phases[Phase::Stream.index()]), auction_wait_placement: inner.auction_wait_placement, resp_bytes: inner.resp_bytes, + auction_dispatched_ms: duration_ms(inner.auction_dispatched), + auction_resolved_ms: duration_ms(inner.auction_resolved), + auction_committed_ms: duration_ms(inner.auction_committed), + auction_id: inner.auction_id.clone(), } } } @@ -379,6 +447,18 @@ pub struct TimingSnapshot { /// Response body size in bytes, set via /// [`RequestTimings::set_resp_bytes`]. pub resp_bytes: Option, + /// T0 offset at which the auction dispatched (bid requests left the + /// edge), or `None` when no auction ran. + pub auction_dispatched_ms: Option, + /// T0 offset at which the auction resolved (final bid or timeout), or + /// `None` when no auction ran. + pub auction_resolved_ms: Option, + /// T0 offset at which winning bids were committed into page state, or + /// `None` when no auction ran. + pub auction_committed_ms: Option, + /// Telemetry auction UUID joining this row to the auction dataset, or + /// `None` when no auction ran. + pub auction_id: Option, } #[cfg(test)] @@ -419,6 +499,60 @@ mod tests { ); } + #[test] + fn auction_marks_are_first_call_wins_and_snapshot_maps_them() { + let timings = RequestTimings::new(); + timings.mark_auction_dispatched("11111111-1111-1111-1111-111111111111".to_owned()); + timings.mark_auction_resolved(); + timings.mark_auction_committed(); + // Second calls must not overwrite the first-recorded values. + timings.mark_auction_dispatched("22222222-2222-2222-2222-222222222222".to_owned()); + timings.mark_auction_resolved(); + timings.mark_auction_committed(); + + let snapshot = timings.snapshot(); + assert!( + snapshot.auction_dispatched_ms.is_some(), + "should record the dispatch offset" + ); + assert!( + snapshot.auction_resolved_ms.is_some(), + "should record the resolve offset" + ); + assert!( + snapshot.auction_committed_ms.is_some(), + "should record the commit offset" + ); + assert_eq!( + snapshot.auction_id.as_deref(), + Some("11111111-1111-1111-1111-111111111111"), + "should keep the first-recorded auction id" + ); + } + + #[test] + fn snapshot_without_auction_marks_yields_none_for_all_offsets() { + let timings = RequestTimings::new(); + timings.mark_headers_ready(); + let snapshot = timings.snapshot(); + assert_eq!( + snapshot.auction_dispatched_ms, None, + "should stay None when no auction dispatched" + ); + assert_eq!( + snapshot.auction_resolved_ms, None, + "should stay None when no auction resolved" + ); + assert_eq!( + snapshot.auction_committed_ms, None, + "should stay None when no auction committed" + ); + assert_eq!( + snapshot.auction_id, None, + "should carry no auction id when no auction ran" + ); + } + #[test] fn render_omits_unrecorded_phases_and_orders_total_first() { let timings = RequestTimings::new(); diff --git a/tinybird/datasources/access_logs_raw.datasource b/tinybird/datasources/access_logs_raw.datasource index 062b884bac..4441de3843 100644 --- a/tinybird/datasources/access_logs_raw.datasource +++ b/tinybird/datasources/access_logs_raw.datasource @@ -27,13 +27,17 @@ SCHEMA > `template_cache_state` LowCardinality(String) `json:$.template_cache_state`, `country` LowCardinality(String) `json:$.country`, `ts_version` LowCardinality(String) `json:$.ts_version`, - `pop` LowCardinality(String) `json:$.pop` + `pop` LowCardinality(String) `json:$.pop`, + `auction_dispatched_ms` Nullable(UInt32) `json:$.auction_dispatched_ms`, + `auction_resolved_ms` Nullable(UInt32) `json:$.auction_resolved_ms`, + `auction_committed_ms` Nullable(UInt32) `json:$.auction_committed_ms`, + `auction_id` String `json:$.auction_id` ENGINE "MergeTree" ENGINE_SORTING_KEY "toDate(event_ts), service_id, publisher_domain, env, route_class, pop, status" TTL "toDate(event_ts) + INTERVAL 30 DAY" FORWARD_QUERY > - SELECT event_ts, method, status, time_elapsed_ms, sample_rate, service_id, publisher_domain, env, route_class, route_template, body_mode, auction_wait_placement, appbuild_ms, filter_ms, geo_ms, kv_ms, origin_ms, template_cache_ms, auction_wait_ms, stream_ms, request_elapsed_ms, resp_bytes, template_cache_state, country, ts_version, pop + SELECT event_ts, method, status, time_elapsed_ms, sample_rate, service_id, publisher_domain, env, route_class, route_template, body_mode, auction_wait_placement, appbuild_ms, filter_ms, geo_ms, kv_ms, origin_ms, template_cache_ms, auction_wait_ms, stream_ms, request_elapsed_ms, resp_bytes, template_cache_state, country, ts_version, pop, CAST(NULL AS Nullable(UInt32)) AS auction_dispatched_ms, CAST(NULL AS Nullable(UInt32)) AS auction_resolved_ms, CAST(NULL AS Nullable(UInt32)) AS auction_committed_ms, 'none' AS auction_id TOKEN ts_access_ingest APPEND From 1fa9f8cc09df889aec42039b13b7a108d1df9d47 Mon Sep 17 00:00:00 2001 From: Jason Evans Date: Wed, 26 Aug 2026 11:31:29 -0700 Subject: [PATCH 024/194] Remove generated integration wrangler config from tracking The Cloudflare integration harness writes wrangler.integration.generated.toml at test time; it was swept into the previous commit by accident. Ignore it so local CI=1 runs cannot commit it again. --- .gitignore | 1 + .../wrangler.integration.generated.toml | 16 ---------------- 2 files changed, 1 insertion(+), 16 deletions(-) delete mode 100644 crates/trusted-server-adapter-cloudflare/wrangler.integration.generated.toml diff --git a/.gitignore b/.gitignore index 24b9e06aa1..96ffa2a5cd 100644 --- a/.gitignore +++ b/.gitignore @@ -63,3 +63,4 @@ src/*.html # leftover local build artifacts (node_modules, target, dist) that remain on disk. /crates/js/ /crates/integration-tests/ +wrangler.integration.generated.toml diff --git a/crates/trusted-server-adapter-cloudflare/wrangler.integration.generated.toml b/crates/trusted-server-adapter-cloudflare/wrangler.integration.generated.toml deleted file mode 100644 index 263403193a..0000000000 --- a/crates/trusted-server-adapter-cloudflare/wrangler.integration.generated.toml +++ /dev/null @@ -1,16 +0,0 @@ -name = "trusted-server" -main = "build/index.js" -compatibility_date = "2024-09-23" -# Keep in sync with wrangler.toml. `cache_option_enabled` is required for the -# outbound `CacheMode::NoStore` cache bypass under this compatibility date. -compatibility_flags = ["nodejs_compat", "cache_option_enabled"] -# No [build] section — bundle is pre-built in CI; wrangler dev must not rebuild. - -[[kv_namespaces]] -binding = "TRUSTED_SERVER_KV" -id = "ci-local-kv" - -[vars] -# Placeholder replaced by the integration test harness with a JSON object that -# contains the runtime Trusted Server app-config blob envelope. -TRUSTED_SERVER_CONFIG = '''{"app_config":"{\"data\":{\"auction\":{\"allowed_context_keys\":[],\"creative_store\":\"creative_store\",\"enabled\":false,\"mediator\":null,\"providers\":[],\"timeout_ms\":2000},\"cache\":{\"asset_rules\":[]},\"consent\":{\"check_expiration\":true,\"conflict_resolution\":{\"freshness_threshold_days\":30,\"mode\":\"restrictive\"},\"gdpr\":{\"applies_in\":[\"AT\",\"BE\",\"BG\",\"HR\",\"CY\",\"CZ\",\"DK\",\"EE\",\"FI\",\"FR\",\"DE\",\"GR\",\"HU\",\"IE\",\"IT\",\"LV\",\"LT\",\"LU\",\"MT\",\"NL\",\"PL\",\"PT\",\"RO\",\"SK\",\"SI\",\"ES\",\"SE\",\"IS\",\"LI\",\"NO\",\"GB\"]},\"max_consent_age_days\":395,\"mode\":\"interpreter\",\"us_privacy_defaults\":{\"gpc_implies_optout\":true,\"lspa_covered\":false,\"notice_given\":true},\"us_states\":{\"privacy_states\":[\"CA\",\"VA\",\"CO\",\"CT\",\"UT\",\"MT\",\"OR\",\"TX\",\"FL\",\"DE\",\"IA\",\"NE\",\"NH\",\"NJ\",\"TN\",\"MN\",\"MD\",\"IN\",\"KY\",\"RI\"]}},\"creative_opportunities\":null,\"debug\":{\"auction_html_comment\":false,\"inject_adm_for_testing\":false,\"ja4_endpoint_enabled\":false},\"ec\":{\"cluster_recheck_secs\":3600,\"cluster_trust_threshold\":10,\"ec_store\":\"ec_identity_store\",\"partners\":[{\"api_token\":\"integration-test-token-alpha-32-bytes-ok\",\"batch_rate_limit\":60,\"bidstream_enabled\":true,\"name\":\"Integration Test Partner\",\"openrtb_atype\":3,\"pull_sync_allowed_domains\":[],\"pull_sync_enabled\":false,\"pull_sync_rate_limit\":10,\"pull_sync_ttl_sec\":86400,\"pull_sync_url\":null,\"source_domain\":\"inttest.example.com\",\"ts_pull_token\":null},{\"api_token\":\"integration-test-token-bravo-32-bytes-ok\",\"batch_rate_limit\":60,\"bidstream_enabled\":true,\"name\":\"Integration Test Partner 2\",\"openrtb_atype\":3,\"pull_sync_allowed_domains\":[],\"pull_sync_enabled\":false,\"pull_sync_rate_limit\":10,\"pull_sync_ttl_sec\":86400,\"pull_sync_url\":null,\"source_domain\":\"inttest2.example.com\",\"ts_pull_token\":null}],\"passphrase\":\"integration-test-ec-secret-padded-32\",\"pull_sync_concurrency\":3},\"handlers\":[{\"password\":\"integration-admin-password-32-bytes-ok\",\"path\":\"^/_ts/admin\",\"username\":\"admin\"}],\"image_optimizer\":{\"profile_sets\":{}},\"integrations\":{\"adserver_mock\":{\"context_query_params\":{\"example_segments\":\"segments\"},\"enabled\":false,\"endpoint\":\"https://adserver.example.com/mediate\",\"timeout_ms\":1000},\"aps\":{\"account_id\":\"example-aps-account-id\",\"allow_script_creatives\":false,\"enabled\":true,\"endpoint\":\"https://aps.example.com/e/pb/bid\",\"timeout_ms\":1000},\"datadome\":{\"api_origin\":\"https://api.example.com\",\"cache_ttl_seconds\":3600,\"enabled\":false,\"rewrite_sdk\":true,\"sdk_origin\":\"https://sdk.example.com\"},\"didomi\":{\"api_origin\":\"https://api.example.com\",\"enabled\":false,\"sdk_origin\":\"https://sdk.example.com\"},\"google_tag_manager\":{\"container_id\":\"GTM-EXAMPLE\",\"enabled\":false,\"upstream_url\":\"https://tags.example.com\"},\"gpt\":{\"cache_ttl_seconds\":3600,\"enabled\":false,\"gam_attribution_enabled\":false,\"rewrite_script\":true,\"script_url\":\"https://ads.example.com/gpt.js\"},\"gpt_diagnostics\":{\"enabled\":true},\"lockr\":{\"api_endpoint\":\"https://identity.example.com\",\"app_id\":\"\",\"cache_ttl_seconds\":3600,\"enabled\":false,\"rewrite_sdk\":true,\"sdk_url\":\"https://identity.example.com/trusted-server.js\"},\"nextjs\":{\"enabled\":false,\"max_combined_payload_bytes\":10485760,\"rewrite_attributes\":[\"href\",\"link\",\"siteBaseUrl\",\"siteProductionDomain\",\"url\"]},\"permutive\":{\"api_endpoint\":\"https://api.example.com\",\"enabled\":false,\"organization_id\":\"\",\"project_id\":\"\",\"secure_signals_endpoint\":\"https://secure-signals.example.com\",\"workspace_id\":\"\"},\"prebid\":{\"bidders\":[],\"client_side_bidders\":[],\"debug\":false,\"enabled\":false,\"server_url\":\"https://prebid.example.com/openrtb2/auction\",\"timeout_ms\":1000},\"sourcepoint\":{\"cache_ttl_seconds\":3600,\"cdn_origin\":\"https://cdn.example.com\",\"enabled\":false,\"rewrite_sdk\":true},\"testlight\":{\"enabled\":false,\"endpoint\":\"https://testlight.example.com/openrtb2/auction\",\"rewrite_scripts\":true,\"timeout_ms\":1200}},\"proxy\":{\"allowed_domains\":[],\"asset_routes\":[],\"certificate_check\":false},\"publisher\":{\"cookie_domain\":\"localhost\",\"domain\":\"localhost\",\"max_buffered_body_bytes\":16777216,\"origin_host_header_override\":null,\"origin_url\":\"http://127.0.0.1:8888\",\"proxy_secret\":\"integration-test-proxy-secret\"},\"request_signing\":{\"config_store_id\":\"app_config\",\"enabled\":false,\"secret_store_id\":\"secrets\"},\"response_headers\":{},\"rewrite\":{\"exclude_domains\":[]},\"tester_cookie\":{\"enabled\":false},\"tinybird\":{\"access_dataset\":\"access_logs_raw\",\"access_enabled\":false,\"access_sample_rate\":0.0,\"access_token_secret\":\"tinybird_access_append_token\",\"api_host\":\"\",\"auction_dataset\":\"auction_events_raw\",\"auction_enabled\":true,\"auction_token_secret\":\"tinybird_auction_append_token\",\"enabled\":false,\"max_body_bytes\":1048576,\"secret_store\":\"ts_secrets\"}},\"generated_at\":\"2026-06-23T00:00:00Z\",\"sha256\":\"895f7fad0ce924476d1c04c68b0bf95f463d1fb631a4f639dcc7cf0c510383b4\",\"version\":1}"}''' From 2ef7635437d50b1c4ec3eac57321aeb168970c0c Mon Sep 17 00:00:00 2001 From: Jason Evans Date: Sat, 29 Aug 2026 07:50:22 +1000 Subject: [PATCH 025/194] Reject single-segment publisher paths in route templates The first path segment is only a section name when the path has depth: under a /%postname%/ permalink structure every article is a single-segment path, so keeping those segments verbatim put full article slugs into the 30-day dataset, against spec section 9. Depth is now required for a named template; single-segment paths, root landing pages included, bucket to /other/*. Route slicing keeps route_class and multi-segment templates like /news/*. --- .../src/access_telemetry.rs | 69 ++++++++++++++----- 1 file changed, 50 insertions(+), 19 deletions(-) diff --git a/crates/trusted-server-core/src/access_telemetry.rs b/crates/trusted-server-core/src/access_telemetry.rs index 2aac6d7d36..aaad466336 100644 --- a/crates/trusted-server-core/src/access_telemetry.rs +++ b/crates/trusted-server-core/src/access_telemetry.rs @@ -130,27 +130,27 @@ pub struct RouteMetadata { /// content-free route template. /// /// Returns `/` plus the first path segment, lowercased and restricted to -/// `[a-z0-9_-]`, with a trailing `/*` appended when the path has -/// additional segments beyond the first. The root path `/` maps to itself. -/// A first segment is rejected to `/other/*` — outright, never filtered or -/// truncated, so no fragment of it ever reaches the row — when it: +/// `[a-z0-9_-]`, plus `/*`, only when the path has at least two segments: +/// depth is what makes the first segment a section name (`/news/*`) rather +/// than the document itself. The root path `/` maps to itself. Everything +/// else is rejected to `/other/*` — outright, never filtered or truncated, +/// so no fragment of a rejected path ever reaches the row: /// -/// - is empty, or contains any character outside the allowlist after -/// lowercasing (an email address, a search phrase); -/// - is longer than [`MAX_SEGMENT_LEN`] characters (UUIDs, long hex -/// tokens, and full article slugs all exceed it — a truncated prefix of -/// any of these would still be identifying); or -/// - contains more than [`MAX_SEGMENT_DIGITS`] ASCII digits. Opaque -/// identifiers (hex ids, base36 ids, reset tokens) are digit-heavy; -/// publisher section names are words, at most a year or a version -/// number. +/// - single-segment paths (`/my-post-title` under a `/%postname%/` +/// permalink structure is a per-article slug that no shape heuristic +/// can separate from a section name); +/// - an empty first segment, or one containing any character outside the +/// allowlist after lowercasing (an email address, a search phrase); +/// - a first segment longer than [`MAX_SEGMENT_LEN`] characters (a +/// truncated prefix of a UUID or token would still be identifying); or +/// - a first segment with more than [`MAX_SEGMENT_DIGITS`] ASCII digits +/// (hex ids, base36 ids, and reset tokens are digit-heavy; section +/// names carry at most a year or a version number). /// /// This is deliberately coarser than the auction-telemetry path /// normalizer, which redacts long tokens but preserves short identifiers /// and arbitrary slugs; that normalizer is not sufficient for a dataset -/// this broad. Short all-alpha slugs on single-segment paths are -/// indistinguishable from section names and still pass; the bound here is -/// shape-based, not semantic. +/// this broad. /// /// # Examples /// @@ -193,7 +193,11 @@ pub fn publisher_route_template(path: &str) -> String { if has_more_depth { format!("/{lowered}/*") } else { - format!("/{lowered}") + // A single-segment path's first segment is the document, not a + // section: `/my-post-title` under WordPress `/%postname%/` is a + // per-article slug, and no shape heuristic can separate it from a + // section name. Only depth >= 2 makes the first segment a section. + "/other/*".to_owned() } } @@ -436,11 +440,37 @@ mod tests { } #[test] - fn publisher_route_template_uppercases_lowercase_before_allowlisting() { + fn publisher_route_template_rejects_single_segment_paths() { + // A single-segment path's first segment is the document itself + // (WordPress `/%postname%/` puts every article at depth 1), so no + // shape heuristic can separate a slug from a section name; depth + // is the only safe signal. Root-level landing pages pay for this + // deliberately. + assert_eq!(publisher_route_template("/my-post-title"), "/other/*"); + assert_eq!( + publisher_route_template("/my-post-title/"), + "/other/*", + "a trailing slash should not count as depth" + ); + assert_eq!( + publisher_route_template("/1234567"), + "/other/*", + "a numeric post id at the digit boundary should still reject" + ); + assert_eq!(publisher_route_template("/user-8f3a9c2b"), "/other/*"); + assert_eq!( + publisher_route_template("/about"), + "/other/*", + "root-level landing pages reject too; only depth makes a section" + ); + } + + #[test] + fn publisher_route_template_lowercases_before_allowlisting() { assert_eq!( publisher_route_template("/News/Article"), "/news/*", - "should lowercase before validating and truncating" + "should lowercase before validating" ); } @@ -524,6 +554,7 @@ mod tests { assert_eq!(RouteClass::IntegrationProxy.as_str(), "integration_proxy"); assert_eq!(RouteClass::Ec.as_str(), "ec"); assert_eq!(RouteClass::AuctionApi.as_str(), "auction_api"); + assert_eq!(RouteClass::Asset.as_str(), "asset"); assert_eq!(RouteClass::Other.as_str(), "other"); } From 082461d10eeabf1e5658dd27ebd89f4b6068af3d Mon Sep 17 00:00:00 2001 From: Jason Evans Date: Sat, 29 Aug 2026 07:50:22 +1000 Subject: [PATCH 026/194] Sample access rows with real randomness at the snapshot rate The bucket-quantized sampler truncated rates below one in a million to a zero threshold (silently emitting nothing) and quantized other low rates downward while rows still carried the configured rate, biasing the sum(1.0 / sample_rate) volume estimator. Its no-rand premise was also wrong: rand::thread_rng() is WASI-backed on this target and the EC generation path already relies on it. The sampler is now a direct uniform-roll comparison, and the roll gates on the rate stored in the snapshot itself, so emission probability and the row's sample_rate column cannot diverge; the divergence guard and its tests are removed. Also per review: the settings-reload fallback in the post-send path could never emit (no snapshot exists when settings were absent) and is removed; the dead_code allow on DeliveryOutcome narrows to the one collected-but-unemitted field; and the post-send ordering test is narrowed to the leg it actually proves, that request_elapsed is stamped when send returns. --- Cargo.lock | 1 + .../trusted-server-adapter-fastly/Cargo.toml | 1 + .../trusted-server-adapter-fastly/src/main.rs | 243 +++--------------- .../src/tinybird.rs | 83 +++--- 4 files changed, 85 insertions(+), 243 deletions(-) diff --git a/Cargo.lock b/Cargo.lock index e29380b77d..201060a527 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -5372,6 +5372,7 @@ dependencies = [ "futures", "log", "log-fastly", + "rand 0.8.6", "serde", "serde_json", "sha2 0.10.9", diff --git a/crates/trusted-server-adapter-fastly/Cargo.toml b/crates/trusted-server-adapter-fastly/Cargo.toml index 47cc609b26..3835d6469c 100644 --- a/crates/trusted-server-adapter-fastly/Cargo.toml +++ b/crates/trusted-server-adapter-fastly/Cargo.toml @@ -29,6 +29,7 @@ serde = { workspace = true } serde_json = { workspace = true } sha2 = { workspace = true } trusted-server-core = { workspace = true } +rand = { workspace = true } url = { workspace = true } urlencoding = { workspace = true } diff --git a/crates/trusted-server-adapter-fastly/src/main.rs b/crates/trusted-server-adapter-fastly/src/main.rs index 934f984027..b5b08ab2ee 100644 --- a/crates/trusted-server-adapter-fastly/src/main.rs +++ b/crates/trusted-server-adapter-fastly/src/main.rs @@ -1,4 +1,6 @@ use std::sync::Arc; + +use rand::Rng as _; use std::time::{Instant, SystemTime, UNIX_EPOCH}; use edgezero_adapter_fastly::config_store::FastlyConfigStore as EdgeZeroFastlyConfigStore; @@ -346,17 +348,12 @@ fn edgezero_main(mut req: FastlyRequest) { ); // The asset/admin/error fallback path: no `EcFinalizeState` (or the ec // finalize branch above failed), so there is no pull-sync dispatch here - // at all — telemetry is the only post-send step. Reload settings when - // `app_state` never built, matching the fallback used earlier in this - // function for entry-point finalize headers. - match settings_snapshot.as_deref() { - Some(settings) => emit_access_telemetry_after_send(settings, &outcome, &timings), - None => match load_settings_from_config_store() { - Ok(settings) => emit_access_telemetry_after_send(&settings, &outcome, &timings), - Err(e) => { - log::warn!("access telemetry emission skipped: failed to reload settings: {e:?}"); - } - }, + // at all — telemetry is the only post-send step. When `app_state` never + // built there is nothing to emit either: `access_telemetry_enabled` was + // necessarily false without a settings snapshot, so the outcome carries + // no access snapshot, and reloading settings here could not change that. + if let Some(settings) = settings_snapshot.as_deref() { + emit_access_telemetry_after_send(settings, &outcome, &timings); } } @@ -458,9 +455,9 @@ fn run_edgezero_pull_sync_after_send( /// either of those per-route types, so every response class can emit. /// /// Sampled-out requests return silently — that is the expected, high-volume -/// case and not worth a log line. A snapshot carrying a degraded -/// `sample_rate` (see [`should_sample_access_row`]) also returns silently, -/// since it only occurs on an already-degraded path. Every other drop (row +/// case and not worth a log line. The sampling roll uses the rate stored on +/// the snapshot itself, so the emission probability always matches the +/// row's `sample_rate` column by construction. Every other drop (row /// build, token load, send, or non-2xx status — all folded into /// `emit_access_event`'s `Result`) logs exactly one warning naming the /// reason. @@ -483,20 +480,12 @@ fn emit_access_telemetry_after_send( .duration_since(UNIX_EPOCH) .unwrap_or_default(); let epoch_ms = u64::try_from(since_epoch.as_millis()).unwrap_or(u64::MAX); - // Entropy for the sampling decision: the timestamp's nanosecond - // resolution XORed with a per-request value already on hand - // (`outcome.bytes`), so two requests handled in the same instance never - // collide on sampling decisions purely because they read the same - // millisecond. There is no `rand` crate dependency here — see - // `tinybird::sampled_in`. - let entropy_nanos = u64::try_from(since_epoch.as_nanos()).unwrap_or(u64::MAX); - let entropy = entropy_nanos ^ outcome.bytes; - - if !should_sample_access_row( - snapshot.sample_rate, - settings.tinybird.access_sample_rate, - entropy, - ) { + // Sample with the rate stored on the snapshot itself — the same value + // serialized into the row's `sample_rate` column — so the emission + // probability and the row's claimed rate cannot diverge, which the + // documented `sum(1.0 / sample_rate)` volume estimator depends on. + let roll = rand::thread_rng().r#gen::(); + if !tinybird::sampled_in(snapshot.sample_rate, roll) { return; } @@ -512,39 +501,6 @@ fn emit_access_telemetry_after_send( } } -/// Whether one response's access-telemetry row should be emitted, combining -/// the degraded-snapshot guard with the sampling roll. -/// -/// `snapshot_sample_rate` is the rate recorded on the [`AccessTelemetrySnapshot`] -/// itself (the value serialized into the row's `sample_rate` column, which -/// the documented volume estimator divides by as `1.0 / sample_rate`). -/// `settings_sample_rate` is the rate used for the sampling decision at -/// call time. The two can diverge: when `app_state` fails to build, -/// [`edgezero_main`] captures a snapshot with `sample_rate` defaulted to -/// `0.0` before any settings ever load, but the two settings-reload -/// emission sites still gate and sample using the *reloaded* settings' -/// (nonzero) rate. Without this guard, such a row could be sampled in and -/// emitted while carrying `sample_rate: 0.0`, corrupting the volume -/// estimator. Dropping these rows is acceptable: they only occur on an -/// already-degraded path, consistent with this pipeline's fail-quiet -/// telemetry policy. Split out of [`emit_access_telemetry_after_send`] so -/// the guard is unit-testable without a network seam. -/// -/// Callers must already have applied the coarse -/// `tinybird.enabled`/`access_enabled` gate. -#[must_use] -fn should_sample_access_row( - snapshot_sample_rate: f64, - settings_sample_rate: f64, - entropy: u64, -) -> bool { - if snapshot_sample_rate <= 0.0 { - return false; - } - - tinybird::sampled_in(settings_sample_rate, entropy) -} - /// Per-response context threaded into [`send_edgezero_response`] so the /// function stays at or under seven parameters. struct SendContext { @@ -568,11 +524,12 @@ struct SendContext { } /// Outcome of handing a finalized response to the client. -#[allow(dead_code)] pub(crate) struct DeliveryOutcome { /// Response body size in bytes. pub bytes: u64, - /// Whether delivery completed or failed partway. + /// Whether delivery completed or failed partway. Collected as + /// groundwork; not yet emitted on any surface. + #[allow(dead_code)] pub result: DeliveryResult, /// Access-telemetry dimensions captured for this response at the /// freeze point. `None` when access telemetry was disabled at snapshot @@ -672,6 +629,11 @@ fn drive_streaming_body( /// A drive that failed after writing at least one byte delivered a truncated /// response rather than nothing at all, so it is [`DeliveryResult::Partial`], /// not [`DeliveryResult::Error`]. +/// +/// The `Ok(())` arm exists for the classifier's totality, not for the +/// production caller: `send_edgezero_response` consumes this value only in +/// its `Err` branch and re-derives the success outcome from +/// `streaming_body.finish()`. fn classify_stream_delivery( drive_result: &Result<(), Report>, bytes: u64, @@ -1036,7 +998,6 @@ mod tests { use edgezero_core::http::HeaderValue; use edgezero_core::http::response_builder; use fastly::mime; - use std::sync::Mutex; use std::time::Duration; use trusted_server_core::integrations::HeaderMutation; use trusted_server_core::request_timing::AuctionWaitPlacement; @@ -1760,83 +1721,14 @@ mod tests { ); } - /// Records `"telemetry"` into a shared order log instead of sending a - /// real request, standing in for the adapter's platform HTTP client in - /// [`post_send_order_is_elapsed_then_pull_sync_then_telemetry`]. - struct OrderingHttpClient { - log: Arc>>, - } - - #[async_trait::async_trait(?Send)] - impl trusted_server_core::platform::PlatformHttpClient for OrderingHttpClient { - async fn send( - &self, - _request: trusted_server_core::platform::PlatformHttpRequest, - ) -> Result< - trusted_server_core::platform::PlatformResponse, - Report, - > { - self.log - .lock() - .expect("should lock order log") - .push("telemetry"); - let response = response_builder() - .status(edgezero_core::http::StatusCode::ACCEPTED) - .body(EdgeBody::empty()) - .expect("should build ordering test response"); - Ok(trusted_server_core::platform::PlatformResponse::new( - response, - )) - } - - async fn send_async( - &self, - _request: trusted_server_core::platform::PlatformHttpRequest, - ) -> Result< - trusted_server_core::platform::PlatformPendingRequest, - Report, - > { - Err(Report::new( - trusted_server_core::platform::PlatformError::Unsupported, - )) - } - - async fn select( - &self, - _pending_requests: Vec, - ) -> Result< - trusted_server_core::platform::PlatformSelectResult, - Report, - > { - Err(Report::new( - trusted_server_core::platform::PlatformError::Unsupported, - )) - } - } - #[test] - fn post_send_order_is_elapsed_then_pull_sync_then_telemetry() { - // `edgezero_main` cannot be driven directly in a unit test (it - // consumes a live `fastly::Request::from_client()`), and - // `run_edgezero_pull_sync_after_send` has no injectable seam of its - // own — it dispatches through the real identity-graph pull-sync - // path, which needs a configured EC KV store, partner registry, and - // rate limiter wired together. This test instead exercises the two - // REAL functions `edgezero_main` calls that DO have a testable seam - // — `send_edgezero_response` (which stamps `request_elapsed` before - // returning, per Task 6/7) and `tinybird::emit_access_event` (the - // telemetry send added by this task) — around an instrumented - // stand-in for the pull-sync dispatch call, in the exact order - // `edgezero_main` places them. - // - // This proves the elapsed-before-telemetry leg from real production - // code (the assertion below reads the real `timings` snapshot - // between the two calls). The pull-sync-before-telemetry leg is a - // source-order invariant in `edgezero_main`'s three call sites - // (verified by code review, not by this test) because - // `run_edgezero_pull_sync_after_send` itself has no seam to - // instrument — see task-8-report.md for this residual. - let log: Arc>> = Arc::new(Mutex::new(Vec::new())); + fn request_elapsed_is_stamped_when_send_returns() { + // `edgezero_main`'s post-send ordering (pull-sync before telemetry) + // is a source-order invariant with no injectable seam, so this test + // deliberately proves only the leg that has one: by the time + // `send_edgezero_response` returns, `request_elapsed` is already + // stamped, so everything `edgezero_main` runs afterwards (pull-sync, + // telemetry emission) is excluded from `request_elapsed_ms`. let timings = RequestTimings::new(); let response = response_builder() .body(EdgeBody::from("ok")) @@ -1854,77 +1746,14 @@ mod tests { access_telemetry_enabled: true, }, ); - assert!( - timings.snapshot().request_elapsed_ms.is_some(), - "request_elapsed should already be stamped before pull-sync/telemetry run" - ); - - // Stand-in for `run_edgezero_pull_sync_after_send`, which has no - // injectable seam (see the test doc comment above). - log.lock().expect("should lock order log").push("pull_sync"); - let http_client = OrderingHttpClient { - log: Arc::clone(&log), - }; - let target = tinybird::TinybirdEventsTarget::from_access_config( - trusted_server_core::settings::TinybirdSettings { - api_host: "api.us-east.aws.tinybird.co".to_owned(), - ..trusted_server_core::settings::TinybirdSettings::default() - }, - ); - let snapshot = outcome - .snapshot - .as_ref() - .expect("should build a snapshot when access telemetry is enabled"); - let row = access_event_row(snapshot, &timings.snapshot(), 0); - - futures::executor::block_on(tinybird::emit_access_event(&http_client, &target, row)) - .expect("should send access telemetry"); - - assert_eq!( - *log.lock().expect("should lock order log"), - vec!["pull_sync", "telemetry"], - "pull-sync must dispatch before telemetry emits" - ); - } - - #[test] - fn should_sample_access_row_rejects_a_degraded_zero_sample_rate() { - // A snapshot captured on the app-state-build-failure fallback path - // carries `sample_rate: 0.0`. Even when the reloaded settings' rate - // would sample every request in (1.0), the row must not emit — - // otherwise it would claim `sample_rate: 0.0` and corrupt the - // `sum(1.0 / sample_rate)` volume estimator. - assert!( - !should_sample_access_row(0.0, 1.0, 0), - "a snapshot with sample_rate 0.0 must never emit, regardless of entropy or settings' rate" - ); assert!( - !should_sample_access_row(0.0, 1.0, u64::MAX), - "the degraded-rate guard must not depend on the entropy value" - ); - } - - #[test] - fn should_sample_access_row_rejects_a_negative_sample_rate() { - assert!( - !should_sample_access_row(-1.0, 1.0, 0), - "a negative snapshot sample_rate is equally degraded and must not emit" - ); - } - - #[test] - fn should_sample_access_row_defers_to_the_settings_sampling_roll_when_not_degraded() { - // With a healthy (nonzero) snapshot sample_rate, the outcome should - // match `tinybird::sampled_in` exactly, since that is the only - // remaining decision. - assert!( - should_sample_access_row(0.25, 1.0, 0), - "a settings rate of 1.0 always samples in, independent of entropy" + timings.snapshot().request_elapsed_ms.is_some(), + "request_elapsed should be stamped by the time send returns" ); assert!( - !should_sample_access_row(0.25, 0.0, 0), - "a settings rate of 0.0 always samples out, independent of the snapshot's rate" + outcome.snapshot.is_some(), + "the access snapshot should exist for the enabled context" ); } } diff --git a/crates/trusted-server-adapter-fastly/src/tinybird.rs b/crates/trusted-server-adapter-fastly/src/tinybird.rs index 1158ada68c..97a1ffd3ea 100644 --- a/crates/trusted-server-adapter-fastly/src/tinybird.rs +++ b/crates/trusted-server-adapter-fastly/src/tinybird.rs @@ -236,39 +236,26 @@ impl AuctionTelemetrySink for FastlyTinybirdAuctionTelemetrySink { // Access telemetry: confirmed-delivery emitter // --------------------------------------------------------------------------- -/// Bucket count [`sampled_in`] maps `entropy` into. -/// -/// Large enough that `rate` values with several significant digits (e.g. -/// `0.015`) still land in a distinct bucket instead of rounding away, while -/// staying well inside `u64` range once multiplied by `rate`. -const ACCESS_SAMPLE_BUCKETS: u64 = 1_000_000; - /// Decides whether one request's access-telemetry row should be emitted. /// -/// `entropy` should vary from request to request — callers derive it from -/// the wall-clock event timestamp `XORed` with a cheap per-request value (see -/// the call site in `main.rs`). There is no `rand` crate dependency here: -/// the wasm32-wasip1 guest has no equivalent to `Math.random()`. Mapping -/// `entropy % ACCESS_SAMPLE_BUCKETS` into `[0, 1)` and comparing against -/// `rate` is not cryptographically uniform (the low bits of a timestamp are -/// not perfectly evenly distributed), but access-telemetry sampling only -/// needs an approximately even sample, not a provably unbiased one. +/// `roll` is a uniform draw from `[0, 1)`; callers pass +/// `rand::thread_rng().r#gen::()`, which the wasm32-wasip1 guest backs with +/// real WASI randomness (the EC generation path already relies on this and +/// the CI wasm release build verifies it). Comparing the draw directly +/// against `rate` keeps the sampling probability exactly `rate` for every +/// positive value: there is no bucket quantization, so rates below one in a +/// million sample proportionally instead of never, and emitted rows' +/// `sample_rate` matches the probability they were sampled at, which the +/// `sum(1.0 / sample_rate)` volume estimator depends on. /// -/// `rate <= 0.0` always returns `false` and `rate >= 1.0` always returns -/// `true`, independent of `entropy`, so both boundary configurations behave -/// predictably. `0.0` cannot actually occur while `access_enabled` is `true` -/// (`Settings` validation requires `access_sample_rate > 0.0` in that case), -/// but this function stays total rather than leaning on that invariant. +/// `rate <= 0.0` never samples and `rate >= 1.0` always samples, for any +/// `roll` in `[0, 1)`. `0.0` cannot actually occur while `access_enabled` +/// is `true` (`Settings` validation requires `access_sample_rate > 0.0` in +/// that case), but this function stays total rather than leaning on that +/// invariant. #[must_use] -pub(crate) fn sampled_in(rate: f64, entropy: u64) -> bool { - if rate >= 1.0 { - return true; - } - if rate <= 0.0 { - return false; - } - let threshold = (rate * ACCESS_SAMPLE_BUCKETS as f64) as u64; - entropy % ACCESS_SAMPLE_BUCKETS < threshold +pub(crate) fn sampled_in(rate: f64, roll: f64) -> bool { + roll < rate } /// Loads and validates the access-log APPEND token from the Fastly secret store. @@ -963,20 +950,44 @@ mod tests { #[test] fn sampled_in_boundary_rates_are_unconditional() { assert!( - sampled_in(1.0, 0), + sampled_in(1.0, 0.0), "a 1.0 sample rate should always sample in" ); assert!( - sampled_in(1.0, u64::MAX), - "a 1.0 sample rate should always sample in regardless of entropy" + sampled_in(1.0, 0.999_999), + "a 1.0 sample rate should sample in for the largest roll" + ); + assert!( + !sampled_in(0.0, 0.0), + "a 0.0 sample rate should never sample in, even on a zero roll" + ); + assert!( + !sampled_in(-1.0, 0.0), + "a negative rate should never sample in" + ); + } + + #[test] + fn sampled_in_keeps_exact_probability_for_tiny_rates() { + // The previous bucket-quantized sampler truncated rates below one + // in a million to a zero threshold, silently emitting nothing. + // Direct comparison keeps every positive rate proportional. + let rate = 0.000_000_1; + assert!( + sampled_in(rate, rate / 2.0), + "a roll below a tiny positive rate should sample in" + ); + assert!( + !sampled_in(rate, rate * 2.0), + "a roll above a tiny positive rate should sample out" ); assert!( - !sampled_in(0.0, 0), - "a 0.0 sample rate should never sample in" + !sampled_in(0.000_001_9, 0.000_001_95), + "no downward quantization: the boundary sits exactly at the rate" ); assert!( - !sampled_in(0.0, u64::MAX), - "a 0.0 sample rate should never sample in regardless of entropy" + sampled_in(0.000_001_9, 0.000_001_85), + "rolls just under the rate should sample in" ); } From f3ee47381858e2030ee3e040f773d565f39be0ca Mon Sep 17 00:00:00 2001 From: Jason Evans Date: Sat, 29 Aug 2026 07:50:22 +1000 Subject: [PATCH 027/194] Drop the origin span before error-path auction telemetry On origin failure with a dispatched auction, the origin span guard stayed alive through the emit_abandoned_auction await, so ts-origin and origin_ms absorbed Tinybird emission time. The span now closes when the send resolves, before either branch, with an error-path regression test. --- crates/trusted-server-core/src/publisher.rs | 55 ++++++++++++++++++++- 1 file changed, 53 insertions(+), 2 deletions(-) diff --git a/crates/trusted-server-core/src/publisher.rs b/crates/trusted-server-core/src/publisher.rs index 53d09bfc61..4c7933795e 100644 --- a/crates/trusted-server-core/src/publisher.rs +++ b/crates/trusted-server-core/src/publisher.rs @@ -4658,8 +4658,13 @@ pub async fn handle_publisher_request( platform_request = platform_request.with_cache_bypass(); } + // The span must end when the origin responds (or fails), before any + // abandonment-telemetry await below — otherwise `ts-origin` would + // absorb Tinybird emission time on the error path. let origin_span = timings.span(Phase::Origin); - let mut response = match services.http_client().send(platform_request).await { + let origin_send_result = services.http_client().send(platform_request).await; + drop(origin_span); + let mut response = match origin_send_result { Ok(platform_response) => platform_response.response, Err(err) => { if let Some(dispatched) = dispatched_auction.take() { @@ -4676,7 +4681,6 @@ pub async fn handle_publisher_request( })); } }; - drop(origin_span); log::debug!( "Publisher origin response received: status={}, header_count={}", @@ -7851,6 +7855,53 @@ mod tests { ); } + #[tokio::test] + async fn origin_span_is_recorded_when_the_origin_send_fails() { + // Regression guard for the review finding that the origin span + // guard stayed alive through the error branch (and its + // abandonment-telemetry await): the span must close when the send + // resolves, so a failed fetch still records `origin_ms` and the + // error branch's own work is excluded from it. + let settings = create_test_settings(); + // No queued response: the stub client fails the origin send. + let stub = Arc::new(StubHttpClient::new()); + let services = + build_services_with_http_client(stub as Arc); + let mut request = HttpRequest::builder() + .method(Method::GET) + .uri("https://publisher.example/some-page") + .header(header::HOST, "publisher.example") + .body(EdgeBody::empty()) + .expect("should build request"); + let timings = RequestTimings::new(); + request.extensions_mut().insert(timings.clone()); + + let orchestrator = AuctionOrchestrator::new(settings.auction.clone()); + let mut ec_context = EcContext::read_from_request(&settings, &request, &services) + .expect("should read EC context"); + let result = handle_publisher_request( + &settings, + &services, + None, + &mut ec_context, + AuctionDispatch { + orchestrator: &orchestrator, + slots: &[], + registry: None, + }, + request, + EdgeCacheHeader::SurrogateControl, + ) + .await; + + assert!(result.is_err(), "should surface the origin failure"); + timings.mark_headers_ready(); + assert!( + timings.snapshot().origin_ms.is_some(), + "should record the Origin span even when the origin send fails" + ); + } + mod rendered_template_identity_tests { //! The gate the plan's Task 3 Step 2 actually asks for. //! From 46b3c7f99ab6bf9335d20e17922f0ddeb5866690 Mon Sep 17 00:00:00 2001 From: Jason Evans Date: Sat, 29 Aug 2026 07:50:22 +1000 Subject: [PATCH 028/194] Address remaining review feedback on timing surfaces and docs - Pin HEADER_PHASES against Phase::header_name() in the phase-index test, closing the second hand-synced list. - Add RouteClass::Asset to the snake_case rendering test; rename the lowercasing test to say what it does. - Give the Axum adapter a named, fully configured construction path (TrustedServerApp::dev_server_service) so server_timing_enabled is never silently discarded; the tuple API is private now. - Document that Server-Timing is client-visible when enabled, in the configuration guide's observability section. - Replace stale event_date references in the spec, plan, and dashboard guidance with the toDate(event_ts) sorting-key expression, and state the single-segment rejection rule in spec section 9. --- crates/trusted-server-adapter-axum/src/app.rs | 19 ++++++++++++++++++- .../trusted-server-adapter-axum/src/main.rs | 14 +++++--------- .../trusted-server-core/src/request_timing.rs | 13 +++++++++++++ docs/guide/configuration.md | 13 +++++++++++++ .../plans/2026-08-24-request-phase-timing.md | 5 +++-- .../2026-08-24-request-phase-timing-design.md | 16 ++++++++++------ 6 files changed, 62 insertions(+), 18 deletions(-) diff --git a/crates/trusted-server-adapter-axum/src/app.rs b/crates/trusted-server-adapter-axum/src/app.rs index 4bdf27d011..aeda26a848 100644 --- a/crates/trusted-server-adapter-axum/src/app.rs +++ b/crates/trusted-server-adapter-axum/src/app.rs @@ -1,6 +1,7 @@ use core::future::Future; use std::sync::Arc; +use edgezero_adapter_axum::service::EdgeZeroAxumService; use edgezero_core::app::Hooks; use edgezero_core::context::RequestContext; use edgezero_core::error::EdgeError; @@ -587,6 +588,22 @@ impl TrustedServerApp { Ok(build_router(&state)) } + /// The dev server's fully configured tower service: the application + /// router wrapped in the terminal timing layer + /// ([`crate::timing::TimingService`]), with `server_timing_enabled` + /// read from the same settings snapshot that built the router. + /// + /// This is the standard construction path for serving this adapter. + /// [`Hooks::routes`] satisfies the `Hooks` trait contract and returns + /// the bare router without the timing layer; callers who serve traffic + /// should use this instead so `server_timing_enabled` is never + /// silently discarded. + #[must_use] + pub fn dev_server_service() -> crate::timing::TimingService { + let (router, server_timing_enabled) = Self::routes_with_server_timing_flag(); + crate::timing::TimingService::new(EdgeZeroAxumService::new(router), server_timing_enabled) + } + /// Build the router alongside whether `Server-Timing` emission is /// enabled, read from the same settings snapshot used to build the /// router. @@ -596,7 +613,7 @@ impl TrustedServerApp { /// `Settings` per request, the Axum dev server builds its application /// state once and reuses the same [`RouterService`] for every request. #[must_use] - pub fn routes_with_server_timing_flag() -> (RouterService, bool) { + fn routes_with_server_timing_flag() -> (RouterService, bool) { let state = match build_state() { Ok(s) => s, Err(ref e) => { diff --git a/crates/trusted-server-adapter-axum/src/main.rs b/crates/trusted-server-adapter-axum/src/main.rs index b8bc28ae21..4e360ea417 100644 --- a/crates/trusted-server-adapter-axum/src/main.rs +++ b/crates/trusted-server-adapter-axum/src/main.rs @@ -3,7 +3,6 @@ use std::net::SocketAddr; use axum::Router; use edgezero_adapter_axum::dev_server::AxumDevServerConfig; use edgezero_adapter_axum::service::EdgeZeroAxumService; -use edgezero_core::router::RouterService; use tokio::net::TcpListener; use tokio::runtime::Builder as RuntimeBuilder; use tokio::signal; @@ -30,8 +29,8 @@ fn main() { }; log::info!("Listening on http://{}", config.addr); - let (router, server_timing_enabled) = TrustedServerApp::routes_with_server_timing_flag(); - if let Err(err) = run(router, server_timing_enabled, config) { + let service = TrustedServerApp::dev_server_service(); + if let Err(err) = run(service, config) { log::error!("trusted-server-adapter-axum failed: {err}"); std::process::exit(1); } @@ -56,22 +55,19 @@ fn main() { /// Returns an error if the Tokio runtime fails to start, the listener fails /// to bind, or the underlying serve loop errors. fn run( - router: RouterService, - server_timing_enabled: bool, + service: TimingService, config: AxumDevServerConfig, ) -> std::io::Result<()> { let runtime = RuntimeBuilder::new_multi_thread().enable_all().build()?; - runtime.block_on(serve(router, server_timing_enabled, config)) + runtime.block_on(serve(service, config)) } async fn serve( - router: RouterService, - server_timing_enabled: bool, + service: TimingService, config: AxumDevServerConfig, ) -> std::io::Result<()> { let listener = TcpListener::bind(config.addr).await?; - let service = TimingService::new(EdgeZeroAxumService::new(router), server_timing_enabled); let axum_router = Router::new().fallback_service(service_fn(move |req| { let mut svc = service.clone(); async move { svc.call(req).await } diff --git a/crates/trusted-server-core/src/request_timing.rs b/crates/trusted-server-core/src/request_timing.rs index 8d1b92cc1c..cbb269b1eb 100644 --- a/crates/trusted-server-core/src/request_timing.rs +++ b/crates/trusted-server-core/src/request_timing.rs @@ -417,6 +417,19 @@ mod tests { seen.iter().all(|slot| *slot), "should cover every phases-array slot" ); + + // `HEADER_PHASES` is a second hand-synced list: a phase that gains + // a `header_name()` but is never added there silently renders + // nothing in the `Server-Timing` value. + let header_bearing: Vec = phases + .into_iter() + .filter(|phase| phase.header_name().is_some()) + .collect(); + assert_eq!( + header_bearing, + HEADER_PHASES.to_vec(), + "should render every header-bearing phase, in declaration order" + ); } #[test] diff --git a/docs/guide/configuration.md b/docs/guide/configuration.md index 675aaf40ac..729801ecb0 100644 --- a/docs/guide/configuration.md +++ b/docs/guide/configuration.md @@ -1846,6 +1846,19 @@ send access-telemetry rows. server_timing_enabled = true ``` +::: warning Client-visible latency disclosure +The `Server-Timing` header is sent to every client on eligible responses, +not only to operators: browsers expose the values to same-origin JavaScript +via `PerformanceResourceTiming.serverTiming`, and any caller can read the +raw header. Enabling it publishes measured per-phase server latency, +including KV read timing on the public identity endpoints (`ts-kv`) and +origin/cache behaviour on publisher pages (`ts-origin`, +`ts-template-cache`). This is standard `Server-Timing` practice and the +values are durations only, but treat the flag as a diagnostic aid to enable +deliberately, not a general always-on toggle, unless disclosing those +timings to all clients is acceptable for the deployment. +::: + **Environment Override**: ```bash diff --git a/docs/superpowers/plans/2026-08-24-request-phase-timing.md b/docs/superpowers/plans/2026-08-24-request-phase-timing.md index 9ff3905f42..02cda64d82 100644 --- a/docs/superpowers/plans/2026-08-24-request-phase-timing.md +++ b/docs/superpowers/plans/2026-08-24-request-phase-timing.md @@ -928,10 +928,11 @@ git commit -m "Emit confirmed access telemetry rows after pull-sync post-send" - [ ] **Step 1: Rewrite the schema** per spec section 9: keep `event_ts DateTime64(3)`, `method`, `status UInt16`, `time_elapsed_ms UInt32`, - `sample_rate Float64`, `event_date` + 30-day TTL; add the columns from spec 9 with + `sample_rate Float64` + 30-day TTL; add the columns from spec 9 with dimension columns non-nullable `LowCardinality(String)` and phase columns `Nullable(UInt32)`; drop `path` and `cache_state`; set - `ENGINE_SORTING_KEY "event_date, service_id, publisher_domain, env, route_class, pop, status"`. + `ENGINE_SORTING_KEY "toDate(event_ts), service_id, publisher_domain, env, route_class, pop, status"` + (`event_date` was dropped for the sorting-key expression; see spec section 9). - [ ] **Step 2: Validate** with the tinybird toolchain if available locally (`tb check` / project tests under `tinybird/tests`); otherwise assert the file diff --git a/docs/superpowers/specs/2026-08-24-request-phase-timing-design.md b/docs/superpowers/specs/2026-08-24-request-phase-timing-design.md index 214d146a1a..2892386f42 100644 --- a/docs/superpowers/specs/2026-08-24-request-phase-timing-design.md +++ b/docs/superpowers/specs/2026-08-24-request-phase-timing-design.md @@ -276,7 +276,8 @@ Extends the reserved `tinybird/datasources/access_logs_raw.datasource`. Kept columns: `event_ts`, `method`, `status`, `time_elapsed_ms` (defined as the `mark_headers_ready()` snapshot; nullable because a contended lock drop can lose the -snapshot), `sample_rate`, `event_date`, 30-day TTL. +snapshot), `sample_rate`, 30-day TTL. (`event_date` was later dropped for the +`toDate(event_ts)` sorting-key expression; see section 9's schema note.) Removed: raw `path`. Route identifiers like `/_ts/admin/ec/{id}` would otherwise put EC identifiers into a 30-day dataset, and publisher paths carry unbounded cardinality @@ -290,8 +291,11 @@ and user-generated content (search terms, usernames, emails in slugs). Replaced `/news/*`). The auction-telemetry normalizer is explicitly not sufficient here: it redacts long tokens but preserves short identifiers and arbitrary slugs. - Rejection is whole-segment, never truncation: a segment is dropped to `/other/*` - when it fails the charset allowlist, exceeds 32 characters, or carries more than 7 - ASCII digits. The character allowlist alone does not bound identity (`[a-z0-9_-]` + when it fails the charset allowlist, exceeds 32 characters, carries more than 7 + ASCII digits, or is the only segment in the path. Depth is what makes a first + segment a section name: single-segment paths are documents (WordPress + `/%postname%/` puts every article at depth 1), so they reject wholesale, root + landing pages included. The character allowlist alone does not bound identity (`[a-z0-9_-]` is exactly the alphabet of UUIDs, hex ids, and reset tokens), and a truncated prefix of any of those is still identifying, so the length and digit bounds reject the segment outright. @@ -352,7 +356,7 @@ pop, status)`. Every column carries a `json:$.` path (the Events API rejec NDJSON into a datasource without JSONPaths, discovered live); `event_date` was dropped in favor of the sorting-key expression because a DEFAULT column cannot carry a JSONPath the producer never sends. Grafana time filtering uses `$__timeFilter(event_ts)` and every panel query -also carries an `event_date` predicate so the primary index prunes; rollout validates +also carries a `toDate(event_ts)` predicate so the primary index prunes; rollout validates the panel queries with `EXPLAIN` before the dashboard is committed. This replaces the reserved key `(event_date, path, status, method)`. Rollout step 4 verifies whether the reserved datasource was ever deployed to the remote workspace; if it was, this schema @@ -409,8 +413,8 @@ ships as a versioned replacement datasource with a cutover, not an in-place edit ## 11. Dashboard and query model No endpoint pipe in v1. Grafana queries `access_logs_raw` directly through the -ClickHouse connector with `$__timeFilter(event_ts)` plus an `event_date` predicate, -matching the auction dashboards. +ClickHouse connector with `$__timeFilter(event_ts)` plus a `toDate(event_ts)` +predicate, matching the auction dashboards. Dashboard: a new standalone `grafana/dashboards/edge-performance.json` in the telemetry repo (`trusted-server-tinybird`), performance only, no panels shared with From c6235b91fcd247a57d27fa8703734340d85d0c22 Mon Sep 17 00:00:00 2001 From: Christian Date: Mon, 31 Aug 2026 14:09:26 -0500 Subject: [PATCH 029/194] Enforce access telemetry body limit The access emitter carried the configured body limit without enforcing it, allowing oversized rows to bypass the intended transport safeguard. --- .../src/tinybird.rs | 42 +++++++++++++++++-- 1 file changed, 39 insertions(+), 3 deletions(-) diff --git a/crates/trusted-server-adapter-fastly/src/tinybird.rs b/crates/trusted-server-adapter-fastly/src/tinybird.rs index 97a1ffd3ea..84dfc37d8a 100644 --- a/crates/trusted-server-adapter-fastly/src/tinybird.rs +++ b/crates/trusted-server-adapter-fastly/src/tinybird.rs @@ -319,14 +319,25 @@ fn build_access_events_request( /// /// # Errors /// -/// Returns `Err` when the access-log APPEND token cannot be loaded, the -/// backend cannot be registered, the request cannot be built or sent, or the -/// Tinybird Events API responds with a non-2xx status. +/// Returns `Err` when the row exceeds the configured request-body limit, the +/// access-log APPEND token cannot be loaded, the backend cannot be registered, +/// the request cannot be built or sent, or the Tinybird Events API responds +/// with a non-2xx status. pub(crate) async fn emit_access_event( client: &dyn PlatformHttpClient, target: &TinybirdEventsTarget, row: String, ) -> Result<(), Report> { + let body_len = row.len(); + if body_len > target.max_body_bytes { + return Err(Report::new(TrustedServerError::Proxy { + message: format!( + "Tinybird access telemetry request body has {body_len} bytes, exceeding {} byte limit", + target.max_body_bytes + ), + })); + } + let token = load_access_token(target)?; let auth_header = FastlyTinybirdAuctionTelemetrySink::authorization_header(&token)?; let backend_name = FastlyPlatformBackend @@ -884,6 +895,31 @@ mod tests { ); } + #[test] + fn access_emitter_rejects_oversized_row_before_sending() { + let mut config = enabled_config(); + config.max_body_bytes = 1024; + let target = TinybirdEventsTarget::from_access_config(config); + let http_client = RecordingHttpClient::respond_with(202); + let row = "x".repeat(1025); + + let result = futures::executor::block_on(emit_access_event(&http_client, &target, row)); + + let error = result.expect_err("should reject a row above the configured body limit"); + assert!( + error.to_string().contains("1024"), + "error should name the configured body limit: {error}" + ); + assert!( + http_client + .requests + .lock() + .expect("should lock recorded requests") + .is_empty(), + "should not send an oversized access row" + ); + } + #[test] fn access_emitter_posts_ndjson_and_validates_2xx() { // `ts_secrets`/`tinybird_access_append_token` is seeded in From ca46e4f7393826f0e33725299bb08f8bc57ad998 Mon Sep 17 00:00:00 2001 From: Christian Date: Fri, 4 Sep 2026 12:38:55 -0500 Subject: [PATCH 030/194] Format auction timeline implementation plan --- .../superpowers/plans/2026-08-26-auction-timeline-offsets.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/docs/superpowers/plans/2026-08-26-auction-timeline-offsets.md b/docs/superpowers/plans/2026-08-26-auction-timeline-offsets.md index 08e14dabc2..6dfff6c8cc 100644 --- a/docs/superpowers/plans/2026-08-26-auction-timeline-offsets.md +++ b/docs/superpowers/plans/2026-08-26-auction-timeline-offsets.md @@ -23,9 +23,11 @@ ### Task 1: RequestTimings marks and snapshot fields **Files:** + - Modify: `crates/trusted-server-core/src/request_timing.rs` **Interfaces:** + - Produces: `mark_auction_dispatched(&self, auction_id: String)`, `mark_auction_resolved(&self)`, `mark_auction_committed(&self)`; `TimingSnapshot { auction_dispatched_ms, auction_resolved_ms, auction_committed_ms: Option, auction_id: Option, .. }` - [ ] Add `auction_dispatched`, `auction_resolved`, `auction_committed: Option` and `auction_id: Option` to `Inner`; initialize `None`. @@ -37,9 +39,11 @@ ### Task 2: Publisher call sites **Files:** + - Modify: `crates/trusted-server-core/src/publisher.rs` **Interfaces:** + - Consumes: Task 1 methods; `observation.auction_id` (`AuctionObservationContext`), in scope at the dispatch site. - [ ] In the `DispatchAuctionOutcome::Dispatched` arm (~line 4341): `timings.mark_auction_dispatched(observation.auction_id.to_string());` @@ -50,6 +54,7 @@ ### Task 3: Row columns and datasource **Files:** + - Modify: `crates/trusted-server-core/src/access_telemetry.rs` - Modify: `tinybird/datasources/access_logs_raw.datasource` From bcc1475f7d458addd2c21745f85456302a9c8ed3 Mon Sep 17 00:00:00 2001 From: Jason Evans Date: Tue, 8 Sep 2026 13:55:24 -0400 Subject: [PATCH 031/194] Bound route templates by allowlist and harden the telemetry config surface Review round 3, the blocking route finding plus the config items: - Publisher route templates now come from an operator allowlist (observability.route_sections, default empty): a first segment that matches an entry and has further depth emits the lowercased allowlist entry as /{section}/*; everything else emits /other/*. The shape heuristics (charset, length, digit bounds) are gone because they could not bound identity: depth-2 first segments are usernames under /{username}/posts shapes and single-segment paths are documents. The emitted value set is now fixed by configuration, so no request-derived byte reaches the row. - Integration-proxy responses carry the registered route pattern verbatim (bounded, integration-defined) instead of a classifier output; the registry stores the pattern at registration. - auction_enabled serializes only when false, so a pushed config cannot silently re-enable auction telemetry on rollback; with a serialization test alongside the observability one. - The secret-store validator is renamed to validate_secret_store_key_name with a key_name parameter: it validates an identifier, never a credential, and the old name tainted the key name as a secret value in CodeQL, lighting up eleven pre-existing log sites. - Docs: the tinybird table gains its three missing rows, max_body_bytes states the 1024 floor the code enforces, the rollback guidance now describes the real compatibility boundary (push a compat config first: drop [observability], access_enabled = false, and enabled = false for access-only deployments), and the fixture uses the example-domain convention. Co-Authored-By: Claude Fable 5 --- .../trusted-server-adapter-fastly/src/app.rs | 41 ++- .../src/access_telemetry.rs | 248 +++++++----------- .../src/integrations/registry.rs | 35 ++- crates/trusted-server-core/src/settings.rs | 83 +++++- docs/guide/configuration.md | 51 ++-- .../2026-08-24-request-phase-timing-design.md | 33 ++- tinybird/fixtures/access_logs_raw.ndjson | 2 +- 7 files changed, 289 insertions(+), 204 deletions(-) diff --git a/crates/trusted-server-adapter-fastly/src/app.rs b/crates/trusted-server-adapter-fastly/src/app.rs index a12484ec54..b2221de8d7 100644 --- a/crates/trusted-server-adapter-fastly/src/app.rs +++ b/crates/trusted-server-adapter-fastly/src/app.rs @@ -881,9 +881,16 @@ async fn dispatch_fallback( // Integration-proxy responses are not bounded by // publisher.max_buffered_body_bytes. Publisher fallback below uses the // publisher-specific streaming finalizer instead. + // The matched route pattern is an integration-defined literal + // (bounded and content-free by construction), so telemetry keeps it + // verbatim instead of running the request path through the lossy + // publisher classifier. route_metadata = Some(RouteMetadata { route_class: RouteClass::IntegrationProxy, - route_template: publisher_route_template(&path), + route_template: state + .registry + .matched_route_pattern(&method, &path) + .map_or_else(|| "/other/*".to_owned(), str::to_owned), }); state .registry @@ -934,7 +941,10 @@ async fn dispatch_fallback( route_metadata = Some(RouteMetadata { route_class: RouteClass::PublisherHtml, - route_template: publisher_route_template(&path), + route_template: publisher_route_template( + &path, + &state.settings.observability.route_sections, + ), }); // Generate an EC ID if needed — mirrors the legacy catch-all arm. @@ -1517,7 +1527,7 @@ mod tests { NamedRouteHandler, PAGE_BIDS_LEGACY_PATH, PAGE_BIDS_PATH, RouteClass, RouteMetadata, TSJS_ROUTE_TEMPLATE, TrustedServerApp, build_per_request_services, build_state_from_settings, handle_publisher_request, - publisher_response_into_streaming_response, publisher_route_template, startup_error_router, + publisher_response_into_streaming_response, startup_error_router, }; use base64::Engine as _; use bytes::Bytes; @@ -2451,8 +2461,8 @@ mod tests { .expect("integration-proxy fallback responses should carry RouteMetadata"); assert_eq!(metadata.route_class, RouteClass::IntegrationProxy); assert_eq!( - metadata.route_template, - publisher_route_template("/integrations/prebid/bundle.js") + metadata.route_template, "/integrations/prebid/bundle.js", + "should carry the registered route pattern verbatim, not a classifier output" ); } @@ -2466,7 +2476,10 @@ mod tests { .get::() .expect("publisher fallback responses should carry RouteMetadata"); assert_eq!(metadata.route_class, RouteClass::PublisherHtml); - assert_eq!(metadata.route_template, "/news/*"); + assert_eq!( + metadata.route_template, "/other/*", + "the default empty section allowlist should collapse publisher paths" + ); } #[test] @@ -3853,6 +3866,22 @@ mod tests { ); } + #[test] + fn server_timing_absent_when_no_cache_control_header_exists() { + // The fail-closed case: absence of Cache-Control is not evidence of + // privacy, so emission must be suppressed rather than defaulted on. + let mut response = response_builder() + .body(Body::empty()) + .expect("should build a response with no cache-control header"); + + crate::apply_server_timing_header(&mut response, &RequestTimings::new(), true); + + assert!( + response_header(&response, "server-timing").is_none(), + "should not emit when the response carries no Cache-Control at all" + ); + } + #[test] fn preexisting_server_timing_values_survive() { let mut response = response_builder() diff --git a/crates/trusted-server-core/src/access_telemetry.rs b/crates/trusted-server-core/src/access_telemetry.rs index aaad466336..e468b109c6 100644 --- a/crates/trusted-server-core/src/access_telemetry.rs +++ b/crates/trusted-server-core/src/access_telemetry.rs @@ -12,20 +12,6 @@ use serde_json::json; use crate::request_timing::{AuctionWaitPlacement, TimingSnapshot}; -/// Maximum length of a publisher path's first segment before -/// [`publisher_route_template`] rejects it to `/other/*`. Longer segments -/// are opaque-identifier or slug shaped (a UUID is 36 characters), and a -/// truncated prefix of either would still be identifying, so the segment -/// is rejected whole rather than truncated. -const MAX_SEGMENT_LEN: usize = 32; - -/// Maximum number of ASCII digits in a publisher path's first segment -/// before [`publisher_route_template`] rejects it to `/other/*`. Hex ids, -/// base36 ids, and reset tokens are digit-heavy; real section names carry -/// at most a year (`2026`) or a small version number, so a segment with -/// more digits than this is treated as an identifier, not a name. -const MAX_SEGMENT_DIGITS: usize = 7; - /// Normalizes an HTTP method token into the bounded set of values stored in /// the `method` `LowCardinality` column. /// @@ -127,46 +113,43 @@ pub struct RouteMetadata { } /// Normalizes a publisher-fallback request path into a bounded, -/// content-free route template. -/// -/// Returns `/` plus the first path segment, lowercased and restricted to -/// `[a-z0-9_-]`, plus `/*`, only when the path has at least two segments: -/// depth is what makes the first segment a section name (`/news/*`) rather -/// than the document itself. The root path `/` maps to itself. Everything -/// else is rejected to `/other/*` — outright, never filtered or truncated, -/// so no fragment of a rejected path ever reaches the row: +/// content-free route template using an operator-configured allowlist of +/// section names. /// -/// - single-segment paths (`/my-post-title` under a `/%postname%/` -/// permalink structure is a per-article slug that no shape heuristic -/// can separate from a section name); -/// - an empty first segment, or one containing any character outside the -/// allowlist after lowercasing (an email address, a search phrase); -/// - a first segment longer than [`MAX_SEGMENT_LEN`] characters (a -/// truncated prefix of a UUID or token would still be identifying); or -/// - a first segment with more than [`MAX_SEGMENT_DIGITS`] ASCII digits -/// (hex ids, base36 ids, and reset tokens are digit-heavy; section -/// names carry at most a year or a version number). +/// The root path `/` maps to itself. A path whose first segment matches an +/// entry in `sections` (ASCII case-insensitive) and that has at least one +/// further segment maps to `/{section}/*`, emitting the lowercased +/// allowlist entry rather than anything taken from the request. Every +/// other path maps to `/other/*`. /// -/// This is deliberately coarser than the auction-telemetry path -/// normalizer, which redacts long tokens but preserves short identifiers -/// and arbitrary slugs; that normalizer is not sufficient for a dataset -/// this broad. +/// This construction makes the template content-free by definition: the +/// set of emitted values is exactly `{"/", "/other/*"}` plus one +/// `/{section}/*` per configured entry, so no request-derived byte ever +/// reaches the row and cardinality is bounded by operator configuration. +/// With the default empty allowlist, every publisher path collapses to +/// `/other/*`. Depth alone was rejected as a signal in review: under +/// `/{username}/posts` shapes the first segment is user data, and under +/// single-segment permalink structures it is the document itself. /// /// # Examples /// /// ``` /// use trusted_server_core::access_telemetry::publisher_route_template; /// -/// assert_eq!(publisher_route_template("/news/some-article-slug"), "/news/*"); -/// assert_eq!(publisher_route_template("/"), "/"); -/// assert_eq!(publisher_route_template("/user@example.com/profile"), "/other/*"); +/// let sections = vec!["news".to_owned()]; +/// assert_eq!( +/// publisher_route_template("/news/some-article-slug", §ions), +/// "/news/*" +/// ); +/// assert_eq!(publisher_route_template("/", §ions), "/"); /// assert_eq!( -/// publisher_route_template("/550e8400-e29b-41d4-a716-446655440000"), +/// publisher_route_template("/alice/orders", §ions), /// "/other/*" /// ); +/// assert_eq!(publisher_route_template("/news/x", &[]), "/other/*"); /// ``` #[must_use] -pub fn publisher_route_template(path: &str) -> String { +pub fn publisher_route_template(path: &str, sections: &[String]) -> String { if path == "/" { return "/".to_owned(); } @@ -178,31 +161,18 @@ pub fn publisher_route_template(path: &str) -> String { }; let has_more_depth = !rest.is_empty(); - let lowered = first_segment.to_ascii_lowercase(); - let is_allowlisted = !lowered.is_empty() - && lowered - .chars() - .all(|c| c.is_ascii_lowercase() || c.is_ascii_digit() || c == '_' || c == '-'); - let within_length = lowered.chars().count() <= MAX_SEGMENT_LEN; - let digit_count = lowered.chars().filter(char::is_ascii_digit).count(); - - if !is_allowlisted || !within_length || digit_count > MAX_SEGMENT_DIGITS { - return "/other/*".to_owned(); - } - - if has_more_depth { - format!("/{lowered}/*") - } else { - // A single-segment path's first segment is the document, not a - // section: `/my-post-title` under WordPress `/%postname%/` is a - // per-article slug, and no shape heuristic can separate it from a - // section name. Only depth >= 2 makes the first segment a section. - "/other/*".to_owned() + let matched_section = sections + .iter() + .find(|section| section.eq_ignore_ascii_case(first_segment)); + match matched_section { + Some(section) if has_more_depth => format!("/{}/*", section.to_ascii_lowercase()), + _ => "/other/*".to_owned(), } } /// A point-in-time view of the access-log dimensions for one response, -/// captured unconditionally at the `Server-Timing` freeze point. +/// captured at the `Server-Timing` freeze point when access telemetry is +/// enabled (a disabled deployment skips the build entirely). /// /// Built from typed response extensions ([`RouteMetadata`], the geo /// lookup state, and the template-cache response state) plus adapter-owned @@ -358,120 +328,84 @@ mod tests { } #[test] - fn publisher_paths_normalize_to_coarse_templates() { + fn publisher_route_template_emits_only_allowlisted_sections() { + let sections = vec!["news".to_owned(), "Sports".to_owned()]; assert_eq!( - publisher_route_template("/news/some-article-slug"), + publisher_route_template("/news/some-article-slug", §ions), "/news/*" ); - assert_eq!(publisher_route_template("/"), "/"); - assert_eq!( - publisher_route_template("/user@example.com/profile"), - "/other/*", - "should reject non-allowlisted characters" - ); - assert_eq!( - publisher_route_template(&format!("/{}", "a".repeat(500))), - "/other/*", - "should reject overlong segments whole rather than truncate" - ); - assert_eq!(publisher_route_template("/search terms here"), "/other/*"); - } - - #[test] - fn publisher_route_template_rejects_opaque_identifier_segments() { - // Every row here passes the character allowlist (`[a-z0-9_-]` is - // exactly what UUIDs, hex ids, and tokens are built from) and must - // be caught by the length and digit-count bounds instead. A - // truncated prefix of any of these would still be identifying, so - // rejection must be whole-segment. - assert_eq!( - publisher_route_template("/550e8400-e29b-41d4-a716-446655440000"), - "/other/*", - "should reject a UUID (36 chars) by length" - ); - assert_eq!( - publisher_route_template("/550e8400-e29b-41d4-a716-446655440000/profile"), - "/other/*", - "should reject a UUID first segment on deeper paths too" - ); - assert_eq!( - publisher_route_template("/8f3a9c2b1d4e5f6a7b8c9d0e1f2a3b4c"), - "/other/*", - "should reject a 32-char hex id by digit count" - ); assert_eq!( - publisher_route_template(&format!("/{}", "a1".repeat(32))), - "/other/*", - "should reject a 64-char token by length" + publisher_route_template("/NEWS/some-article-slug", §ions), + "/news/*", + "matching should be case-insensitive on the request side" ); assert_eq!( - publisher_route_template("/reset-password-token-9f2b1c7d4e8a"), - "/other/*", - "should reject a reset token by length" + publisher_route_template("/sports/scores/today", §ions), + "/sports/*", + "the emitted value should be the lowercased allowlist entry" ); + assert_eq!(publisher_route_template("/", §ions), "/"); assert_eq!( - publisher_route_template("/how-to-treat-my-recent-hiv-diagnosis"), + publisher_route_template("/news", §ions), "/other/*", - "should reject a full article slug by length" + "an allowlisted section with no further depth is the document, not a section" ); - } - - #[test] - fn publisher_route_template_keeps_digit_light_section_names() { - assert_eq!( - publisher_route_template("/2026/08/some-article"), - "/2026/*", - "a year archive segment should pass the digit bound" - ); - assert_eq!( - publisher_route_template("/wp-content/themes/site/app.css"), - "/wp-content/*", - "a hyphenated section name should pass" - ); - } - - #[test] - fn publisher_route_template_rejects_empty_first_segment() { assert_eq!( - publisher_route_template("//double-slash"), + publisher_route_template("/opinion/some-slug", §ions), "/other/*", - "an empty first segment should not be treated as allowlisted" + "a section absent from the allowlist should collapse" ); } #[test] - fn publisher_route_template_rejects_single_segment_paths() { - // A single-segment path's first segment is the document itself - // (WordPress `/%postname%/` puts every article at depth 1), so no - // shape heuristic can separate a slug from a section name; depth - // is the only safe signal. Root-level landing pages pay for this - // deliberately. - assert_eq!(publisher_route_template("/my-post-title"), "/other/*"); - assert_eq!( - publisher_route_template("/my-post-title/"), - "/other/*", - "a trailing slash should not count as depth" - ); - assert_eq!( - publisher_route_template("/1234567"), - "/other/*", - "a numeric post id at the digit boundary should still reject" - ); - assert_eq!(publisher_route_template("/user-8f3a9c2b"), "/other/*"); - assert_eq!( - publisher_route_template("/about"), - "/other/*", - "root-level landing pages reject too; only depth makes a section" - ); + fn publisher_route_template_collapses_everything_by_default() { + // The default allowlist is empty, so no request-derived byte can + // reach the row: the only emitted values are `/` and `/other/*`. + let empty: Vec = Vec::new(); + for path in [ + "/news/some-article-slug", + "/alice/orders", + "/jane-doe/posts", + "/johnsmith1985/settings", + "/how-to-treat-my-hiv-today/comments", + "/abcdefabcdefabcdefabcdefabcdefab/x", + "/8f3a9c2b/x", + "/my-post-title", + "/user@example.com/profile", + "/search terms here", + "//double-slash", + "/550e8400-e29b-41d4-a716-446655440000/profile", + ] { + assert_eq!( + publisher_route_template(path, &empty), + "/other/*", + "should collapse with an empty allowlist: {path}" + ); + } + assert_eq!(publisher_route_template("/", &empty), "/"); } #[test] - fn publisher_route_template_lowercases_before_allowlisting() { - assert_eq!( - publisher_route_template("/News/Article"), - "/news/*", - "should lowercase before validating" - ); + fn publisher_route_template_never_emits_user_data_even_when_allowlisted() { + // Adversarial depth-2 shapes from review: even with sections + // configured, a first segment that is not an exact allowlist match + // collapses, and the emitted value on a match is the allowlist + // entry itself, never request bytes. + let sections = vec!["news".to_owned()]; + for path in [ + "/jane-doe/posts", + "/johnsmith1985/settings", + "/how-to-treat-my-hiv-today/comments", + "/abcdefabcdefabcdefabcdefabcdefab/x", + "/8f3a9c2b/x", + "/newsy/article", + ] { + assert_eq!( + publisher_route_template(path, §ions), + "/other/*", + "should collapse non-allowlisted first segments: {path}" + ); + } } #[test] diff --git a/crates/trusted-server-core/src/integrations/registry.rs b/crates/trusted-server-core/src/integrations/registry.rs index 1acb6fdb6d..6d65afef3a 100644 --- a/crates/trusted-server-core/src/integrations/registry.rs +++ b/crates/trusted-server-core/src/integrations/registry.rs @@ -688,7 +688,17 @@ impl IntegrationRegistrationBuilder { } } -type RouteValue = (Arc, &'static str); +/// Proxy handler, integration id, and the registered route pattern (kept so +/// telemetry can label responses with the integration-defined literal, e.g. +/// `/integrations/prebid/*`, instead of deriving anything from the request +/// path). +type RouteValue = (Arc, &'static str, String); + +/// A test-constructor route entry: method, path pattern, and the proxy with +/// its integration id ([`IntegrationRegistry::from_routes`] fills the +/// pattern into [`RouteValue`] itself). +#[cfg(test)] +type RouteEntry<'a> = (Method, &'a str, (Arc, &'static str)); struct IntegrationRegistryInner { // Method-specific routers for O(log n) lookups @@ -805,7 +815,11 @@ impl IntegrationRegistry { for proxy in registration.proxies { for route in proxy.routes() { - let value = (proxy.clone(), registration.integration_id); + let value = ( + proxy.clone(), + registration.integration_id, + route.path.clone(), + ); // Convert /* wildcard to matchit's {*rest} syntax let matchit_path = if route.path.ends_with("/*") { @@ -894,6 +908,16 @@ impl IntegrationRegistry { self.find_route(method, path).is_some() } + /// The registered route pattern matched by `method` and `path`, if any. + /// + /// Patterns are integration-defined literals (for example + /// `/integrations/prebid/*`), so they are bounded and content-free and + /// safe to store as a telemetry dimension, unlike the request path. + #[must_use] + pub fn matched_route_pattern(&self, method: &Method, path: &str) -> Option<&str> { + self.find_route(method, path).map(|value| value.2.as_str()) + } + /// Return true when at least one integration request filter is /// registered. /// @@ -980,7 +1004,7 @@ impl IntegrationRegistry { services, mut req, } = input; - if let Some((proxy, _)) = self.find_route(method, path) { + if let Some((proxy, _, _)) = self.find_route(method, path) { // Organic proxy handler: generate if needed (best effort). // Only generate for document navigations — subresource requests // may lack consent signals such as the Sec-GPC header. @@ -1296,7 +1320,7 @@ impl IntegrationRegistry { /// # Panics /// /// Panics if route registration fails due to duplicate or invalid paths. - pub fn from_routes(routes: Vec<(Method, &str, RouteValue)>) -> Self { + pub fn from_routes(routes: Vec>) -> Self { let mut get_router = Router::new(); let mut post_router = Router::new(); let mut put_router = Router::new(); @@ -1305,7 +1329,8 @@ impl IntegrationRegistry { let mut head_router = Router::new(); let mut options_router = Router::new(); - for (method, path, value) in routes { + for (method, path, (proxy, integration_id)) in routes { + let value: RouteValue = (proxy, integration_id, path.to_owned()); // Convert /* wildcard to matchit's {*rest} syntax let matchit_path = if path.ends_with("/*") { format!( diff --git a/crates/trusted-server-core/src/settings.rs b/crates/trusted-server-core/src/settings.rs index 216a7fd7ac..19951b8d4b 100644 --- a/crates/trusted-server-core/src/settings.rs +++ b/crates/trusted-server-core/src/settings.rs @@ -1795,7 +1795,11 @@ pub struct TinybirdSettings { /// configs preserve their current auction-emission behavior after /// upgrading; set `false` to silence auction events while keeping /// `enabled` on for other Tinybird telemetry (e.g. `access_enabled`). - #[serde(default = "default_true")] + /// Serialized only when `false`: older binaries ignore the key rather + /// than reject it, so writing the default `true` into every pushed + /// config would let a rollback silently resume auction telemetry after + /// an operator disabled it. + #[serde(default = "default_true", skip_serializing_if = "is_true")] pub auction_enabled: bool, /// Regional Tinybird API host, without scheme or path. #[serde(default)] @@ -1924,11 +1928,17 @@ impl TinybirdSettings { } if self.auction_enabled { validate_tinybird_dataset(&self.auction_dataset, "tinybird.auction_dataset")?; - validate_tinybird_secret(&self.auction_token_secret, "tinybird.auction_token_secret")?; + validate_secret_store_key_name( + &self.auction_token_secret, + "tinybird.auction_token_secret", + )?; } if self.access_enabled { validate_tinybird_dataset(&self.access_dataset, "tinybird.access_dataset")?; - validate_tinybird_secret(&self.access_token_secret, "tinybird.access_token_secret")?; + validate_secret_store_key_name( + &self.access_token_secret, + "tinybird.access_token_secret", + )?; if self.access_sample_rate <= 0.0 { return Err(Report::new(TrustedServerError::Configuration { message: "tinybird.access_sample_rate must be > 0 when tinybird.access_enabled is true".to_owned(), @@ -1973,8 +1983,15 @@ fn validate_tinybird_dataset(value: &str, setting: &str) -> Result<(), Report Result<(), Report> { - if value.is_empty() || value.chars().any(char::is_control) { +// Named to make the key-name-vs-secret-value distinction legible to static +// analysis: the argument is a Secret Store KEY NAME (an identifier such as +// `tinybird_access_append_token`), never a credential value, so formatting +// it into an error message discloses nothing. +fn validate_secret_store_key_name( + key_name: &str, + setting: &str, +) -> Result<(), Report> { + if key_name.is_empty() || key_name.chars().any(char::is_control) { return Err(Report::new(TrustedServerError::Configuration { message: format!("{setting} must be a non-empty Secret Store key"), })); @@ -2554,6 +2571,14 @@ pub(crate) const AUCTION_DEBUG_UPSTREAM_METADATA_KEYS: &[&str] = &[ "upstream_message_truncated", ]; +/// `skip_serializing_if` helper: true is the serde default for the fields +/// that use it, so serializing it would only widen the pushed config's +/// rollback surface. +#[allow(clippy::trivially_copy_pass_by_ref)] +fn is_true(value: &bool) -> bool { + *value +} + fn default_true() -> bool { true } @@ -2714,6 +2739,13 @@ pub struct ObservabilitySettings { /// timing. Defaults to `false` (off). #[serde(default)] pub server_timing_enabled: bool, + /// Section names whose publisher paths keep a named route template in + /// access telemetry (`/{section}/*`); everything else collapses to + /// `/other/*`. Matching is ASCII case-insensitive on the first path + /// segment, and a match requires at least one further segment. Defaults + /// to empty, which collapses every publisher path. + #[serde(default, skip_serializing_if = "Vec::is_empty")] + pub route_sections: Vec, } impl ObservabilitySettings { @@ -4448,6 +4480,47 @@ mod tests { ); } + #[test] + fn auction_enabled_serializes_only_when_disabled() { + let mut settings = create_test_settings(); + assert!(settings.tinybird.auction_enabled, "should default on"); + let toml = toml::to_string(&settings).expect("should serialize settings"); + assert!( + !toml.contains("auction_enabled"), + "should omit the default-true key: a rollback must not silently \ + re-enable auction telemetry an operator disabled" + ); + + settings.tinybird.auction_enabled = false; + let toml = toml::to_string(&settings).expect("should serialize settings"); + assert!( + toml.contains("auction_enabled = false"), + "should serialize the operator's explicit disable" + ); + } + + #[test] + fn route_sections_serialize_only_when_configured() { + let mut settings = create_test_settings(); + assert!( + settings.observability.route_sections.is_empty(), + "should default to the collapse-everything allowlist" + ); + settings.observability.server_timing_enabled = true; + let toml = toml::to_string(&settings).expect("should serialize settings"); + assert!( + !toml.contains("route_sections"), + "should omit the empty allowlist so a prior binary can parse the config" + ); + + settings.observability.route_sections = vec!["news".to_owned()]; + let toml = toml::to_string(&settings).expect("should serialize settings"); + assert!( + toml.contains("route_sections"), + "should serialize a configured allowlist" + ); + } + #[test] fn test_settings_from_valid_toml() { let toml_str = crate_test_settings_str(); diff --git a/docs/guide/configuration.md b/docs/guide/configuration.md index 2b154b8a3e..8cc3f73243 100644 --- a/docs/guide/configuration.md +++ b/docs/guide/configuration.md @@ -1926,9 +1926,10 @@ enabling one does not enable the other. ### `[observability]` -| Field | Type | Required | Default | Description | -| ----------------------- | ------- | -------- | ------- | ------------------------------------------------------------------- | -| `server_timing_enabled` | Boolean | No | `false` | Append request-phase timings to the `Server-Timing` response header | +| Field | Type | Required | Default | Description | +| ----------------------- | -------- | -------- | ------- | --------------------------------------------------------------------------------------------------------- | +| `server_timing_enabled` | Boolean | No | `false` | Append request-phase timings to the `Server-Timing` response header | +| `route_sections` | String[] | No | `[]` | Section names kept as publisher route templates (`/{section}/*`) in access telemetry; empty collapses all | **Purpose**: Surfaces per-phase request timing (`ts-total` plus recorded phases such as `ts-appbuild`, `ts-filter`, `ts-geo`, `ts-kv`, `ts-origin`, and @@ -1957,6 +1958,12 @@ send access-telemetry rows. server_timing_enabled = true ``` +::: tip The Axum dev server reads this flag once at startup +Unlike the Fastly adapter, which reads settings per request, the Axum dev +server bakes `server_timing_enabled` into its service when it starts. +Flipping the flag there requires a restart to take effect. +::: + ::: warning Client-visible latency disclosure The `Server-Timing` header is sent to every client on eligible responses, not only to operators: browsers expose the values to same-origin JavaScript @@ -1993,12 +2000,15 @@ independent emitters: auction telemetry (`auction_dataset`, `auction_token_secret`) and access telemetry. The keys below cover the access-telemetry sink and the shared enable flags. -| Field | Type | Required | Default | Description | -| -------------------- | ------- | ------------------------------------ | ------- | ------------------------------------------------------------------------------- | -| `enabled` | Boolean | Yes, when `access_enabled` | `false` | Master switch for the shared Tinybird transport (host, store, credentials) | -| `auction_enabled` | Boolean | No | `true` | Independently gates auction telemetry emission, decoupled from access telemetry | -| `access_enabled` | Boolean | No | `false` | Enables the sampled access-telemetry row sent after each response is delivered | -| `access_sample_rate` | Float | Yes (`> 0.0`), when `access_enabled` | `0.0` | Fraction (`0.0`-`1.0`) of requests to emit an access-telemetry row for | +| Field | Type | Required | Default | Description | +| --------------------- | ------- | ------------------------------------ | ------------------------------ | ------------------------------------------------------------------------------- | +| `enabled` | Boolean | Yes, when `access_enabled` | `false` | Master switch for the shared Tinybird transport (host, store, credentials) | +| `auction_enabled` | Boolean | No | `true` | Independently gates auction telemetry emission, decoupled from access telemetry | +| `access_enabled` | Boolean | No | `false` | Enables the sampled access-telemetry row sent after each response is delivered | +| `access_dataset` | String | Yes, when `access_enabled` | `access_logs_raw` | Access-log Events API datasource name | +| `access_token_secret` | String | Yes, when `access_enabled` | `tinybird_access_append_token` | Secret Store key holding the access APPEND token | +| `max_body_bytes` | Integer | No | `1048576` | Maximum NDJSON request body size; must be at least 1024 | +| `access_sample_rate` | Float | Yes (`> 0.0`), when `access_enabled` | `0.0` | Fraction (`0.0`-`1.0`) of requests to emit an access-telemetry row for | **Purpose**: `access_enabled` and `auction_enabled` gate the two Tinybird sinks separately so that turning on one does not silently turn on (or leave @@ -2007,8 +2017,8 @@ Setting `access_enabled = true` with `access_sample_rate = 0.0` is rejected at config load as an armed-but-silent configuration; use `access_enabled` itself to turn the sink off, not the sample rate. Enabling `access_enabled` also requires the shared transport fields (`enabled`, non-empty `api_host`, -`secret_store`, `access_dataset`, `access_token_secret`, and a positive -`max_body_bytes`) to already be set. +`secret_store`, `access_dataset`, `access_token_secret`, and a +`max_body_bytes` of at least 1024) to already be set. **Example**: @@ -2041,18 +2051,23 @@ the response the reader sees. ### Deploy and rollback ordering -::: warning `Settings` rejects unknown fields; order matters -Both `[observability].server_timing_enabled` and the new `[tinybird]` access -keys are new fields on a config schema that uses `deny_unknown_fields`, so an -older binary fails to load a config that carries them. +::: warning Push a compatibility config before rolling back +The compatibility boundary is uneven. The top-level `Settings` schema uses +`deny_unknown_fields`, so an older binary rejects a config carrying the +`[observability]` table. The nested `[tinybird]` table does not: an older +binary accepts unknown keys there, rejects `access_enabled = true` through +validation, ignores `auction_enabled` entirely, and reads `enabled = true` +as "auction telemetry on". **Deploying**: upgrade the binary first, then push a config containing the new fields second. Never push a config with these fields while a pre-observability binary can still receive it. -**Rolling back**: reverse the order. Remove the `[observability]` table and -any new `[tinybird]` access keys from the config and push that first, then -roll back the binary second. +**Rolling back**: push a compatibility config first, then roll the binary +back. The compatibility config removes the `[observability]` table, sets +`access_enabled = false`, and, for a deployment that only used the access +sink, sets `enabled = false` as well; otherwise the older binary would +interpret the leftover `enabled = true` as enabling auction telemetry. ::: ## Validation diff --git a/docs/superpowers/specs/2026-08-24-request-phase-timing-design.md b/docs/superpowers/specs/2026-08-24-request-phase-timing-design.md index 2892386f42..3833e81eb3 100644 --- a/docs/superpowers/specs/2026-08-24-request-phase-timing-design.md +++ b/docs/superpowers/specs/2026-08-24-request-phase-timing-design.md @@ -85,7 +85,7 @@ auction wait, buffered mode ..... auction_wait_ms (row only; pre-header in th | send_edgezero_response, immediately before into_parts(): mark_headers_ready() snapshot (unconditional) - build AccessTelemetrySnapshot (unconditional) + build AccessTelemetrySnapshot (only when tinybird.access_enabled) append Server-Timing header (flag-gated, only on conclusively private responses) | headers committed; body streams @@ -125,9 +125,11 @@ New module `crates/trusted-server-core/src/request_timing.rs`. - Sharing: `RequestTimings` is a cheap-clone handle, `Arc>`. It crosses three boundaries: adapter entry to core handlers, the streaming body closure (records body-phase spans after the response object has been handed off), and the adapter's - post-send emission read. Access is exclusively `try_lock()`: a contended or - poisoned lock drops the sample immediately rather than waiting, so recording can - never delay a request. + post-send emission read. Access is exclusively `try_lock()`: a contended lock + drops the one sample rather than waiting, so recording can never delay a + request, and a poisoned lock is recovered (the guarded data are plain counters + with no invariant a panic can break) so one panic cannot silence the rest of + the request's timing. - Recording API: `timings.record(Phase::Geo, dur)` and a scope guard `timings.span(Phase::Origin)` that records on drop. Guards use saturating duration math; a non-monotonic reading records zero rather than panicking. The auction-wait @@ -214,7 +216,8 @@ marks middleware finalization, not header commitment, and must not be treated as timing boundary. At the freeze point, in order: `mark_headers_ready()` (unconditional), the -`AccessTelemetrySnapshot` build (unconditional, section 10), then, gated on +`AccessTelemetrySnapshot` build (gated on `tinybird.access_enabled`, +section 10), then, gated on `observability.server_timing_enabled`, append one `Server-Timing` header from `server_timing_value()`. Append semantics, never insert: an origin-supplied Server-Timing survives, and the fronting delivery layer's own entries (`time-elapsed`, @@ -290,12 +293,16 @@ and user-generated content (search terms, usernames, emails in slugs). Replaced restricted to a bounded allowlisted charset, plus `/*` when deeper (for example `/news/*`). The auction-telemetry normalizer is explicitly not sufficient here: it redacts long tokens but preserves short identifiers and arbitrary slugs. -- Rejection is whole-segment, never truncation: a segment is dropped to `/other/*` - when it fails the charset allowlist, exceeds 32 characters, carries more than 7 - ASCII digits, or is the only segment in the path. Depth is what makes a first - segment a section name: single-segment paths are documents (WordPress - `/%postname%/` puts every article at depth 1), so they reject wholesale, root - landing pages included. The character allowlist alone does not bound identity (`[a-z0-9_-]` +- Publisher templates come from an operator-configured allowlist of section names + (`observability.route_sections`, default empty), not from the request: a path + whose first segment matches an allowlist entry and that has at least one further + segment emits `/{section}/*` (the lowercased allowlist entry itself); everything + else emits `/other/*`, and the root path emits `/`. This replaces the earlier + shape heuristics (charset, length, digit bounds), which review showed cannot + bound identity: depth-2 first segments are usernames on `/{username}/posts` + shapes, and single-segment paths are documents under `/%postname%/` permalinks. + With the allowlist, the emitted value set is fixed by configuration, so no + request-derived byte ever reaches the row. The character allowlist alone does not bound identity (`[a-z0-9_-]` is exactly the alphabet of UUIDs, hex ids, and reset tokens), and a truncated prefix of any of those is still identifying, so the length and digit bounds reject the segment outright. @@ -364,7 +371,9 @@ ships as a versioned replacement datasource with a cutover, not an in-place edit ## 10. Emission mechanics -- `AccessTelemetrySnapshot`: built unconditionally at the freeze point, before +- `AccessTelemetrySnapshot`: built at the freeze point when + `tinybird.access_enabled` is set (revised in review from the original + unconditional build, so a disabled deployment pays nothing here), before `into_parts()` consumes the response. It captures method, status, route metadata (from the `RouteMetadata` extension), and typed dimension states (`env`, `template_cache_state`, geo country). It exists because nothing else survives to diff --git a/tinybird/fixtures/access_logs_raw.ndjson b/tinybird/fixtures/access_logs_raw.ndjson index 3c82c5ca6e..983b3acd1e 100644 --- a/tinybird/fixtures/access_logs_raw.ndjson +++ b/tinybird/fixtures/access_logs_raw.ndjson @@ -1 +1 @@ -{"event_ts":"2026-06-23 12:00:00.000","method":"GET","status":200,"time_elapsed_ms":145,"sample_rate":0.1,"service_id":"abc123","publisher_domain":"test-publisher.com","env":"production","route_class":"publisher_html","route_template":"/news/*","body_mode":"streamed","auction_wait_placement":"in_stream","appbuild_ms":12,"filter_ms":5,"geo_ms":3,"kv_ms":8,"origin_ms":25,"template_cache_ms":10,"auction_wait_ms":45,"stream_ms":18,"request_elapsed_ms":145,"resp_bytes":8192,"template_cache_state":"hit","country":"US","ts_version":"v1.2.3","pop":"SFO"} +{"event_ts":"2026-06-23 12:00:00.000","method":"GET","status":200,"time_elapsed_ms":145,"sample_rate":0.1,"service_id":"abc123","publisher_domain":"test-publisher.example","env":"production","route_class":"publisher_html","route_template":"/news/*","body_mode":"streamed","auction_wait_placement":"in_stream","appbuild_ms":12,"filter_ms":5,"geo_ms":3,"kv_ms":8,"origin_ms":25,"template_cache_ms":10,"auction_wait_ms":45,"stream_ms":18,"request_elapsed_ms":145,"resp_bytes":8192,"template_cache_state":"hit","country":"US","ts_version":"v1.2.3","pop":"SFO"} From a1523675a21f5bbd3a327de45a95385d26716fc2 Mon Sep 17 00:00:00 2001 From: Jason Evans Date: Tue, 8 Sep 2026 13:55:24 -0400 Subject: [PATCH 032/194] Address round-3 telemetry robustness findings - The access sink streams the Tinybird response (the body is never consumed; buffered conversion materializes chunked bodies before the limit check) and newline-terminates rows to match the auction sink's NDJSON framing, with the recording client now asserting both. - Poisoned RequestTimings locks recover via into_inner instead of silently dropping every subsequent sample: the guarded data are plain counters, so one panic cannot blank the header and row for the rest of the request. Module and spec docs updated to stop conflating poisoning with contention. - The geo write-back skips 401 responses through a shared helper: resolve_geo_for_response short-circuits on 401 before consulting the carried state, so the old unconditional write downgraded a carried Resolved to Attempted and cost the row its country. - TimedKvStore forwards exists, so decorating a store with a cheap metadata probe (Spin) no longer downgrades it to the get-and-discard default body; with a contradiction-stub delegation test. - Post-send ordering is owned by run_post_send_steps, which both production sites route through, and the instrumented sequence test drives the real seam: elapsed stamped by send, then pull-sync, then telemetry. - Axum: dev_server_service remains the standard path; new tests pin flag-off suppression and the extension round trip (a phase recorded in the handler must surface as ts-filter in the header); the outer-wrapper rationale is reworded to the terminal-freeze-point argument; the configuration guide notes the flag is read once at startup. - The no-Cache-Control fail-closed case is pinned by a test, and t0's boundary (constructed after the adapter prologue) is documented. Co-Authored-By: Claude Fable 5 --- .../trusted-server-adapter-axum/src/timing.rs | 82 +++++++++++++++-- .../wrangler.integration.generated.toml | 16 ++++ .../trusted-server-adapter-fastly/src/main.rs | 91 ++++++++++++++----- .../src/middleware.rs | 80 +++++++++++++--- .../src/tinybird.rs | 26 +++++- .../src/platform/timed_kv.rs | 68 ++++++++++++++ .../trusted-server-core/src/request_timing.rs | 89 +++++++++++++----- .../plans/2026-08-24-request-phase-timing.md | 3 +- 8 files changed, 385 insertions(+), 70 deletions(-) create mode 100644 crates/trusted-server-adapter-cloudflare/wrangler.integration.generated.toml diff --git a/crates/trusted-server-adapter-axum/src/timing.rs b/crates/trusted-server-adapter-axum/src/timing.rs index 832823c153..315ebb5efc 100644 --- a/crates/trusted-server-adapter-axum/src/timing.rs +++ b/crates/trusted-server-adapter-axum/src/timing.rs @@ -9,13 +9,15 @@ //! [`append_server_timing_if_private`](trusted_server_core::request_timing::append_server_timing_if_private). //! //! This wraps *outside* `RouterService` rather than registering as -//! `RouterBuilder::middleware`. A router-generated 404/405 short-circuits -//! `RouterInner::dispatch` before its middleware chain ever runs, so -//! middleware never sees those responses. By the time a response reaches -//! this layer -- after `RouterService::oneshot` inside -//! `EdgeZeroAxumService::call` has already converted any dispatch error into -//! a plain response -- every response is covered uniformly, router-generated -//! or not. +//! `RouterBuilder::middleware` because the tower boundary is the terminal +//! freeze point: by the time a response reaches this layer -- after +//! `RouterService::oneshot` inside `EdgeZeroAxumService::call` has +//! converted any dispatch error into a plain response -- every response is +//! covered uniformly regardless of how routing produced it, and the +//! position survives future routing changes. (In this application's router +//! a catch-all fallback spans every path and publisher method, so +//! router-generated 404/405s that bypass middleware are close to +//! unreachable today; the outer position does not depend on them.) //! //! `/health` is excluded by path match before a //! [`RequestTimings`](trusted_server_core::request_timing::RequestTimings) @@ -163,6 +165,72 @@ mod tests { ); } + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] + async fn axum_suppresses_header_when_flag_is_off() { + let router = RouterService::builder() + .get("/private", |_ctx: RequestContext| async { + private_ok_response() + }) + .build(); + let mut service = TimingService::new(EdgeZeroAxumService::new(router), false); + + let request = Request::builder() + .uri("/private") + .body(AxumBody::empty()) + .expect("should build request"); + let response = service + .ready() + .await + .expect("should be ready") + .call(request) + .await + .expect("should not fail"); + + assert!( + header(&response, "server-timing").is_none(), + "should not emit server-timing when server_timing_enabled is false" + ); + } + + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] + async fn axum_round_trips_phase_timings_recorded_in_the_handler() { + // The collector crosses the adapter boundary as a request extension; + // this pins the round trip end to end: a phase recorded inside a + // core-style handler must come back out in the rendered header, so + // a future adapter conversion that drops request extensions fails + // here instead of silently losing every phase. + let router = RouterService::builder() + .get("/private", |ctx: RequestContext| async move { + if let Some(timings) = ctx.request().extensions().get::() { + timings.record( + trusted_server_core::request_timing::Phase::Filter, + std::time::Duration::from_millis(7), + ); + } + private_ok_response() + }) + .build(); + let mut service = TimingService::new(EdgeZeroAxumService::new(router), true); + + let request = Request::builder() + .uri("/private") + .body(AxumBody::empty()) + .expect("should build request"); + let response = service + .ready() + .await + .expect("should be ready") + .call(request) + .await + .expect("should not fail"); + + let server_timing = header(&response, "server-timing").expect("should emit header"); + assert!( + server_timing.contains("ts-filter;dur=7.0"), + "a phase recorded in the handler should survive the adapter round trip: {server_timing}" + ); + } + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] async fn axum_404_carries_header_when_private() { // An empty router has no routes at all, so any path dispatches diff --git a/crates/trusted-server-adapter-cloudflare/wrangler.integration.generated.toml b/crates/trusted-server-adapter-cloudflare/wrangler.integration.generated.toml new file mode 100644 index 0000000000..263403193a --- /dev/null +++ b/crates/trusted-server-adapter-cloudflare/wrangler.integration.generated.toml @@ -0,0 +1,16 @@ +name = "trusted-server" +main = "build/index.js" +compatibility_date = "2024-09-23" +# Keep in sync with wrangler.toml. `cache_option_enabled` is required for the +# outbound `CacheMode::NoStore` cache bypass under this compatibility date. +compatibility_flags = ["nodejs_compat", "cache_option_enabled"] +# No [build] section — bundle is pre-built in CI; wrangler dev must not rebuild. + +[[kv_namespaces]] +binding = "TRUSTED_SERVER_KV" +id = "ci-local-kv" + +[vars] +# Placeholder replaced by the integration test harness with a JSON object that +# contains the runtime Trusted Server app-config blob envelope. +TRUSTED_SERVER_CONFIG = '''{"app_config":"{\"data\":{\"auction\":{\"allowed_context_keys\":[],\"creative_store\":\"creative_store\",\"enabled\":false,\"mediator\":null,\"providers\":[],\"timeout_ms\":2000},\"cache\":{\"asset_rules\":[]},\"consent\":{\"check_expiration\":true,\"conflict_resolution\":{\"freshness_threshold_days\":30,\"mode\":\"restrictive\"},\"gdpr\":{\"applies_in\":[\"AT\",\"BE\",\"BG\",\"HR\",\"CY\",\"CZ\",\"DK\",\"EE\",\"FI\",\"FR\",\"DE\",\"GR\",\"HU\",\"IE\",\"IT\",\"LV\",\"LT\",\"LU\",\"MT\",\"NL\",\"PL\",\"PT\",\"RO\",\"SK\",\"SI\",\"ES\",\"SE\",\"IS\",\"LI\",\"NO\",\"GB\"]},\"max_consent_age_days\":395,\"mode\":\"interpreter\",\"us_privacy_defaults\":{\"gpc_implies_optout\":true,\"lspa_covered\":false,\"notice_given\":true},\"us_states\":{\"privacy_states\":[\"CA\",\"VA\",\"CO\",\"CT\",\"UT\",\"MT\",\"OR\",\"TX\",\"FL\",\"DE\",\"IA\",\"NE\",\"NH\",\"NJ\",\"TN\",\"MN\",\"MD\",\"IN\",\"KY\",\"RI\"]}},\"creative_opportunities\":null,\"debug\":{\"auction_html_comment\":false,\"inject_adm_for_testing\":false,\"ja4_endpoint_enabled\":false},\"ec\":{\"cluster_recheck_secs\":3600,\"cluster_trust_threshold\":10,\"ec_store\":\"ec_identity_store\",\"partners\":[{\"api_token\":\"integration-test-token-alpha-32-bytes-ok\",\"batch_rate_limit\":60,\"bidstream_enabled\":true,\"name\":\"Integration Test Partner\",\"openrtb_atype\":3,\"pull_sync_allowed_domains\":[],\"pull_sync_enabled\":false,\"pull_sync_rate_limit\":10,\"pull_sync_ttl_sec\":86400,\"pull_sync_url\":null,\"source_domain\":\"inttest.example.com\",\"ts_pull_token\":null},{\"api_token\":\"integration-test-token-bravo-32-bytes-ok\",\"batch_rate_limit\":60,\"bidstream_enabled\":true,\"name\":\"Integration Test Partner 2\",\"openrtb_atype\":3,\"pull_sync_allowed_domains\":[],\"pull_sync_enabled\":false,\"pull_sync_rate_limit\":10,\"pull_sync_ttl_sec\":86400,\"pull_sync_url\":null,\"source_domain\":\"inttest2.example.com\",\"ts_pull_token\":null}],\"passphrase\":\"integration-test-ec-secret-padded-32\",\"pull_sync_concurrency\":3},\"handlers\":[{\"password\":\"integration-admin-password-32-bytes-ok\",\"path\":\"^/_ts/admin\",\"username\":\"admin\"}],\"image_optimizer\":{\"profile_sets\":{}},\"integrations\":{\"adserver_mock\":{\"context_query_params\":{\"example_segments\":\"segments\"},\"enabled\":false,\"endpoint\":\"https://adserver.example.com/mediate\",\"timeout_ms\":1000},\"aps\":{\"account_id\":\"example-aps-account-id\",\"allow_script_creatives\":false,\"enabled\":true,\"endpoint\":\"https://aps.example.com/e/pb/bid\",\"timeout_ms\":1000},\"datadome\":{\"api_origin\":\"https://api.example.com\",\"cache_ttl_seconds\":3600,\"enabled\":false,\"rewrite_sdk\":true,\"sdk_origin\":\"https://sdk.example.com\"},\"didomi\":{\"api_origin\":\"https://api.example.com\",\"enabled\":false,\"sdk_origin\":\"https://sdk.example.com\"},\"google_tag_manager\":{\"container_id\":\"GTM-EXAMPLE\",\"enabled\":false,\"upstream_url\":\"https://tags.example.com\"},\"gpt\":{\"cache_ttl_seconds\":3600,\"enabled\":false,\"gam_attribution_enabled\":false,\"rewrite_script\":true,\"script_url\":\"https://ads.example.com/gpt.js\"},\"gpt_diagnostics\":{\"enabled\":true},\"lockr\":{\"api_endpoint\":\"https://identity.example.com\",\"app_id\":\"\",\"cache_ttl_seconds\":3600,\"enabled\":false,\"rewrite_sdk\":true,\"sdk_url\":\"https://identity.example.com/trusted-server.js\"},\"nextjs\":{\"enabled\":false,\"max_combined_payload_bytes\":10485760,\"rewrite_attributes\":[\"href\",\"link\",\"siteBaseUrl\",\"siteProductionDomain\",\"url\"]},\"permutive\":{\"api_endpoint\":\"https://api.example.com\",\"enabled\":false,\"organization_id\":\"\",\"project_id\":\"\",\"secure_signals_endpoint\":\"https://secure-signals.example.com\",\"workspace_id\":\"\"},\"prebid\":{\"bidders\":[],\"client_side_bidders\":[],\"debug\":false,\"enabled\":false,\"server_url\":\"https://prebid.example.com/openrtb2/auction\",\"timeout_ms\":1000},\"sourcepoint\":{\"cache_ttl_seconds\":3600,\"cdn_origin\":\"https://cdn.example.com\",\"enabled\":false,\"rewrite_sdk\":true},\"testlight\":{\"enabled\":false,\"endpoint\":\"https://testlight.example.com/openrtb2/auction\",\"rewrite_scripts\":true,\"timeout_ms\":1200}},\"proxy\":{\"allowed_domains\":[],\"asset_routes\":[],\"certificate_check\":false},\"publisher\":{\"cookie_domain\":\"localhost\",\"domain\":\"localhost\",\"max_buffered_body_bytes\":16777216,\"origin_host_header_override\":null,\"origin_url\":\"http://127.0.0.1:8888\",\"proxy_secret\":\"integration-test-proxy-secret\"},\"request_signing\":{\"config_store_id\":\"app_config\",\"enabled\":false,\"secret_store_id\":\"secrets\"},\"response_headers\":{},\"rewrite\":{\"exclude_domains\":[]},\"tester_cookie\":{\"enabled\":false},\"tinybird\":{\"access_dataset\":\"access_logs_raw\",\"access_enabled\":false,\"access_sample_rate\":0.0,\"access_token_secret\":\"tinybird_access_append_token\",\"api_host\":\"\",\"auction_dataset\":\"auction_events_raw\",\"auction_enabled\":true,\"auction_token_secret\":\"tinybird_auction_append_token\",\"enabled\":false,\"max_body_bytes\":1048576,\"secret_store\":\"ts_secrets\"}},\"generated_at\":\"2026-06-23T00:00:00Z\",\"sha256\":\"895f7fad0ce924476d1c04c68b0bf95f463d1fb631a4f639dcc7cf0c510383b4\",\"version\":1}"}''' diff --git a/crates/trusted-server-adapter-fastly/src/main.rs b/crates/trusted-server-adapter-fastly/src/main.rs index 333df736ae..8cd2b39d06 100644 --- a/crates/trusted-server-adapter-fastly/src/main.rs +++ b/crates/trusted-server-adapter-fastly/src/main.rs @@ -290,8 +290,16 @@ fn edgezero_main(mut req: FastlyRequest, env: &EnvConfig) { access_telemetry_enabled, }, ); - run_edgezero_pull_sync_after_send(settings, &partner_registry, &ec_state); - emit_access_telemetry_after_send(settings, &outcome, &timings); + run_post_send_steps( + || { + run_edgezero_pull_sync_after_send( + settings, + &partner_registry, + &ec_state, + ) + }, + || emit_access_telemetry_after_send(settings, &outcome, &timings), + ); return; } Err(e) => { @@ -322,12 +330,16 @@ fn edgezero_main(mut req: FastlyRequest, env: &EnvConfig) { access_telemetry_enabled, }, ); - run_edgezero_pull_sync_after_send( - &settings, - &partner_registry, - &ec_state, + run_post_send_steps( + || { + run_edgezero_pull_sync_after_send( + &settings, + &partner_registry, + &ec_state, + ); + }, + || emit_access_telemetry_after_send(&settings, &outcome, &timings), ); - emit_access_telemetry_after_send(&settings, &outcome, &timings); return; } Err(e) => { @@ -408,13 +420,9 @@ fn apply_entry_point_finalize_headers( // router-level 404/405 for an unregistered method), so `geo_state` may // still be `NotAttempted` even after a fresh lookup just ran above. // Write the resolved outcome back so the access-telemetry snapshot built - // later in `send_edgezero_response` sees what was actually looked up, - // not the stale carried-in state. - let resolved_state = match &geo_info { - Some(info) => GeoLookupState::Resolved(info.clone()), - None => GeoLookupState::Attempted, - }; - response.extensions_mut().insert(resolved_state); + // later in `send_edgezero_response` sees what was actually looked up; + // 401 handling lives in the shared helper. + middleware::write_back_geo_lookup_state(response, geo_info.as_ref()); } fn apply_edgezero_ec_finalize( @@ -453,6 +461,19 @@ fn run_edgezero_pull_sync_after_send( } } +/// Runs the post-send steps in their contract order: EC identity pull-sync +/// first, then access-telemetry emission. +/// +/// Every `edgezero_main` site that has both steps routes through this +/// function, so the ordering is owned in exactly one place and the +/// sequence test can instrument it; `request_elapsed` is already stamped +/// before either step because `send_edgezero_response` stamps it before +/// returning. +fn run_post_send_steps(pull_sync: impl FnOnce(), emit_access_telemetry: impl FnOnce()) { + pull_sync(); + emit_access_telemetry(); +} + /// Builds and emits the access-telemetry row for one delivered response, /// when access telemetry is enabled and this request is sampled in. /// @@ -1003,6 +1024,8 @@ pub(crate) fn derive_device_signals(req: &FastlyRequest) -> DeviceSignals { #[cfg(test)] mod tests { + use std::sync::Mutex; + use super::*; use base64::Engine as _; use edgezero_core::body::Body as EdgeBody; @@ -1733,13 +1756,13 @@ mod tests { } #[test] - fn request_elapsed_is_stamped_when_send_returns() { - // `edgezero_main`'s post-send ordering (pull-sync before telemetry) - // is a source-order invariant with no injectable seam, so this test - // deliberately proves only the leg that has one: by the time - // `send_edgezero_response` returns, `request_elapsed` is already - // stamped, so everything `edgezero_main` runs afterwards (pull-sync, - // telemetry emission) is excluded from `request_elapsed_ms`. + fn post_send_order_is_elapsed_then_pull_sync_then_telemetry() { + // The full contract sequence, instrumented through the real seams: + // `send_edgezero_response` stamps `request_elapsed` before + // returning, and `run_post_send_steps` (which every production + // site with both steps routes through) owns pull-sync-then- + // telemetry ordering. + let log: Arc>> = Arc::new(Mutex::new(Vec::new())); let timings = RequestTimings::new(); let response = response_builder() .body(EdgeBody::from("ok")) @@ -1757,14 +1780,36 @@ mod tests { access_telemetry_enabled: true, }, ); - assert!( timings.snapshot().request_elapsed_ms.is_some(), - "request_elapsed should be stamped by the time send returns" + "request_elapsed should be stamped before any post-send step runs" ); assert!( outcome.snapshot.is_some(), "the access snapshot should exist for the enabled context" ); + + let pull_log = Arc::clone(&log); + let emit_log = Arc::clone(&log); + run_post_send_steps( + move || { + pull_log + .lock() + .expect("should lock order log") + .push("pull_sync") + }, + move || { + emit_log + .lock() + .expect("should lock order log") + .push("telemetry") + }, + ); + + assert_eq!( + *log.lock().expect("should lock order log"), + vec!["pull_sync", "telemetry"], + "pull-sync must dispatch before telemetry emits" + ); } } diff --git a/crates/trusted-server-adapter-fastly/src/middleware.rs b/crates/trusted-server-adapter-fastly/src/middleware.rs index 79ded83628..7bd5773634 100644 --- a/crates/trusted-server-adapter-fastly/src/middleware.rs +++ b/crates/trusted-server-adapter-fastly/src/middleware.rs @@ -100,17 +100,10 @@ impl Middleware for FinalizeResponseMiddleware { }) }); - // Write the resolved outcome back so a downstream access-telemetry - // snapshot (built from response extensions after finalize) sees - // what was actually looked up here rather than the stale carried-in - // state — mirrors the entry-point finalize site in `main.rs` - // (`apply_entry_point_finalize_headers`), which writes back for the - // same reason. - let resolved_state = match &geo_info { - Some(geo) => GeoLookupState::Resolved(geo.clone()), - None => GeoLookupState::Attempted, - }; - response.extensions_mut().insert(resolved_state); + // Mirrors the entry-point finalize site in `main.rs` + // (`apply_entry_point_finalize_headers`); 401 handling lives in the + // shared helper. + write_back_geo_lookup_state(&mut response, geo_info.as_ref()); apply_finalize_headers(&self.settings, geo_info.as_ref(), &mut response); response @@ -193,6 +186,26 @@ impl Middleware for AuthMiddleware { /// is intentionally more conservative: geo data is not sent to any /// unauthenticated caller regardless of whether the 401 originated from this /// server or the upstream origin. +/// Writes the resolved geo outcome back onto the response as a +/// [`GeoLookupState`] extension, so a downstream access-telemetry snapshot +/// sees what was actually looked up rather than the stale carried-in state. +/// +/// Skips the write on a 401: [`resolve_geo_for_response`] returns `None` +/// for unauthorized responses before consulting the carried state, so +/// writing `Attempted` there would overwrite a carried `Resolved` with a +/// value that was never looked up, and the row would lose a country it +/// legitimately had. +pub(crate) fn write_back_geo_lookup_state(response: &mut Response, geo_info: Option<&GeoInfo>) { + if response.status() == StatusCode::UNAUTHORIZED { + return; + } + let resolved_state = match geo_info { + Some(geo) => GeoLookupState::Resolved(geo.clone()), + None => GeoLookupState::Attempted, + }; + response.extensions_mut().insert(resolved_state); +} + pub(crate) fn resolve_geo_for_response( response: &Response, carried: &GeoLookupState, @@ -285,6 +298,51 @@ pub(crate) use trusted_server_core::response_privacy::{ mod tests { use super::*; + #[test] + fn geo_write_back_preserves_resolved_state_on_401() { + // A 401 short-circuits geo resolution before the carried state is + // consulted, so the write-back must not downgrade a carried + // Resolved to Attempted (which would cost the row its country). + let mut response = response_builder() + .status(StatusCode::UNAUTHORIZED) + .body(Body::empty()) + .expect("should build a 401 response"); + response + .extensions_mut() + .insert(GeoLookupState::Resolved(sample_geo_info())); + + write_back_geo_lookup_state(&mut response, None); + + match response.extensions().get::() { + Some(GeoLookupState::Resolved(geo)) => { + assert_eq!( + geo.country, + sample_geo_info().country, + "should keep the carried country" + ); + } + other => panic!("should keep the Resolved state on a 401, got {other:?}"), + } + } + + #[test] + fn geo_write_back_records_attempted_on_non_401_miss() { + let mut response = response_builder() + .status(StatusCode::OK) + .body(Body::empty()) + .expect("should build a 200 response"); + + write_back_geo_lookup_state(&mut response, None); + + assert!( + matches!( + response.extensions().get::(), + Some(GeoLookupState::Attempted) + ), + "should record an attempted-but-missed lookup on ordinary responses" + ); + } + use std::collections::HashMap; use std::net::IpAddr; use std::sync::{Arc, Mutex}; diff --git a/crates/trusted-server-adapter-fastly/src/tinybird.rs b/crates/trusted-server-adapter-fastly/src/tinybird.rs index 84dfc37d8a..dfcf65f982 100644 --- a/crates/trusted-server-adapter-fastly/src/tinybird.rs +++ b/crates/trusted-server-adapter-fastly/src/tinybird.rs @@ -326,8 +326,13 @@ fn build_access_events_request( pub(crate) async fn emit_access_event( client: &dyn PlatformHttpClient, target: &TinybirdEventsTarget, - row: String, + mut row: String, ) -> Result<(), Report> { + // Match the auction sink's NDJSON framing: every row is + // newline-terminated, and the terminator counts toward the body limit. + if !row.ends_with('\n') { + row.push('\n'); + } let body_len = row.len(); if body_len > target.max_body_bytes { return Err(Report::new(TrustedServerError::Proxy { @@ -354,8 +359,11 @@ pub(crate) async fn emit_access_event( backend_name ); + // The response body is never consumed, so stream it: buffered + // conversion on Fastly materializes the body before the size limit is + // enforced, which a chunked response could abuse. let response = client - .send(PlatformHttpRequest::new(request, backend_name)) + .send(PlatformHttpRequest::new(request, backend_name).with_stream_response()) .await .change_context(TrustedServerError::Proxy { message: "failed to send Tinybird access telemetry request".to_owned(), @@ -490,6 +498,7 @@ mod tests { uri: String, headers: Vec<(String, String)>, body: Vec, + stream_response: bool, } /// Records outbound requests and, for [`PlatformHttpClient::send`] (the @@ -515,6 +524,7 @@ mod tests { fn record(&self, request: PlatformHttpRequest) { let backend_name = request.backend_name; + let stream_response = request.stream_response; let (parts, body) = request.request.into_parts(); let headers = parts .headers @@ -532,6 +542,7 @@ mod tests { uri: parts.uri.to_string(), headers, body: body.into_bytes().unwrap_or_default().to_vec(), + stream_response, }; self.requests .lock() @@ -949,10 +960,15 @@ mod tests { header_value(&requests[0].headers, header::AUTHORIZATION.as_str()), Some("Bearer test-tinybird-access-append-token") ); + let body = std::str::from_utf8(&requests[0].body).expect("should record utf8 body"); assert_eq!( - std::str::from_utf8(&requests[0].body).expect("should record utf8 body"), - row, - "should send the row verbatim as the request body" + body, + format!("{row}\n"), + "should send newline-delimited JSON, matching the auction sink's framing" + ); + assert!( + requests[0].stream_response, + "should stream the Tinybird response: the body is never consumed" ); } diff --git a/crates/trusted-server-core/src/platform/timed_kv.rs b/crates/trusted-server-core/src/platform/timed_kv.rs index 0594b7cef1..faac0ad2c1 100644 --- a/crates/trusted-server-core/src/platform/timed_kv.rs +++ b/crates/trusted-server-core/src/platform/timed_kv.rs @@ -53,6 +53,15 @@ impl PlatformKvStore for TimedKvStore> { self.inner.put_bytes(key, value).await } + // Forwarded explicitly: the trait's default body falls back to + // `get_bytes`, which would silently downgrade a backend's cheap + // metadata-only existence probe (the Spin adapter has one) into a full + // value transfer just because the store was decorated. + async fn exists(&self, key: &str) -> Result { + let _span = self.timings.span(Phase::EcKv); + self.inner.exists(key).await + } + async fn put_bytes_with_ttl( &self, key: &str, @@ -145,6 +154,65 @@ mod tests { ); } + #[test] + fn exists_delegates_to_the_inner_store_not_get_bytes() { + // A stub whose `exists` answer contradicts its `get_bytes` answer: + // if the decorator fell back to the trait's get-and-discard default + // body, this would return `false`. + struct ExistsOnlyStore; + + #[async_trait::async_trait(?Send)] + impl PlatformKvStore for ExistsOnlyStore { + async fn get_bytes(&self, _key: &str) -> Result, KvError> { + Ok(None) + } + async fn put_bytes(&self, _key: &str, _value: Bytes) -> Result<(), KvError> { + Ok(()) + } + async fn put_bytes_with_ttl( + &self, + _key: &str, + _value: Bytes, + _ttl: StdDuration, + ) -> Result<(), KvError> { + Ok(()) + } + async fn delete(&self, _key: &str) -> Result<(), KvError> { + Ok(()) + } + async fn list_keys_page( + &self, + _prefix: &str, + _cursor: Option<&str>, + _limit: usize, + ) -> Result { + Ok(KvPage { + keys: Vec::new(), + cursor: None, + }) + } + async fn exists(&self, _key: &str) -> Result { + Ok(true) + } + } + + let timings = RequestTimings::new(); + let inner: Arc = Arc::new(ExistsOnlyStore); + let store = TimedKvStore::new(inner, timings.clone()); + + let exists = futures::executor::block_on(store.exists("key")) + .expect("should forward the existence probe"); + assert!( + exists, + "should delegate to the inner exists, not the get_bytes default body" + ); + timings.mark_headers_ready(); + assert!( + timings.snapshot().kv_ms.is_some(), + "should time the existence probe like any other store operation" + ); + } + #[test] fn store_name_is_not_timed() { let timings = RequestTimings::new(); diff --git a/crates/trusted-server-core/src/request_timing.rs b/crates/trusted-server-core/src/request_timing.rs index cbb269b1eb..3b26c3ca25 100644 --- a/crates/trusted-server-core/src/request_timing.rs +++ b/crates/trusted-server-core/src/request_timing.rs @@ -1,10 +1,12 @@ //! Per-request phase timing collection and Server-Timing rendering. //! -//! Collection is always-on and infallible: saturating math, lock failure -//! drops the sample, no panics. See the design spec +//! Collection is always-on and infallible: saturating math, no panics. A +//! contended lock drops the one sample; a poisoned lock recovers (the +//! guarded data are plain counters), so a panic elsewhere cannot silence +//! the rest of the request's timing. See the design spec //! `docs/superpowers/specs/2026-08-24-request-phase-timing-design.md`. -use std::sync::{Arc, Mutex}; +use std::sync::{Arc, Mutex, TryLockError}; use std::time::Duration; use http::{HeaderName, HeaderValue, Response}; @@ -91,7 +93,12 @@ pub enum AuctionWaitPlacement { /// Mutable state behind [`RequestTimings`], guarded by a [`Mutex`]. struct Inner { - /// Instant the request started; the reference point for elapsed marks. + /// Instant the collector was constructed; the reference point for + /// elapsed marks. Adapters construct the collector after their own + /// prologue (client-request acquisition, logger init, and any + /// short-circuit routes on Fastly), so `ts-total` and + /// `time_elapsed_ms` exclude that prologue rather than measuring + /// wall-clock-from-accept. t0: Instant, /// Accumulated duration per [`Phase`], indexed by [`Phase::index`]. phases: [Option; PHASE_COUNT], @@ -111,9 +118,9 @@ struct Inner { /// Per-request phase timing collector. /// /// Cheap to clone (an [`Arc`] handle) and safe to share across threads and -/// async tasks handling the same request. Every method is infallible: lock -/// contention or poisoning silently drops the sample rather than blocking or -/// panicking. +/// async tasks handling the same request. Every method is infallible: +/// contention silently drops the one sample rather than blocking, and a +/// poisoned lock is recovered rather than treated as permanent loss. #[derive(Clone)] pub struct RequestTimings(Arc>); @@ -134,10 +141,16 @@ impl RequestTimings { /// Accumulates `dur` into `phase`'s running total. /// /// Repeated calls for the same phase saturate-add rather than overwrite. - /// Drops the sample silently on lock contention or poisoning. + /// Drops the sample silently on lock contention; a poisoned lock is recovered. pub fn record(&self, phase: Phase, dur: Duration) { - let Ok(mut inner) = self.0.try_lock() else { - return; + let mut inner = match self.0.try_lock() { + Ok(guard) => guard, + // Poisoning is recoverable here: the guarded data are plain + // counters with no invariant a panic can break, so recording + // keeps working for the rest of the request instead of going + // silently dark. Contention still drops the one sample. + Err(TryLockError::Poisoned(poisoned)) => poisoned.into_inner(), + Err(TryLockError::WouldBlock) => return, }; let index = phase.index(); let accumulated = inner.phases[index] @@ -149,10 +162,16 @@ impl RequestTimings { /// Records an auction wait duration under [`Phase::AuctionWait`] and /// stores its placement. /// - /// Drops the sample silently on lock contention or poisoning. + /// Drops the sample silently on lock contention; a poisoned lock is recovered. pub fn record_auction_wait(&self, placement: AuctionWaitPlacement, dur: Duration) { - let Ok(mut inner) = self.0.try_lock() else { - return; + let mut inner = match self.0.try_lock() { + Ok(guard) => guard, + // Poisoning is recoverable here: the guarded data are plain + // counters with no invariant a panic can break, so recording + // keeps working for the rest of the request instead of going + // silently dark. Contention still drops the one sample. + Err(TryLockError::Poisoned(poisoned)) => poisoned.into_inner(), + Err(TryLockError::WouldBlock) => return, }; let index = Phase::AuctionWait.index(); let accumulated = inner.phases[index] @@ -178,8 +197,14 @@ impl RequestTimings { /// Subsequent calls are no-ops (first call wins). Drops the sample /// silently on lock contention or poisoning. pub fn mark_headers_ready(&self) { - let Ok(mut inner) = self.0.try_lock() else { - return; + let mut inner = match self.0.try_lock() { + Ok(guard) => guard, + // Poisoning is recoverable here: the guarded data are plain + // counters with no invariant a panic can break, so recording + // keeps working for the rest of the request instead of going + // silently dark. Contention still drops the one sample. + Err(TryLockError::Poisoned(poisoned)) => poisoned.into_inner(), + Err(TryLockError::WouldBlock) => return, }; if inner.headers_ready_total.is_none() { inner.headers_ready_total = Some(inner.t0.elapsed()); @@ -192,8 +217,14 @@ impl RequestTimings { /// Subsequent calls are no-ops (first call wins). Drops the sample /// silently on lock contention or poisoning. pub fn mark_request_elapsed(&self) { - let Ok(mut inner) = self.0.try_lock() else { - return; + let mut inner = match self.0.try_lock() { + Ok(guard) => guard, + // Poisoning is recoverable here: the guarded data are plain + // counters with no invariant a panic can break, so recording + // keeps working for the rest of the request instead of going + // silently dark. Contention still drops the one sample. + Err(TryLockError::Poisoned(poisoned)) => poisoned.into_inner(), + Err(TryLockError::WouldBlock) => return, }; if inner.request_elapsed.is_none() { inner.request_elapsed = Some(inner.t0.elapsed()); @@ -202,10 +233,16 @@ impl RequestTimings { /// Records the response body size in bytes. /// - /// Drops the sample silently on lock contention or poisoning. + /// Drops the sample silently on lock contention; a poisoned lock is recovered. pub fn set_resp_bytes(&self, bytes: u64) { - let Ok(mut inner) = self.0.try_lock() else { - return; + let mut inner = match self.0.try_lock() { + Ok(guard) => guard, + // Poisoning is recoverable here: the guarded data are plain + // counters with no invariant a panic can break, so recording + // keeps working for the rest of the request instead of going + // silently dark. Contention still drops the one sample. + Err(TryLockError::Poisoned(poisoned)) => poisoned.into_inner(), + Err(TryLockError::WouldBlock) => return, }; inner.resp_bytes = Some(bytes); } @@ -221,7 +258,11 @@ impl RequestTimings { /// silently (returning `None`) on lock contention or poisoning. #[must_use] pub fn server_timing_value(&self) -> Option { - let inner = self.0.try_lock().ok()?; + let inner = match self.0.try_lock() { + Ok(guard) => guard, + Err(TryLockError::Poisoned(poisoned)) => poisoned.into_inner(), + Err(TryLockError::WouldBlock) => return None, + }; let total = inner.headers_ready_total?; let mut entries = vec![format_entry("ts-total", total)]; for phase in HEADER_PHASES { @@ -241,8 +282,10 @@ impl RequestTimings { /// consistent with the infallibility of every other method. #[must_use] pub fn snapshot(&self) -> TimingSnapshot { - let Ok(inner) = self.0.try_lock() else { - return TimingSnapshot::default(); + let inner = match self.0.try_lock() { + Ok(guard) => guard, + Err(TryLockError::Poisoned(poisoned)) => poisoned.into_inner(), + Err(TryLockError::WouldBlock) => return TimingSnapshot::default(), }; TimingSnapshot { time_elapsed_ms: duration_ms(inner.headers_ready_total), diff --git a/docs/superpowers/plans/2026-08-24-request-phase-timing.md b/docs/superpowers/plans/2026-08-24-request-phase-timing.md index 02cda64d82..1b25b70d14 100644 --- a/docs/superpowers/plans/2026-08-24-request-phase-timing.md +++ b/docs/superpowers/plans/2026-08-24-request-phase-timing.md @@ -819,7 +819,8 @@ plus the coarse template; the freeze point consumes the extension (no `RouteClas column in `NAMED_ROUTES`, no reconstruction from a handler enum). Also in this task: make `TemplateCacheResponseState` a typed response extension in `publisher.rs`, set at every point that writes `x-ts-template-cache` so header and extension cannot -drift; the row reads the extension. The snapshot is built unconditionally in +drift; the row reads the extension. The snapshot is built (when access +telemetry is enabled) in `send_edgezero_response` right after `mark_headers_ready()` and returned inside `DeliveryOutcome` (add field `pub snapshot: AccessTelemetrySnapshot`). From 58f8a354c9794ac5f36566fef0f64a8976ad33a4 Mon Sep 17 00:00:00 2001 From: Jason Evans Date: Thu, 17 Sep 2026 04:04:17 -0500 Subject: [PATCH 033/194] Correct the section 18 status line and give the wrangler ignore its own entry The status said the implementation was a follow-up PR; it is in this one. The generated Cloudflare wrangler config was ignored under a comment about defunct pre-rename crate directories, which it has nothing to do with. Co-Authored-By: Claude Opus 5 (1M context) --- .gitignore | 3 +++ .../specs/2026-08-24-request-phase-timing-design.md | 5 +++-- 2 files changed, 6 insertions(+), 2 deletions(-) diff --git a/.gitignore b/.gitignore index 96ffa2a5cd..238e6b1333 100644 --- a/.gitignore +++ b/.gitignore @@ -63,4 +63,7 @@ src/*.html # leftover local build artifacts (node_modules, target, dist) that remain on disk. /crates/js/ /crates/integration-tests/ + +# Wrangler config generated by the Cloudflare integration-test harness from +# wrangler.toml at run time; regenerated on every run. wrangler.integration.generated.toml diff --git a/docs/superpowers/specs/2026-08-24-request-phase-timing-design.md b/docs/superpowers/specs/2026-08-24-request-phase-timing-design.md index 23c45101ec..61d43e2730 100644 --- a/docs/superpowers/specs/2026-08-24-request-phase-timing-design.md +++ b/docs/superpowers/specs/2026-08-24-request-phase-timing-design.md @@ -585,8 +585,9 @@ streamed. No allocation in the hot path beyond the one `Arc` at entry, the ## 18. Auction timeline offsets (follow-up increment) -Status: spec amendment for a follow-up PR; not part of the initial implementation -(#1074). Builds only on machinery that spec sections 5, 9, and 10 already define. +Status: spec amendment written ahead of implementation, then implemented in the +same PR on top of the initial implementation (#1074). Builds only on machinery +that spec sections 5, 9, and 10 already define. ### Problem From dae88a0b786c7031ef57fafa036c0e85285afb05 Mon Sep 17 00:00:00 2001 From: Christian Date: Fri, 4 Sep 2026 12:37:03 -0500 Subject: [PATCH 034/194] Improve GPT auction diagnostics observability --- .../src/integrations/gpt_bootstrap.js | 5 +- .../src/integrations/gpt_diagnostics.rs | 38 ++- crates/trusted-server-core/src/publisher.rs | 299 +++++++++++++++++- .../tests/nextjs/gpt-diagnostics.spec.ts | 7 +- .../trusted-server-js/lib/src/core/types.ts | 36 ++- .../lib/src/integrations/gpt/index.ts | 57 +++- .../src/integrations/gpt_diagnostics/api.ts | 18 +- .../integrations/gpt_diagnostics/badges.ts | 23 +- .../integrations/gpt_diagnostics/overlay.ts | 48 ++- .../gpt_diagnostics/presentation_helpers.ts | 21 +- .../src/integrations/gpt_diagnostics/store.ts | 110 ++++++- .../lib/test/integrations/gpt/ad_init.test.ts | 62 +++- .../integrations/gpt/gpt_bootstrap.test.ts | 10 +- .../gpt/schedule_initial_ad_init.test.ts | 19 +- .../integrations/gpt_diagnostics/api.test.ts | 17 +- .../gpt_diagnostics/badges.test.ts | 14 +- .../gpt_diagnostics/overlay.test.ts | 29 +- .../gpt_diagnostics/store.test.ts | 93 +++++- docs/guide/integrations/gpt-diagnostics.md | 111 +++++-- 19 files changed, 927 insertions(+), 90 deletions(-) diff --git a/crates/trusted-server-core/src/integrations/gpt_bootstrap.js b/crates/trusted-server-core/src/integrations/gpt_bootstrap.js index 2475c50828..4d887b970c 100644 --- a/crates/trusted-server-core/src/integrations/gpt_bootstrap.js +++ b/crates/trusted-server-core/src/integrations/gpt_bootstrap.js @@ -117,7 +117,7 @@ // and deliberately identical to the bundle scheduler — the impression is // spent on a viewed tab, and the post-hydration guarantee holds whenever // the request is actually issued. - ts.scheduleInitialAdInit = function (initialBids, initialSlots) { + ts.scheduleInitialAdInit = function (initialBids, initialSlots, initialAuctionDiagnostics) { // The bundle may replace this scheduler after the fallback claims the initial // pass. Keep the latch on the shared document API so replacement cannot reset it. if ((ts.navGeneration || 0) !== 0 || ts.initialAdInitScheduled) return; @@ -127,6 +127,9 @@ // would overwrite a committed SPA navigation's slots. if (initialSlots !== undefined) ts.adSlots = initialSlots; if (initialBids !== undefined) ts.bids = initialBids; + if (initialAuctionDiagnostics !== undefined) { + ts.auctionDiagnostics = initialAuctionDiagnostics; + } var fire = function () { if ((ts.navGeneration || 0) !== 0) return; if (typeof ts.adInit === "function") ts.adInit(); diff --git a/crates/trusted-server-core/src/integrations/gpt_diagnostics.rs b/crates/trusted-server-core/src/integrations/gpt_diagnostics.rs index b4a188f2ab..b98e3cbbb3 100644 --- a/crates/trusted-server-core/src/integrations/gpt_diagnostics.rs +++ b/crates/trusted-server-core/src/integrations/gpt_diagnostics.rs @@ -62,6 +62,7 @@ pub enum GptDiagnosticsCookieAction { #[derive(Clone, Debug, Default, PartialEq, Eq)] pub struct GptDiagnosticsRequestDecision { active: bool, + browser_session_active: bool, clean_browser_path_and_query: Option, cookie_action: GptDiagnosticsCookieAction, } @@ -73,6 +74,16 @@ impl GptDiagnosticsRequestDecision { self.active } + /// Whether this request came from an activated diagnostics browser session. + /// + /// Unlike [`Self::active`], this remains true for non-document requests such + /// as the SPA page-bids fetch. It is captured before the private activation + /// cookie is stripped from the request. + #[must_use] + pub(crate) fn browser_session_active(&self) -> bool { + self.browser_session_active + } + /// Whether the response must be private and non-storeable. #[must_use] pub fn requires_private_no_store(&self) -> bool { @@ -121,6 +132,7 @@ impl GptDiagnosticsRequestDecision { pub(crate) fn active_for_tests() -> Self { Self { active: true, + browser_session_active: true, clean_browser_path_and_query: None, cookie_action: GptDiagnosticsCookieAction::None, } @@ -143,6 +155,7 @@ mod head_seam_invariant_tests { ] { out.push(GptDiagnosticsRequestDecision { active, + browser_session_active: active, clean_browser_path_and_query: clean.clone(), cookie_action, }); @@ -279,12 +292,19 @@ pub fn prepare_request( replace_path_and_query(request, &clean_path)?; } - let mut decision = GptDiagnosticsRequestDecision::default(); + let mut decision = GptDiagnosticsRequestDecision { + browser_session_active: integration_enabled + && directive == QueryDirective::Absent + && cookie_state.occurrences == 1 + && cookie_state.canonical, + ..GptDiagnosticsRequestDecision::default() + }; if integration_enabled && eligible_navigation && had_reserved_query { decision.clean_browser_path_and_query = Some(clean_path); match directive { QueryDirective::Enable => { decision.active = true; + decision.browser_session_active = true; decision.cookie_action = GptDiagnosticsCookieAction::SetSession; } QueryDirective::Disable => { @@ -542,6 +562,22 @@ mod tests { assert_eq!(duplicate.headers()[header::COOKIE], "other=value"); } + #[test] + fn active_cookie_marks_non_document_requests_without_activating_document_behavior() { + let mut request = Request::builder() + .method(Method::GET) + .uri("https://publisher.example/_ts/page-bids?path=/article") + .header(header::COOKIE, "__Host-ts-console=1; other=value") + .body(EdgeBody::empty()) + .expect("should build page-bids request"); + + let decision = prepare_request(&settings(true), &mut request).expect("should prepare"); + + assert!(!decision.active()); + assert!(decision.browser_session_active()); + assert_eq!(request.headers()[header::COOKIE], "other=value"); + } + #[test] fn invalid_duplicate_and_disable_directives_fail_closed() { for query in [ diff --git a/crates/trusted-server-core/src/publisher.rs b/crates/trusted-server-core/src/publisher.rs index e367552e82..064197cc69 100644 --- a/crates/trusted-server-core/src/publisher.rs +++ b/crates/trusted-server-core/src/publisher.rs @@ -3163,6 +3163,44 @@ fn request_origin(scheme: &str, host: &str) -> String { /// JSON for every non-empty map; `serde_json::from_str` failed and `unwrap_or_default()` /// turned the failure into `{}`. Shared modes therefore served **zero bids**, silently, /// on every request that had any. Every fixture had empty bids, so nothing caught it. +#[derive(Clone, Debug, serde::Serialize)] +#[serde(rename_all = "camelCase")] +struct BrowserAuctionDiagnostics { + #[serde(skip_serializing_if = "Option::is_none")] + auction_dispatched_ms: Option, + #[serde(skip_serializing_if = "Option::is_none")] + auction_resolved_ms: Option, + #[serde(skip_serializing_if = "Option::is_none")] + auction_committed_ms: Option, + #[serde(skip_serializing_if = "Option::is_none")] + auction_wait_ms: Option, + #[serde(skip_serializing_if = "Option::is_none")] + auction_wait_placement: Option<&'static str>, +} + +impl BrowserAuctionDiagnostics { + fn from_request_timings(timings: &RequestTimings) -> Option { + let snapshot = timings.snapshot(); + snapshot.auction_dispatched_ms?; + Some(Self { + auction_dispatched_ms: snapshot.auction_dispatched_ms, + auction_resolved_ms: snapshot.auction_resolved_ms, + auction_committed_ms: snapshot.auction_committed_ms, + auction_wait_ms: snapshot.auction_wait_ms, + auction_wait_placement: snapshot.auction_wait_placement.map( + |placement| match placement { + AuctionWaitPlacement::PreHeader => "pre_header", + AuctionWaitPlacement::InStream => "in_stream", + }, + ), + }) + } +} + +fn elapsed_millis(started: &web_time::Instant) -> u32 { + started.elapsed().as_millis().min(u128::from(u32::MAX)) as u32 +} + #[derive(Clone, Default)] pub(crate) struct AdBidsState { /// Rendered bids `` sequences inside the string. pub(crate) fn build_bids_script(bid_map: &serde_json::Map) -> String { + build_bids_script_with_diagnostics(bid_map, None) +} + +fn build_bids_script_with_diagnostics( + bid_map: &serde_json::Map, + auction_diagnostics: Option<&BrowserAuctionDiagnostics>, +) -> String { let json = serde_json::to_string(bid_map) .expect("serde_json::to_string of Map should be infallible"); let escaped = html_escape_for_script(&json); @@ -5614,6 +5706,23 @@ pub(crate) fn build_bids_script(bid_map: &serde_json::Map(function(){{\ +var t=window.tsjs=window.tsjs||{{}};\ +var b=JSON.parse(\"{}\");\ +var d=JSON.parse(\"{}\");\ +var s=t.scheduleInitialAdInit;\ +if(typeof s===\"function\")s(b,void 0,d);\ +else{{t.bids=b;t.auctionDiagnostics=d;}}\ +}})();", + escaped, + html_escape_for_script(&diagnostics) + ); + } + format!( "", + html_escape_for_script(slots_json), + html_escape_for_script(&bids), + html_escape_for_script(&diagnostics) + ); + } + format!( "`; + const script = template.content.querySelector('script')!; + const attribute = raw.replaceAll('\0', '\ufffd'); + expect(script.getAttribute('src')).toBe(attribute); + + container.appendChild(script); + expect(script.getAttribute('src')).toBe(attribute); + }); + + it.each([ + ['http:', 'aim.loc.kr/identity-lockr.js', 'lockr'], + ['https:', 'aim.loc.kr/identity-lockr.js', 'lockr'], + ['http:', 'cdn.permutive.com/app-web.js', 'permutive'], + ['https:', 'cdn.permutive.com/app-web.js', 'permutive'], + ])('resolves %s%s before the %s guard claims it', (scheme, path, owner) => { + const raw = `${scheme}${path}`; + expect(parseSdkNetworkUrl(raw)).toBeNull(); + + const script = document.createElement('script'); + script.setAttribute('src', raw); + expect(script.src).toBe( + scheme === baseScheme ? `${baseScheme}//publisher.example.com/${path}` : `${scheme}//${path}` + ); + container.appendChild(script); + + expect(script.getAttribute('src')).toBe( + scheme === baseScheme ? raw : `${window.location.origin}/integrations/${owner}/sdk` + ); + }); +}); diff --git a/docs/guide/integrations/lockr.md b/docs/guide/integrations/lockr.md index e63648e60a..1c76e0f77f 100644 --- a/docs/guide/integrations/lockr.md +++ b/docs/guide/integrations/lockr.md @@ -148,7 +148,7 @@ The Lockr integration is implemented in [crates/trusted-server-core/src/integrat - Route: `/integrations/lockr/api/*` (GET, POST) proxies Lockr API calls - Purpose: Keep SDK delivery and identity API traffic on the publisher origin -### SDK URL rewriting +**SDK URL Rewriting**: Server-side attribute rewriting and the browser script guard recognize only `aim.loc.kr` and `identity.loc.kr` URLs whose pathname contains diff --git a/docs/superpowers/specs/2026-10-16-1199-sdk-url-ownership-design.md b/docs/superpowers/specs/2026-10-16-1199-sdk-url-ownership-design.md index 3f9cf7086b..adb4e22930 100644 --- a/docs/superpowers/specs/2026-10-16-1199-sdk-url-ownership-design.md +++ b/docs/superpowers/specs/2026-10-16-1199-sdk-url-ownership-design.md @@ -50,14 +50,30 @@ guards. ### Input forms -- Trim ASCII whitespace before parsing: space, tab, LF, CR, form feed, and vertical - tab. Do not trim Unicode whitespace such as U+FEFF or U+0085. Include those - boundary cases in the shared corpus. -- Require an explicit `http://`, `https://`, or `//` prefix, case-insensitive for - schemes. Reject shorthand such as `https:host/path` and `https:/host/path`. +- Reject literal NULs in raw matcher inputs: HTML replaces them with U+FFFD, + unlike direct URL-property assignment, which can trim them. This prevents raw + HTML recognition from claiming a URL that the browser would not load as an SDK. +- Before checking the prefix, apply the browser URL preprocessing rules: trim + leading/trailing C0 controls and space (U+0001–U+0020), then remove tab, LF, and + CR throughout the input. Do not trim Unicode whitespace such as U+FEFF or + U+0085. These boundary cases remain in the shared corpus. +- Require an explicit HTTP(S) scheme followed by two authority separators, or + two separators for a protocol-relative URL. Each separator may be `/` or `\`, + matching browser HTTP(S) backslash handling. Scheme matching is case-insensitive. + Leave path/query interpretation to the native parser rather than replacing + backslashes globally. +- Reject base-dependent shorthand such as `https:host/path` and `https:/host/path` + in raw/server matching. Whether those forms resolve to a vendor host or a + publisher path depends on the document scheme and base URL. Dynamic insertion + guards can recognize the DOM-resolved URL, but static markup using these forms + can still load the vendor SDK directly. - Accept absolute HTTP and HTTPS URLs, including uppercase schemes and hosts. - After the prefix check, malformed means rejected by the native URL parser; - normal parser normalization is allowed. + After preprocessing and the authority check, malformed means rejected by the + native URL parser. Normal parser normalization is allowed, but runtimes differ: + Node and Chromium accept some punycode labels rejected by Rust's `url` crate + and jsdom; Chromium also normalizes some fullwidth-percent encoded-dot host + spellings rejected by those runtimes. These remain fail-open recognition + limits, not a promise to recognize every browser-loadable spelling. - Accept protocol-relative URLs by adding `https:` before parsing. Ownership is independent of the page's HTTP/HTTPS scheme. - Reject empty, malformed, non-network, relative-path, and bare-host inputs. @@ -119,8 +135,10 @@ Direct private-matcher assertions remain in each integration's tests. Create `crates/trusted-server-js/lib/test/fixtures/sdk-url-ownership.json`. Each case records a descriptive name, input URL, and expected integration ID or null. An optional `insertionOwner` records a different expectation after native DOM URL -resolution. In particular, raw matchers reject shorthand schemes, while a script -or link URL getter normalizes them to absolute vendor URLs before insertion. +resolution. In particular, raw matchers reject shorthand schemes. With the +fixture's `https:` shorthand and jsdom's `http:` page, script and link URL getters +resolve them to absolute vendor URLs before insertion. On an `https:` publisher +page, the same inputs resolve to publisher paths and remain unowned. Rust tests and raw browser claimant tests use `owner`; DOM insertion tests use `insertionOwner` when present. This distinguishes parser input policy from the browser's existing attribute-resolution behavior. @@ -142,7 +160,10 @@ The corpus must cover: - Already-rewritten first-party GPT URLs whose query mentions a vendor host. - Empty/malformed, relative, bare-host, shorthand-scheme, and non-network inputs. - Trailing-dot hostnames, percent-encoded SDK path text, the bare - `edge.permutive.app` host, and ASCII/Unicode whitespace boundaries. + `edge.permutive.app` host, and C0/space/Unicode whitespace boundaries. +- Tab/CR/LF preprocessing, backslash and mixed authority separators, and + normalized lookalike/user-information/path/query negatives. +- NUL-containing raw attributes versus DOM URL properties and HTML tokenization. Replace stale comments pointing to Rust line numbers with pointers to this shared corpus. @@ -155,7 +176,10 @@ For each corpus case, run both Rust matchers and assert whether the expected owner equals that integration. Run the corresponding JS guards against the same input URL so parser decisions cannot drift behind duplicated case lists. Also assert server attribute rewrite actions for matching and nonmatching cases, -including `src` and `href`, while retaining disabled-rewrite coverage. +including `src` and `href`, while retaining disabled-rewrite coverage. Check +attribute-routing eligibility before calling the rewriter. For owned corpus +cases, also tokenize actual script/link markup with lol_html and assert the +rewritten attributes in the output. ### All-guard ownership and insertion tests @@ -170,7 +194,9 @@ production dispatcher API. Separately exercise actual insertion through both `script.src = url` and native `Element.prototype.setAttribute.call(script, 'src', url)`, then append the node. Assert the final first-party route or unchanged attribute. Account for ordinary -DOM relative-URL resolution separately from the raw matcher assertion. +DOM relative-URL resolution separately from the raw matcher assertion. Exercise +shorthand URLs with both HTTP and HTTPS document bases in +`test/shared/sdk_url.test.ts`; same-scheme shorthand must remain a publisher path. Include the issue's three reproduced Lockr collision cases in `test/integrations/lockr/url_collision.test.ts`. Force Lockr ahead of GPT/GTM in @@ -196,10 +222,10 @@ query-bearing valid SDKs and false positives, through append and insertBefore. Focused checks from the JS library directory: ```sh -npx vitest run test/integrations/lockr test/integrations/permutive test/shared/dom_insertion_dispatcher.test.ts +npx vitest run test/integrations/lockr test/integrations/permutive test/shared/sdk_url.test.ts test/shared/dom_insertion_dispatcher.test.ts ``` -Focused checks from the repository root, using the pinned Rust 1.95 toolchain +Focused checks from the repository root, using the pinned Rust toolchain and core's default feature set: ```sh @@ -216,7 +242,7 @@ coordinate those commands instead of running competing dependency installs. ## Independent review -A fresh Pi process using `openai-codex/gpt-6.1-sol` reviewed this spec, the full +A fresh-context reviewer read this spec, the full issue, and current code without conversation history or edits. It found no blocking defects. Its suggestions were accepted: bypass GPT assignment patches in raw claimant tests, add explicit trailing-dot/encoded-path/bare-edge-host @@ -230,13 +256,15 @@ focused JS tests. It reported no findings. ## Validation results -The regression tests failed against the original matchers before the fixes. -The final shared corpus contains 83 cases. The all-guard test expands them into -1,068 ownership and insertion checks across both handler orders and DOM paths. +The original regression tests failed against the substring matchers before the +ownership fixes. The normalization regressions also failed before expanding the +common preprocessing policy. The shared corpus now contains 105 cases, including +22 normalization positives and negatives. The all-guard test expands them into +1,376 ownership and insertion checks across both handler orders and DOM paths. | Check | Result | | ------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------- | -| Full JS suite, `npx vitest run` | Passed, 2,279 tests across 51 files; no reported type errors | +| Full JS suite, `npx vitest run --maxWorkers=4` | Passed, 2,605 tests across 52 files; no reported type errors | | JS `npm run lint`, `npm run format`, `npm run build` | Passed | | Native Lockr and Permutive inline test modules | Passed, 11 and 9 tests, including the shared corpus and attribute rewriting | | `cargo test-fastly`, `cargo test-fastly-reuse`, `cargo test-axum`, `cargo test-cloudflare`, `cargo test-spin` | Passed; existing ignored tests remain ignored | @@ -247,14 +275,8 @@ The final shared corpus contains 83 cases. The all-guard test expands them into | Docs and repository Markdown formatting | Passed | | URL-host lint on added lines | Passed | -An additional full native-core run, `cargo test -p trusted-server-core`, had -2,940 passing tests and 12 pre-existing legacy settings-test failures. Cargo -injects `TRUSTED_SERVER__GIT_VERSION`, which the legacy environment settings -loader rejects as the unknown field `git_version`. This was reproduced in an -untouched archive of `5bc388625` with -`cargo test -p trusted-server-core settings::tests::test_origin_host_header_override_env -- --exact`. -No settings or version-handling changes are included here. The required Fastly -core suite and focused native integration tests pass. +Default-worker JS runs hit timeouts in core, creative, and Prebid tests locally. +The full four-worker run passed without changing test timeouts or assertions. No live vendor SDK fetch or production deployment smoke test was performed. From ce2247a9f073ae5c0598479f8c0b99dca261f47b Mon Sep 17 00:00:00 2001 From: dhruv8sh Date: Tue, 27 Oct 2026 02:41:58 +0530 Subject: [PATCH 127/194] Apply owner precedence and clear provider opt-outs on re-grant Add the owned row of the precedence table, the re-grant clear on the writer's flush, and an admin DELETE route that clears a provider's opt-out on every adapter. Signed-off-by: dhruv8sh --- crates/trusted-server-adapter-axum/src/app.rs | 9 +- .../tests/routes.rs | 50 +++ .../src/app.rs | 8 + .../tests/routes.rs | 44 ++ .../trusted-server-adapter-fastly/src/app.rs | 52 +++ crates/trusted-server-adapter-spin/src/app.rs | 12 +- .../tests/routes.rs | 43 ++ .../src/auction/endpoints.rs | 387 +++++++++++++++++- crates/trusted-server-core/src/ec/admin.rs | 238 +++++++++++ crates/trusted-server-core/src/ec/eids.rs | 8 + .../src/ec/identity/coordinator.rs | 250 ++++++++++- .../src/ec/identity/flush.rs | 106 ++++- .../src/ec/identity/policy.rs | 2 - crates/trusted-server-core/src/publisher.rs | 42 ++ crates/trusted-server-core/src/settings.rs | 52 +++ 15 files changed, 1277 insertions(+), 26 deletions(-) diff --git a/crates/trusted-server-adapter-axum/src/app.rs b/crates/trusted-server-adapter-axum/src/app.rs index 7c5051c3db..76300c48b0 100644 --- a/crates/trusted-server-adapter-axum/src/app.rs +++ b/crates/trusted-server-adapter-axum/src/app.rs @@ -328,7 +328,7 @@ const LEGACY_ADMIN_DENY_METHODS: &[Method] = &[ Method::DELETE, ]; -fn named_routes() -> [NamedRoute; 17] { +fn named_routes() -> [NamedRoute; 18] { [ NamedRoute { path: "/.well-known/trusted-server.json", @@ -374,6 +374,13 @@ fn named_routes() -> [NamedRoute; 17] { primary_methods: &[Method::GET], handler: NamedRouteHandler::AdminEcNotSupported, }, + // The provider opt-out clear writes the EC identity graph, so it + // answers like the lookup: not available on this adapter. + NamedRoute { + path: "/_ts/admin/ec/{id}/providers/{provider}/opt-out", + primary_methods: &[Method::DELETE], + handler: NamedRouteHandler::AdminEcNotSupported, + }, // Admin EIDs echo: pure request inspection (no KV), so the dev // server serves the real handler. NamedRoute { diff --git a/crates/trusted-server-adapter-axum/tests/routes.rs b/crates/trusted-server-adapter-axum/tests/routes.rs index bcb31b7e44..fa445e5a2d 100644 --- a/crates/trusted-server-adapter-axum/tests/routes.rs +++ b/crates/trusted-server-adapter-axum/tests/routes.rs @@ -114,6 +114,7 @@ fn all_explicit_routes_are_registered() { ("POST", "/_ts/admin/keys/deactivate"), ("GET", "/_ts/admin/ec"), ("GET", "/_ts/admin/ec/{id}"), + ("DELETE", "/_ts/admin/ec/{id}/providers/{provider}/opt-out"), ("GET", "/_ts/admin/eids"), ("POST", "/admin/keys/rotate"), ("POST", "/admin/keys/deactivate"), @@ -388,6 +389,55 @@ async fn authenticated_admin_ec_routes_return_501() { } } +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn authenticated_admin_opt_out_clear_route_returns_501() { + // The clear writes the EC identity graph, which is Fastly KV backed, so + // the Axum dev server answers the DELETE locally with 501 like the EC + // lookup. + let path = format!( + "/_ts/admin/ec/{}.abc123/providers/pull-ssp-example-com/opt-out", + "a".repeat(64) + ); + let send = |method: &'static str| { + let path = path.clone(); + async move { + let req = Request::builder() + .method(method) + .uri(&path) + .header("authorization", "Basic YWRtaW46YWRtaW4tcGFzcw==") + .body(AxumBody::empty()) + .expect("should build request"); + make_service() + .ready() + .await + .expect("should be ready") + .call(req) + .await + .expect("should respond") + } + }; + + let deleted = send("DELETE").await; + assert_eq!( + deleted.status().as_u16(), + 501, + "DELETE {path} should report that the opt-out clear is unsupported" + ); + assert_eq!( + deleted + .headers() + .get("content-type") + .and_then(|v| v.to_str().ok()), + Some("application/json"), + "should answer with the same JSON as the EC lookup" + ); + assert_eq!( + send("GET").await.status().as_u16(), + 404, + "control: GET {path} should be denied locally as an unknown admin route" + ); +} + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] async fn admin_ec_route_without_credentials_returns_401() { let mut svc = make_service(); diff --git a/crates/trusted-server-adapter-cloudflare/src/app.rs b/crates/trusted-server-adapter-cloudflare/src/app.rs index f4278f3e50..5e3571c289 100644 --- a/crates/trusted-server-adapter-cloudflare/src/app.rs +++ b/crates/trusted-server-adapter-cloudflare/src/app.rs @@ -646,6 +646,14 @@ fn build_router(state: &Arc) -> RouterService { .get("/_ts/admin/ec/{id}", |_ctx: RequestContext| async { Ok::(admin_ec_lookup_not_supported()) }) + // The provider opt-out clear writes the EC identity graph, so it + // answers like the lookup. + .delete( + "/_ts/admin/ec/{id}/providers/{provider}/opt-out", + |_ctx: RequestContext| async { + Ok::(admin_ec_lookup_not_supported()) + }, + ) // Admin EIDs echo: pure request inspection (no KV), so this // adapter serves the real handler. .get( diff --git a/crates/trusted-server-adapter-cloudflare/tests/routes.rs b/crates/trusted-server-adapter-cloudflare/tests/routes.rs index f385bf5295..b4ba6a457c 100644 --- a/crates/trusted-server-adapter-cloudflare/tests/routes.rs +++ b/crates/trusted-server-adapter-cloudflare/tests/routes.rs @@ -254,6 +254,7 @@ fn all_explicit_routes_are_registered() { ("POST", "/_ts/admin/keys/deactivate"), ("GET", "/_ts/admin/ec"), ("GET", "/_ts/admin/ec/{id}"), + ("DELETE", "/_ts/admin/ec/{id}/providers/{provider}/opt-out"), ("GET", "/_ts/admin/eids"), ("POST", "/auction"), // SPA re-auction endpoint, plus its deprecated `/__ts/` alias. Both @@ -353,6 +354,49 @@ async fn authenticated_admin_ec_routes_return_501() { } } +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn authenticated_admin_opt_out_clear_route_returns_501() { + // The clear writes the EC identity graph, which is Fastly KV backed, so + // Cloudflare answers the DELETE locally with 501 like the EC lookup. + let path = format!( + "/_ts/admin/ec/{}.abc123/providers/pull-ssp-example-com/opt-out", + "a".repeat(64) + ); + let delete = request_builder() + .method("DELETE") + .uri(&path) + .header("authorization", "Basic YWRtaW46YWRtaW4tcGFzcw==") + .body(edgezero_core::body::Body::empty()) + .expect("should build request"); + let resp = route(test_router(), delete).await; + + assert_eq!( + resp.status().as_u16(), + 501, + "DELETE {path} should report that the opt-out clear is unsupported" + ); + assert_eq!( + resp.headers() + .get("content-type") + .and_then(|v| v.to_str().ok()), + Some("application/json"), + "should answer with the same JSON as the EC lookup" + ); + + let get = request_builder() + .method("GET") + .uri(&path) + .header("authorization", "Basic YWRtaW46YWRtaW4tcGFzcw==") + .body(edgezero_core::body::Body::empty()) + .expect("should build request"); + let resp = route(test_router(), get).await; + assert_eq!( + resp.status().as_u16(), + 404, + "control: GET {path} should be denied locally as an unknown admin route" + ); +} + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] async fn admin_ec_route_without_credentials_returns_401() { let req = request_builder() diff --git a/crates/trusted-server-adapter-fastly/src/app.rs b/crates/trusted-server-adapter-fastly/src/app.rs index a200cdc134..3e289e7313 100644 --- a/crates/trusted-server-adapter-fastly/src/app.rs +++ b/crates/trusted-server-adapter-fastly/src/app.rs @@ -24,6 +24,7 @@ //! | POST | `/_ts/admin/keys/deactivate` | [`handle_deactivate_key`] | //! | GET | `/_ts/admin/ec` | [`handle_admin_ec_lookup`] | //! | GET | `/_ts/admin/ec/{id}` | [`handle_admin_ec_lookup`] | +//! | DELETE | `/_ts/admin/ec/{id}/providers/{provider}/opt-out` | [`handle_admin_opt_out_clear`] | //! | GET | `/_ts/admin/eids` | [`handle_admin_eids_lookup`] | //! | POST | `/_ts/api/v1/batch-sync` | [`handle_batch_sync`] | //! | GET | `/_ts/api/v1/identify` | [`handle_identify`] | @@ -108,6 +109,7 @@ use trusted_server_core::config_payload::DEFAULT_SECRET_STORE_ID; use trusted_server_core::constants::{COOKIE_SHAREDID, COOKIE_TS_EIDS}; use trusted_server_core::ec::admin::{ deny_admin_diagnostic_fallback, handle_admin_ec_lookup, handle_admin_eids_lookup, + handle_admin_opt_out_clear, }; use trusted_server_core::ec::batch_sync::handle_batch_sync; use trusted_server_core::ec::consent::ec_consent_withdrawn; @@ -715,6 +717,15 @@ async fn execute_named( // These diagnostics are read-only. Running the normal EC lifecycle would // attach finalization state and could ingest request cookies into KV after // the handler returns, violating that contract. + // The opt-out clear writes one KV row through the removal write, never the + // request's cookies, so it skips the EC lifecycle for the same reason. + if matches!(handler, NamedRouteHandler::AdminOptOutClear) { + return Ok(handle_admin_opt_out_clear( + state.identity_graph.as_ref(), + &req, + )); + } + if matches!( handler, NamedRouteHandler::AdminEcLookup | NamedRouteHandler::AdminEidsLookup @@ -795,6 +806,9 @@ async fn run_named_route( NamedRouteHandler::AdminCachePurge => { unreachable!("cache purge should be handled before EC setup") } + NamedRouteHandler::AdminOptOutClear => { + unreachable!("opt-out clear should be handled before EC setup") + } NamedRouteHandler::LegacyAdminDenied => Ok(legacy_admin_alias_denied()), NamedRouteHandler::BatchSync => { // Dispatched by execute_named before EC state is built. @@ -1235,6 +1249,7 @@ enum NamedRouteHandler { DeactivateKey, AdminEcLookup, AdminEidsLookup, + AdminOptOutClear, AdminCachePurge, /// Legacy `/admin/keys/*` aliases — denied locally with 404 so they never /// reach the publisher fallback (which would leak admin credentials). @@ -1314,6 +1329,14 @@ const NAMED_ROUTES: &[NamedRoute] = &[ primary_methods: &[Method::GET], handler: NamedRouteHandler::AdminEcLookup, }, + // Operator clear of a provider's sticky opt-out (`ps.o`). Only DELETE is + // claimed: any other method reaches the publisher fallback, which denies + // the reserved admin namespace locally. + NamedRoute { + path: "/_ts/admin/ec/{id}/providers/{provider}/opt-out", + primary_methods: &[Method::DELETE], + handler: NamedRouteHandler::AdminOptOutClear, + }, // Admin EIDs echo: decodes the request's ts-eids/sharedId cookies with // an ingestion preview. Pure request inspection — no KV access. NamedRoute { @@ -2124,6 +2147,35 @@ mod tests { ); } + #[test] + fn admin_opt_out_clear_route_is_registered() { + // The operator clear is a DELETE on its own route. The route must call + // the core handler, and `/_ts/admin/ec/{id}` must stay GET-only so the + // clear is reachable only through the dedicated path. + let route = NAMED_ROUTES + .iter() + .find(|route| route.path == "/_ts/admin/ec/{id}/providers/{provider}/opt-out") + .expect("should register the provider opt-out clear route"); + assert!( + matches!(route.handler, NamedRouteHandler::AdminOptOutClear), + "the route must map to the opt-out clear handler" + ); + assert_eq!( + route.primary_methods, + &[Method::DELETE], + "the route must have DELETE as its only primary method" + ); + let lookup = NAMED_ROUTES + .iter() + .find(|route| route.path == "/_ts/admin/ec/{id}") + .expect("should register the lookup route"); + assert_eq!( + lookup.primary_methods, + &[Method::GET], + "control: the lookup route must stay GET-only" + ); + } + #[test] fn page_bids_serves_canonical_path_and_deprecated_alias() { // The SPA re-auction endpoint lives at the canonical single-underscore diff --git a/crates/trusted-server-adapter-spin/src/app.rs b/crates/trusted-server-adapter-spin/src/app.rs index 27e40fdb71..03bcbfb323 100644 --- a/crates/trusted-server-adapter-spin/src/app.rs +++ b/crates/trusted-server-adapter-spin/src/app.rs @@ -222,7 +222,7 @@ const LEGACY_ADMIN_DENY_METHODS: &[Method] = &[ Method::DELETE, ]; -fn named_fallback_paths() -> [(&'static str, &'static [Method]); 17] { +fn named_fallback_paths() -> [(&'static str, &'static [Method]); 18] { [ ("/.well-known/trusted-server.json", &[Method::GET]), ("/verify-signature", &[Method::POST]), @@ -230,6 +230,10 @@ fn named_fallback_paths() -> [(&'static str, &'static [Method]); 17] { ("/_ts/admin/keys/deactivate", &[Method::POST]), ("/_ts/admin/ec", &[Method::GET]), ("/_ts/admin/ec/{id}", &[Method::GET]), + ( + "/_ts/admin/ec/{id}/providers/{provider}/opt-out", + &[Method::DELETE], + ), ("/_ts/admin/eids", &[Method::GET]), ("/_ts/admin/cache/purge", LEGACY_ADMIN_DENY_METHODS), ("/admin/keys/rotate", LEGACY_ADMIN_DENY_METHODS), @@ -922,6 +926,12 @@ fn build_router(state: &Arc) -> RouterService { // adapter has no store to read. .get("/_ts/admin/ec", admin_ec_not_supported_handler) .get("/_ts/admin/ec/{id}", admin_ec_not_supported_handler) + // The provider opt-out clear writes the EC identity graph, so it + // answers like the lookup. + .delete( + "/_ts/admin/ec/{id}/providers/{provider}/opt-out", + admin_ec_not_supported_handler, + ) .get("/_ts/admin/eids", admin_eids_handler) .post("/auction", auction_handler) .get(PAGE_BIDS_PATH, page_bids_handler.clone()) diff --git a/crates/trusted-server-adapter-spin/tests/routes.rs b/crates/trusted-server-adapter-spin/tests/routes.rs index 613832b059..cb62546dec 100644 --- a/crates/trusted-server-adapter-spin/tests/routes.rs +++ b/crates/trusted-server-adapter-spin/tests/routes.rs @@ -151,6 +151,49 @@ async fn authenticated_admin_ec_routes_return_501() { } } +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn authenticated_admin_opt_out_clear_route_returns_501() { + // The clear writes the EC identity graph, which is Fastly KV backed, so + // Spin answers the DELETE locally with 501 like the EC lookup. + let path = format!( + "/_ts/admin/ec/{}.abc123/providers/pull-ssp-example-com/opt-out", + "a".repeat(64) + ); + let delete = request_builder() + .method("DELETE") + .uri(&path) + .header("authorization", "Basic YWRtaW46YWRtaW4tcGFzcw==") + .body(edgezero_core::body::Body::empty()) + .expect("should build request"); + let resp = route(test_router(), delete).await; + + assert_eq!( + resp.status().as_u16(), + 501, + "DELETE {path} should report that the opt-out clear is unsupported" + ); + assert_eq!( + resp.headers() + .get("content-type") + .and_then(|v| v.to_str().ok()), + Some("application/json"), + "should answer with the same JSON as the EC lookup" + ); + + let get = request_builder() + .method("GET") + .uri(&path) + .header("authorization", "Basic YWRtaW46YWRtaW4tcGFzcw==") + .body(edgezero_core::body::Body::empty()) + .expect("should build request"); + let resp = route(test_router(), get).await; + assert_eq!( + resp.status().as_u16(), + 404, + "control: GET {path} should be denied locally as an unknown admin route" + ); +} + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] async fn admin_ec_route_without_credentials_returns_401() { let req = request_builder() diff --git a/crates/trusted-server-core/src/auction/endpoints.rs b/crates/trusted-server-core/src/auction/endpoints.rs index feb2f2e0e7..5fd104c5c8 100644 --- a/crates/trusted-server-core/src/auction/endpoints.rs +++ b/crates/trusted-server-core/src/auction/endpoints.rs @@ -706,32 +706,51 @@ fn parse_cookie_auction_eids(cookie_value: Option<&str>) -> Option> { /// are forwarded as parsed, in wire order, never altered beyond a denied /// UID: they are what Prebid sent the `trustedServer` adapter, and a source /// may repeat with different EID-level fields. A server source's whole entry -/// set is appended after them, only when no browser entry carries the source, -/// so the two are never mixed. Sources are compared as the partner registry -/// normalizes them (lowercase, trailing dot removed). Entries without UIDs are -/// dropped. +/// set is appended after them, and the two are never mixed: +/// +/// - For a source whose owner wrote a valid value, the owner's set replaces +/// every browser entry of the source, however often the browser repeated +/// it. +/// - For any other source, the server's set goes out only when no browser +/// entry carries the source, so the browser's current value wins over a +/// browser-seeded record and over any unowned record. +/// +/// Sources are compared as the partner registry normalizes them (lowercase, +/// trailing dot removed). Entries without UIDs are dropped. pub(crate) fn merge_auction_eids( client_eids: &[Eid], server: &ServerSet, denials: &Denials, ) -> Vec { + let owned_sources: HashSet = server + .sources() + .iter() + .filter(|source| source.owned && source.eids.iter().any(|eid| !eid.uids.is_empty())) + .filter_map(|source| normalize_partner_source_domain(&source.source_domain).ok()) + .collect(); + let is_owned = |eid: &Eid| { + normalize_partner_source_domain(&eid.source) + .is_ok_and(|source| owned_sources.contains(&source)) + }; let mut merged: Vec = filter_browser_candidates(client_eids, denials) .into_iter() - .filter(|eid| !eid.uids.is_empty()) + .filter(|eid| !eid.uids.is_empty() && !is_owned(eid)) .collect(); let browser_sources: HashSet = merged .iter() .filter_map(|eid| normalize_partner_source_domain(&eid.source).ok()) .collect(); - for eid in server.sources().iter().flat_map(|source| &source.eids) { - if eid.uids.is_empty() { - continue; - } - let carried_by_browser = normalize_partner_source_domain(&eid.source) - .is_ok_and(|source| browser_sources.contains(&source)); - if !carried_by_browser { - merged.push(eid.clone()); + for source in server.sources() { + for eid in &source.eids { + if eid.uids.is_empty() { + continue; + } + let carried_by_browser = normalize_partner_source_domain(&eid.source) + .is_ok_and(|source| browser_sources.contains(&source)); + if source.owned || !carried_by_browser { + merged.push(eid.clone()); + } } } merged @@ -746,8 +765,11 @@ mod tests { use crate::auction::types::{AuctionRequest, AuctionResponse}; use crate::consent::jurisdiction::Jurisdiction; use crate::consent::types::ConsentContext; + use crate::ec::identity::descriptor::ConsentPolicy; use crate::ec::identity::revision::uid_hash; - use crate::ec::identity::test_support::{ec_partner, example_entry_set, full_record}; + use crate::ec::identity::test_support::{ + ec_partner, example_entry_set, full_record, pid, provider_from, source, + }; use crate::ec::kv_types::{KvEid, KvEntry, KvPartnerId, KvProviderState, KvUid}; use crate::error::IntoHttpResponse as _; use crate::openrtb::Uid; @@ -761,6 +783,7 @@ mod tests { use base64::engine::general_purpose::STANDARD as BASE64; use serde_json::Value as JsonValue; use serde_json::json; + use std::collections::BTreeMap; use std::sync::{Arc, Mutex}; #[derive(Default)] @@ -2522,6 +2545,342 @@ mod tests { ); } + /// The request identity for `entry` when `provider` owns each of `owned`. + fn owned_identity( + entry: KvEntry, + registry: &PartnerRegistry, + provider: Arc, + owned: &[&str], + consent: ConsentContext, + ) -> (AuctionIdentity, EcContext) { + let owners = owned + .iter() + .map(|name| (source(name), provider.descriptor().id().clone())) + .collect(); + let runtime = Arc::new( + IdentityRuntime::from_settings_with_providers( + &create_test_settings(), + vec![provider], + owners, + ) + .expect("should build the runtime"), + ); + let ec_id = merge_ec_id(); + let mut ec_context = EcContext::new_for_test(Some(ec_id.clone()), consent); + ec_context.set_identity_runtime(runtime); + let snapshot = EcKvSnapshot::Present { + ec_id, + entry: Box::new(entry), + generation: None, + }; + let identity = auction_identity( + &snapshot, + Some(registry), + &ec_context, + PolicySignals::default(), + ); + (identity, ec_context) + } + + fn non_regulated_consent() -> ConsentContext { + ConsentContext { + jurisdiction: Jurisdiction::NonRegulated, + ..ConsentContext::default() + } + } + + fn owner_provider(expiring: bool) -> Arc { + let mut builder = + crate::ec::identity::descriptor::ProviderDescriptor::builder(pid("owner")) + .with_source(source(SOURCE_S)) + .with_resolves( + crate::ec::identity::descriptor::Resolves::new() + .with_endpoint("https://api.example.com/fetch"), + ) + .with_consent_policy(ConsentPolicy::strict()); + if expiring { + builder = builder.with_expiring_ids(); + } + provider_from(builder.build().expect("should build the descriptor")) + } + + fn owner_record(uid: &str) -> KvPartnerId { + KvPartnerId { + w: Some("owner".to_owned()), + ..KvPartnerId::new(uid) + } + } + + #[test] + fn owner_valid_value_beats_browser_value() { + let registry = merge_registry(&[SOURCE_S, SOURCE_T]); + let (identity, _) = owned_identity( + merge_entry(vec![(SOURCE_S, owner_record("owner-uid"))]), + ®istry, + owner_provider(false), + &[SOURCE_S], + non_regulated_consent(), + ); + + let merged = identity.merge(Some(&[ + plain_eid(SOURCE_S, "browser", Some(1)), + plain_eid(SOURCE_T, "other", Some(1)), + ])); + + assert_eq!( + merged, + Some(vec![ + plain_eid(SOURCE_T, "other", Some(1)), + plain_eid(SOURCE_S, "owner-uid", Some(default_atype())), + ]), + "the owner's value should replace the browser's, after the browser entries" + ); + } + + #[test] + fn owner_valid_set_replaces_every_repeated_browser_entry() { + let registry = merge_registry(&[SOURCE_S, SOURCE_T]); + let (identity, _) = owned_identity( + merge_entry(vec![( + SOURCE_S, + full_record(Some("owner"), example_entry_set()), + )]), + ®istry, + owner_provider(false), + &[SOURCE_S], + non_regulated_consent(), + ); + + let merged = identity.merge(Some(&[ + plain_eid(SOURCE_S, "browser-one", Some(1)), + plain_eid(SOURCE_T, "other", Some(1)), + Eid { + inserter: Some("inserter.example.com".to_owned()), + ..plain_eid(SOURCE_S, "browser-two", Some(3)) + }, + ])); + + assert_eq!( + merged, + Some(vec![ + plain_eid(SOURCE_T, "other", Some(1)), + Eid { + source: SOURCE_S.to_owned(), + uids: vec![ + Uid { + id: "example-a".to_owned(), + atype: Some(1), + ext: None, + }, + Uid { + id: "example-b".to_owned(), + atype: Some(3), + ext: Some(json!({ "stype": "ppuid" })), + }, + ], + inserter: Some("example.com".to_owned()), + matcher: None, + mm: Some(3), + ext: None, + }, + Eid { + source: SOURCE_S.to_owned(), + uids: vec![Uid { + id: "example-c".to_owned(), + atype: Some(1), + ext: None, + }], + inserter: None, + matcher: Some("partner.example.com".to_owned()), + mm: None, + ext: None, + }, + ]), + "exactly the owner's two entries should go out, in stored order, and no browser entry of the source" + ); + } + + #[test] + fn browser_seeded_value_never_beats_browser_current_value() { + let registry = merge_registry(&[SOURCE_S]); + let (identity, _) = owned_identity( + merge_entry(vec![(SOURCE_S, KvPartnerId::new("seeded"))]), + ®istry, + owner_provider(false), + &[SOURCE_S], + non_regulated_consent(), + ); + + let sent = identity.merge(Some(&[plain_eid(SOURCE_S, "current", Some(1))])); + let silent = identity.merge(None); + + assert_eq!( + sent, + Some(vec![plain_eid(SOURCE_S, "current", Some(1))]), + "the browser's current value should beat a browser-seeded record" + ); + assert_eq!( + silent, + Some(vec![plain_eid(SOURCE_S, "seeded", Some(default_atype()))]), + "control: a seeded value of a source that does not expire should go out when the browser is silent" + ); + } + + #[test] + fn expiring_owned_source_never_serves_browser_seeded_value() { + let registry = merge_registry(&[SOURCE_S]); + let (expiring, _) = owned_identity( + merge_entry(vec![(SOURCE_S, KvPartnerId::new("seeded"))]), + ®istry, + owner_provider(true), + &[SOURCE_S], + non_regulated_consent(), + ); + let (owned_expiring, _) = owned_identity( + merge_entry(vec![( + SOURCE_S, + KvPartnerId { + exp: Some(current_timestamp() + 1_000), + ..owner_record("owner-uid") + }, + )]), + ®istry, + owner_provider(true), + &[SOURCE_S], + non_regulated_consent(), + ); + + assert_eq!( + expiring.merge(None), + None, + "should not serve a browser-seeded value of an expiring owned source" + ); + assert_eq!( + owned_expiring.merge(None), + Some(vec![plain_eid( + SOURCE_S, + "owner-uid", + Some(default_atype()) + )]), + "control: the owner's unexpired value should go out" + ); + } + + #[test] + fn unowned_source_keeps_the_browser_first_rule() { + let registry = merge_registry(&[SOURCE_S, SOURCE_T]); + let (identity, _) = owned_identity( + merge_entry(vec![ + (SOURCE_S, owner_record("owner-uid")), + (SOURCE_T, owner_record("claimed-uid")), + ]), + ®istry, + owner_provider(false), + &[SOURCE_S], + non_regulated_consent(), + ); + + let merged = identity.merge(Some(&[plain_eid(SOURCE_T, "browser", Some(1))])); + + assert_eq!( + merged, + Some(vec![ + plain_eid(SOURCE_T, "browser", Some(1)), + plain_eid(SOURCE_S, "owner-uid", Some(default_atype())), + ]), + "the browser should still win for a source with no owner" + ); + } + + #[test] + fn targeted_opt_out_blocks_server_ids_flush_and_mirror() { + let registry = merge_registry(&[SOURCE_S]); + let consent_with = |targeted: bool| ConsentContext { + jurisdiction: Jurisdiction::NonRegulated, + gpp: Some(crate::consent::types::GppConsent { + version: 1, + section_ids: vec![7], + eu_tcf: None, + us_sale_opt_out: Some(false), + us_sharing_opt_out: None, + us_targeted_advertising_opt_out: Some(targeted), + us_gpc: None, + us_known_child: None, + us_privacy_section: None, + us_section_decode_failed: false, + }), + ..ConsentContext::default() + }; + let provider = owner_provider(false); + let entry = || merge_entry(vec![(SOURCE_S, owner_record("owner-uid"))]); + let (blocked, blocked_context) = owned_identity( + entry(), + ®istry, + provider.clone(), + &[SOURCE_S], + consent_with(true), + ); + let (allowed, allowed_context) = owned_identity( + entry(), + ®istry, + provider.clone(), + &[SOURCE_S], + consent_with(false), + ); + + assert_eq!( + blocked.merge(None), + None, + "a targeted-advertising opt-out should leave no server-side ID in user.eids" + ); + assert_eq!( + blocked.mirror_member(&blocked_context), + Some(json!({ "clear": true })), + "the ts-eids-s mirror should carry none of the provider's entries" + ); + assert!( + allowed.merge(None).is_some() + && allowed + .mirror_member(&allowed_context) + .is_some_and(|member| member.get("value").is_some()), + "control: without the opt-out the ID should go out and be mirrored" + ); + + let catalog = ProviderCatalog::new(vec![provider.clone()], BTreeMap::new()); + let settings = crate::settings::ProviderSettings::default(); + let identity_settings = crate::settings::IdentitySettings::default(); + let check = |consent: &ConsentContext| { + crate::ec::identity::coordinator::eligibility( + &crate::ec::identity::coordinator::EligibilityInput { + provider: provider.as_ref(), + settings: &settings, + identity: &identity_settings, + catalog: &catalog, + consent, + }, + &crate::ec::identity::coordinator::EligibilityView::empty(), + &crate::ec::identity::test_support::facts_for_writer(), + crate::ec::identity::coordinator::Phase::AfterSend, + ) + }; + assert!( + matches!( + check(&consent_with(true)), + crate::ec::identity::coordinator::Eligibility::Skip( + crate::ec::identity::coordinator::SkipReason::ConsentPolicy + ) + ), + "the provider should not be dispatched, so nothing of it is flushed" + ); + assert!( + matches!( + check(&consent_with(false)), + crate::ec::identity::coordinator::Eligibility::Eligible { .. } + ), + "control: without the opt-out the provider should be eligible" + ); + } + #[test] fn golden_merge_one_uid_row_goes_out_as_on_main() { let registry = merge_registry(&["a.example.com", "b.example.com"]); diff --git a/crates/trusted-server-core/src/ec/admin.rs b/crates/trusted-server-core/src/ec/admin.rs index a51bb469c4..50d5b01a8d 100644 --- a/crates/trusted-server-core/src/ec/admin.rs +++ b/crates/trusted-server-core/src/ec/admin.rs @@ -6,6 +6,9 @@ //! attach, so operators can debug KV-to-auction propagation without KV //! console access. //! +//! Also serves `DELETE /_ts/admin/ec/{id}/providers/{provider}/opt-out`, +//! which clears a provider's sticky opt-out. +//! //! Also serves `GET /_ts/admin/eids`, which echoes the request's `ts-eids` //! and `sharedId` cookies with an ingestion preview — the client-side half //! of EID propagation that is never stored server-side. @@ -36,10 +39,14 @@ use crate::openrtb::Eid; use super::checked_current_timestamp; use super::eids::{resolve_partner_ids, to_eids}; use super::generation::is_valid_ec_id; +use super::identity::IdentityProviderId; use super::identity::IdentityRuntime; use super::identity::UnixTime; use super::identity::catalog::ProviderCatalog; +use super::identity::input::ThreadRandom; +use super::identity::mutation::{ApplyContext, IdentityMutation, ProviderStateChange}; use super::identity::policy::PolicySignals; +use super::identity::removal::{RemovalWriteOutcome, write_removals}; use super::identity::request::opted_out; use super::identity::servable::ServeContext; use super::kv::KvIdentityGraph; @@ -315,6 +322,104 @@ pub fn handle_admin_ec_lookup( Ok(json_response(StatusCode::OK, body)) } +/// Handles `DELETE /_ts/admin/ec/{id}/providers/{provider}/opt-out`. +/// +/// Clears the provider's sticky opt-out `ps.o` by writing `oc = now`, which +/// also empties `ps.ov` (spec 8). It runs the removal write's CAS loop and +/// never reads the request's cookies or consent. Responds: +/// +/// - `200 OK` with `{"cleared": true}` when the clear was written; +/// - `400 Bad Request` for an invalid EC ID or provider ID; +/// - `404 Not Found` when the row is missing or the provider holds no opt-out; +/// - `405 Method Not Allowed` for any method but `DELETE`; +/// - `501 Not Implemented` when no EC identity graph is configured; +/// - `503 Service Unavailable` when the write did not land. +#[must_use] +pub fn handle_admin_opt_out_clear( + kv: Option<&KvIdentityGraph>, + req: &Request, +) -> Response { + if req.method() != Method::DELETE { + let mut response = json_error(StatusCode::METHOD_NOT_ALLOWED, "method not allowed"); + response + .headers_mut() + .insert(header::ALLOW, HeaderValue::from_static("DELETE")); + return response; + } + let Some(kv) = kv else { + return admin_ec_lookup_not_supported(); + }; + let Some((ec_id, provider)) = opt_out_path_parts(req.uri().path()) else { + return json_error(StatusCode::NOT_FOUND, "admin diagnostic route not found"); + }; + if !is_valid_ec_id(ec_id) { + return json_error( + StatusCode::BAD_REQUEST, + "invalid EC ID format (expected {64hex}.{6alnum})", + ); + } + let Ok(provider) = provider.parse::() else { + return json_error(StatusCode::BAD_REQUEST, "invalid identity provider ID"); + }; + + let Some(now) = checked_current_timestamp() else { + return json_error( + StatusCode::SERVICE_UNAVAILABLE, + "the clock is unavailable, so the opt-out was not cleared", + ); + }; + let mut clear = IdentityMutation::default(); + clear.merge_state( + provider.clone(), + ProviderStateChange { + opt_out_clear: Some(now), + ..ProviderStateChange::default() + }, + ); + let catalog = ProviderCatalog::empty(); + let context = ApplyContext { + now, + random: &ThreadRandom, + catalog: &catalog, + }; + match write_removals(kv, ec_id, &clear, &context) { + RemovalWriteOutcome::Written => { + log::info!( + "Admin cleared the opt-out of identity provider '{provider}' for '{}'", + log_id(ec_id) + ); + json_response(StatusCode::OK, r#"{"cleared":true}"#.to_owned()) + } + RemovalWriteOutcome::NothingToWrite => json_error( + StatusCode::NOT_FOUND, + "no EC entry, or the provider holds no opt-out to clear", + ), + RemovalWriteOutcome::Failed => json_error( + StatusCode::SERVICE_UNAVAILABLE, + "the opt-out clear could not be written; retry", + ), + } +} + +/// Splits `/_ts/admin/ec/{id}/providers/{provider}/opt-out` into its EC ID +/// and provider ID, or `None` for any other shape. +fn opt_out_path_parts(path: &str) -> Option<(&str, &str)> { + let mut segments = path + .strip_prefix(ADMIN_EC_PATH)? + .strip_prefix('/')? + .split('/'); + let ec_id = segments.next()?; + let (Some("providers"), Some(provider), Some("opt-out"), None) = ( + segments.next(), + segments.next(), + segments.next(), + segments.next(), + ) else { + return None; + }; + Some((ec_id, provider)) +} + /// Returns the portable response used when an adapter has no EC KV backend. #[must_use] pub fn admin_ec_lookup_not_supported() -> Response { @@ -1774,4 +1879,137 @@ mod tests { assert_eq!(matched[0]["source_domain"], "sharedid.org"); assert_eq!(matched[0]["uid"], "shared-uid-123"); } + + const OPT_OUT_PROVIDER: &str = "pull-ssp-example-com"; + + fn opt_out_path(ec_id: &str, provider: &str) -> String { + format!("/_ts/admin/ec/{ec_id}/providers/{provider}/opt-out") + } + + fn opted_out_entry() -> KvEntry { + let mut entry = sample_entry(); + entry.ps.insert( + OPT_OUT_PROVIDER.to_owned(), + KvProviderState { + o: Some(crate::ec::kv_types::KvOptOut { + at: 1_000, + fp: "800".to_owned(), + }), + ov: vec!["0123456789abcdef".to_owned()], + ..KvProviderState::default() + }, + ); + entry.ps.insert( + "other-provider".to_owned(), + KvProviderState { + ov: vec!["fedcba9876543210".to_owned()], + ..KvProviderState::default() + }, + ); + entry + } + + #[test] + fn admin_clears_provider_opt_out() { + let ec_id = test_ec_id(); + let kv = kv_with_entry(&ec_id, &opted_out_entry()); + + let response = handle_admin_opt_out_clear( + Some(&kv), + &request_with_method( + http::Method::DELETE, + &opt_out_path(&ec_id, OPT_OUT_PROVIDER), + ), + ); + + assert_eq!( + response.status(), + StatusCode::OK, + "should clear the opt-out" + ); + assert_eq!( + response_json(response), + serde_json::json!({ "cleared": true }), + "should report the clear" + ); + let (stored, _) = kv + .get(&ec_id) + .expect("should read the row") + .expect("should hold the row"); + let state = stored + .ps + .get(OPT_OUT_PROVIDER) + .expect("should keep the provider state"); + assert!( + state.oc.is_some_and(|clear| clear > 1_000), + "should store the clear as oc" + ); + assert!(state.ov.is_empty(), "should empty ps.ov"); + assert_eq!( + stored + .ps + .get("other-provider") + .map(|other| other.ov.clone()), + Some(vec!["fedcba9876543210".to_owned()]), + "control: another provider's state should be untouched" + ); + } + + #[test] + fn admin_opt_out_clear_rejects_what_it_cannot_clear() { + let ec_id = test_ec_id(); + let kv = kv_with_entry(&ec_id, &opted_out_entry()); + let delete = |path: &str| { + handle_admin_opt_out_clear(Some(&kv), &request_with_method(http::Method::DELETE, path)) + .status() + }; + + assert_eq!( + delete(&opt_out_path(&ec_id, "other-provider")), + StatusCode::NOT_FOUND, + "should not find an opt-out the provider does not hold" + ); + assert_eq!( + delete(&opt_out_path("not-an-ec-id", OPT_OUT_PROVIDER)), + StatusCode::BAD_REQUEST, + "should reject an invalid EC ID" + ); + assert_eq!( + delete(&opt_out_path(&ec_id, "Not_A_Provider")), + StatusCode::BAD_REQUEST, + "should reject an invalid provider ID" + ); + let get = handle_admin_opt_out_clear( + Some(&kv), + &get_request(&opt_out_path(&ec_id, OPT_OUT_PROVIDER)), + ); + assert_eq!( + get.status(), + StatusCode::METHOD_NOT_ALLOWED, + "should accept only DELETE" + ); + let (stored, _) = kv + .get(&ec_id) + .expect("should read the row") + .expect("should hold the row"); + assert!( + stored + .ps + .get(OPT_OUT_PROVIDER) + .is_some_and(|state| state.oc.is_none() && !state.ov.is_empty()), + "control: no rejected request should clear the opt-out" + ); + assert_eq!( + handle_admin_opt_out_clear( + None, + &request_with_method( + http::Method::DELETE, + &opt_out_path(&ec_id, OPT_OUT_PROVIDER) + ), + ) + .status(), + StatusCode::NOT_IMPLEMENTED, + "should report an unconfigured identity graph" + ); + } } diff --git a/crates/trusted-server-core/src/ec/eids.rs b/crates/trusted-server-core/src/ec/eids.rs index 8a0196d802..98af796571 100644 --- a/crates/trusted-server-core/src/ec/eids.rs +++ b/crates/trusted-server-core/src/ec/eids.rs @@ -134,6 +134,9 @@ pub(crate) struct ServerSource { pub(crate) writer: Option, /// Unix seconds at which the record stops being servable. pub(crate) expires_at: Option, + /// Whether the record is the source owner's valid value: the source has + /// an owner and the record's writer is that owner (spec 7.1). + pub(crate) owned: bool, /// One EID per stored entry, in stored order, provenance defaults filled. pub(crate) eids: Vec, } @@ -175,6 +178,7 @@ impl ServerSet { source_domain: eid.source.clone(), writer: None, expires_at: None, + owned: false, eids: vec![eid], }) .collect(), @@ -200,13 +204,17 @@ pub(crate) fn server_set( .into_iter() .map(|record| { let mut eids = to_eids(core::slice::from_ref(&record)); + let mut owned = false; if let Ok(source) = record.source_domain.parse::() { fill_provenance(&mut eids, record.writer.as_ref(), &source, serve.catalog); + owned = record.writer.is_some() + && serve.catalog.owner_of(&source) == record.writer.as_ref(); } ServerSource { source_domain: record.source_domain, writer: record.writer, expires_at: record.expires_at, + owned, eids, } }) diff --git a/crates/trusted-server-core/src/ec/identity/coordinator.rs b/crates/trusted-server-core/src/ec/identity/coordinator.rs index 1cc97a21b2..8053f00e12 100644 --- a/crates/trusted-server-core/src/ec/identity/coordinator.rs +++ b/crates/trusted-server-core/src/ec/identity/coordinator.rs @@ -17,13 +17,13 @@ use super::exchange::{ProviderRequest, ProviderResponse}; use super::ids::{IdentityProviderId, SourceDomain, UnixTime}; use super::input::{RandomSource, ResolveInput, ResolveSignals}; use super::mutation::{PartnerIdChange, ProviderResult, ProviderStateChange, StateUpdate}; -use super::policy::{PolicySignals, policy_allows}; +use super::policy::{FP_US_OPT_OUT_MASK, PolicySignals, fingerprint_bits, policy_allows}; use super::provider::IdentityProvider; use super::resolution::{ EntrySet, EntrySets, IdentityProviderError, Resolution, ResolvedEid, StateChange, entry_sets, }; use super::revision::{IdentityKeys, StateRevision, record_revision, uid_hash, visitor_key}; -use crate::consent::ConsentContext; +use crate::consent::{ConsentContext, effective_tcf}; use crate::ec::kv_types::{KvEntry, KvOptOut, KvPartnerId}; use crate::ec::rate_limiter::RateLimiter; use crate::platform::{ @@ -33,6 +33,8 @@ use crate::platform::{ use crate::settings::{IdentitySettings, ProviderMode, ProviderSettings, Settings}; use crate::streaming_processor::{BodyStreamDecoder, Compression}; +/// Deciseconds in a second: `LastUpdated` of a TC string counts deciseconds. +const DECISECONDS_PER_SECOND: u64 = 10; /// How long a burst-capped provider waits: one limiter window. const BURST_WAIT_SECS: u64 = 60; /// The wait after a `no_match` that gives no `retry_in`. @@ -1497,6 +1499,103 @@ pub(crate) fn dispatch( outcomes } +/// Whether the visitor's consent now re-grants `descriptor`'s provider after +/// the sticky opt-out `opt_out` (spec 8). +/// +/// - A provider with a GVL ID: a TC string whose `LastUpdated` is after +/// `opt_out.at` and that grants the vendor and every purpose. +/// - Any provider, under a GPP string with a readable US section: `fp` records +/// at least one US opt-out bit and the current fingerprint holds none of +/// the US opt-out bits `fp` recorded. Other bits may differ. +/// +/// A jurisdiction change, a newly present GPP string without a US section, a +/// relaxed signal or an `fp` that does not parse never clears. +#[must_use] +pub(crate) fn regrant_clears_opt_out( + descriptor: &ProviderDescriptor, + opt_out: &KvOptOut, + consent: &ConsentContext, + signals: PolicySignals, +) -> bool { + tcf_regrants(descriptor, opt_out, consent) + || gpp_regrants(descriptor, opt_out, consent, signals) +} + +/// Whether a TC string newer than the opt-out grants the provider's GVL +/// vendor and every purpose its policy requires. +fn tcf_regrants( + descriptor: &ProviderDescriptor, + opt_out: &KvOptOut, + consent: &ConsentContext, +) -> bool { + let policy = descriptor.consent_policy(); + let (Some(vendor), Some(tcf)) = (policy.tcf_vendor(), effective_tcf(consent)) else { + return false; + }; + tcf.last_updated_ds / DECISECONDS_PER_SECOND > opt_out.at + && tcf.has_vendor_consent(vendor) + && policy + .tcf_purposes() + .iter() + .all(|purpose| tcf.has_purpose_consent(usize::from(*purpose))) +} + +/// Whether the GPP US section now shows none of the US opt-outs `fp` +/// recorded. GPP carries no creation time, so this is judged by section +/// values; without a readable US section nothing is judged. +fn gpp_regrants( + descriptor: &ProviderDescriptor, + opt_out: &KvOptOut, + consent: &ConsentContext, + signals: PolicySignals, +) -> bool { + let Some(gpp) = consent.gpp.as_ref() else { + return false; + }; + let has_us_section = gpp.us_sale_opt_out.is_some() + || gpp.us_sharing_opt_out.is_some() + || gpp.us_targeted_advertising_opt_out.is_some() + || gpp.us_privacy_section.is_some(); + if gpp.us_section_decode_failed || !has_us_section { + return false; + } + let Ok(recorded) = u64::from_str_radix(&opt_out.fp, 16) else { + return false; + }; + if recorded & FP_US_OPT_OUT_MASK == 0 { + return false; + } + let Ok(current) = u64::from_str_radix( + &fingerprint_bits(descriptor.consent_policy(), consent, signals), + 16, + ) else { + return false; + }; + current & recorded & FP_US_OPT_OUT_MASK == 0 +} + +/// The providers whose sticky opt-out the request's consent re-grants, each +/// to be cleared at `now`. +#[must_use] +pub(crate) fn opt_out_clears( + view: &EligibilityView, + catalog: &ProviderCatalog, + consent: &ConsentContext, + signals: PolicySignals, +) -> Vec { + catalog + .providers() + .filter_map(|provider| { + let descriptor = provider.descriptor(); + let state = view.providers.get(descriptor.id())?; + let opt_out = state.o.as_ref()?; + (view.is_opted_out(descriptor.id()) + && regrant_clears_opt_out(descriptor, opt_out, consent, signals)) + .then(|| descriptor.id().clone()) + }) + .collect() +} + #[cfg(test)] mod tests { use std::collections::BTreeMap; @@ -4375,4 +4474,151 @@ mod tests { "should sort generic pull by source, rotate the start each hour, and send other providers after" ); } + + fn regrant_descriptor(policy: ConsentPolicy) -> ProviderDescriptor { + custom_provider("regrant", &["a.example.com"], Resolves::new(), policy) + .descriptor() + .clone() + } + + fn stored_opt_out(at: u64, fp: &str) -> KvOptOut { + KvOptOut { + at, + fp: fp.to_owned(), + } + } + + fn gpp_consent(sale: Option, targeted: Option) -> ConsentContext { + let mut consent = allowing_consent(); + consent.gpp = Some(GppConsent { + version: 1, + section_ids: vec![7], + eu_tcf: None, + us_sale_opt_out: sale, + us_sharing_opt_out: None, + us_targeted_advertising_opt_out: targeted, + us_gpc: None, + us_known_child: None, + us_privacy_section: None, + us_section_decode_failed: false, + }); + consent + } + + #[test] + fn tcf_regrant_after_opt_out_clears_it() { + let descriptor = regrant_descriptor(ConsentPolicy::strict().with_tcf_vendor(5)); + let opt_out = stored_opt_out(1_000, "0"); + let regranted = |last_updated_ds: u64, vendor: u16| { + let mut consent = test_consent_gdpr_without_vendor(vendor); + consent + .tcf + .as_mut() + .expect("should carry a TC string") + .last_updated_ds = last_updated_ds; + regrant_clears_opt_out(&descriptor, &opt_out, &consent, PolicySignals::default()) + }; + + assert!( + regranted(10_010, 99), + "should clear on a newer TC string that grants the vendor and purposes" + ); + assert!( + !regranted(10_000, 99), + "control: a TC string not after the opt-out should not clear" + ); + assert!( + !regranted(10_010, 5), + "control: a newer TC string without the vendor should not clear" + ); + } + + #[test] + fn jurisdiction_change_never_clears_it() { + let descriptor = regrant_descriptor(ConsentPolicy::strict()); + let signals = PolicySignals::default(); + let sale = stored_opt_out(1_000, "800"); + + let mut eu_without_gpp = allowing_consent(); + eu_without_gpp.jurisdiction = crate::consent::jurisdiction::Jurisdiction::Gdpr; + let mut header_only = allowing_consent(); + header_only.gpp = Some(GppConsent { + version: 1, + section_ids: vec![2], + eu_tcf: None, + us_sale_opt_out: None, + us_sharing_opt_out: None, + us_targeted_advertising_opt_out: None, + us_gpc: None, + us_known_child: None, + us_privacy_section: None, + us_section_decode_failed: false, + }); + + assert!( + !regrant_clears_opt_out(&descriptor, &sale, &eu_without_gpp, signals), + "should not clear when the visitor moved to a jurisdiction with no GPP string" + ); + assert!( + !regrant_clears_opt_out(&descriptor, &sale, &header_only, signals), + "should not clear on a newly present GPP string with no US section" + ); + assert!( + regrant_clears_opt_out(&descriptor, &sale, &gpp_consent(Some(false), None), signals), + "control: a US section showing no opt-out should clear" + ); + } + + #[test] + fn gpp_regrant_clears_when_none_of_the_recorded_opt_outs_remain() { + let descriptor = regrant_descriptor(ConsentPolicy::strict()); + let signals = PolicySignals::default(); + let sale_opt_out = gpp_consent(Some(true), None); + let stored = fingerprint_bits(descriptor.consent_policy(), &sale_opt_out, signals); + assert_eq!(stored, "800", "should record the sale opt-out bit"); + // Purpose 4 was also granted when the opt-out was stored (bit 3). + let with_purpose_four = format!("{:x}", 0x800_u64 | 0x8); + let opt_out = stored_opt_out(1_000, &with_purpose_four); + + assert!( + regrant_clears_opt_out( + &descriptor, + &opt_out, + &gpp_consent(Some(false), None), + signals + ), + "should clear when the section shows no opt-out, even with purpose 4 changed" + ); + assert!( + !regrant_clears_opt_out(&descriptor, &opt_out, &sale_opt_out, signals), + "control: a section that still shows the sale opt-out should not clear" + ); + assert!( + !regrant_clears_opt_out( + &descriptor, + &stored_opt_out(1_000, "8"), + &gpp_consent(Some(false), None), + signals + ), + "control: an fp with no US opt-out bit should never clear by GPP" + ); + assert!( + !regrant_clears_opt_out( + &descriptor, + &stored_opt_out(1_000, "not-hex"), + &gpp_consent(Some(false), None), + signals + ), + "control: an fp that does not parse should never clear" + ); + assert!( + regrant_clears_opt_out( + &descriptor, + &opt_out, + &gpp_consent(Some(false), Some(true)), + signals + ), + "should judge only the opt-outs the fingerprint recorded, not a new one" + ); + } } diff --git a/crates/trusted-server-core/src/ec/identity/flush.rs b/crates/trusted-server-core/src/ec/identity/flush.rs index bf68986a80..964cd427ce 100644 --- a/crates/trusted-server-core/src/ec/identity/flush.rs +++ b/crates/trusted-server-core/src/ec/identity/flush.rs @@ -13,12 +13,12 @@ use http::StatusCode; use super::coordinator::{ Call, Eligibility, EligibilityInput, EligibilityView, OutcomeChanges, OutcomeContext, Phase, - RequestFacts, dispatch, eligibility, outcome_to_changes, + RequestFacts, dispatch, eligibility, opt_out_clears, outcome_to_changes, }; use super::descriptor::{Input, Resolves}; use super::ids::{IdentityProviderId, SourceDomain, UnixTime}; use super::input::{ProviderConsent, Region, ResolveSignals, StoredId, ThreadRandom}; -use super::mutation::{ApplyContext, IdentityMutation, apply}; +use super::mutation::{ApplyContext, IdentityMutation, ProviderStateChange, apply}; use super::policy::{PolicySignals, fingerprint_bits}; use super::provider::IdentityProvider; use super::removal::{log_removal_lost, write_removals}; @@ -123,8 +123,15 @@ pub fn run_after_send(input: AfterSendInput<'_>) -> FlushOutcome { Vec::new() }; let needs_cluster_size = phase_one_entry.is_some_and(lacks_cluster_size); + // A re-grant clears a sticky opt-out on the writer's one flush (spec 8). + let regranted = opt_out_clears( + &phase_one_view, + runtime.catalog(), + context.consent(), + facts.policy_signals, + ); - if mutation.is_empty() && due.is_empty() && !needs_cluster_size { + if mutation.is_empty() && due.is_empty() && !needs_cluster_size && regranted.is_empty() { return FlushOutcome::Skipped; } @@ -186,6 +193,16 @@ pub fn run_after_send(input: AfterSendInput<'_>) -> FlushOutcome { ); } + for provider in regranted { + mutation.merge_state( + provider, + ProviderStateChange { + opt_out_clear: Some(now), + ..ProviderStateChange::default() + }, + ); + } + if lacks_cluster_size(&live) { match graph.compute_cluster_size(&ec_id) { Ok(size) => mutation.set_cluster_size(size), @@ -493,9 +510,11 @@ mod tests { use error_stack::Report; use super::*; - use crate::ec::identity::mutation::{PartnerIdChange, ProviderStateChange}; + use crate::consent::types::GppConsent; + use crate::ec::identity::denials::{denials, filter_browser_candidates}; + use crate::ec::identity::mutation::PartnerIdChange; use crate::ec::identity::request::WriterDecision; - use crate::ec::identity::revision::{record_revision, visitor_key}; + use crate::ec::identity::revision::{record_revision, uid_hash, visitor_key}; use crate::ec::identity::test_support::fake_http::{ FakeBackend, FakeHttpClient, fake_services, }; @@ -503,8 +522,9 @@ mod tests { TEST_EC_ID, WriteCountingEcKv, allowing_consent, ec_partner, example_entry_set, full_record, pid, record_for_test, source, }; - use crate::ec::kv_types::{KvNetwork, KvOptOut, KvPartnerId}; + use crate::ec::kv_types::{KvNetwork, KvOptOut, KvPartnerId, KvProviderState}; use crate::error::TrustedServerError; + use crate::openrtb::{Eid, Uid}; use crate::settings::{ProviderMode, ProviderSettings}; use crate::test_support::tests::create_test_settings; @@ -1238,4 +1258,78 @@ mod tests { ); assert_eq!(fixture.store.writes().lookups, 0, "should not re-read"); } + + #[test] + fn regrant_clear_empties_ov() { + let held = "browser-held-uid"; + let mut row = live_row(true); + row.ps.insert( + PULL_ID.to_owned(), + KvProviderState { + o: Some(KvOptOut { + at: 1_000, + fp: "800".to_owned(), + }), + ov: vec![uid_hash(held)], + ..KvProviderState::default() + }, + ); + let browser = Eid { + source: PULL_SOURCE.to_owned(), + uids: vec![Uid { + id: held.to_owned(), + atype: Some(1), + ext: None, + }], + inserter: None, + matcher: None, + mm: None, + ext: None, + }; + let mut fixture = Fixture::new(Some(&row), true); + let before = denials(&row, fixture.runtime.catalog()); + let mut consent = allowing_consent(); + consent.gpp = Some(GppConsent { + version: 1, + section_ids: vec![7], + eu_tcf: None, + us_sale_opt_out: Some(false), + us_sharing_opt_out: None, + us_targeted_advertising_opt_out: None, + us_gpc: None, + us_known_child: None, + us_privacy_section: None, + us_section_decode_failed: false, + }); + *fixture.context.consent_mut() = consent; + + let outcome = fixture.run(&ScriptedLimiter::allowing()); + + assert_eq!(outcome, FlushOutcome::Written, "should flush the clear"); + assert_eq!( + fixture.store.writes().if_generation_match, + 1, + "should ride the one write" + ); + let stored = fixture.row(); + let state = stored + .ps + .get(PULL_ID) + .expect("should keep the provider state"); + assert!( + state.oc.is_some_and(|clear| clear > 1_000), + "should store the clear as oc" + ); + assert!(state.ov.is_empty(), "should empty ps.ov"); + assert!( + filter_browser_candidates(core::slice::from_ref(&browser), &before).is_empty(), + "control: before the clear the browser UID should be denied" + ); + let after = denials(&stored, fixture.runtime.catalog()); + assert_eq!( + filter_browser_candidates(core::slice::from_ref(&browser), &after), + vec![browser], + "should send the browser UID again after the clear" + ); + } } diff --git a/crates/trusted-server-core/src/ec/identity/policy.rs b/crates/trusted-server-core/src/ec/identity/policy.rs index 6d69b10cb9..5c38c3e810 100644 --- a/crates/trusted-server-core/src/ec/identity/policy.rs +++ b/crates/trusted-server-core/src/ec/identity/policy.rs @@ -39,8 +39,6 @@ pub(crate) const FP_US_SHARING: u32 = 14; /// Fingerprint bit of the US targeted-advertising opt-out. pub(crate) const FP_US_TARGETED: u32 = 15; /// The US opt-out bits: sale, sharing, targeted advertising and GPC. -// Temporary: Task 36's consent re-gate, the first non-test reader, arrives later. -#[allow(dead_code)] pub(crate) const FP_US_OPT_OUT_MASK: u64 = (1 << FP_US_SALE) | (1 << FP_US_SHARING) | (1 << FP_US_TARGETED) | (1 << FP_GPC); diff --git a/crates/trusted-server-core/src/publisher.rs b/crates/trusted-server-core/src/publisher.rs index 37aee9ad56..2148e20ade 100644 --- a/crates/trusted-server-core/src/publisher.rs +++ b/crates/trusted-server-core/src/publisher.rs @@ -8177,6 +8177,48 @@ mod tests { ); } + #[test] + fn merge_publisher_owner_value_beats_the_browser_cookie_value() { + let mut entry = KvEntry::minimal("other.example.com", "kv-other", 1000); + entry.ids.insert( + "ids.example.com".to_owned(), + crate::ec::kv_types::KvPartnerId { + w: Some("owner".to_owned()), + ..crate::ec::kv_types::KvPartnerId::new("owner-uid") + }, + ); + let (snapshot, registry, mut ec_context) = publisher_merge_fixture(entry); + let owner = + crate::ec::identity::test_support::fixed_provider("owner", &["ids.example.com"]); + ec_context.set_identity_runtime(Arc::new( + crate::ec::identity::IdentityRuntime::from_settings_with_providers( + &create_test_settings(), + vec![owner], + std::collections::BTreeMap::from([( + crate::ec::identity::test_support::source("ids.example.com"), + crate::ec::identity::test_support::pid("owner"), + )]), + ) + .expect("should build the runtime"), + )); + let cookie_entries = serde_json::json!([ + { "source": "ids.example.com", "uids": [{ "id": "browser-uid", "atype": 1 }] }, + { "source": "other.example.com", "uids": [{ "id": "browser-other", "atype": 1 }] } + ]); + + let merged = publisher_merge(&snapshot, ®istry, &ec_context, &cookie_entries, None); + + let atype = crate::settings::EcPartner::default_openrtb_atype(); + assert_eq!( + merged, + Some(vec![ + publisher_eid("other.example.com", "browser-other", 1), + publisher_eid("ids.example.com", "owner-uid", atype), + ]), + "the owner's value should replace the browser's, and the unowned browser value should still win" + ); + } + fn pull_partner_config(source_domain: &str) -> crate::settings::EcPartner { crate::settings::EcPartner { name: format!("Partner {source_domain}"), diff --git a/crates/trusted-server-core/src/settings.rs b/crates/trusted-server-core/src/settings.rs index 6db706673b..24585fdd60 100644 --- a/crates/trusted-server-core/src/settings.rs +++ b/crates/trusted-server-core/src/settings.rs @@ -3535,6 +3535,7 @@ impl Settings { "/_ts/admin/keys/deactivate", "/_ts/admin/ec", "/_ts/admin/ec/{id}", + "/_ts/admin/ec/{id}/providers/{provider}/opt-out", "/_ts/admin/eids", "/_ts/admin/cache/purge", ]; @@ -3562,9 +3563,28 @@ impl Settings { ), ]; + /// Probes for the dynamic provider opt-out route: two concrete IDs and + /// providers, so a handler anchored to one shape does not count as + /// covering the route. + const ADMIN_OPT_OUT_AUTH_PROBES: [&str; 2] = [ + concat!( + "/_ts/admin/ec/", + "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + ".Ab12Z9", + "/providers/pull-ssp-example-com/opt-out", + ), + concat!( + "/_ts/admin/ec/", + "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", + ".Zz09aB", + "/providers/other_provider-2/opt-out", + ), + ]; + fn admin_auth_probes(path: &'static str) -> [&'static str; 2] { match path { "/_ts/admin/ec/{id}" => Self::ADMIN_EC_ID_AUTH_PROBES, + "/_ts/admin/ec/{id}/providers/{provider}/opt-out" => Self::ADMIN_OPT_OUT_AUTH_PROBES, path => [path, path], } } @@ -8088,6 +8108,7 @@ source_domain = "partner.example.com" "/_ts/admin/keys/deactivate", "/_ts/admin/ec", "/_ts/admin/ec/{id}", + "/_ts/admin/ec/{id}/providers/{provider}/opt-out", "/_ts/admin/eids", "/_ts/admin/cache/purge", ], @@ -8095,6 +8116,36 @@ source_domain = "partner.example.com" ); } + #[test] + fn admin_opt_out_route_is_in_admin_endpoints() { + assert!( + Settings::ADMIN_ENDPOINTS.contains(&"/_ts/admin/ec/{id}/providers/{provider}/opt-out"), + "should list the provider opt-out clear route" + ); + let narrow = settings_str_without_admin_handler() + + r#" + [[handlers]] + path = "^/_ts/admin/ec/[^/]+$" + username = "admin" + password = "secret" + "#; + let settings: Settings = toml::from_str(&narrow).expect("should deserialize TOML"); + let uncovered = settings + .uncovered_admin_endpoints() + .expect("should check admin coverage"); + assert!( + uncovered.contains(&"/_ts/admin/ec/{id}/providers/{provider}/opt-out"), + "should require auth coverage of the opt-out route" + ); + assert!( + create_test_settings() + .uncovered_admin_endpoints() + .expect("should check admin coverage") + .is_empty(), + "control: the admin prefix handler should cover the opt-out route" + ); + } + #[test] fn uncovered_admin_endpoints_returns_empty_when_handler_covers_admin() { let settings = create_test_settings(); @@ -8128,6 +8179,7 @@ source_domain = "partner.example.com" "/_ts/admin/keys/deactivate", "/_ts/admin/ec", "/_ts/admin/ec/{id}", + "/_ts/admin/ec/{id}/providers/{provider}/opt-out", "/_ts/admin/eids", "/_ts/admin/cache/purge", ], From 0be0c53b2ce0566fd8b14c6a30ce2645fbeb04f5 Mon Sep 17 00:00:00 2001 From: dhruv8sh Date: Tue, 27 Oct 2026 03:08:34 +0530 Subject: [PATCH 128/194] Run hold-mode identity providers before the writer navigation's auction Answers feed that auction and join the writer's single flush. Stagers run no provider, and a navigation without an auction runs hold-mode providers after send instead. The navigation records that it ran an auction, so phase 4 skips hold-mode providers it already dispatched. An error status drops a found outcome's scheduling with its positive answer. Signed-off-by: dhruv8sh --- .../trusted-server-adapter-fastly/src/app.rs | 2 + .../src/auction/endpoints.rs | 15 + crates/trusted-server-core/src/ec/eids.rs | 65 ++ .../src/ec/identity/coordinator.rs | 662 +++++++++++++++++- .../src/ec/identity/flush.rs | 100 +-- .../src/ec/identity/mutation.rs | 132 +++- .../src/ec/identity/request.rs | 27 + .../src/ec/identity/test_support.rs | 276 +++++++- crates/trusted-server-core/src/ec/mod.rs | 7 + crates/trusted-server-core/src/publisher.rs | 420 ++++++++++- 10 files changed, 1650 insertions(+), 56 deletions(-) diff --git a/crates/trusted-server-adapter-fastly/src/app.rs b/crates/trusted-server-adapter-fastly/src/app.rs index 3e289e7313..08b7641267 100644 --- a/crates/trusted-server-adapter-fastly/src/app.rs +++ b/crates/trusted-server-adapter-fastly/src/app.rs @@ -577,6 +577,8 @@ fn build_ec_request_state( context.set_device_signals(device_signals); context.set_real_browser(is_real_browser); context.set_identity_runtime(Arc::clone(&state.identity)); + context + .set_identity_rate_limiter(Arc::new(FastlyRateLimiter::new(RATE_COUNTER_NAME))); // Orphan-recovery eligibility is intentionally left false here. // Authorizing it during generic pre-routing would let named // routes and request-filter short circuits (e.g. a DataDome diff --git a/crates/trusted-server-core/src/auction/endpoints.rs b/crates/trusted-server-core/src/auction/endpoints.rs index 5fd104c5c8..d7904fa11d 100644 --- a/crates/trusted-server-core/src/auction/endpoints.rs +++ b/crates/trusted-server-core/src/auction/endpoints.rs @@ -23,6 +23,7 @@ use crate::ec::eids::{ServerSet, server_set}; use crate::ec::identity::IdentityRuntime; use crate::ec::identity::UnixTime; use crate::ec::identity::catalog::ProviderCatalog; +use crate::ec::identity::coordinator::HoldResults; use crate::ec::identity::denials::{Denials, denials, filter_browser_candidates}; use crate::ec::identity::mirror::{MirrorPayload, MirrorRead, MirrorSource, mirror_payload}; use crate::ec::identity::policy::PolicySignals; @@ -489,6 +490,20 @@ impl AuctionIdentity { self.server.as_ref().map(ServerSet::eids) } + /// Overlays the answers of the hold dispatch before this navigation's + /// auction onto the server set. A request that may not use KV IDs stays + /// without a set. + pub(crate) fn add_hold_results( + &mut self, + results: HoldResults, + registry: &PartnerRegistry, + catalog: &ProviderCatalog, + ) { + if let Some(server) = self.server.as_mut() { + server.with_hold_results(results, registry, catalog); + } + } + /// The request's bidstream EIDs before consent gating: the browser's /// candidates after the denials, then the server set (spec 7.1). #[must_use] diff --git a/crates/trusted-server-core/src/ec/eids.rs b/crates/trusted-server-core/src/ec/eids.rs index 98af796571..5d7e763f73 100644 --- a/crates/trusted-server-core/src/ec/eids.rs +++ b/crates/trusted-server-core/src/ec/eids.rs @@ -13,6 +13,7 @@ use crate::error::TrustedServerError; use crate::openrtb::{Eid, Uid}; use super::identity::catalog::ProviderCatalog; +use super::identity::coordinator::HoldResults; use super::identity::servable::{ServeContext, is_servable}; use super::identity::{IdentityProviderId, SourceDomain}; use super::kv_types::{KvEid, KvEntry}; @@ -186,6 +187,70 @@ impl ServerSet { } } +impl ServerSet { + /// Overlays the answers of this navigation's hold dispatch (spec 4.7), + /// replacing the stored value of each source they name. + /// + /// Each source gets one `Eid` per entry, in the order the provider + /// returned them, with the writer's provenance default applied + /// ([`fill_provenance`]). A source the partner registry does not list for + /// the bidstream is left out, as a stored record would be. + pub(crate) fn with_hold_results( + &mut self, + results: HoldResults, + registry: &PartnerRegistry, + catalog: &ProviderCatalog, + ) { + let HoldResults { + additions, + mut writer, + } = results; + for (source, entries) in additions { + let Some(partner) = registry + .get(source.as_str()) + .filter(|partner| partner.bidstream_enabled) + else { + continue; + }; + let writer = writer.remove(&source); + let mut eids: Vec = entries + .iter() + .map(|entry| Eid { + source: partner.source_domain.clone(), + uids: entry + .uids + .iter() + .map(|uid| Uid { + id: uid.id.clone(), + atype: Some(uid.atype.unwrap_or(partner.openrtb_atype)), + ext: uid.ext.clone(), + }) + .collect(), + inserter: entry.inserter.clone(), + matcher: entry.matcher.clone(), + mm: entry.mm, + ext: entry.ext.clone(), + }) + .collect(); + fill_provenance(&mut eids, writer.as_ref(), &source, catalog); + let held = ServerSource { + source_domain: partner.source_domain.clone(), + owned: writer.is_some() && catalog.owner_of(&source) == writer.as_ref(), + writer, + expires_at: None, + eids, + }; + match self + .sources + .binary_search_by(|stored| stored.source_domain.cmp(&held.source_domain)) + { + Ok(index) => self.sources[index] = held, + Err(index) => self.sources.insert(index, held), + } + } + } +} + /// The servable server set of `entry` (spec 7.1). /// /// Without a read row, or for a row that is not live and consenting, the set diff --git a/crates/trusted-server-core/src/ec/identity/coordinator.rs b/crates/trusted-server-core/src/ec/identity/coordinator.rs index 8053f00e12..84b56d34f1 100644 --- a/crates/trusted-server-core/src/ec/identity/coordinator.rs +++ b/crates/trusted-server-core/src/ec/identity/coordinator.rs @@ -14,9 +14,13 @@ use web_time::Instant; use super::catalog::ProviderCatalog; use super::descriptor::{Input, ProviderDescriptor, ResolveKind, Resolves}; use super::exchange::{ProviderRequest, ProviderResponse}; +use super::flush::{DispatchInput, DispatchRun, add_dispatch_results, may_flush, request_facts}; use super::ids::{IdentityProviderId, SourceDomain, UnixTime}; -use super::input::{RandomSource, ResolveInput, ResolveSignals}; -use super::mutation::{PartnerIdChange, ProviderResult, ProviderStateChange, StateUpdate}; +use super::input::{RandomSource, ResolveInput, ResolveSignals, ThreadRandom}; +use super::mutation::{ + ApplyContext, IdentityMutation, PartnerIdChange, ProviderResult, ProviderStateChange, + StateUpdate, apply, +}; use super::policy::{FP_US_OPT_OUT_MASK, PolicySignals, fingerprint_bits, policy_allows}; use super::provider::IdentityProvider; use super::resolution::{ @@ -24,8 +28,9 @@ use super::resolution::{ }; use super::revision::{IdentityKeys, StateRevision, record_revision, uid_hash, visitor_key}; use crate::consent::{ConsentContext, effective_tcf}; -use crate::ec::kv_types::{KvEntry, KvOptOut, KvPartnerId}; +use crate::ec::kv_types::{KvEid, KvEntry, KvOptOut, KvPartnerId}; use crate::ec::rate_limiter::RateLimiter; +use crate::ec::{EcContext, current_timestamp}; use crate::platform::{ PlatformBackendSpec, PlatformHttpRequest, PlatformPendingRequest, PlatformResponse, RuntimeServices, @@ -479,6 +484,19 @@ pub(crate) enum CallOutcome { NotDispatched, } +impl CallOutcome { + /// Whether the call produced a positive answer: a resolution that carries + /// at least one ID. + #[must_use] + pub(crate) fn is_found(&self) -> bool { + matches!( + self, + Self::Resolved(resolution) | Self::InvalidState(Some(resolution)) + if !resolution.eids().is_empty() + ) + } +} + /// What an outcome needs besides the provider and the view. // Temporary: the request wiring, the first non-test caller, arrives in a later task. #[allow(dead_code)] @@ -1596,6 +1614,153 @@ pub(crate) fn opt_out_clears( .collect() } +/// Runs [`run_hold`] with the request's runtime and limiter. `None` when the +/// adapter set no identity runtime, so no provider is configured. +pub(crate) fn hold_before_auction( + settings: &Settings, + services: &RuntimeServices, + context: &mut EcContext, +) -> Option { + let runtime = context.identity().runtime().cloned()?; + let limiter = context.identity().rate_limiter(); + let random = ThreadRandom; + let deps = runtime.deps( + settings, + services, + &random, + limiter + .as_deref() + .map(|limiter| -> &dyn RateLimiter { limiter }), + ); + Some(run_hold(&deps, context)) +} + +/// What the hold dispatch before a navigation's auction learned (spec 4.7). +/// +/// The answers this auction's server set adds, per source, and the provider +/// that wrote each, so the output rules can apply that provider's provenance +/// default. Nothing here is stored: the same answers ride the writer's one +/// flush. +#[derive(Debug, Clone, Default, PartialEq)] +pub(crate) struct HoldResults { + /// The entry set each source now holds, in the order the provider + /// returned it. + pub(crate) additions: BTreeMap>, + /// The provider that wrote each source in `additions`. + pub(crate) writer: BTreeMap, +} + +/// Dispatches the hold-mode providers of a returning writer navigation that +/// runs an auction (spec 4.7). +/// +/// Every provider that [`eligibility`] passes in [`Phase::Hold`] on the phase 1 +/// view is called within `hold_budget_ms`, measured from the first dispatch, +/// each waiting at most `min(hold_timeout_ms, remaining budget)`. The phase 1 +/// read has already confirmed a live, consenting row, so no EC ID is +/// disclosed before consent is confirmed. Each outcome joins the writer's +/// mutation, and every answer that mutation would store is returned for this +/// auction. No answer goes into a cookie. +/// +/// A request that is not a returning writer, runs no auction, or whose phase 1 +/// read is not a live, consenting row dispatches nothing. +pub(crate) fn run_hold(deps: &IdentityDeps<'_>, context: &mut EcContext) -> HoldResults { + let mut results = HoldResults::default(); + if !context.identity().runs_auction() || !may_flush(context) { + return results; + } + let Some(ec_id) = context.ec_value().map(str::to_owned) else { + return results; + }; + let Some(entry) = context + .kv_snapshot() + .entry_for(&ec_id) + .filter(|entry| entry.consent.ok) + .cloned() + else { + return results; + }; + + let now = current_timestamp(); + let facts = request_facts(context, now); + let view = EligibilityView::from_kv(&entry); + let calls: Vec = deps + .catalog + .providers() + .filter_map(|provider| { + let id = provider.descriptor().id(); + let provider_settings = deps.settings.ec.identity.provider(id); + let Eligibility::Eligible { kinds } = eligibility( + &EligibilityInput { + provider: provider.as_ref(), + settings: &provider_settings, + identity: &deps.settings.ec.identity, + catalog: deps.catalog, + consent: context.consent(), + }, + &view, + &facts, + Phase::Hold, + ) else { + return None; + }; + Some(Call { + provider: id.clone(), + kinds, + state_json: view + .state(id) + .and_then(|state| serde_json::to_string(state).ok()), + }) + }) + .collect(); + if calls.is_empty() { + return results; + } + + let mut held = IdentityMutation::default(); + add_dispatch_results( + &mut held, + deps, + &DispatchInput { + context, + ec_id: &ec_id, + live: &entry, + view: &view, + facts: &facts, + }, + DispatchRun { + calls, + budget: Some(Duration::from_millis( + deps.settings.ec.identity.hold_budget_ms, + )), + }, + ); + + let apply_context = ApplyContext { + now, + random: deps.random, + catalog: deps.catalog, + }; + if let Some(applied) = apply(&held, &entry, &apply_context) { + for (source, record) in &applied.entry.ids { + if entry.ids.get(source) == Some(record) { + continue; + } + let (Ok(source), Some(writer)) = ( + source.parse::(), + record.w.as_deref().and_then(|writer| writer.parse().ok()), + ) else { + continue; + }; + results + .additions + .insert(source.clone(), record.eids.clone()); + results.writer.insert(source, writer); + } + } + context.identity_mut().mutation_mut().absorb(held); + results +} + #[cfg(test)] mod tests { use std::collections::BTreeMap; @@ -4621,4 +4786,495 @@ mod tests { "should judge only the opt-outs the fingerprint recorded, not a new one" ); } + + /// Hold dispatch before a writer navigation's auction (spec 4.7). + mod hold_mode { + use http::StatusCode; + + use super::*; + use crate::ec::identity::flush::FlushOutcome; + use crate::ec::identity::request::WriterDecision; + use crate::ec::identity::test_support::TEST_EC_ID; + use crate::ec::identity::test_support::fake_http::AnsweringProvider; + use crate::ec::identity::test_support::hold_fixture::{HoldFixture, live_row}; + use crate::ec::kv_types::{KvOptOut, KvProviderState, KvUid}; + + const HOLD_HOST: &str = "p.example.com"; + const HOLD_SOURCE: &str = "ids.example.com"; + + fn hold_provider( + id: &str, + host: &str, + source: &str, + ) -> (Arc, ProviderMode) { + ( + AnsweringProvider::new(id, host, source, None).into_provider(), + ProviderMode::Hold, + ) + } + + fn found_body(uid: &str) -> Vec { + serde_json::to_vec(&serde_json::json!({ + "answer": "found", + "entries": [{ "uids": [{ "id": uid, "atype": 1 }] }], + })) + .expect("should serialize the answer") + } + + fn single_entry(uid: &str) -> Vec { + vec![KvEid { + uids: vec![KvUid { + id: uid.to_owned(), + atype: Some(1), + ext: None, + }], + ..KvEid::default() + }] + } + + #[test] + fn hold_answer_feeds_this_auction_and_joins_the_mutation() { + let mut fixture = HoldFixture::new( + vec![hold_provider("p", HOLD_HOST, HOLD_SOURCE)], + &live_row(), + true, + ); + fixture + .client + .respond(HOLD_HOST, 200, &found_body("held-uid")); + + let results = fixture.run_hold(); + + assert_eq!( + results, + HoldResults { + additions: BTreeMap::from([(source(HOLD_SOURCE), single_entry("held-uid"))]), + writer: BTreeMap::from([(source(HOLD_SOURCE), pid("p"))]), + }, + "should hand this auction the answer and its writer" + ); + let staged = fixture.context.identity().mutation().positive_changes(); + assert!( + matches!( + staged, + [PartnerIdChange::FillMissing { source: staged_source, record }] + if *staged_source == source(HOLD_SOURCE) + && record.w.as_deref() == Some("p") + && record.eids == single_entry("held-uid") + ), + "should stage the answer in the writer's mutation: {staged:?}" + ); + + let outcome = fixture.flush(StatusCode::OK); + + assert_eq!(outcome, FlushOutcome::Written, "should write the answer"); + assert_eq!( + fixture.row().ids[HOLD_SOURCE].eids, + single_entry("held-uid"), + "should store the answer" + ); + assert_eq!( + fixture.client.sent_hosts(), + [HOLD_HOST], + "should not dispatch the provider again after send" + ); + let writes = fixture.store.writes(); + assert_eq!( + (writes.if_generation_match, writes.total()), + (1, 1), + "should make the one flush write and no other" + ); + } + + #[test] + fn auction_starts_at_the_hold_budget_when_a_provider_is_slow() { + let mut fixture = HoldFixture::new( + vec![ + hold_provider("a", "a.example.com", "a.example.com"), + hold_provider("b", "b.example.com", "b.example.com"), + ], + &live_row(), + true, + ); + fixture.settings.ec.identity.hold_budget_ms = 50; + fixture + .client + .respond("a.example.com", 200, &found_body("slow-uid")); + fixture + .client + .respond("b.example.com", 200, &found_body("late-uid")); + fixture + .client + .delay("a.example.com", Duration::from_millis(120)); + + let started = Instant::now(); + let results = fixture.run_hold(); + let elapsed = started.elapsed(); + + assert!( + elapsed >= Duration::from_millis(50), + "should wait out the budget before the auction starts: {elapsed:?}" + ); + assert!( + elapsed < Duration::from_millis(1_000), + "should not wait past the slow provider: {elapsed:?}" + ); + assert_eq!( + results.additions.keys().collect::>(), + [&source("a.example.com")], + "should keep the answer that arrived and drop the one the budget cut off" + ); + assert_eq!( + fixture.client.sent_hosts().len(), + 2, + "control: should have sent both calls up front" + ); + } + + #[test] + fn cut_off_hold_call_sets_hold_retry_without_counting_a_failure() { + let mut row = live_row(); + row.ps.insert( + "p".to_owned(), + KvProviderState { + f: 2, + ..KvProviderState::default() + }, + ); + let mut fixture = + HoldFixture::new(vec![hold_provider("p", HOLD_HOST, HOLD_SOURCE)], &row, true); + fixture.settings.ec.identity.hold_budget_ms = 0; + fixture.client.respond(HOLD_HOST, 200, &found_body("never")); + + let before = current_timestamp(); + let results = fixture.run_hold(); + let outcome = fixture.flush(StatusCode::OK); + + assert_eq!(results, HoldResults::default(), "should add nothing"); + assert_eq!( + outcome, + FlushOutcome::Written, + "should write the retry wait" + ); + let state = fixture.row().ps["p"].clone(); + let retry = fixture.settings.ec.identity.hold_retry_sec; + assert!( + state + .n + .is_some_and(|n| n >= before + retry && n <= before + retry + 5), + "should wait hold_retry_sec: {state:?}" + ); + assert_eq!(state.f, 2, "should leave the failure count as it was"); + assert_eq!( + fixture.limiter.count_of("identity-fail:p"), + 0, + "should not feed the breaker" + ); + } + + #[test] + fn hold_kv_opt_out_blocks_dispatch() { + let mut opted_out = live_row(); + opted_out.ps.insert( + "p".to_owned(), + KvProviderState { + o: Some(KvOptOut { + at: 900, + fp: "0".to_owned(), + }), + ..KvProviderState::default() + }, + ); + let mut blocked = HoldFixture::new( + vec![hold_provider("p", HOLD_HOST, HOLD_SOURCE)], + &opted_out, + true, + ); + blocked + .client + .respond(HOLD_HOST, 200, &found_body("blocked")); + let mut control = HoldFixture::new( + vec![hold_provider("p", HOLD_HOST, HOLD_SOURCE)], + &live_row(), + true, + ); + control + .client + .respond(HOLD_HOST, 200, &found_body("allowed")); + + let blocked_results = blocked.run_hold(); + let control_results = control.run_hold(); + + assert_eq!( + (blocked_results, blocked.client.sent_hosts()), + (HoldResults::default(), Vec::::new()), + "should call no provider the row opted out of" + ); + assert_eq!( + (control_results.additions.len(), control.client.sent_hosts()), + (1, vec![HOLD_HOST.to_owned()]), + "control: should call the provider with no opt-out" + ); + } + + #[test] + fn no_hold_on_a_stager() { + let mut stager = HoldFixture::new( + vec![hold_provider("p", HOLD_HOST, HOLD_SOURCE)], + &live_row(), + true, + ); + stager.client.respond(HOLD_HOST, 200, &found_body("stager")); + stager + .context + .identity_mut() + .set_writer_decision(WriterDecision { + writer: false, + is_real_browser: true, + user_agent: Some("ExampleBrowser/1.0".to_owned()), + dnt: false, + page_url: None, + }); + let mut writer = HoldFixture::new( + vec![hold_provider("p", HOLD_HOST, HOLD_SOURCE)], + &live_row(), + true, + ); + writer.client.respond(HOLD_HOST, 200, &found_body("writer")); + + let stager_results = stager.run_hold(); + let writer_results = writer.run_hold(); + + assert_eq!( + (stager_results, stager.client.sent_hosts()), + (HoldResults::default(), Vec::::new()), + "should make no provider call on a request that is not the writer" + ); + assert_eq!( + (writer_results.additions.len(), writer.client.sent_hosts()), + (1, vec![HOLD_HOST.to_owned()]), + "control: should call the provider on the writer" + ); + } + + #[test] + fn hold_no_match_on_a_writer_stops_recalls_from_the_spa_session() { + let mut fixture = HoldFixture::new( + vec![hold_provider("p", HOLD_HOST, HOLD_SOURCE)], + &live_row(), + true, + ); + fixture + .client + .respond(HOLD_HOST, 200, br#"{"answer":"no_match"}"#); + fixture.run_hold(); + assert_eq!( + fixture.flush(StatusCode::OK), + FlushOutcome::Written, + "should write the no_match wait" + ); + assert_eq!( + fixture.client.sent_hosts(), + [HOLD_HOST], + "control: the writer navigation called the provider once" + ); + assert!( + fixture.row().ps["p"] + .n + .is_some_and(|n| n > current_timestamp()), + "should hold the provider back with ps.n" + ); + + // The SPA session's stagers (`/auction`, `/_ts/page-bids`) read + // the stored row and are never the writer. + for _ in 0..3 { + fixture + .context + .set_kv_snapshot(fixture.graph.load_snapshot(TEST_EC_ID)); + fixture + .context + .identity_mut() + .set_writer_decision(WriterDecision { + writer: false, + is_real_browser: true, + user_agent: Some("ExampleBrowser/1.0".to_owned()), + dnt: false, + page_url: None, + }); + fixture.run_hold(); + } + // A second writer navigation inside ps.n is held back too. + fixture + .context + .identity_mut() + .set_writer_decision(WriterDecision { + writer: true, + is_real_browser: true, + user_agent: Some("ExampleBrowser/1.0".to_owned()), + dnt: false, + page_url: None, + }); + fixture.run_hold(); + + assert_eq!( + fixture.client.sent_hosts(), + [HOLD_HOST], + "should never call the provider again before ps.n" + ); + } + + #[test] + fn hold_provider_moves_after_send_on_a_navigation_without_an_auction() { + let mut fixture = HoldFixture::new( + vec![hold_provider("p", HOLD_HOST, HOLD_SOURCE)], + &live_row(), + false, + ); + fixture.client.time_out(HOLD_HOST); + + let held = fixture.run_hold(); + let before_send = fixture.client.sent_hosts(); + let outcome = fixture.flush(StatusCode::OK); + + assert_eq!( + (held, before_send), + (HoldResults::default(), Vec::::new()), + "should dispatch nothing before the response" + ); + assert_eq!(outcome, FlushOutcome::Written, "should write the failure"); + assert_eq!( + fixture.client.sent_hosts(), + [HOLD_HOST], + "should dispatch after send" + ); + assert_eq!( + fixture.backend.first_byte_timeouts(), + [Duration::from_millis( + fixture + .settings + .ec + .identity + .provider(&pid("p")) + .after_send_timeout_ms + )], + "should wait the after-send timeout, not the hold budget" + ); + assert_eq!( + fixture.row().ps["p"].f, + 1, + "should count a miss as a transport failure" + ); + assert_eq!( + fixture.limiter.count_of("identity-fail:p"), + 1, + "should feed the breaker, which a cut-off never does" + ); + } + + #[test] + fn stateful_hold_answer_flushes_its_state_with_its_ids() { + let mut fixture = HoldFixture::new( + vec![( + AnsweringProvider::stateful("p", HOLD_HOST, HOLD_SOURCE).into_provider(), + ProviderMode::Hold, + )], + &live_row(), + true, + ); + fixture + .client + .respond(HOLD_HOST, 200, &found_body("stateful-uid")); + + let results = fixture.run_hold(); + let outcome = fixture.flush(StatusCode::OK); + + assert_eq!( + results.additions.get(&source(HOLD_SOURCE)), + Some(&single_entry("stateful-uid")), + "should feed this auction" + ); + assert_eq!(outcome, FlushOutcome::Written, "should write once"); + let row = fixture.row(); + assert_eq!( + row.ids[HOLD_SOURCE].eids, + single_entry("stateful-uid"), + "should store the IDs" + ); + assert_eq!( + row.ps["p"].st, + Some(serde_json::json!({ "seen": true })), + "should store the state with its IDs" + ); + assert_eq!(fixture.store.writes().total(), 1, "should make one write"); + } + + #[test] + fn writer_navigation_with_non_2xx_status_flushes_hold_removals_and_backoff_only() { + let mut row = live_row(); + row.ids.insert( + "i.example.com".to_owned(), + KvPartnerId { + exp: Some(600), + ..record_for_test("old-uid", Some("i"), 500) + }, + ); + row.ps.insert( + "f".to_owned(), + KvProviderState { + f: 2, + ..KvProviderState::default() + }, + ); + let mut fixture = HoldFixture::new( + vec![ + hold_provider("f", "f.example.com", "f.example.com"), + hold_provider("n", "n.example.com", "n.example.com"), + hold_provider("i", "i.example.com", "i.example.com"), + ], + &row, + true, + ); + fixture + .client + .respond("f.example.com", 200, &found_body("found-uid")); + fixture + .client + .respond("n.example.com", 200, br#"{"answer":"no_match"}"#); + fixture + .client + .respond("i.example.com", 200, br#"{"answer":"invalidate"}"#); + + let results = fixture.run_hold(); + let outcome = fixture.flush(StatusCode::NOT_FOUND); + + assert_eq!( + results.additions.keys().collect::>(), + [&source("f.example.com")], + "should still feed this auction the found answer" + ); + assert_eq!( + outcome, + FlushOutcome::Written, + "should write the cut mutation" + ); + let stored = fixture.row(); + assert!( + !stored.ids.contains_key("f.example.com"), + "should store no ID from the found" + ); + assert!( + !stored.ids.contains_key("i.example.com"), + "should store the invalidate's removal" + ); + assert!( + stored.ps["n"].n.is_some_and(|n| n > current_timestamp()), + "should store the no_match backoff" + ); + assert_eq!( + stored.ps["f"].f, 2, + "should leave the found's failure count alone" + ); + assert_eq!(fixture.store.writes().total(), 1, "should make one write"); + } + } } diff --git a/crates/trusted-server-core/src/ec/identity/flush.rs b/crates/trusted-server-core/src/ec/identity/flush.rs index 964cd427ce..04cf65c878 100644 --- a/crates/trusted-server-core/src/ec/identity/flush.rs +++ b/crates/trusted-server-core/src/ec/identity/flush.rs @@ -8,12 +8,13 @@ use std::collections::{BTreeMap, BTreeSet}; use std::net::IpAddr; +use std::time::Duration; use http::StatusCode; use super::coordinator::{ - Call, Eligibility, EligibilityInput, EligibilityView, OutcomeChanges, OutcomeContext, Phase, - RequestFacts, dispatch, eligibility, opt_out_clears, outcome_to_changes, + Call, Eligibility, EligibilityInput, EligibilityView, IdentityDeps, OutcomeChanges, + OutcomeContext, Phase, RequestFacts, dispatch, eligibility, opt_out_clears, outcome_to_changes, }; use super::descriptor::{Input, Resolves}; use super::ids::{IdentityProviderId, SourceDomain, UnixTime}; @@ -176,20 +177,22 @@ pub fn run_after_send(input: AfterSendInput<'_>) -> FlushOutcome { .filter_map(|id| live_call(runtime, settings, context, &live_view, &facts, id)) .collect(); if !calls.is_empty() { + let random = ThreadRandom; + let deps = runtime.deps(settings, services, &random, Some(rate_limiter)); add_dispatch_results( &mut mutation, + &deps, &DispatchInput { - runtime, - settings, - services, context, - rate_limiter, ec_id: &ec_id, live: &live, view: &live_view, facts: &facts, }, - calls, + DispatchRun { + calls, + budget: None, + }, ); } @@ -290,7 +293,7 @@ fn write_failed( } /// What eligibility reads about the request itself. -fn request_facts(context: &EcContext, now: u64) -> RequestFacts { +pub(super) fn request_facts(context: &EcContext, now: u64) -> RequestFacts { let identity = context.identity(); let mut available_inputs = BTreeSet::new(); for (input, present) in [ @@ -311,20 +314,20 @@ fn request_facts(context: &EcContext, now: u64) -> RequestFacts { }, writer: identity.writer(), new_visitor: context.ec_generated(), - // Hold dispatch before the auction is not wired yet, so no hold-mode - // provider has run before phase 4: each runs here, as on a navigation - // that runs no auction (spec 4.7 step 7). - runs_auction: false, + // A navigation that ran an auction dispatched its hold-mode providers + // before it, so phase 4 runs them only for one that ran none + // (spec 4.7 step 7). + runs_auction: identity.runs_auction(), now: UnixTime::from_secs(now), available_inputs, } } -fn client_ip(context: &EcContext) -> Option { +pub(super) fn client_ip(context: &EcContext) -> Option { context.client_ip().and_then(|ip| ip.parse().ok()) } -fn region(context: &EcContext) -> Option { +pub(super) fn region(context: &EcContext) -> Option { context .geo_info() .and_then(|geo| Region::from_geo(&geo.country, geo.region.as_deref())) @@ -400,25 +403,29 @@ fn live_call( } /// What dispatching the due providers reads. -struct DispatchInput<'a> { - runtime: &'a IdentityRuntime, - settings: &'a Settings, - services: &'a RuntimeServices, - context: &'a EcContext, - rate_limiter: &'a dyn RateLimiter, - ec_id: &'a str, - live: &'a KvEntry, - view: &'a EligibilityView, - facts: &'a RequestFacts, +pub(super) struct DispatchInput<'a> { + pub(super) context: &'a EcContext, + pub(super) ec_id: &'a str, + pub(super) live: &'a KvEntry, + pub(super) view: &'a EligibilityView, + pub(super) facts: &'a RequestFacts, } -/// Dispatches `calls` after send and adds their outcomes to `mutation`. -fn add_dispatch_results( +/// The calls to send and, for a hold dispatch before the auction, the budget +/// they share. +pub(super) struct DispatchRun { + pub(super) calls: Vec, + pub(super) budget: Option, +} + +/// Dispatches `run.calls` and adds their outcomes to `mutation`. +pub(super) fn add_dispatch_results( mutation: &mut IdentityMutation, + deps: &IdentityDeps<'_>, input: &DispatchInput<'_>, - calls: Vec, + run: DispatchRun, ) { - let random = ThreadRandom; + let DispatchRun { calls, budget } = run; let resolving: BTreeSet<&SourceDomain> = calls.iter().flat_map(|call| call.kinds.keys()).collect(); let stored: BTreeMap = input @@ -437,26 +444,20 @@ fn add_dispatch_results( region: region(input.context), hashed_email: None, consent: ProviderConsent::from_context(input.context.consent()), - publisher_domain: &input.settings.publisher.domain, + publisher_domain: &deps.settings.publisher.domain, now: input.facts.now, - random: &random, + random: deps.random, ec_id: Some(input.ec_id.to_owned()), stored, }; - let deps = input.runtime.deps( - input.settings, - input.services, - &random, - Some(input.rate_limiter), - ); - let catalog = input.runtime.catalog(); - for (id, outcome) in dispatch(&deps, calls, &signals, None) { + let catalog = deps.catalog; + for (id, outcome) in dispatch(deps, calls, &signals, budget) { let Some(provider) = catalog.get(&id) else { continue; }; let provider: &dyn IdentityProvider = provider.as_ref(); let descriptor = provider.descriptor(); - let provider_settings = input.settings.ec.identity.provider(&id); + let provider_settings = deps.settings.ec.identity.provider(&id); let fingerprint = fingerprint_bits( descriptor.consent_policy(), input.context.consent(), @@ -469,7 +470,7 @@ fn add_dispatch_results( now: input.facts.now, catalog, settings: &provider_settings, - identity: &input.settings.ec.identity, + identity: &deps.settings.ec.identity, expected_state: stateful.then(|| { StateRevision::of( input @@ -481,13 +482,22 @@ fn add_dispatch_results( }), fingerprint: &fingerprint, }; + let found = outcome.is_found(); let changes = outcome_to_changes(provider, outcome, input.view, &outcome_context); - add_outcome(mutation, &id, changes); + add_outcome(mutation, &id, changes, found); } } /// Adds one outcome's changes to the mutation. -fn add_outcome(mutation: &mut IdentityMutation, id: &IdentityProviderId, changes: OutcomeChanges) { +/// +/// A `found` outcome's state change is tagged, so an error status drops its +/// `n` and `f` with its positive answer (spec 5.1). +fn add_outcome( + mutation: &mut IdentityMutation, + id: &IdentityProviderId, + changes: OutcomeChanges, + found: bool, +) { let OutcomeChanges { changes, result, @@ -499,7 +509,11 @@ fn add_outcome(mutation: &mut IdentityMutation, id: &IdentityProviderId, changes if let Some(result) = result { mutation.add_result(result); } - mutation.merge_state(id.clone(), state); + if found { + mutation.merge_found_state(id.clone(), state); + } else { + mutation.merge_state(id.clone(), state); + } } #[cfg(test)] diff --git a/crates/trusted-server-core/src/ec/identity/mutation.rs b/crates/trusted-server-core/src/ec/identity/mutation.rs index c4de568201..73b7686435 100644 --- a/crates/trusted-server-core/src/ec/identity/mutation.rs +++ b/crates/trusted-server-core/src/ec/identity/mutation.rs @@ -1,7 +1,7 @@ //! One request's identity changes as a single mutation, and its merge into //! the live KV row (spec 5.4). -use std::collections::BTreeMap; +use std::collections::{BTreeMap, BTreeSet}; use super::catalog::ProviderCatalog; use super::ids::{IdentityProviderId, SourceDomain}; @@ -136,6 +136,8 @@ pub(crate) struct IdentityMutation { provider_state: BTreeMap, results: Vec, cluster_size: Option, + /// Providers whose stand-alone state change comes from a `found` outcome. + found_states: BTreeSet, } impl IdentityMutation { @@ -183,6 +185,37 @@ impl IdentityMutation { .merge(change); } + /// Merges the state change of a `found` outcome and tags it, so + /// [`retain_for_error_status`](Self::retain_for_error_status) can drop + /// its `n` and `f` with the positive answer it belongs to. + pub(crate) fn merge_found_state( + &mut self, + provider: IdentityProviderId, + change: ProviderStateChange, + ) { + self.found_states.insert(provider.clone()); + self.merge_state(provider, change); + } + + /// Adds everything `other` holds, as if it had been staged here after + /// the changes already present. + pub(crate) fn absorb(&mut self, other: Self) { + let Self { + partner_ids, + provider_state, + results, + cluster_size, + found_states, + } = other; + self.partner_ids.extend(partner_ids); + for (provider, change) in provider_state { + self.merge_state(provider, change); + } + self.results.extend(results); + self.cluster_size = cluster_size.or(self.cluster_size); + self.found_states.extend(found_states); + } + /// Sets the cluster size to write. // Temporary: first used by a later identity task. #[allow(dead_code)] @@ -292,20 +325,24 @@ impl IdentityMutation { /// 2xx or 304 (spec 5.1). /// /// Keeps every removal, opt-out and clear, the `n` and `f` of each - /// stand-alone state change, and the cluster size. Drops browser ingest, - /// every other positive change, any stand-alone `st`, and each provider - /// result whole, keeping only its opt-out and clear. + /// stand-alone state change that does not come from a `found` outcome, + /// and the cluster size. Drops browser ingest, every other positive + /// change, any stand-alone `st`, the `n` and `f` of a `found` outcome, and + /// each provider result whole, keeping only its opt-out and clear. /// /// A stand-alone state change carries the `n` and `f` of an outcome other /// than a stateful `found`, whose scheduling rides in its result. A - /// stateless provider's `found` also lands here, so its `f = 0` is kept; - /// Part 1 dispatches no provider before the status is known, so no such - /// change reaches this cut yet. + /// stateless provider's `found` lands here too, and its tag + /// ([`merge_found_state`](Self::merge_found_state)) marks it for the cut. pub(crate) fn retain_for_error_status(&mut self) { self.partner_ids.retain(PartnerIdChange::is_removal); - for state in self.provider_state.values_mut() { + for (provider, state) in &mut self.provider_state { + if self.found_states.contains(provider) { + *state = state.removals(); + } state.st = StateUpdate::Keep; } + self.found_states.clear(); for result in core::mem::take(&mut self.results) { if result.state.has_removal() { self.merge_state(result.provider, result.state.removals()); @@ -972,6 +1009,85 @@ mod tests { ); } + #[test] + fn retain_for_error_status_drops_the_scheduling_of_a_found_and_keeps_its_opt_out() { + let opt_out = KvOptOut { + at: NOW, + fp: "1".to_owned(), + }; + let found = ProviderStateChange { + n: Some(NOW + 60), + f: Some(0), + opt_out: Some(opt_out.clone()), + ..ProviderStateChange::default() + }; + let backoff = ProviderStateChange { + n: Some(NOW + 86_400), + f: Some(3), + ..ProviderStateChange::default() + }; + let mut mutation = IdentityMutation::default(); + mutation.merge_found_state(pid("found"), found); + mutation.merge_state(pid("other"), backoff.clone()); + + mutation.retain_for_error_status(); + + let mut expected = IdentityMutation::default(); + expected.merge_state( + pid("found"), + ProviderStateChange { + opt_out: Some(opt_out), + ..ProviderStateChange::default() + }, + ); + expected.merge_state(pid("other"), backoff); + assert_eq!( + mutation, expected, + "should keep the found's opt-out and the other outcome's n and f, and drop the found's n and f" + ); + } + + #[test] + fn absorb_adds_changes_states_and_found_tags_after_the_existing_ones() { + let mut base = IdentityMutation::default(); + base.add_change(fill("a.example.com", KvPartnerId::browser("u", Some(NOW)))); + base.merge_state( + pid("p"), + ProviderStateChange { + n: Some(10), + ..ProviderStateChange::default() + }, + ); + let mut other = IdentityMutation::default(); + other.add_change(fill("b.example.com", KvPartnerId::browser("v", Some(NOW)))); + other.merge_found_state( + pid("p"), + ProviderStateChange { + n: Some(20), + f: Some(0), + ..ProviderStateChange::default() + }, + ); + + base.absorb(other); + + let mut expected = IdentityMutation::default(); + expected.add_change(fill("a.example.com", KvPartnerId::browser("u", Some(NOW)))); + expected.add_change(fill("b.example.com", KvPartnerId::browser("v", Some(NOW)))); + expected.merge_found_state( + pid("p"), + ProviderStateChange { + n: Some(20), + f: Some(0), + ..ProviderStateChange::default() + }, + ); + assert_eq!( + base, expected, + "should hold both mutations, the found tag included" + ); + } + #[test] fn tombstone_drops_everything() { let mut mutation = IdentityMutation::default(); diff --git a/crates/trusted-server-core/src/ec/identity/request.rs b/crates/trusted-server-core/src/ec/identity/request.rs index 22a6e845a9..6ae01ae59a 100644 --- a/crates/trusted-server-core/src/ec/identity/request.rs +++ b/crates/trusted-server-core/src/ec/identity/request.rs @@ -9,6 +9,7 @@ use super::ids::{IdentityProviderId, SourceDomain}; use super::mutation::{IdentityMutation, is_opted_out}; use super::runtime::IdentityRuntime; use crate::ec::kv_types::KvEntry; +use crate::ec::rate_limiter::RateLimiter; /// The providers with a sticky opt-out (`ps.o`) in `entry`, unless a later /// `oc` clears it (spec 7.1). @@ -37,6 +38,8 @@ pub(crate) struct RequestIdentity { dnt: bool, page_url: Option, runtime: Option>, + rate_limiter: Option>, + runs_auction: bool, mutation: IdentityMutation, pending_removals: Option, removal_lost: u32, @@ -98,6 +101,30 @@ impl RequestIdentity { self.runtime.as_ref() } + /// The limiter the provider burst cap and circuit breaker count in, when + /// the adapter set one. + #[must_use] + pub(crate) fn rate_limiter(&self) -> Option> { + self.rate_limiter.clone() + } + + /// Sets the limiter the hold dispatch counts in. + pub(crate) fn set_rate_limiter(&mut self, limiter: Arc) { + self.rate_limiter = Some(limiter); + } + + /// Whether this navigation runs an auction, so hold-mode providers were + /// dispatched before it (or not at all) and phase 4 must not run them. + #[must_use] + pub(crate) fn runs_auction(&self) -> bool { + self.runs_auction + } + + /// Records that this navigation runs an auction (spec 4.7 step 7). + pub(crate) fn set_runs_auction(&mut self, runs_auction: bool) { + self.runs_auction = runs_auction; + } + /// The writer's one flush. #[must_use] pub(crate) fn mutation(&self) -> &IdentityMutation { diff --git a/crates/trusted-server-core/src/ec/identity/test_support.rs b/crates/trusted-server-core/src/ec/identity/test_support.rs index f9c9ff6144..ca3549f168 100644 --- a/crates/trusted-server-core/src/ec/identity/test_support.rs +++ b/crates/trusted-server-core/src/ec/identity/test_support.rs @@ -951,11 +951,15 @@ pub mod fake_http { use error_stack::Report; + use crate::ec::identity::descriptor::EidProvenance; use crate::ec::identity::descriptor::{ConsentPolicy, ProviderDescriptor, Resolves}; use crate::ec::identity::exchange::{ProviderRequest, ProviderResponse}; + use crate::ec::identity::ids::SourceDomain; use crate::ec::identity::input::ResolveInput; use crate::ec::identity::provider::IdentityProvider; - use crate::ec::identity::resolution::{IdentityProviderError, Resolution}; + use crate::ec::identity::resolution::{ + IdentityProviderError, Resolution, ResolvedEid, ResolvedUid, + }; #[cfg(test)] use crate::platform::test_support::{NoopConfigStore, NoopGeo, NoopSecretStore}; use crate::platform::{ @@ -965,6 +969,7 @@ pub mod fake_http { }; #[cfg(test)] use crate::platform::{ClientInfo, RuntimeServices}; + use serde_json::json; /// What the fake does for one host. #[derive(Clone)] @@ -1304,4 +1309,273 @@ pub mod fake_http { response.success_text().map(|_| Resolution::no_match()) } } + + /// A resolver that calls its endpoint and answers as the 2xx JSON body + /// says: `{"answer":"found","entries":[{"uids":[{"id":"x","atype":1}], + /// "inserter":"example.com"}]}`, `{"answer":"invalidate"}` or anything + /// else for `no_match`. + pub(crate) struct AnsweringProvider { + descriptor: ProviderDescriptor, + stateful: bool, + } + + impl AnsweringProvider { + /// A provider `id` calling `host`, claiming `source`, with `default` + /// as its provenance default for the source when given. + pub(crate) fn new( + id: &str, + host: &str, + source: &str, + default: Option, + ) -> Self { + Self::build(id, host, source, default, false) + } + + /// As [`Self::new`], declaring private state that a `found` replaces. + pub(crate) fn stateful(id: &str, host: &str, source: &str) -> Self { + Self::build(id, host, source, None, true) + } + + fn build( + id: &str, + host: &str, + source: &str, + default: Option, + stateful: bool, + ) -> Self { + let mut resolves = Resolves::new().with_endpoint(&format!("https://{host}/resolve")); + if stateful { + resolves = resolves.with_private_state(); + } + let source: SourceDomain = source.parse().expect("should parse the source"); + let mut builder = + ProviderDescriptor::builder(id.parse().expect("should parse the provider ID")) + .with_source(source.clone()) + .with_resolves(resolves) + .with_consent_policy(ConsentPolicy::eid_persistence_only()); + if let Some(default) = default { + builder = builder.with_provenance(&source, default); + } + Self { + descriptor: builder.build().expect("should build the descriptor"), + stateful, + } + } + + pub(crate) fn into_provider(self) -> Arc { + Arc::new(self) + } + } + + impl IdentityProvider for AnsweringProvider { + fn descriptor(&self) -> &ProviderDescriptor { + &self.descriptor + } + + fn request( + &self, + input: &ResolveInput<'_>, + ) -> Result, Report> { + if self.stateful { + input.state::()?; + } + Ok(Some(ProviderRequest::get(input.endpoint()?))) + } + + fn response( + &self, + response: &ProviderResponse, + _input: &ResolveInput<'_>, + ) -> Result> { + let body: serde_json::Value = response.success_json()?; + let source = self.descriptor.primary_source(); + match body["answer"].as_str() { + Some("found") => { + let eids = body["entries"] + .as_array() + .map(Vec::as_slice) + .unwrap_or_default() + .iter() + .map(|entry| { + let mut uids = + entry["uids"].as_array().into_iter().flatten().map(|uid| { + let mut resolved = ResolvedUid::new( + uid["id"].as_str().expect("should have a UID"), + ); + if let Some(atype) = uid["atype"].as_i64() { + resolved = resolved.with_atype( + i32::try_from(atype).expect("should fit an atype"), + ); + } + resolved + }); + let mut eid = ResolvedEid::new( + source, + uids.next().expect("should have a first UID"), + ); + for uid in uids { + eid = eid.with_uid(uid); + } + if let Some(inserter) = entry["inserter"].as_str() { + eid = eid.with_inserter(inserter); + } + eid + }); + let mut resolution = Resolution::found(eids); + if self.stateful { + resolution = resolution.replacing_state(&json!({ "seen": true })); + } + Ok(resolution) + } + Some("invalidate") => Ok(Resolution::invalidate(source)), + _ => Ok(Resolution::no_match()), + } + } + } +} + +/// A returning writer navigation with a live row, hold-mode providers and a +/// fake client, for the tests of hold dispatch and the flush that follows. +#[cfg(test)] +pub(crate) mod hold_fixture { + use std::collections::BTreeMap; + use std::sync::Arc; + + use http::StatusCode; + + use super::fake_http::{FakeBackend, FakeHttpClient, fake_services}; + use super::{FakeRateLimiter, TEST_EC_ID, WriteCountingEcKv, allowing_consent}; + use crate::ec::EcContext; + use crate::ec::identity::coordinator::{HoldResults, run_hold}; + use crate::ec::identity::flush::{AfterSendInput, FlushOutcome, run_after_send}; + use crate::ec::identity::ids::{IdentityProviderId, SourceDomain}; + use crate::ec::identity::input::ThreadRandom; + use crate::ec::identity::provider::IdentityProvider; + use crate::ec::identity::request::WriterDecision; + use crate::ec::identity::runtime::IdentityRuntime; + use crate::ec::kv::KvIdentityGraph; + use crate::ec::kv_types::KvEntry; + use crate::platform::RuntimeServices; + use crate::settings::{ProviderMode, ProviderSettings, Settings}; + use crate::test_support::tests::create_test_settings; + + /// A live, consenting row sized for the flush, with no IDs. + pub(crate) fn live_row() -> KvEntry { + let mut entry = KvEntry::new(&allowing_consent(), None, 1_000, "test-publisher.com"); + entry.network = Some(crate::ec::kv_types::KvNetwork { + cluster_size: Some(1), + }); + entry + } + + /// Everything one hold navigation reads. + pub(crate) struct HoldFixture { + pub(crate) settings: Settings, + pub(crate) runtime: Arc, + pub(crate) client: Arc, + pub(crate) backend: Arc, + pub(crate) services: RuntimeServices, + pub(crate) graph: KvIdentityGraph, + pub(crate) store: Arc, + pub(crate) context: EcContext, + pub(crate) limiter: Arc, + } + + impl HoldFixture { + /// A writer navigation with `row` read in phase 1. Each provider runs + /// in the given mode. With `runs_auction` the navigation runs one. + pub(crate) fn new( + providers: Vec<(Arc, ProviderMode)>, + row: &KvEntry, + runs_auction: bool, + ) -> Self { + let mut settings = create_test_settings(); + for (provider, mode) in &providers { + settings.ec.identity.providers.insert( + provider.descriptor().id().to_string(), + ProviderSettings { + mode: *mode, + ..ProviderSettings::default() + }, + ); + } + let runtime = Arc::new( + IdentityRuntime::from_settings_with_providers( + &settings, + providers + .into_iter() + .map(|(provider, _)| provider) + .collect(), + BTreeMap::::new(), + ) + .expect("should build the runtime"), + ); + let client = Arc::new(FakeHttpClient::new()); + let backend = Arc::new(FakeBackend::new()); + let services = fake_services(&client, &backend); + let (graph, store) = KvIdentityGraph::write_counting("hold-store"); + graph.create(TEST_EC_ID, row).expect("should seed the row"); + let mut context = EcContext::new_for_test_with_ip( + Some(TEST_EC_ID.to_owned()), + allowing_consent(), + Some("192.0.2.10".to_owned()), + ); + context.set_identity_runtime(Arc::clone(&runtime)); + context.identity_mut().set_writer_decision(WriterDecision { + writer: true, + is_real_browser: true, + user_agent: Some("ExampleBrowser/1.0".to_owned()), + dnt: false, + page_url: None, + }); + context.identity_mut().set_runs_auction(runs_auction); + context.set_kv_snapshot(graph.load_snapshot(TEST_EC_ID)); + store.reset_counts(); + Self { + settings, + runtime, + client, + backend, + services, + graph, + store, + context, + limiter: Arc::new(FakeRateLimiter::new()), + } + } + + /// Runs the hold dispatch before the auction. + pub(crate) fn run_hold(&mut self) -> HoldResults { + let random = ThreadRandom; + let deps = self.runtime.deps( + &self.settings, + &self.services, + &random, + Some(self.limiter.as_ref()), + ); + run_hold(&deps, &mut self.context) + } + + /// Runs phase 4 for a response with `status`. + pub(crate) fn flush(&mut self, status: StatusCode) -> FlushOutcome { + run_after_send(AfterSendInput { + runtime: &self.runtime, + settings: &self.settings, + services: &self.services, + graph: &self.graph, + context: &mut self.context, + status, + rate_limiter: self.limiter.as_ref(), + }) + } + + /// The row as KV holds it. + pub(crate) fn row(&self) -> KvEntry { + self.graph + .get(TEST_EC_ID) + .expect("should read the row") + .expect("should hold the row") + .0 + } + } } diff --git a/crates/trusted-server-core/src/ec/mod.rs b/crates/trusted-server-core/src/ec/mod.rs index 4ca631ead0..e084affdb3 100644 --- a/crates/trusted-server-core/src/ec/mod.rs +++ b/crates/trusted-server-core/src/ec/mod.rs @@ -93,6 +93,7 @@ use self::identity::request::{RequestIdentity, WriterDecision}; use self::identity::runtime::IdentityRuntime; use self::kv::{CreateIfAbsentOutcome, KvIdentityGraph, apply_partner_id_updates}; use self::kv_types::KvEntry; +use self::rate_limiter::RateLimiter; /// Bounded request classifications that may stage browser EID cookies. /// @@ -575,6 +576,12 @@ impl EcContext { self.identity.set_runtime(runtime); } + /// Gives the request the limiter that counts provider calls for the burst + /// cap and the circuit breaker before the response is sent. + pub fn set_identity_rate_limiter(&mut self, limiter: Arc) { + self.identity.set_rate_limiter(limiter); + } + /// The identity runtime, when the adapter set one. #[must_use] pub(crate) fn identity_runtime(&self) -> Option<&IdentityRuntime> { diff --git a/crates/trusted-server-core/src/publisher.rs b/crates/trusted-server-core/src/publisher.rs index 2148e20ade..1c9daeb66d 100644 --- a/crates/trusted-server-core/src/publisher.rs +++ b/crates/trusted-server-core/src/publisher.rs @@ -66,6 +66,7 @@ use crate::creative_opportunities::{ AdStackGateInput, AssemblyMode, CreativeOpportunitiesConfig, RuntimeAdStackExpected, evaluate_ad_stack_gate, }; +use crate::ec::identity::coordinator::{HoldResults, hold_before_auction}; use crate::ec::identity::mirror::MirrorPayload; use crate::ec::identity::policy::PolicySignals; use crate::ec::kv::KvIdentityGraph; @@ -4902,6 +4903,10 @@ pub async fn handle_publisher_request( observation.set_origin_cache_shareable(origin_response_is_shareable && request_is_document); if should_run_auction { + // This navigation runs an auction: hold-mode providers run now, + // within their budget, and phase 4 leaves them alone (spec 4.7). + ec_context.identity_mut().set_runs_auction(true); + let hold = hold_before_auction(settings, services, ec_context); let slots_ctx = MatchedSlotsContext { matched_slots: &matched_slots, request_path: &request_path, @@ -4925,6 +4930,7 @@ pub async fn handle_publisher_request( signals: PolicySignals::from_headers(auction_client_request.headers()), services, geo: geo.as_ref(), + hold: hold.as_ref(), path_label: "Server-side", }, ); @@ -5524,6 +5530,9 @@ struct AuctionEidTargeting<'a> { signals: PolicySignals, services: &'a RuntimeServices, geo: Option<&'a GeoInfo>, + /// What the hold dispatch before this auction learned, on a writer + /// navigation that dispatched one. Page-bids and the tests pass `None`. + hold: Option<&'a HoldResults>, /// Prefix for the consent-stripped warning (e.g. `"Server-side"`). path_label: &'a str, } @@ -5548,12 +5557,19 @@ fn apply_auction_eids_and_device( } else { None }; - let identity = auction_identity( + let mut identity = auction_identity( targeting.kv_snapshot, targeting.partner_registry, targeting.ec_context, targeting.signals, ); + if let (Some(hold), Some(registry), Some(runtime)) = ( + targeting.hold, + targeting.partner_registry, + targeting.ec_context.identity_runtime(), + ) { + identity.add_hold_results(hold.clone(), registry, runtime.catalog()); + } let merged_eids = identity.merge(client_eids.as_deref()); let had_eids = merged_eids.as_ref().is_some_and(|v| !v.is_empty()); auction_request.user.eids = @@ -7110,6 +7126,7 @@ pub async fn handle_page_bids( signals: PolicySignals::from_headers(req.headers()), services, geo: geo.as_ref(), + hold: None, path_label: "Page-bids", }, ); @@ -8007,6 +8024,7 @@ mod tests { signals: PolicySignals::default(), services: &services, geo: None, + hold: None, path_label: "Test", }, ); @@ -8029,6 +8047,405 @@ mod tests { ); } + /// Sends the origin's requests to a [`StubHttpClient`] and a hold + /// provider's to a [`FakeHttpClient`], so one navigation can reach both. + struct HoldRoutingClient { + origin: Arc, + providers: Arc, + provider_hosts: Vec, + } + + struct RoutedPending { + provider: bool, + inner: crate::platform::PlatformPendingRequest, + } + + #[async_trait::async_trait(?Send)] + impl crate::platform::PlatformHttpClient for HoldRoutingClient { + fn supports_concurrent_fanout(&self) -> bool { + false + } + + async fn send( + &self, + request: crate::platform::PlatformHttpRequest, + ) -> Result> + { + self.origin.send(request).await + } + + async fn send_async( + &self, + request: crate::platform::PlatformHttpRequest, + ) -> Result> + { + let host = request.request.uri().host().unwrap_or_default().to_owned(); + let provider = self.provider_hosts.contains(&host); + let inner = if provider { + self.providers.send_async(request).await? + } else { + self.origin.send_async(request).await? + }; + Ok(crate::platform::PlatformPendingRequest::new( + RoutedPending { provider, inner }, + )) + } + + async fn select( + &self, + mut pending_requests: Vec, + ) -> Result> + { + let first = pending_requests.remove(0); + let ready = self.wait(first).await; + Ok(crate::platform::PlatformSelectResult { + ready, + remaining: pending_requests, + failed_backend_name: None, + }) + } + + async fn wait( + &self, + pending: crate::platform::PlatformPendingRequest, + ) -> Result> + { + let routed = pending + .downcast::() + .map_err(|_| Report::new(crate::platform::PlatformError::HttpClient))?; + if routed.provider { + self.providers.wait(routed.inner).await + } else { + self.origin.wait(routed.inner).await + } + } + } + + /// What one writer navigation with a hold provider did. + struct HoldNavigation { + provider_hosts: Vec, + auction_eids: serde_json::Value, + set_cookies: Vec, + runs_auction: bool, + staged_sources: Vec, + } + + /// Drives a returning writer navigation that runs an auction, with one + /// hold-mode provider that answers `found` for `ids.example.com`. + async fn run_hold_navigation() -> HoldNavigation { + use crate::ec::identity::test_support::fake_http::{ + AnsweringProvider, FakeBackend, FakeHttpClient, + }; + + let mut settings = scheduling_settings(); + settings.ec.identity.providers.insert( + "hold-provider".to_owned(), + crate::settings::ProviderSettings { + mode: crate::settings::ProviderMode::Hold, + ..crate::settings::ProviderSettings::default() + }, + ); + let origin = Arc::new(StubHttpClient::new()); + origin.push_response_with_headers( + 200, + b"ok".to_vec(), + vec![("content-type", "text/html; charset=utf-8")], + ); + let providers = Arc::new(FakeHttpClient::new()); + providers.respond( + "hold.example.com", + 200, + &serde_json::to_vec(&serde_json::json!({ + "answer": "found", + "entries": [{ "uids": [{ "id": "nav-uid", "atype": 1 }] }] + })) + .expect("should serialize the answer"), + ); + let client = Arc::new(HoldRoutingClient { + origin: Arc::clone(&origin), + providers: Arc::clone(&providers), + provider_hosts: vec!["hold.example.com".to_owned()], + }); + let services = crate::platform::RuntimeServices::builder() + .config_store(Arc::new(crate::platform::test_support::NoopConfigStore)) + .secret_store(Arc::new(NoopSecretStore)) + .kv_store(Arc::new(edgezero_core::key_value_store::NoopKvStore)) + .backend(Arc::new(FakeBackend::new())) + .http_client(client) + .geo(Arc::new(crate::platform::test_support::NoopGeo)) + .client_info(crate::platform::ClientInfo::default()) + .build(); + + let ec_id = format!("{}.CkId01", "b".repeat(64)); + let (graph, _store) = KvIdentityGraph::write_counting("hold-navigation-store"); + graph + .create( + &ec_id, + &KvEntry::new(&scheduling_consent(), None, 1_000, "test-publisher.com"), + ) + .expect("should seed the row"); + let mut ec_context = EcContext::new_writer_for_test_with_ip( + Some(ec_id), + scheduling_consent(), + Some("203.0.113.7".to_owned()), + ); + ec_context.identity_mut().set_writer_decision( + crate::ec::identity::request::WriterDecision { + writer: true, + is_real_browser: true, + user_agent: None, + dnt: false, + page_url: None, + }, + ); + ec_context.set_identity_runtime(Arc::new( + crate::ec::identity::IdentityRuntime::from_settings_with_providers( + &settings, + vec![ + AnsweringProvider::new( + "hold-provider", + "hold.example.com", + "ids.example.com", + None, + ) + .into_provider(), + ], + std::collections::BTreeMap::new(), + ) + .expect("should build the runtime"), + )); + let registry = PartnerRegistry::from_config(&[pull_partner_config("ids.example.com")]) + .expect("should build partner registry"); + let captured = Arc::new(Mutex::new(None)); + let mut orchestrator = AuctionOrchestrator::new(settings.auction.clone()); + orchestrator.register_provider(Arc::new(SchedulingCaptureProvider { + captured: Arc::clone(&captured), + http: Arc::clone(&origin), + lookups: Arc::new(AtomicUsize::new(0)), + })); + let slots = [scheduling_slot()]; + + let response = handle_publisher_request( + &settings, + &services, + Some(&graph), + &mut ec_context, + AuctionDispatch { + orchestrator: &orchestrator, + slots: &slots, + registry: Some(®istry), + }, + navigation_request(), + EdgeCacheHeader::SurrogateControl, + ) + .await + .expect("should proxy the origin response"); + + let headers = match &response { + PublisherResponse::PassThrough { response, .. } + | PublisherResponse::Stream { response, .. } + | PublisherResponse::AssembleTemplate { response, .. } + | PublisherResponse::Buffered(response) => response.headers().clone(), + }; + let auction_eids = captured + .lock() + .expect("should lock captured auction") + .take() + .map(|captured| { + serde_json::to_value(&captured.request.user.eids) + .expect("should serialize the EIDs") + }) + .expect("should have dispatched the auction"); + HoldNavigation { + provider_hosts: providers.sent_hosts(), + auction_eids, + set_cookies: headers + .get_all(header::SET_COOKIE) + .iter() + .map(|value| value.to_str().unwrap_or_default().to_owned()) + .collect(), + runs_auction: ec_context.identity().runs_auction(), + staged_sources: ec_context + .identity() + .mutation() + .positive_changes() + .iter() + .filter_map(|change| match change { + crate::ec::identity::mutation::PartnerIdChange::FillMissing { + source, .. + } => Some(source.to_string()), + _ => None, + }) + .collect(), + } + } + + #[tokio::test] + async fn hold_answer_reaches_the_navigation_auction_and_the_writer_mutation() { + let navigation = run_hold_navigation().await; + + assert_eq!( + navigation.provider_hosts, + ["hold.example.com"], + "should call the hold provider once, before the auction" + ); + assert_eq!( + navigation.auction_eids, + serde_json::json!([ + { "source": "ids.example.com", "uids": [{ "id": "nav-uid", "atype": 1 }] } + ]), + "should send the hold answer in this navigation's auction" + ); + assert!( + navigation.runs_auction, + "should record that the navigation ran an auction, so phase 4 skips hold providers" + ); + assert_eq!( + navigation.staged_sources, + ["ids.example.com"], + "should stage the answer in the writer's mutation" + ); + } + + #[tokio::test] + async fn hold_sets_no_cookie() { + let navigation = run_hold_navigation().await; + + assert_eq!( + navigation.provider_hosts, + ["hold.example.com"], + "control: should have run the hold provider" + ); + assert!( + navigation.set_cookies.is_empty(), + "should set no cookie from a hold answer: {:?}", + navigation.set_cookies + ); + } + + #[test] + fn full_set_hold_answer_is_sent_in_order_and_flushed_whole() { + use crate::ec::identity::descriptor::EidProvenance; + use crate::ec::identity::test_support::fake_http::AnsweringProvider; + use crate::ec::identity::test_support::hold_fixture::{HoldFixture, live_row}; + use crate::ec::kv_types::{KvEid, KvUid}; + + let provider = AnsweringProvider::new( + "hold-provider", + "hold.example.com", + "ids.example.com", + Some( + EidProvenance::new() + .with_inserter("default.example.com") + .with_mm(2), + ), + ) + .into_provider(); + let mut fixture = HoldFixture::new( + vec![(provider, crate::settings::ProviderMode::Hold)], + &live_row(), + true, + ); + fixture.client.respond( + "hold.example.com", + 200, + &serde_json::to_vec(&serde_json::json!({ + "answer": "found", + "entries": [ + { + "uids": [{ "id": "uid-a", "atype": 1 }, { "id": "uid-b", "atype": 3 }], + "inserter": "vendor.example.com" + }, + { "uids": [{ "id": "uid-c", "atype": 1 }] } + ] + })) + .expect("should serialize the answer"), + ); + let snapshot = fixture.context.kv_snapshot().clone(); + let ec_id = fixture + .context + .ec_value() + .expect("should hold an EC ID") + .to_owned(); + let registry = PartnerRegistry::from_config(&[pull_partner_config("ids.example.com")]) + .expect("should build partner registry"); + + let hold = fixture.run_hold(); + let services = noop_services(); + let mut auction_request = canonical_parity_auction_request(); + auction_request.user.consent = None; + auction_request.user.eids = None; + apply_auction_eids_and_device( + &mut auction_request, + &AuctionEidTargeting { + cookie_jar: None, + ec_id: Some(ec_id.as_str()), + kv_snapshot: &snapshot, + partner_registry: Some(®istry), + ec_context: &fixture.context, + signals: PolicySignals::default(), + services: &services, + geo: None, + hold: Some(&hold), + path_label: "Test", + }, + ); + let outcome = fixture.flush(http::StatusCode::OK); + + assert_eq!( + serde_json::to_value(&auction_request.user.eids).expect("should serialize EIDs"), + serde_json::json!([ + { + "source": "ids.example.com", + "inserter": "vendor.example.com", + "uids": [{ "id": "uid-a", "atype": 1 }, { "id": "uid-b", "atype": 3 }] + }, + { + "source": "ids.example.com", + "inserter": "default.example.com", + "mm": 2, + "uids": [{ "id": "uid-c", "atype": 1 }] + } + ]), + "should send both entries in order, the first as returned and the second with the provenance default" + ); + assert_eq!( + outcome, + crate::ec::identity::flush::FlushOutcome::Written, + "should flush the answer" + ); + let uid = |id: &str, atype: i32| KvUid { + id: id.to_owned(), + atype: Some(atype), + ext: None, + }; + assert_eq!( + fixture.row().ids["ids.example.com"].eids, + vec![ + KvEid { + uids: vec![uid("uid-a", 1), uid("uid-b", 3)], + inserter: Some("vendor.example.com".to_owned()), + matcher: None, + mm: None, + ext: None, + }, + KvEid { + uids: vec![uid("uid-c", 1)], + inserter: None, + matcher: None, + mm: None, + ext: None, + }, + ], + "should store the set as returned, without the default" + ); + assert_eq!( + fixture.store.writes().total(), + 1, + "should make the one flush write" + ); + } + /// The navigation and page-bids merge for `snapshot` and a `ts-eids` /// cookie holding `cookie_entries`, plus any `extra_cookie`. fn publisher_merge( @@ -8061,6 +8478,7 @@ mod tests { signals: PolicySignals::default(), services: &services, geo: None, + hold: None, path_label: "Test", }, ); From a13902fca3b82af293dab7a3ed1793e9e452c60d Mon Sep 17 00:00:00 2001 From: dhruv8sh Date: Tue, 27 Oct 2026 03:16:46 +0530 Subject: [PATCH 129/194] Make hold dispatch honour call timeouts and fail closed without a limiter The fake provider client now fails a wait that outlasts the first-byte timeout its backend was given, so the slow-provider test pins a real cut-off. Phase 4 treats hold-mode providers as already handled only when the hold dispatch ran on a usable phase 1 row. Without a rate limiter on the request the hold dispatch sends nothing. Signed-off-by: dhruv8sh --- .../src/ec/identity/coordinator.rs | 101 ++++++++++++++---- .../src/ec/identity/flush.rs | 11 +- .../src/ec/identity/request.rs | 17 ++- .../src/ec/identity/test_support.rs | 36 +++++++ crates/trusted-server-core/src/publisher.rs | 31 +++++- 5 files changed, 164 insertions(+), 32 deletions(-) diff --git a/crates/trusted-server-core/src/ec/identity/coordinator.rs b/crates/trusted-server-core/src/ec/identity/coordinator.rs index 84b56d34f1..731a684e4d 100644 --- a/crates/trusted-server-core/src/ec/identity/coordinator.rs +++ b/crates/trusted-server-core/src/ec/identity/coordinator.rs @@ -1615,22 +1615,26 @@ pub(crate) fn opt_out_clears( } /// Runs [`run_hold`] with the request's runtime and limiter. `None` when the -/// adapter set no identity runtime, so no provider is configured. +/// adapter set no identity runtime (no provider is configured) or no limiter +/// (fails closed). pub(crate) fn hold_before_auction( settings: &Settings, services: &RuntimeServices, context: &mut EcContext, ) -> Option { let runtime = context.identity().runtime().cloned()?; - let limiter = context.identity().rate_limiter(); + let Some(limiter) = context.identity().rate_limiter() else { + // Without a limiter the burst cap and breaker cannot hold, so fail + // closed: dispatch nothing before the response. + log::warn!("Skipping the identity hold dispatch: no rate limiter is set for the request"); + return None; + }; let random = ThreadRandom; let deps = runtime.deps( settings, services, &random, - limiter - .as_deref() - .map(|limiter| -> &dyn RateLimiter { limiter }), + Some(limiter.as_ref() as &dyn RateLimiter), ); Some(run_hold(&deps, context)) } @@ -1681,7 +1685,8 @@ pub(crate) fn run_hold(deps: &IdentityDeps<'_>, context: &mut EcContext) -> Hold }; let now = current_timestamp(); - let facts = request_facts(context, now); + let facts = request_facts(context, now, true); + context.identity_mut().set_hold_dispatched(); let view = EligibilityView::from_kv(&entry); let calls: Vec = deps .catalog @@ -1735,6 +1740,8 @@ pub(crate) fn run_hold(deps: &IdentityDeps<'_>, context: &mut EcContext) -> Hold }, ); + // Computed on the phase 1 row; the flush applies the same mutation to the + // phase 4 live row, which may differ. let apply_context = ApplyContext { now, random: deps.random, @@ -1749,6 +1756,9 @@ pub(crate) fn run_hold(deps: &IdentityDeps<'_>, context: &mut EcContext) -> Hold source.parse::(), record.w.as_deref().and_then(|writer| writer.parse().ok()), ) else { + log::debug!( + "Skipping a hold answer for source '{source}': no parseable writer on the record" + ); continue; }; results @@ -4149,7 +4159,7 @@ mod tests { fixture.client.respond("c.example.com", 200, b"{}"); fixture .client - .delay("a.example.com", Duration::from_millis(60)); + .delay("b.example.com", Duration::from_millis(60)); fixture.client.time_out("c.example.com"); let outcomes = fixture.run( @@ -4899,35 +4909,84 @@ mod tests { fixture.settings.ec.identity.hold_budget_ms = 50; fixture .client - .respond("a.example.com", 200, &found_body("slow-uid")); + .respond("a.example.com", 200, &found_body("fast-uid")); fixture .client .respond("b.example.com", 200, &found_body("late-uid")); - fixture - .client - .delay("a.example.com", Duration::from_millis(120)); + let delay = Duration::from_millis(400); + fixture.client.delay("b.example.com", delay); + let before = current_timestamp(); let started = Instant::now(); let results = fixture.run_hold(); let elapsed = started.elapsed(); + let outcome = fixture.flush(StatusCode::OK); assert!( - elapsed >= Duration::from_millis(50), - "should wait out the budget before the auction starts: {elapsed:?}" - ); - assert!( - elapsed < Duration::from_millis(1_000), - "should not wait past the slow provider: {elapsed:?}" + // The call's timeout is the budget left when it was sent. + elapsed >= Duration::from_millis(45) && elapsed < delay, + "should start the auction at the budget, not at the slow answer: {elapsed:?}" ); assert_eq!( results.additions.keys().collect::>(), [&source("a.example.com")], - "should keep the answer that arrived and drop the one the budget cut off" + "should keep the fast answer and drop the slow one" + ); + assert_eq!(outcome, FlushOutcome::Written, "should write the flush"); + let row = fixture.row(); + let retry = fixture.settings.ec.identity.hold_retry_sec; + assert!( + row.ps["b"] + .n + .is_some_and(|n| n >= before + retry && n <= before + retry + 5), + "should wait hold_retry_sec after the cut-off: {:?}", + row.ps["b"] + ); + assert_eq!(row.ps["b"].f, 0, "should count no failure for the cut-off"); + assert_eq!( + fixture.limiter.count_of("identity-fail:b"), + 0, + "should not feed the breaker" + ); + assert!( + row.ids.contains_key("a.example.com") && !row.ids.contains_key("b.example.com"), + "should store the fast answer and not the slow one" + ); + } + + #[test] + fn hold_providers_still_run_after_send_when_hold_found_no_usable_row() { + let mut fixture = HoldFixture::new( + vec![hold_provider("p", HOLD_HOST, HOLD_SOURCE)], + &live_row(), + true, ); + fixture + .client + .respond(HOLD_HOST, 200, &found_body("late-uid")); + fixture + .context + .set_kv_snapshot(crate::ec::EcKvSnapshot::NotRead); + + let held = fixture.run_hold(); + let before_send = fixture.client.sent_hosts(); + let outcome = fixture.flush(StatusCode::OK); + + assert_eq!( + (held, before_send), + (HoldResults::default(), Vec::::new()), + "should dispatch nothing in hold without a usable row" + ); + assert_eq!(outcome, FlushOutcome::Written, "should write the answer"); assert_eq!( - fixture.client.sent_hosts().len(), - 2, - "control: should have sent both calls up front" + fixture.client.sent_hosts(), + [HOLD_HOST], + "should run the hold-mode provider once, after send" + ); + assert_eq!( + fixture.row().ids[HOLD_SOURCE].eids, + single_entry("late-uid"), + "should store the answer" ); } diff --git a/crates/trusted-server-core/src/ec/identity/flush.rs b/crates/trusted-server-core/src/ec/identity/flush.rs index 04cf65c878..85caddfa36 100644 --- a/crates/trusted-server-core/src/ec/identity/flush.rs +++ b/crates/trusted-server-core/src/ec/identity/flush.rs @@ -113,7 +113,9 @@ pub fn run_after_send(input: AfterSendInput<'_>) -> FlushOutcome { } let now = current_timestamp(); - let facts = request_facts(context, now); + // Hold-mode providers run in phase 4 unless the hold dispatch already + // considered them on a usable row (spec 4.7 step 7). + let facts = request_facts(context, now, context.identity().hold_dispatched()); let phase_one = context.kv_snapshot().clone(); let phase_one_entry = phase_one.entry_for(&ec_id); let phase_one_view = @@ -293,7 +295,7 @@ fn write_failed( } /// What eligibility reads about the request itself. -pub(super) fn request_facts(context: &EcContext, now: u64) -> RequestFacts { +pub(super) fn request_facts(context: &EcContext, now: u64, runs_auction: bool) -> RequestFacts { let identity = context.identity(); let mut available_inputs = BTreeSet::new(); for (input, present) in [ @@ -314,10 +316,7 @@ pub(super) fn request_facts(context: &EcContext, now: u64) -> RequestFacts { }, writer: identity.writer(), new_visitor: context.ec_generated(), - // A navigation that ran an auction dispatched its hold-mode providers - // before it, so phase 4 runs them only for one that ran none - // (spec 4.7 step 7). - runs_auction: identity.runs_auction(), + runs_auction, now: UnixTime::from_secs(now), available_inputs, } diff --git a/crates/trusted-server-core/src/ec/identity/request.rs b/crates/trusted-server-core/src/ec/identity/request.rs index 6ae01ae59a..8bac2b50f2 100644 --- a/crates/trusted-server-core/src/ec/identity/request.rs +++ b/crates/trusted-server-core/src/ec/identity/request.rs @@ -40,6 +40,7 @@ pub(crate) struct RequestIdentity { runtime: Option>, rate_limiter: Option>, runs_auction: bool, + hold_dispatched: bool, mutation: IdentityMutation, pending_removals: Option, removal_lost: u32, @@ -113,13 +114,25 @@ impl RequestIdentity { self.rate_limiter = Some(limiter); } - /// Whether this navigation runs an auction, so hold-mode providers were - /// dispatched before it (or not at all) and phase 4 must not run them. + /// Whether this navigation runs an auction, so hold-mode providers may + /// be dispatched before it. #[must_use] pub(crate) fn runs_auction(&self) -> bool { self.runs_auction } + /// Whether the hold dispatch considered every hold-mode provider on a + /// usable phase 1 row, so phase 4 must not run them again. + #[must_use] + pub(crate) fn hold_dispatched(&self) -> bool { + self.hold_dispatched + } + + /// Records that the hold dispatch ran on a usable phase 1 row. + pub(crate) fn set_hold_dispatched(&mut self) { + self.hold_dispatched = true; + } + /// Records that this navigation runs an auction (spec 4.7 step 7). pub(crate) fn set_runs_auction(&mut self, runs_auction: bool) { self.runs_auction = runs_auction; diff --git a/crates/trusted-server-core/src/ec/identity/test_support.rs b/crates/trusted-server-core/src/ec/identity/test_support.rs index ca3549f168..fa4c73c632 100644 --- a/crates/trusted-server-core/src/ec/identity/test_support.rs +++ b/crates/trusted-server-core/src/ec/identity/test_support.rs @@ -986,6 +986,8 @@ pub mod fake_http { host: String, route: Route, delay: Duration, + /// The first-byte timeout the request's backend was registered with. + timeout: Option, } /// An HTTP client that answers per host, and records what it was sent. @@ -994,6 +996,7 @@ pub mod fake_http { delays: Mutex>, sent: Mutex>, uris: Mutex>, + backend: Mutex>>, } impl FakeHttpClient { @@ -1005,9 +1008,17 @@ pub mod fake_http { delays: Mutex::new(BTreeMap::new()), sent: Mutex::new(Vec::new()), uris: Mutex::new(Vec::new()), + backend: Mutex::new(None), } } + /// Makes a wait that outlasts the first-byte timeout `backend` last + /// registered for the host fail once that timeout has passed, as a + /// real backend does. + pub fn honour_timeouts_of(&self, backend: &Arc) { + *self.backend.lock().expect("should lock the backend") = Some(Arc::clone(backend)); + } + /// Answers `host` with `status` and `body`. pub fn respond(&self, host: &str, status: u16, body: &[u8]) { self.respond_with_headers(host, status, body, &[]); @@ -1101,10 +1112,17 @@ pub mod fake_http { .get(&host) .copied() .unwrap_or_default(); + let timeout = self + .backend + .lock() + .expect("should lock the backend") + .as_ref() + .and_then(|backend| backend.last_first_byte_timeout(&host)); Ok(PlatformPendingRequest::new(FakePending { host, route, delay, + timeout, })) } @@ -1131,6 +1149,12 @@ pub mod fake_http { let pending = pending .downcast::() .map_err(|_| Report::new(PlatformError::HttpClient))?; + if let Some(timeout) = pending.timeout + && pending.delay > timeout + { + std::thread::sleep(timeout); + return Err(Report::new(PlatformError::HttpClient)); + } if !pending.delay.is_zero() { std::thread::sleep(pending.delay); } @@ -1168,6 +1192,17 @@ pub mod fake_http { } } + /// The first-byte timeout `host` was last registered with. + fn last_first_byte_timeout(&self, host: &str) -> Option { + self.specs + .lock() + .expect("should lock the specs") + .iter() + .rev() + .find(|spec| spec.host == host) + .map(|spec| spec.first_byte_timeout) + } + /// The first-byte timeouts the backends were registered with, in order. pub fn first_byte_timeouts(&self) -> Vec { self.specs @@ -1206,6 +1241,7 @@ pub mod fake_http { client: &Arc, backend: &Arc, ) -> RuntimeServices { + client.honour_timeouts_of(backend); RuntimeServices::builder() .config_store(Arc::new(NoopConfigStore)) .secret_store(Arc::new(NoopSecretStore)) diff --git a/crates/trusted-server-core/src/publisher.rs b/crates/trusted-server-core/src/publisher.rs index 1c9daeb66d..0318952834 100644 --- a/crates/trusted-server-core/src/publisher.rs +++ b/crates/trusted-server-core/src/publisher.rs @@ -8132,7 +8132,7 @@ mod tests { /// Drives a returning writer navigation that runs an auction, with one /// hold-mode provider that answers `found` for `ids.example.com`. - async fn run_hold_navigation() -> HoldNavigation { + async fn run_hold_navigation(with_limiter: bool) -> HoldNavigation { use crate::ec::identity::test_support::fake_http::{ AnsweringProvider, FakeBackend, FakeHttpClient, }; @@ -8214,6 +8214,11 @@ mod tests { ) .expect("should build the runtime"), )); + if with_limiter { + ec_context.set_identity_rate_limiter(Arc::new( + crate::ec::identity::test_support::FakeRateLimiter::new(), + )); + } let registry = PartnerRegistry::from_config(&[pull_partner_config("ids.example.com")]) .expect("should build partner registry"); let captured = Arc::new(Mutex::new(None)); @@ -8282,7 +8287,7 @@ mod tests { #[tokio::test] async fn hold_answer_reaches_the_navigation_auction_and_the_writer_mutation() { - let navigation = run_hold_navigation().await; + let navigation = run_hold_navigation(true).await; assert_eq!( navigation.provider_hosts, @@ -8307,9 +8312,29 @@ mod tests { ); } + #[tokio::test] + async fn hold_without_a_rate_limiter_dispatches_nothing() { + let navigation = run_hold_navigation(false).await; + + assert!( + navigation.provider_hosts.is_empty(), + "should fail closed and call no provider without a limiter: {:?}", + navigation.provider_hosts + ); + assert_eq!( + navigation.auction_eids, + serde_json::Value::Null, + "should run the auction without a hold answer" + ); + assert!( + navigation.runs_auction && navigation.staged_sources.is_empty(), + "control: the navigation still ran its auction and staged nothing" + ); + } + #[tokio::test] async fn hold_sets_no_cookie() { - let navigation = run_hold_navigation().await; + let navigation = run_hold_navigation(true).await; assert_eq!( navigation.provider_hosts, From 347b6827ecd0f8d8eeb826afa1e963da86732c11 Mon Sep 17 00:00:00 2001 From: dhruv8sh Date: Tue, 27 Oct 2026 03:44:52 +0530 Subject: [PATCH 130/194] Capture identity revocations from proxied provider responses Capture is revoke-only: a real browser's captured removals, opt-outs and clears are written to KV on the proxied request with a CAS loop, retried once after send, and counted as lost if both fail. Positive IDs are counted, never stored. A non-browser client's capture writes nothing. revoke_scope = "global" runs the consent-withdrawal tombstone from a real browser. The Lockr proxy now logs only the upstream path, not its query. Signed-off-by: dhruv8sh --- .../trusted-server-adapter-fastly/src/main.rs | 173 +++- .../trusted-server-core/src/ec/client_eids.rs | 39 +- crates/trusted-server-core/src/ec/finalize.rs | 179 +++- .../src/ec/identity/coordinator.rs | 853 +++++++++++++++++- .../src/ec/identity/descriptor.rs | 2 - .../src/ec/identity/flush.rs | 49 + .../src/ec/identity/input.rs | 2 - .../src/ec/identity/measurement.rs | 79 +- .../src/ec/identity/mod.rs | 2 - .../src/ec/identity/mutation.rs | 5 +- .../src/ec/identity/request.rs | 12 + .../src/ec/identity/resolution.rs | 2 - .../src/ec/identity/test_support.rs | 296 +++++- crates/trusted-server-core/src/ec/mod.rs | 49 + .../src/integrations/lockr.rs | 41 +- .../src/integrations/registry.rs | 574 +++++++++++- .../trusted-server-core/src/test_support.rs | 56 ++ 17 files changed, 2350 insertions(+), 63 deletions(-) diff --git a/crates/trusted-server-adapter-fastly/src/main.rs b/crates/trusted-server-adapter-fastly/src/main.rs index 76e51957d4..0baa30ee8b 100644 --- a/crates/trusted-server-adapter-fastly/src/main.rs +++ b/crates/trusted-server-adapter-fastly/src/main.rs @@ -18,7 +18,8 @@ use trusted_server_core::ec::device::DeviceSignals; use trusted_server_core::ec::finalize::{ec_finalize_response, retry_failed_withdrawals}; use trusted_server_core::ec::identity::IdentityRuntime; use trusted_server_core::ec::identity::flush::{ - AfterSendInput, FlushOutcome, may_flush, run_after_send, + AfterSendInput, FlushOutcome, lose_pending_removals, may_flush, retry_pending_removals, + run_after_send, }; use trusted_server_core::ec::identity::measurement::{IdentityMeasurement, MeasurementInput}; use trusted_server_core::ec::kv::KvIdentityGraph; @@ -712,6 +713,17 @@ fn finalize_send_and_flush_measured( } let withdrawal_after_send_failed = ec_state.ec_context.has_failed_withdrawals(); + // A capture removal write that failed before send gets one more attempt. + if ec_state.ec_context.has_pending_removals() { + match graph_once() { + Some(graph) => retry_pending_removals(&mut ec_state.ec_context, &graph), + None => { + log::warn!("capture removal retry skipped: identity graph unavailable"); + lose_pending_removals(&mut ec_state.ec_context); + } + } + } + let outcome = flush_after_send( runtime, settings, @@ -3826,6 +3838,165 @@ mod tests { ); } + /// A scenario whose Didomi proxy answers `body`, with a capture-only + /// provider listening on that proxy's path and a row holding a + /// browser-written `ssp.example.com` ID for it to invalidate. + fn capture_scenario(body: &str) -> Scenario { + let Scenario { + state, + store, + network, + } = Scenario::with_settings(DIDOMI_TOML); + let provider = + trusted_server_core::ec::identity::test_support::answering_capture_provider( + "didomi", + &["ssp.example.com"], + None, + ); + let runtime = IdentityRuntime::from_settings_with_providers( + &state.settings, + vec![provider], + BTreeMap::new(), + ) + .expect("should build the identity runtime"); + let scenario = Scenario { + state: crate::app::app_state_with_identity_for_test(state, runtime), + store, + network, + }; + scenario + .network + .respond("sdk.example.com", 200, "application/json", body); + let mut entry = live_entry(); + entry.ids.insert( + "ssp.example.com".to_owned(), + KvPartnerId { + rv: Some("0123456789abcdef".to_owned()), + at: Some(1_000), + ..KvPartnerId::new("probe-uid") + }, + ); + scenario.seed(&entry); + scenario + } + + const CAPTURE_INVALIDATION: &str = r#"{"invalidate":["ssp.example.com"]}"#; + + #[test] + fn capture_removal_is_a_separate_write_beside_the_one_flush() { + let control = capture_scenario(r#"{}"#); + control.drive(didomi_navigation(Some(&returning_with_shared_id()))); + assert_writes( + &control.store, + (0, 1, 0), + "control: a proxied navigation with nothing to capture flushes once", + ); + + let scenario = capture_scenario(CAPTURE_INVALIDATION); + + let (response, outcome) = + scenario.drive(didomi_navigation(Some(&returning_with_shared_id()))); + + assert_eq!(response.status(), 200, "should serve the proxied answer"); + assert_eq!( + outcome, + Some(FlushOutcome::Written), + "should still flush the writer's ingest" + ); + assert_writes( + &scenario.store, + (0, 2, 0), + "the capture removal write, then the writer's one flush", + ); + let row = scenario.row(); + assert_eq!( + stored_uid(&row, "ssp.example.com"), + None, + "should have removed the captured invalidation" + ); + assert_eq!( + stored_uid(&row, "sharedid.org").as_deref(), + Some(SHARED_ID), + "should keep the writer's own ingest in its flush" + ); + } + + #[test] + fn failed_capture_removal_write_runs_once_more_after_send() { + let scenario = capture_scenario(CAPTURE_INVALIDATION); + scenario.store.refuse_conditional_writes(EC_ID, 5); + + let (outcome, measurement) = drive_measured_for_test( + &scenario.state, + didomi_navigation(Some(&returning_with_shared_id())), + ); + + assert_eq!( + outcome, + Some(FlushOutcome::Written), + "should flush the ingest too" + ); + assert_writes( + &scenario.store, + (0, 7, 0), + "five refused attempts before send, the retry after send, the flush", + ); + assert_eq!( + stored_uid(&scenario.row(), "ssp.example.com"), + None, + "should have written the removal after send" + ); + assert_eq!(measurement.removal_lost(), 0, "should lose no removal"); + } + + #[test] + fn capture_removal_lost_after_both_attempts_is_counted() { + let scenario = capture_scenario(CAPTURE_INVALIDATION); + scenario.store.refuse_conditional_writes(EC_ID, 10); + + let (_, measurement) = drive_measured_for_test( + &scenario.state, + didomi_navigation(Some(&returning_with_shared_id())), + ); + + assert_eq!( + measurement.removal_lost(), + 1, + "should count the removal lost after both attempts" + ); + assert!( + measurement.to_string().contains("removal_lost=1"), + "should print the count on the measurement line" + ); + } + + #[test] + fn capture_positive_ids_are_counted_on_the_measurement_line_and_not_stored() { + let scenario = capture_scenario( + r#"{"tokens":[{"source":"ssp.example.com","value":"captured-secret-uid"}]}"#, + ); + + let (_, measurement) = drive_measured_for_test( + &scenario.state, + didomi_navigation(Some(&returning_with_shared_id())), + ); + + let line = measurement.to_string(); + assert!( + line.contains("capture_unpersisted=didomi/ssp.example.com:1"), + "should count the positive ID: {line}" + ); + assert!( + !line.contains("captured-secret-uid"), + "should print no value: {line}" + ); + assert_eq!( + stored_uid(&scenario.row(), "ssp.example.com").as_deref(), + Some("probe-uid"), + "should not store the captured ID" + ); + } + #[test] fn registered_source_beyond_todays_trim_reaches_kv_on_next_navigation() { use base64::Engine as _; diff --git a/crates/trusted-server-core/src/ec/client_eids.rs b/crates/trusted-server-core/src/ec/client_eids.rs index 30951b9d21..91e824f916 100644 --- a/crates/trusted-server-core/src/ec/client_eids.rs +++ b/crates/trusted-server-core/src/ec/client_eids.rs @@ -216,11 +216,10 @@ fn parse_client_eid_ext(value: &JsonValue) -> Option { #[cfg(test)] mod tests { - use std::sync::{Mutex, Once}; - use serde_json::json; use super::*; + use crate::test_support::logs::{captured_logs, install_capturing_logger}; fn parse(text: &str) -> Vec { let raw = RawValue::from_string(text.to_owned()).expect("should capture raw JSON"); @@ -245,42 +244,6 @@ mod tests { format!(r#"{{"k":"{}"}}"#, "x".repeat(bytes - 8)) } - static CAPTURED_LOGS: Mutex> = Mutex::new(Vec::new()); - static INSTALL_LOGGER: Once = Once::new(); - - struct CapturingLogger; - - impl log::Log for CapturingLogger { - fn enabled(&self, metadata: &log::Metadata<'_>) -> bool { - metadata.level() <= log::Level::Debug - } - - fn log(&self, record: &log::Record<'_>) { - if self.enabled(record.metadata()) { - CAPTURED_LOGS - .lock() - .expect("should lock captured logs") - .push(record.args().to_string()); - } - } - - fn flush(&self) {} - } - - fn install_capturing_logger() { - INSTALL_LOGGER.call_once(|| { - log::set_logger(&CapturingLogger).expect("should install the capturing test logger"); - log::set_max_level(log::LevelFilter::Debug); - }); - } - - fn captured_logs() -> Vec { - CAPTURED_LOGS - .lock() - .expect("should lock captured logs") - .clone() - } - #[test] fn keeps_a_valid_entry_with_every_field() { let parsed = parsed_json( diff --git a/crates/trusted-server-core/src/ec/finalize.rs b/crates/trusted-server-core/src/ec/finalize.rs index ac8b32c8ec..c7e8130c1d 100644 --- a/crates/trusted-server-core/src/ec/finalize.rs +++ b/crates/trusted-server-core/src/ec/finalize.rs @@ -60,9 +60,16 @@ pub fn ec_finalize_response( response: &mut Response, ) { let consent_allows_ec = ec_consent_granted(ec_context.consent()); - let consent_withdrawn = ec_consent_withdrawn(ec_context.consent()); - - if !consent_allows_ec { + // A global revoke captured from a real browser (spec 8) is a withdrawal: + // the same tombstone, cookie expiry and after-send run, with whatever the + // request staged dropped. + let global_revoke = ec_context.global_revoke_requested(); + let consent_withdrawn = ec_consent_withdrawn(ec_context.consent()) || global_revoke; + + if !consent_allows_ec || global_revoke { + if global_revoke { + *ec_context.identity_mut().mutation_mut() = IdentityMutation::default(); + } finalize_unusable_consent( settings, ec_context, @@ -2977,6 +2984,172 @@ mod tests { } } + mod capture_revoke { + use super::*; + use crate::ec::identity::mutation::{IdentityMutation, PartnerIdChange}; + use crate::ec::identity::revision::Revision; + use crate::ec::identity::test_support::{pid, source}; + + fn finalize(context: &mut EcContext, graph: &KvIdentityGraph) -> Response { + let mut response = empty_response(); + ec_finalize_response( + &create_test_settings(), + context, + Some(graph), + &PartnerRegistry::empty(), + None, + None, + &mut response, + ); + response + } + + fn consenting_context(ec_id: &str) -> EcContext { + let mut context = make_context_with_consent( + Some(ec_id), + Some(ec_id), + true, + false, + granting_consent(), + ); + context + .identity_mut() + .mutation_mut() + .add_change(PartnerIdChange::OwnerRemove { + source: source("a.example.com"), + provider: pid("lockr"), + expected: Revision::Unset, + }); + context + } + + fn expires_the_ec_cookie(response: &Response) -> bool { + response + .headers() + .get_all("set-cookie") + .iter() + .filter_map(|value| value.to_str().ok()) + .any(|cookie| cookie.starts_with("ts-ec=") && cookie.contains("Max-Age=0")) + } + + #[test] + fn capture_global_revoke_from_a_real_browser_tombstones_the_row_and_clears_the_ec_cookie() { + let (graph, store) = KvIdentityGraph::write_counting("ec-store"); + let ec_id = sample_ec_id("revoke"); + graph + .create(&ec_id, &KvEntry::minimal("p.example.com", "uid", 1_000)) + .expect("should seed the row"); + store.reset_counts(); + let mut context = consenting_context(&ec_id); + context.set_kv_snapshot(graph.load_snapshot(&ec_id)); + context.request_global_revoke(); + + let response = finalize(&mut context, &graph); + + let (stored, _) = graph + .get(&ec_id) + .expect("should read the row") + .expect("should keep the tombstone"); + assert!(!stored.consent.ok, "should tombstone the row"); + assert!(stored.ids.is_empty(), "should remove every ID"); + assert_eq!( + store.writes().if_generation_match, + 1, + "should write one tombstone" + ); + assert!( + expires_the_ec_cookie(&response), + "should expire the EC cookie" + ); + assert!( + context.failed_withdrawals().is_empty(), + "should record no failed tombstone" + ); + assert_eq!( + context.identity().mutation(), + &IdentityMutation::default(), + "should drop the request's mutation" + ); + + let (graph, store) = KvIdentityGraph::write_counting("ec-store"); + graph + .create(&ec_id, &KvEntry::minimal("p.example.com", "uid", 1_000)) + .expect("should seed the row"); + store.reset_counts(); + let mut control = consenting_context(&ec_id); + control.set_kv_snapshot(graph.load_snapshot(&ec_id)); + let response = finalize(&mut control, &graph); + assert!( + graph + .get(&ec_id) + .expect("should read the row") + .is_some_and(|(row, _)| row.consent.ok), + "control: without a revoke the row stays live" + ); + assert!( + !expires_the_ec_cookie(&response), + "control: without a revoke the cookie stays" + ); + assert_eq!(store.writes().total(), 0, "control: writes nothing"); + } + + #[test] + fn capture_global_revoke_failed_tombstone_gets_the_after_send_run() { + let (graph, store) = KvIdentityGraph::write_counting("ec-store"); + let ec_id = sample_ec_id("rvfail"); + graph + .create(&ec_id, &KvEntry::minimal("p.example.com", "uid", 1_000)) + .expect("should seed the row"); + store.refuse_conditional_writes(&ec_id, 5); + let mut context = consenting_context(&ec_id); + context.set_kv_snapshot(graph.load_snapshot(&ec_id)); + context.request_global_revoke(); + + let response = finalize(&mut context, &graph); + + assert_eq!( + context.failed_withdrawals(), + std::slice::from_ref(&ec_id), + "should record the failed tombstone for the after-send run" + ); + assert!( + expires_the_ec_cookie(&response), + "should still expire the cookie" + ); + + retry_failed_withdrawals(&mut context, &graph); + + assert!( + graph + .get(&ec_id) + .expect("should read the row") + .is_some_and(|(row, _)| !row.consent.ok), + "should tombstone the row after send" + ); + } + + #[test] + fn capture_global_revoke_without_an_ec_cookie_expires_nothing() { + let (graph, store) = KvIdentityGraph::write_counting("ec-store"); + let ec_id = sample_ec_id("nocook"); + graph + .create(&ec_id, &KvEntry::minimal("p.example.com", "uid", 1_000)) + .expect("should seed the row"); + store.reset_counts(); + let mut context = + make_context_with_consent(Some(&ec_id), None, false, true, granting_consent()); + context.request_global_revoke(); + + let response = finalize(&mut context, &graph); + + assert!( + !expires_the_ec_cookie(&response), + "should expire no cookie the request did not carry" + ); + assert_eq!(store.writes().total(), 0, "should tombstone no row"); + } + } + fn live_entry() -> KvEntry { let mut entry = KvEntry::tombstone(1000); entry.consent.ok = true; diff --git a/crates/trusted-server-core/src/ec/identity/coordinator.rs b/crates/trusted-server-core/src/ec/identity/coordinator.rs index 731a684e4d..f6811e6954 100644 --- a/crates/trusted-server-core/src/ec/identity/coordinator.rs +++ b/crates/trusted-server-core/src/ec/identity/coordinator.rs @@ -16,26 +16,31 @@ use super::descriptor::{Input, ProviderDescriptor, ResolveKind, Resolves}; use super::exchange::{ProviderRequest, ProviderResponse}; use super::flush::{DispatchInput, DispatchRun, add_dispatch_results, may_flush, request_facts}; use super::ids::{IdentityProviderId, SourceDomain, UnixTime}; -use super::input::{RandomSource, ResolveInput, ResolveSignals, ThreadRandom}; +use super::input::{ + CaptureInput, ProviderConsent, RandomSource, ResolveInput, ResolveSignals, StoredId, + ThreadRandom, +}; use super::mutation::{ ApplyContext, IdentityMutation, PartnerIdChange, ProviderResult, ProviderStateChange, StateUpdate, apply, }; use super::policy::{FP_US_OPT_OUT_MASK, PolicySignals, fingerprint_bits, policy_allows}; use super::provider::IdentityProvider; +use super::removal::{RemovalWriteOutcome, write_removals}; use super::resolution::{ EntrySet, EntrySets, IdentityProviderError, Resolution, ResolvedEid, StateChange, entry_sets, }; use super::revision::{IdentityKeys, StateRevision, record_revision, uid_hash, visitor_key}; use crate::consent::{ConsentContext, effective_tcf}; +use crate::ec::kv::KvIdentityGraph; use crate::ec::kv_types::{KvEid, KvEntry, KvOptOut, KvPartnerId}; use crate::ec::rate_limiter::RateLimiter; -use crate::ec::{EcContext, current_timestamp}; +use crate::ec::{EcContext, EcKvSnapshot, current_timestamp, log_id}; use crate::platform::{ PlatformBackendSpec, PlatformHttpRequest, PlatformPendingRequest, PlatformResponse, RuntimeServices, }; -use crate::settings::{IdentitySettings, ProviderMode, ProviderSettings, Settings}; +use crate::settings::{IdentitySettings, ProviderMode, ProviderSettings, RevokeScope, Settings}; use crate::streaming_processor::{BodyStreamDecoder, Compression}; /// Deciseconds in a second: `LastUpdated` of a TC string counts deciseconds. @@ -1771,6 +1776,294 @@ pub(crate) fn run_hold(deps: &IdentityDeps<'_>, context: &mut EcContext) -> Hold results } +/// One proxied response offered for capture (spec 4.7). +/// +/// There is no request body here: capture never reads it. +pub(crate) struct CaptureCall<'a> { + pub(crate) method: &'a http::Method, + pub(crate) path: &'a str, + pub(crate) is_real_browser: bool, + pub(crate) status: StatusCode, + pub(crate) headers: &'a http::HeaderMap, + /// The decoded body, at most `max_response_bytes`. + pub(crate) body: &'a [u8], +} + +/// Whether any enabled provider's capture rule matches the request. +#[must_use] +pub(crate) fn has_capture_rule( + catalog: &ProviderCatalog, + method: &http::Method, + path: &str, +) -> bool { + catalog.providers().any(|provider| { + provider + .descriptor() + .captures() + .is_some_and(|captures| captures.matches(method, path)) + }) +} + +/// The row a capture reads its view from: the request's own snapshot when it +/// holds one for this EC ID, else one read. Empty when the row is missing, +/// unreadable or not a live row. +fn capture_view( + graph: Option<&KvIdentityGraph>, + context: &EcContext, + ec_id: Option<&str>, +) -> EligibilityView { + let (Some(graph), Some(ec_id)) = (graph, ec_id) else { + return EligibilityView::empty(); + }; + let snapshot = if context.kv_snapshot().belongs_to(ec_id) { + context.kv_snapshot().clone() + } else { + graph.load_snapshot(ec_id) + }; + match snapshot { + EcKvSnapshot::Present { entry, .. } if entry.consent.ok => EligibilityView::from_kv(&entry), + EcKvSnapshot::Failed { .. } => { + log::warn!( + "Identity capture for '{}' could not read the row, so its removals cannot be built", + log_id(ec_id) + ); + EligibilityView::empty() + } + _ => EligibilityView::empty(), + } +} + +/// Whether eligibility steps 1 to 3 of spec 4.7 pass for the request. +fn passes_consent_steps(descriptor: &ProviderDescriptor, context: &EcContext, now: u64) -> bool { + let facts = request_facts(context, now, false); + facts.is_real_browser + && facts.ec_persistence_allowed + && policy_allows( + descriptor.consent_policy(), + context.consent(), + facts.policy_signals, + ) +} + +/// What one capture result asks for, once the real-browser gate has passed. +struct CaptureAsk { + removals: Vec, + state: ProviderStateChange, +} + +/// The sources the provider claims that a result carries IDs for, once each. +fn positive_sources( + provider: &IdentityProviderId, + resolution: &Resolution, + catalog: &ProviderCatalog, +) -> BTreeSet { + let claimed = catalog.sources_of(provider); + resolution + .eids() + .iter() + .map(|eid| eid.source().clone()) + .filter(|source| claimed.contains(source)) + .collect() +} + +/// Reads one result for its removals, opt-out and clear, and nothing else. +fn capture_ask( + deps: &IdentityDeps<'_>, + context: &EcContext, + provider: &dyn IdentityProvider, + resolution: Resolution, + view: &EligibilityView, + positive: bool, +) -> CaptureAsk { + let descriptor = provider.descriptor(); + let id = descriptor.id(); + let now = UnixTime::from_secs(current_timestamp()); + if resolution.changes_state() { + log::warn!("Identity provider '{id}' returned a state change from capture, ignoring it"); + } + let opted_out = resolution.is_opted_out(); + let facts = request_facts(context, now.as_secs(), false); + let fingerprint = fingerprint_bits( + descriptor.consent_policy(), + context.consent(), + facts.policy_signals, + ); + let provider_settings = deps.settings.ec.identity.provider(id); + let changes = outcome_to_changes( + provider, + CallOutcome::Resolved(resolution), + view, + &OutcomeContext { + now, + catalog: deps.catalog, + settings: &provider_settings, + identity: &deps.settings.ec.identity, + expected_state: None, + fingerprint: &fingerprint, + }, + ); + let OutcomeChanges { + changes, + result, + state, + } = changes; + let resulting = result.map_or(state, |result| result.state); + let mut ask = CaptureAsk { + removals: changes + .into_iter() + .filter(PartnerIdChange::is_removal) + .collect(), + state: ProviderStateChange::default(), + }; + if let Some(opt_out) = resulting.opt_out.filter(|_| opted_out) { + ask.state.opt_out = Some(opt_out); + ask.state.st = StateUpdate::Clear; + } + // Only a capture-only provider without a GVL ID has no other way to clear: + // a vendor that issues again has been re-consented through (spec 8). + let only_capture_clears = + deps.catalog.is_capture_only(id) && descriptor.consent_policy().tcf_vendor().is_none(); + if positive + && only_capture_clears + && view.is_opted_out(id) + && passes_consent_steps(descriptor, context, now.as_secs()) + { + ask.state.opt_out_clear = Some(now.as_secs()); + } + ask +} + +/// Offers a proxied 2xx response to each provider whose capture rule matches +/// (spec 4.7). +/// +/// Capture is revoke-only. A real browser's removals, opt-outs and clears are +/// written at once with the removal write, whatever the consent; a request +/// that is not a real browser writes and stages nothing. A positive ID is +/// counted and never stored. A `revoke_scope = "global"` opt-out asks +/// finalize for the consent-withdrawal tombstone path instead. +pub(crate) fn run_capture( + deps: &IdentityDeps<'_>, + graph: Option<&KvIdentityGraph>, + context: &mut EcContext, + call: &CaptureCall<'_>, +) { + let matching: Vec<&std::sync::Arc> = deps + .catalog + .providers() + .filter(|provider| { + provider + .descriptor() + .captures() + .is_some_and(|captures| captures.matches(call.method, call.path)) + }) + .collect(); + if matching.is_empty() { + return; + } + let ec_id = context.ec_value().map(str::to_owned); + let view = if call.is_real_browser { + capture_view(graph, context, ec_id.as_deref()) + } else { + EligibilityView::empty() + }; + let response = + ProviderResponse::new(call.status, call.headers.clone(), call.body.to_vec(), None); + let consent = ProviderConsent::from_context(context.consent()); + let now = UnixTime::from_secs(current_timestamp()); + + let mut mutation = IdentityMutation::default(); + for provider in matching { + let descriptor = provider.descriptor(); + let id = descriptor.id(); + let stored: BTreeMap = view + .records() + .filter(|(source, _)| deps.catalog.sources_of(id).contains(source)) + .filter_map(|(source, record)| { + StoredId::from_record(source, record).map(|stored| (source.clone(), stored)) + }) + .collect(); + let input = CaptureInput::new( + descriptor, + call.path, + &response, + &stored, + &consent, + &deps.settings.publisher.domain, + now, + ); + let resolution = match provider.capture(&input) { + Ok(resolution) => resolution, + Err(error) => { + if !error.current_context().is_not_implemented() { + log::warn!( + "Identity provider '{id}' failed to capture: {}", + error.current_context() + ); + } + continue; + } + }; + + let positive_sources = positive_sources(id, &resolution, deps.catalog); + for source in &positive_sources { + context + .identity_mut() + .record_unpersisted(id.clone(), source.clone()); + } + if !call.is_real_browser { + for source in resolution.invalidated() { + log::debug!( + "Dropping a capture removal of provider '{id}', source '{source}': the client is not a real browser" + ); + } + if resolution.is_opted_out() { + log::debug!( + "Dropping a capture opt-out of provider '{id}': the client is not a real browser" + ); + } + continue; + } + if resolution.is_opted_out() + && deps.settings.ec.identity.provider(id).revoke_scope() == RevokeScope::Global + { + context.request_global_revoke(); + continue; + } + + let ask = capture_ask( + deps, + context, + provider.as_ref(), + resolution, + &view, + !positive_sources.is_empty(), + ); + for removal in ask.removals { + mutation.add_change(removal); + } + mutation.merge_state(id.clone(), ask.state); + } + + if !mutation.has_removals() { + return; + } + let (Some(graph), Some(ec_id)) = (graph, ec_id.as_deref()) else { + log::debug!("Dropping capture removals: no identity graph or EC ID for the request"); + return; + }; + let apply_context = ApplyContext { + now: now.as_secs(), + random: deps.random, + catalog: deps.catalog, + }; + if write_removals(graph, ec_id, &mutation, &apply_context) == RemovalWriteOutcome::Failed { + if let Some(earlier) = context.identity().pending_removals().cloned() { + mutation.absorb(earlier); + } + context.identity_mut().set_pending_removals(Some(mutation)); + } +} + #[cfg(test)] mod tests { use std::collections::BTreeMap; @@ -5336,4 +5629,558 @@ mod tests { assert_eq!(fixture.store.writes().total(), 1, "should make one write"); } } + + /// Revoke-only capture from proxied responses (spec 4.7). + mod capture_run { + use serde_json::json; + + use super::*; + use crate::consent::jurisdiction::Jurisdiction; + use crate::ec::identity::revision::set_hashes; + use crate::ec::identity::test_support::capture_fixture::{CaptureFixture, capture_path}; + use crate::ec::identity::test_support::hold_fixture::live_row; + use crate::ec::identity::test_support::{ + TEST_EC_ID, answering_capture_provider, record_for_test, + }; + use crate::ec::kv_types::KvProviderState; + use crate::test_support::logs::{captured_logs, install_capturing_logger}; + + const FIRST: &str = "a.example.com"; + const SECOND: &str = "b.example.com"; + + fn lockr() -> Arc { + answering_capture_provider("lockr", &[FIRST, SECOND], None) + } + + fn lockr_path() -> String { + capture_path("lockr") + } + + fn row_with(records: &[(&str, &str, Option<&str>)]) -> KvEntry { + let mut row = live_row(); + for (source, uid, writer) in records { + row.ids + .insert((*source).to_owned(), record_for_test(uid, *writer, 500)); + } + row + } + + fn opted_out_row(at: u64) -> KvEntry { + let mut row = live_row(); + row.ps.insert( + "lockr".to_owned(), + KvProviderState { + o: Some(KvOptOut { + at, + fp: "0".to_owned(), + }), + ..KvProviderState::default() + }, + ); + row + } + + fn denied_consent() -> ConsentContext { + ConsentContext { + gdpr_applies: true, + jurisdiction: Jurisdiction::Gdpr, + ..ConsentContext::default() + } + } + + #[test] + fn capture_stores_no_positive_id_and_counts_it() { + let mut fixture = CaptureFixture::new(vec![lockr()], Some(&live_row())); + + fixture.capture( + &lockr_path(), + &json!({"tokens": [ + {"source": FIRST, "value": "secret-uid-1"}, + {"source": SECOND, "value": "secret-uid-2"}, + ]}), + ); + + assert_eq!( + fixture.row(), + live_row(), + "should store neither positive ID" + ); + assert_eq!(fixture.store.writes().total(), 0, "should make no KV write"); + let counted = fixture.context.identity().capture_unpersisted(); + assert_eq!( + counted, + &BTreeMap::from([ + ((pid("lockr"), source(FIRST)), 1), + ((pid("lockr"), source(SECOND)), 1), + ]), + "should count one per source" + ); + assert!( + !format!("{counted:?}").contains("secret-uid"), + "should hold no value" + ); + } + + #[test] + fn capture_removal_is_written_before_send() { + let seeded = row_with(&[(FIRST, "secret-uid-a", None)]); + let mut fixture = CaptureFixture::new(vec![lockr()], Some(&seeded)); + + fixture.capture( + &lockr_path(), + &json!({"tokens": [{"source": SECOND, "value": "secret-uid-b"}], + "invalidate": [FIRST]}), + ); + + let writes = fixture.store.writes(); + assert_eq!( + writes.if_generation_match, 1, + "should write once with a CAS" + ); + assert_eq!(writes.total(), 1, "should make no other write"); + let row = fixture.row(); + assert!(!row.ids.contains_key(FIRST), "should remove the record"); + assert!( + !row.ids.contains_key(SECOND), + "should store no positive ID alongside" + ); + assert_eq!( + row.ps["lockr"].iv, + set_hashes(&seeded.ids[FIRST].eids), + "should hash the removed record into iv" + ); + assert!( + fixture.context.identity().pending_removals().is_none(), + "should leave nothing to retry" + ); + } + + #[test] + fn capture_removal_applies_whatever_the_consent() { + let seeded = row_with(&[(FIRST, "secret-uid-a", None)]); + let body = json!({"invalidate": [FIRST]}); + + let mut granted = CaptureFixture::new(vec![lockr()], Some(&seeded)); + granted.capture(&lockr_path(), &body); + let mut denied = + CaptureFixture::new(vec![lockr()], Some(&seeded)).with_consent(denied_consent()); + denied.capture(&lockr_path(), &body); + + assert_eq!( + denied.row(), + granted.row(), + "should write the same removal under denied consent" + ); + assert!( + !denied.row().ids.contains_key(FIRST), + "should remove the record" + ); + assert_eq!(denied.store.writes().total(), 1, "should write once"); + } + + #[test] + fn capture_removal_write_retries_only_on_generation_mismatch() { + let seeded = row_with(&[(FIRST, "secret-uid-a", None)]); + let body = json!({"invalidate": [FIRST]}); + + let mut mismatched = CaptureFixture::new(vec![lockr()], Some(&seeded)); + mismatched.store.refuse_conditional_writes(TEST_EC_ID, 2); + mismatched.capture(&lockr_path(), &body); + assert_eq!( + mismatched.store.writes().if_generation_match, + 3, + "should retry each generation mismatch" + ); + assert!( + !mismatched.row().ids.contains_key(FIRST), + "should win on the third write" + ); + + let mut failing = CaptureFixture::new(vec![lockr()], Some(&seeded)); + failing.store.fail_conditional_writes(TEST_EC_ID); + failing.capture(&lockr_path(), &body); + assert_eq!( + failing.store.writes().if_generation_match, + 1, + "should not retry a non-CAS error" + ); + } + + #[test] + fn failed_capture_removal_write_runs_once_more_after_send() { + let seeded = row_with(&[(FIRST, "secret-uid-a", None)]); + let mut fixture = CaptureFixture::new(vec![lockr()], Some(&seeded)); + fixture.store.refuse_conditional_writes(TEST_EC_ID, 5); + + fixture.capture(&lockr_path(), &json!({"invalidate": [FIRST]})); + + assert!( + fixture.row().ids.contains_key(FIRST), + "should leave the record after five refusals" + ); + assert!( + fixture.context.identity().pending_removals().is_some(), + "should keep the removal for the retry" + ); + assert_eq!( + fixture.context.identity().removal_lost(), + 0, + "should not count a removal lost yet" + ); + + fixture.retry_after_send(); + + assert!( + !fixture.row().ids.contains_key(FIRST), + "should write the removal after send" + ); + assert!( + fixture.context.identity().pending_removals().is_none(), + "should clear the pending removal" + ); + assert_eq!( + fixture.context.identity().removal_lost(), + 0, + "should count nothing lost" + ); + } + + #[test] + fn capture_removal_lost_after_both_attempts_is_counted_and_logged_without_a_value() { + install_capturing_logger(); + let provider = answering_capture_provider("lostcap", &[FIRST], None); + let seeded = row_with(&[(FIRST, "secret-uid-lost-9137", None)]); + let mut fixture = CaptureFixture::new(vec![provider], Some(&seeded)); + fixture.store.fail_conditional_writes(TEST_EC_ID); + + fixture.capture(&capture_path("lostcap"), &json!({"invalidate": [FIRST]})); + fixture.retry_after_send(); + + assert_eq!( + fixture.context.identity().removal_lost(), + 1, + "should count the lost removal" + ); + assert!( + fixture.context.identity().pending_removals().is_none(), + "should keep nothing after the last attempt" + ); + assert!( + fixture.row().ids.contains_key(FIRST), + "control: the record is still there" + ); + let logs = captured_logs(); + assert!( + logs.iter().any(|line| line.contains("removal lost") + && line.contains("lostcap") + && line.contains(FIRST)), + "should log the provider and source: {logs:?}" + ); + assert!( + !logs + .iter() + .any(|line| line.contains("secret-uid-lost-9137")), + "should log no value" + ); + } + + #[test] + fn capture_removal_write_skips_a_tombstone_a_missing_row_and_an_applied_removal() { + let body = json!({"invalidate": [FIRST]}); + + let mut tombstone = + CaptureFixture::new(vec![lockr()], Some(&KvEntry::tombstone(1_000))); + tombstone.capture(&lockr_path(), &body); + let mut missing = CaptureFixture::new(vec![lockr()], None); + missing.capture(&lockr_path(), &body); + let mut applied = CaptureFixture::new(vec![lockr()], Some(&live_row())); + applied.capture(&lockr_path(), &body); + + for (name, fixture) in [ + ("a tombstone", &tombstone), + ("a missing row", &missing), + ("an applied removal", &applied), + ] { + assert_eq!( + fixture.store.writes().total(), + 0, + "should write nothing for {name}" + ); + assert!( + fixture.context.identity().pending_removals().is_none(), + "should keep nothing to retry for {name}" + ); + } + let mut control = CaptureFixture::new( + vec![lockr()], + Some(&row_with(&[(FIRST, "secret-uid-a", None)])), + ); + control.capture(&lockr_path(), &body); + assert_eq!( + control.store.writes().total(), + 1, + "control: a live record is written" + ); + } + + #[test] + fn capture_opt_out_removes_every_record_the_provider_wrote() { + let seeded = row_with(&[ + (FIRST, "secret-uid-a", Some("lockr")), + (SECOND, "secret-uid-b", Some("other")), + ]); + let mut fixture = CaptureFixture::new(vec![lockr()], Some(&seeded)); + + fixture.capture(&lockr_path(), &json!({"opted_out": true})); + + let row = fixture.row(); + assert!( + !row.ids.contains_key(FIRST), + "should remove the record the provider wrote" + ); + assert!( + row.ids.contains_key(SECOND), + "control: should keep another writer's record" + ); + assert!(row.ps["lockr"].o.is_some(), "should store the opt-out"); + assert_eq!( + row.ps["lockr"].ov, + set_hashes(&seeded.ids[FIRST].eids), + "should hash the removed record into ov" + ); + assert_eq!(fixture.store.writes().total(), 1, "should write once"); + } + + #[test] + fn capture_only_opt_out_also_removes_browser_records_of_its_sources() { + let seeded = row_with(&[ + (FIRST, "secret-uid-a", None), + ("unrelated.example.com", "secret-uid-c", None), + ]); + let mut fixture = CaptureFixture::new(vec![lockr()], Some(&seeded)); + + fixture.capture(&lockr_path(), &json!({"opted_out": true})); + + let row = fixture.row(); + assert!( + !row.ids.contains_key(FIRST), + "should remove the browser record of its source" + ); + assert!( + row.ids.contains_key("unrelated.example.com"), + "control: should keep a browser record of another source" + ); + } + + #[test] + fn capture_only_provider_clears_its_opt_out_on_a_consenting_positive_result() { + let seeded = opted_out_row(100); + let positive = json!({"tokens": [{"source": FIRST, "value": "secret-uid-a"}]}); + + let mut fixture = CaptureFixture::new(vec![lockr()], Some(&seeded)); + fixture.capture(&lockr_path(), &positive); + let row = fixture.row(); + assert!( + row.ps["lockr"].oc.is_some_and(|clear| clear > 100), + "should store the clear" + ); + assert!(row.ids.is_empty(), "should store no ID"); + assert_eq!(fixture.store.writes().total(), 1, "should write once"); + + let mut denied = + CaptureFixture::new(vec![lockr()], Some(&seeded)).with_consent(denied_consent()); + denied.capture(&lockr_path(), &positive); + assert_eq!( + denied.row().ps["lockr"].oc, + None, + "control: should not clear under denied consent" + ); + assert_eq!(denied.store.writes().total(), 0, "should not write"); + + let mut vendor = CaptureFixture::new( + vec![answering_capture_provider("lockr", &[FIRST], Some(7))], + Some(&seeded), + ); + vendor.capture(&lockr_path(), &positive); + assert_eq!( + vendor.row().ps["lockr"].oc, + None, + "control: a provider with a GVL ID never clears by capture" + ); + + let mut empty = CaptureFixture::new(vec![lockr()], Some(&seeded)); + empty.capture(&lockr_path(), &json!({})); + assert_eq!( + empty.row().ps["lockr"].oc, + None, + "control: a result without IDs does not clear" + ); + } + + #[test] + fn capture_global_revoke_from_a_real_browser_requests_the_tombstone_and_writes_nothing() { + let seeded = row_with(&[(FIRST, "secret-uid-a", Some("lockr"))]); + let mut global = CaptureFixture::with_scopes(vec![lockr()], Some(&seeded), &["lockr"]); + + global.capture(&lockr_path(), &json!({"opted_out": true})); + + assert!( + global.context.global_revoke_requested(), + "should request the global revoke" + ); + assert_eq!(global.row(), seeded, "should leave the row to finalize"); + assert_eq!(global.store.writes().total(), 0, "should write nothing"); + + let mut provider = CaptureFixture::new(vec![lockr()], Some(&seeded)); + provider.capture(&lockr_path(), &json!({"opted_out": true})); + assert!( + !provider.context.global_revoke_requested(), + "control: the provider scope requests no global revoke" + ); + assert!( + provider.row().ps["lockr"].o.is_some(), + "control: the provider scope stores the opt-out" + ); + } + + #[test] + fn capture_from_a_non_browser_writes_nothing() { + let seeded = row_with(&[(FIRST, "secret-uid-a", Some("lockr"))]); + for body in [ + json!({"invalidate": [FIRST]}), + json!({"opted_out": true}), + json!({"tokens": [{"source": FIRST, "value": "secret-uid-z"}]}), + ] { + let mut fixture = + CaptureFixture::with_scopes(vec![lockr()], Some(&seeded), &["lockr"]) + .not_a_browser(); + + fixture.capture(&lockr_path(), &body); + + assert_eq!( + fixture.store.writes().total(), + 0, + "should write nothing for {body}" + ); + assert_eq!(fixture.row(), seeded, "should change no row for {body}"); + assert!( + !fixture.context.global_revoke_requested(), + "should request no global revoke for {body}" + ); + assert!( + fixture.context.identity().pending_removals().is_none(), + "should stage nothing for {body}" + ); + assert!( + fixture.context.identity().mutation().is_empty(), + "should add nothing to the mutation for {body}" + ); + } + let mut positive = CaptureFixture::new(vec![lockr()], Some(&seeded)).not_a_browser(); + positive.capture( + &lockr_path(), + &json!({"tokens": [{"source": FIRST, "value": "secret-uid-z"}]}), + ); + assert_eq!( + positive.context.identity().capture_unpersisted().len(), + 1, + "should still count the positive ID" + ); + } + + #[test] + fn capture_on_a_proxy_writer_navigation_uses_the_removal_write_not_the_mutation() { + let seeded = row_with(&[(FIRST, "secret-uid-a", None)]); + let mut fixture = CaptureFixture::new(vec![lockr()], Some(&seeded)); + fixture.context.identity_mut().set_writer_decision( + crate::ec::identity::request::WriterDecision { + writer: true, + is_real_browser: true, + user_agent: None, + dnt: false, + page_url: None, + }, + ); + + fixture.capture(&lockr_path(), &json!({"invalidate": [FIRST]})); + + assert!( + fixture.context.identity().mutation().is_empty(), + "should leave the writer's mutation alone" + ); + assert_eq!( + fixture.store.writes().if_generation_match, + 1, + "should write the removal at once" + ); + assert!( + !fixture.row().ids.contains_key(FIRST), + "should remove the record" + ); + } + + #[test] + fn capture_never_touches_n_f_or_the_breaker() { + let mut seeded = row_with(&[(FIRST, "secret-uid-a", None)]); + seeded.ps.insert( + "lockr".to_owned(), + KvProviderState { + n: Some(9_999_999_999), + f: 2, + ..KvProviderState::default() + }, + ); + for body in [ + json!({"invalidate": [FIRST]}), + json!({"opted_out": true}), + json!({}), + ] { + let mut fixture = CaptureFixture::new(vec![lockr()], Some(&seeded)); + + fixture.capture(&lockr_path(), &body); + + let state = &fixture.row().ps["lockr"]; + assert_eq!(state.n, Some(9_999_999_999), "should keep n for {body}"); + assert_eq!(state.f, 2, "should keep f for {body}"); + } + } + + #[test] + fn capture_state_change_is_ignored() { + let mut seeded = row_with(&[(FIRST, "secret-uid-a", None)]); + seeded.ps.insert( + "statecap".to_owned(), + KvProviderState { + st: Some(json!("old-state")), + ..KvProviderState::default() + }, + ); + install_capturing_logger(); + let provider = answering_capture_provider("statecap", &[FIRST], None); + let mut fixture = CaptureFixture::new(vec![provider], Some(&seeded)); + + fixture.capture( + &capture_path("statecap"), + &json!({"invalidate": [FIRST], "state": "new-state"}), + ); + assert!( + captured_logs() + .iter() + .any(|line| line.contains("'statecap'") + && line.contains("state change from capture")), + "should warn about the ignored state change" + ); + + let row = fixture.row(); + assert_eq!( + row.ps["statecap"].st, + Some(json!("old-state")), + "should keep the stored state" + ); + assert!( + !row.ids.contains_key(FIRST), + "control: should still apply the removal" + ); + } + } } diff --git a/crates/trusted-server-core/src/ec/identity/descriptor.rs b/crates/trusted-server-core/src/ec/identity/descriptor.rs index c49da6d47d..9f7af44bd6 100644 --- a/crates/trusted-server-core/src/ec/identity/descriptor.rs +++ b/crates/trusted-server-core/src/ec/identity/descriptor.rs @@ -461,8 +461,6 @@ impl Captures { /// Whether a proxied request matches a rule. #[must_use] - // Temporary: first used by a later identity task. - #[allow(dead_code)] pub(crate) fn matches(&self, method: &Method, path: &str) -> bool { self.paths .iter() diff --git a/crates/trusted-server-core/src/ec/identity/flush.rs b/crates/trusted-server-core/src/ec/identity/flush.rs index 85caddfa36..e7c9f6eaeb 100644 --- a/crates/trusted-server-core/src/ec/identity/flush.rs +++ b/crates/trusted-server-core/src/ec/identity/flush.rs @@ -294,6 +294,55 @@ fn write_failed( FlushOutcome::WriteFailed { removals: outcome } } +/// Tries once more, after send, the capture removal write that failed before +/// send (spec 5.3). +/// +/// This is the last attempt: the pending removals are taken, and each one +/// that still cannot be written is logged and counted as lost. Without a +/// pending removal it does nothing. +pub fn retry_pending_removals(context: &mut EcContext, graph: &KvIdentityGraph) { + let Some(removals) = context.identity().pending_removals().cloned() else { + return; + }; + let ec_id = context.ec_value().map(str::to_owned); + let runtime = context.identity().runtime().cloned(); + let (Some(ec_id), Some(runtime)) = (ec_id, runtime) else { + log::warn!("Capture removal retry skipped: no EC ID or identity runtime"); + lose_pending_removals(context); + return; + }; + let random = ThreadRandom; + let outcome = write_removals( + graph, + &ec_id, + &removals, + &ApplyContext { + now: current_timestamp(), + random: &random, + catalog: runtime.catalog(), + }, + ); + if outcome == RemovalWriteOutcome::Failed { + lose_pending_removals(context); + } else { + context.identity_mut().set_pending_removals(None); + } +} + +/// Gives up the capture removal write that failed before send: logs and +/// counts each removal as lost. For an after-send run that cannot even try, +/// such as one without an identity graph. +pub fn lose_pending_removals(context: &mut EcContext) { + let Some(removals) = context.identity().pending_removals().cloned() else { + return; + }; + context.identity_mut().set_pending_removals(None); + log_removal_lost(context.ec_value().unwrap_or_default(), &removals); + for _ in removals.removal_labels() { + context.identity_mut().record_removal_lost(); + } +} + /// What eligibility reads about the request itself. pub(super) fn request_facts(context: &EcContext, now: u64, runs_auction: bool) -> RequestFacts { let identity = context.identity(); diff --git a/crates/trusted-server-core/src/ec/identity/input.rs b/crates/trusted-server-core/src/ec/identity/input.rs index b54ab7012b..58237163b0 100644 --- a/crates/trusted-server-core/src/ec/identity/input.rs +++ b/crates/trusted-server-core/src/ec/identity/input.rs @@ -654,8 +654,6 @@ impl fmt::Debug for CaptureInput<'_> { impl<'a> CaptureInput<'a> { /// The input for one capture. #[must_use] - // Temporary: first used by a later identity task. - #[allow(dead_code)] pub(crate) fn new( descriptor: &'a ProviderDescriptor, path: &'a str, diff --git a/crates/trusted-server-core/src/ec/identity/measurement.rs b/crates/trusted-server-core/src/ec/identity/measurement.rs index 7d9edf6190..c65563083f 100644 --- a/crates/trusted-server-core/src/ec/identity/measurement.rs +++ b/crates/trusted-server-core/src/ec/identity/measurement.rs @@ -5,7 +5,10 @@ use core::fmt::Write as _; +use std::collections::BTreeMap; + use super::flush::FlushOutcome; +use super::ids::{IdentityProviderId, SourceDomain}; /// What kind of identity route the request was. #[derive(Debug, Clone, Copy, PartialEq, Eq, derive_more::Display)] @@ -49,6 +52,7 @@ pub struct IdentityMeasurement { pub(crate) removal_lost: u32, pub(crate) withdrawal_after_send: bool, pub(crate) withdrawal_after_send_failed: bool, + pub(crate) capture_unpersisted: BTreeMap<(IdentityProviderId, SourceDomain), u32>, } impl IdentityMeasurement { @@ -100,6 +104,13 @@ impl IdentityMeasurement { self.withdrawal_after_send_failed } + /// Positive capture IDs counted and never stored, per provider and + /// source. Counts only: no value is held. + #[must_use] + pub fn capture_unpersisted(&self) -> &BTreeMap<(IdentityProviderId, SourceDomain), u32> { + &self.capture_unpersisted + } + /// Logs the measurement at info level. pub fn log(&self) { log::info!("identity request measurement: {self}"); @@ -111,7 +122,7 @@ impl core::fmt::Display for IdentityMeasurement { write!( formatter, "cookie_header_bytes={} route={} kv_reads={} kv_writes={} flush={} removal_lost={} \ - withdrawal_after_send={} withdrawal_after_send_failed={}", + withdrawal_after_send={} withdrawal_after_send_failed={} capture_unpersisted={}", self.cookie_header_bytes, self.route, self.kv_reads, @@ -120,10 +131,29 @@ impl core::fmt::Display for IdentityMeasurement { self.removal_lost, u8::from(self.withdrawal_after_send), u8::from(self.withdrawal_after_send_failed), + CaptureUnpersisted(&self.capture_unpersisted), ) } } +/// The capture counts as `provider/source:count`, comma separated, or `none`. +struct CaptureUnpersisted<'a>(&'a BTreeMap<(IdentityProviderId, SourceDomain), u32>); + +impl core::fmt::Display for CaptureUnpersisted<'_> { + fn fmt(&self, formatter: &mut core::fmt::Formatter<'_>) -> core::fmt::Result { + if self.0.is_empty() { + return formatter.write_str("none"); + } + for (index, ((provider, source), count)) in self.0.iter().enumerate() { + if index > 0 { + formatter.write_char(',')?; + } + write!(formatter, "{provider}/{source}:{count}")?; + } + Ok(()) + } +} + /// A phase 4 outcome as a lower snake case label, with a drop reason /// appended. pub(crate) struct FlushLabel(pub(crate) Option); @@ -178,4 +208,51 @@ mod tests { ); } } + + fn measurement( + capture_unpersisted: BTreeMap<(IdentityProviderId, SourceDomain), u32>, + ) -> IdentityMeasurement { + IdentityMeasurement { + cookie_header_bytes: 10, + route: RouteKind::Stager, + kv_reads: 0, + kv_writes: 0, + flush: None, + removal_lost: 0, + withdrawal_after_send: false, + withdrawal_after_send_failed: false, + capture_unpersisted, + } + } + + #[test] + fn measurement_line_counts_capture_unpersisted_per_provider_and_source() { + let line = measurement(BTreeMap::from([ + ( + ( + "lockr".parse().expect("should parse the provider ID"), + "a.example.com".parse().expect("should parse the source"), + ), + 2, + ), + ( + ( + "lockr".parse().expect("should parse the provider ID"), + "b.example.com".parse().expect("should parse the source"), + ), + 1, + ), + ])) + .to_string(); + let empty = measurement(BTreeMap::new()).to_string(); + + assert!( + line.ends_with("capture_unpersisted=lockr/a.example.com:2,lockr/b.example.com:1"), + "should list each provider and source with its count: {line}" + ); + assert!( + empty.ends_with("capture_unpersisted=none"), + "control: should print none without a capture: {empty}" + ); + } } diff --git a/crates/trusted-server-core/src/ec/identity/mod.rs b/crates/trusted-server-core/src/ec/identity/mod.rs index 14fe2c80a0..2cfae1e2ce 100644 --- a/crates/trusted-server-core/src/ec/identity/mod.rs +++ b/crates/trusted-server-core/src/ec/identity/mod.rs @@ -10,9 +10,7 @@ pub(crate) mod catalog; pub(crate) mod coordinator; pub(crate) mod denials; pub mod descriptor; -// Temporary: later identity tasks import the items they use. #[cfg(test)] -#[allow(dead_code)] pub(crate) mod examples; pub mod exchange; pub mod flush; diff --git a/crates/trusted-server-core/src/ec/identity/mutation.rs b/crates/trusted-server-core/src/ec/identity/mutation.rs index 73b7686435..c67e4b5c40 100644 --- a/crates/trusted-server-core/src/ec/identity/mutation.rs +++ b/crates/trusted-server-core/src/ec/identity/mutation.rs @@ -49,9 +49,8 @@ pub(crate) enum PartnerIdChange { } impl PartnerIdChange { - // Temporary: first used by a later identity task. - #[allow(dead_code)] - fn is_removal(&self) -> bool { + /// Whether the change removes a record. + pub(crate) fn is_removal(&self) -> bool { matches!(self, Self::OwnerRemove { .. } | Self::WriterRemove { .. }) } } diff --git a/crates/trusted-server-core/src/ec/identity/request.rs b/crates/trusted-server-core/src/ec/identity/request.rs index 8bac2b50f2..dedbff6341 100644 --- a/crates/trusted-server-core/src/ec/identity/request.rs +++ b/crates/trusted-server-core/src/ec/identity/request.rs @@ -47,6 +47,7 @@ pub(crate) struct RequestIdentity { cookie_header_bytes: usize, create_attempts: u32, capture_unpersisted: BTreeMap<(IdentityProviderId, SourceDomain), u32>, + global_revoke: bool, } impl core::fmt::Debug for RequestIdentity { @@ -212,6 +213,17 @@ impl RequestIdentity { *count = count.saturating_add(1); } + /// Whether a real browser's capture asked for a global revoke (spec 8). + #[must_use] + pub(crate) fn global_revoke_requested(&self) -> bool { + self.global_revoke + } + + /// Asks finalize to run the consent-withdrawal tombstone path. + pub(crate) fn request_global_revoke(&mut self) { + self.global_revoke = true; + } + /// Sets the writer decision and the signals captured with it. pub(crate) fn set_writer_decision(&mut self, decision: WriterDecision) { self.writer = decision.writer; diff --git a/crates/trusted-server-core/src/ec/identity/resolution.rs b/crates/trusted-server-core/src/ec/identity/resolution.rs index ae58a0d245..26ddea379c 100644 --- a/crates/trusted-server-core/src/ec/identity/resolution.rs +++ b/crates/trusted-server-core/src/ec/identity/resolution.rs @@ -542,8 +542,6 @@ impl Resolution { /// Whether this result changes state at all. #[must_use] - // Temporary: first used by a later identity task. - #[allow(dead_code)] pub(crate) fn changes_state(&self) -> bool { self.state != StateOp::Keep } diff --git a/crates/trusted-server-core/src/ec/identity/test_support.rs b/crates/trusted-server-core/src/ec/identity/test_support.rs index fa4c73c632..ad7d916126 100644 --- a/crates/trusted-server-core/src/ec/identity/test_support.rs +++ b/crates/trusted-server-core/src/ec/identity/test_support.rs @@ -31,11 +31,14 @@ use crate::ec::identity::descriptor::{ use crate::ec::identity::exchange::{ProviderRequest, ProviderResponse}; use crate::ec::identity::ids::{IdentityProviderId, SourceDomain, UnixTime}; use crate::ec::identity::input::{ - HashedEmail, ProviderConsent, Region, ResolveInput, ResolveSignals, SeededRandom, StoredId, + CaptureInput, HashedEmail, ProviderConsent, Region, ResolveInput, ResolveSignals, SeededRandom, + StoredId, }; use crate::ec::identity::policy::PolicySignals; use crate::ec::identity::provider::IdentityProvider; -use crate::ec::identity::resolution::{IdentityProviderError, Resolution, When}; +use crate::ec::identity::resolution::{ + IdentityProviderError, Resolution, ResolvedEid, ResolvedUid, When, +}; use crate::ec::identity::revision::{StateRevision, new_revision}; use crate::ec::kv_backend::in_memory::InMemoryEcKv; use crate::ec::kv_backend::{EcKvLookup, EcKvStore, EcKvWrite, EcKvWriteMode, EcKvWriteOutcome}; @@ -164,6 +167,95 @@ pub fn fixed_capture_provider(id: &str, sources: &[&str]) -> Arc &ProviderDescriptor { + &self.descriptor + } + + fn capture( + &self, + input: &CaptureInput<'_>, + ) -> Result> { + let Ok(body) = input.response().json::() else { + return Ok(Resolution::no_match()); + }; + let sources = |name: &str| -> Vec { + body[name] + .as_array() + .into_iter() + .flatten() + .filter_map(|value| value.as_str()?.parse().ok()) + .collect() + }; + let tokens: Vec = body["tokens"] + .as_array() + .into_iter() + .flatten() + .filter_map(|token| { + let source = input.source_named(token["source"].as_str()?)?; + Some(ResolvedEid::new( + source, + ResolvedUid::new(token["value"].as_str()?), + )) + }) + .collect(); + let invalidated = sources("invalidate"); + let mut resolution = if body["opted_out"].as_bool() == Some(true) { + Resolution::opted_out() + } else if !tokens.is_empty() { + Resolution::found(tokens) + } else if let Some(first) = invalidated.first() { + Resolution::invalidate(first) + } else { + Resolution::no_match() + }; + for source in &invalidated { + resolution = resolution.also_invalidate(source); + } + if let Some(state) = body["state"].as_str() { + resolution = resolution.replacing_state(&state); + } + Ok(resolution) + } +} + +/// A capture-only provider `id` over `sources` that answers as the response +/// body says, with a capture rule on `/integrations//`. With `gvl_id` its +/// consent policy names that TCF vendor. +/// +/// # Panics +/// +/// Panics when `id` or a source is not valid. +#[must_use] +pub fn answering_capture_provider( + id: &str, + sources: &[&str], + gvl_id: Option, +) -> Arc { + let policy = match gvl_id { + Some(vendor) => ConsentPolicy::strict().with_tcf_vendor(vendor), + None => ConsentPolicy::eid_persistence_only(), + }; + let mut builder = + ProviderDescriptor::builder(id.parse().expect("should parse the provider ID")) + .with_captures(Captures::new().with_path_prefix(&format!("/integrations/{id}/"))) + .with_consent_policy(policy); + for source in sources { + builder = builder.with_source(source.parse().expect("should parse the source")); + } + Arc::new(AnsweringCapture { + descriptor: builder.build().expect("should build the descriptor"), + }) +} + /// A [`fixed_provider`] that also declares a `with_provenance` default per /// listed source. /// @@ -425,6 +517,7 @@ pub struct WriteCountingEcKv { overwrite: AtomicUsize, lookups: AtomicUsize, refusals: Mutex>, + failing: Mutex>, } impl WriteCountingEcKv { @@ -438,9 +531,24 @@ impl WriteCountingEcKv { overwrite: AtomicUsize::new(0), lookups: AtomicUsize::new(0), refusals: Mutex::new(BTreeMap::new()), + failing: Mutex::new(BTreeSet::new()), } } + /// Fails every generation-checked write to `key` with a store error, the + /// non-retryable failure (Fastly's per-key write limit). A failed write + /// is still counted. + /// + /// # Panics + /// + /// Panics when the failure table is poisoned. + pub fn fail_conditional_writes(&self, key: &str) { + self.failing + .lock() + .expect("should lock the failures") + .insert(key.to_owned()); + } + /// Refuses the next `count` generation-checked writes to `key` with /// [`EcKvWriteOutcome::PreconditionFailed`], the retryable refusal. A /// refused write is still counted. @@ -505,6 +613,17 @@ impl EcKvStore for WriteCountingEcKv { }; count.fetch_add(1, Ordering::SeqCst); if matches!(write.mode, EcKvWriteMode::IfGenerationMatch(_)) { + if self + .failing + .lock() + .expect("should lock the failures") + .contains(key) + { + return Err(Report::new(TrustedServerError::KvStore { + store_name: self.inner.store_name().to_owned(), + message: "per-key write limit".to_owned(), + })); + } let mut refusals = self.refusals.lock().expect("should lock the refusals"); if let Some(remaining) = refusals.get_mut(key) && *remaining > 0 @@ -1615,3 +1734,176 @@ pub(crate) mod hold_fixture { } } } + +/// A proxied API call from a browser with a live row and capture-only +/// providers, for the tests of the capture removal write. +#[cfg(test)] +pub(crate) mod capture_fixture { + use std::collections::BTreeMap; + use std::sync::Arc; + + use http::{HeaderMap, Method, StatusCode}; + + use super::fake_http::{FakeBackend, FakeHttpClient, fake_services}; + use super::{TEST_EC_ID, WriteCountingEcKv, allowing_consent}; + use crate::consent::ConsentContext; + use crate::ec::EcContext; + use crate::ec::identity::coordinator::{CaptureCall, run_capture}; + use crate::ec::identity::flush::retry_pending_removals; + use crate::ec::identity::ids::{IdentityProviderId, SourceDomain}; + use crate::ec::identity::input::ThreadRandom; + use crate::ec::identity::provider::IdentityProvider; + use crate::ec::identity::request::WriterDecision; + use crate::ec::identity::runtime::IdentityRuntime; + use crate::ec::kv::KvIdentityGraph; + use crate::ec::kv_types::KvEntry; + use crate::platform::RuntimeServices; + use crate::settings::{ProviderSettings, RevokeScope, Settings}; + use crate::test_support::tests::create_test_settings; + + /// The path every fixture capture call goes to. + pub(crate) fn capture_path(provider: &str) -> String { + format!("/integrations/{provider}/api/token") + } + + /// Everything one capture call reads. + pub(crate) struct CaptureFixture { + pub(crate) settings: Settings, + pub(crate) runtime: Arc, + pub(crate) services: RuntimeServices, + pub(crate) graph: KvIdentityGraph, + pub(crate) store: Arc, + pub(crate) context: EcContext, + } + + impl CaptureFixture { + /// A real browser's proxied call with `row` stored, read in phase 1, + /// and every provider's `revoke_scope` at its default. + pub(crate) fn new( + providers: Vec>, + row: Option<&KvEntry>, + ) -> Self { + Self::with_scopes(providers, row, &[]) + } + + /// As [`Self::new`], with `revoke_scope = "global"` for each provider + /// listed in `global`. + pub(crate) fn with_scopes( + providers: Vec>, + row: Option<&KvEntry>, + global: &[&str], + ) -> Self { + let mut settings = create_test_settings(); + for id in global { + settings.ec.identity.providers.insert( + (*id).to_owned(), + ProviderSettings { + revoke_scope: Some(RevokeScope::Global), + ..ProviderSettings::default() + }, + ); + } + let runtime = Arc::new( + IdentityRuntime::from_settings_with_providers( + &settings, + providers, + BTreeMap::::new(), + ) + .expect("should build the runtime"), + ); + let client = Arc::new(FakeHttpClient::new()); + let backend = Arc::new(FakeBackend::new()); + let services = fake_services(&client, &backend); + let (graph, store) = KvIdentityGraph::write_counting("capture-store"); + if let Some(row) = row { + graph.create(TEST_EC_ID, row).expect("should seed the row"); + } + let mut context = EcContext::new_for_test_with_ip( + Some(TEST_EC_ID.to_owned()), + allowing_consent(), + Some("192.0.2.10".to_owned()), + ); + context.set_identity_runtime(Arc::clone(&runtime)); + context.identity_mut().set_writer_decision(WriterDecision { + writer: false, + is_real_browser: true, + user_agent: Some("ExampleBrowser/1.0".to_owned()), + dnt: false, + page_url: None, + }); + context.set_kv_snapshot(graph.load_snapshot(TEST_EC_ID)); + store.reset_counts(); + Self { + settings, + runtime, + services, + graph, + store, + context, + } + } + + /// Replaces the request's consent. + pub(crate) fn with_consent(mut self, consent: ConsentContext) -> Self { + let mut context = EcContext::new_for_test_with_ip( + Some(TEST_EC_ID.to_owned()), + consent, + Some("192.0.2.10".to_owned()), + ); + context.set_identity_runtime(Arc::clone(&self.runtime)); + context.set_kv_snapshot(self.context.kv_snapshot().clone()); + context.identity_mut().set_real_browser(true); + self.context = context; + self + } + + /// Makes the request a client that does not look like a browser. + pub(crate) fn not_a_browser(mut self) -> Self { + self.context.identity_mut().set_real_browser(false); + self + } + + /// Runs the capture for a 200 response holding `body` at `path`. + pub(crate) fn capture(&mut self, path: &str, body: &serde_json::Value) { + let bytes = serde_json::to_vec(body).expect("should serialize the body"); + self.capture_bytes(path, StatusCode::OK, &bytes); + } + + /// Runs the capture for a response with `status` holding `bytes`. + pub(crate) fn capture_bytes(&mut self, path: &str, status: StatusCode, bytes: &[u8]) { + let real_browser = self.context.identity().is_real_browser(); + let random = ThreadRandom; + let deps = self + .runtime + .deps(&self.settings, &self.services, &random, None); + let headers = HeaderMap::new(); + run_capture( + &deps, + Some(&self.graph), + &mut self.context, + &CaptureCall { + method: &Method::POST, + path, + is_real_browser: real_browser, + status, + headers: &headers, + body: bytes, + }, + ); + } + + /// Runs the removal write's one retry after send. + pub(crate) fn retry_after_send(&mut self) { + retry_pending_removals(&mut self.context, &self.graph); + } + + /// The row as KV holds it. + pub(crate) fn row(&self) -> KvEntry { + self.graph + .get(TEST_EC_ID) + .expect("should read the row") + .expect("should hold the row") + .0 + } + } +} diff --git a/crates/trusted-server-core/src/ec/mod.rs b/crates/trusted-server-core/src/ec/mod.rs index e084affdb3..b717328271 100644 --- a/crates/trusted-server-core/src/ec/mod.rs +++ b/crates/trusted-server-core/src/ec/mod.rs @@ -555,9 +555,23 @@ impl EcContext { removal_lost: self.identity.removal_lost(), withdrawal_after_send: input.withdrawal_after_send, withdrawal_after_send_failed: input.withdrawal_after_send_failed, + capture_unpersisted: self.identity.capture_unpersisted().clone(), } } + /// Asks finalize to run the consent-withdrawal tombstone path for this + /// request, as a `revoke_scope = "global"` capture from a real browser + /// does (spec 8). + pub(crate) fn request_global_revoke(&mut self) { + self.identity.request_global_revoke(); + } + + /// Whether a capture asked for the consent-withdrawal tombstone path. + #[must_use] + pub(crate) fn global_revoke_requested(&self) -> bool { + self.identity.global_revoke_requested() + } + /// The identity state this request decides and collects. #[must_use] pub(crate) fn identity(&self) -> &RequestIdentity { @@ -599,6 +613,12 @@ impl EcContext { /// Only an organic real-browser top-level document navigation that is not /// a prefetch writes (spec 5.1). Also captures the user agent, `DNT` and /// the page URL without query or fragment. + /// + /// The page URL is read from the request URI, so it needs an absolute + /// URI. The Fastly adapter gives one (`edgezero-adapter-fastly` builds the + /// request from `get_url_str()`); a request with an origin-form URI has no + /// page URL, and a provider that wants the `PageUrl` input is then not + /// due. pub fn decide_writer( &mut self, req: &Request, @@ -674,6 +694,13 @@ impl EcContext { !self.failed_withdrawals.is_empty() } + /// Whether a capture removal write failed before send and awaits its + /// one retry after send. + #[must_use] + pub fn has_pending_removals(&self) -> bool { + self.identity.pending_removals().is_some() + } + /// Forgets a withdrawn EC ID whose retried tombstone was written. pub(crate) fn clear_failed_withdrawal(&mut self, ec_id: &str) { self.failed_withdrawals.retain(|id| id != ec_id); @@ -959,6 +986,28 @@ mod tests { } } + #[test] + fn decide_writer_has_no_page_url_for_an_origin_form_uri() { + let request = Request::builder() + .uri("/a?b=c") + .header("sec-fetch-dest", "document") + .body(EdgeBody::empty()) + .expect("should build the request"); + let mut context = EcContext::default(); + + context.decide_writer(&request, true, true); + + assert!( + context.identity().writer(), + "control: the request is still the writer" + ); + assert_eq!( + context.identity().page_url(), + None, + "should have no page URL without an absolute URI" + ); + } + /// [`EcKvStore`] wrapper whose first `collisions` `Add` writes report a /// precondition failure, forcing generation to retry with a fresh suffix. struct AddCollidingEcKv { diff --git a/crates/trusted-server-core/src/integrations/lockr.rs b/crates/trusted-server-core/src/integrations/lockr.rs index 002182bd13..2f86d30e40 100644 --- a/crates/trusted-server-core/src/integrations/lockr.rs +++ b/crates/trusted-server-core/src/integrations/lockr.rs @@ -201,7 +201,8 @@ impl LockrIntegration { .unwrap_or_default(); let target_url = format!("{}{}{}", self.config.api_endpoint, target_path, query); - log::info!("Forwarding to Lockr API: {}", target_url); + // The query string can carry identifiers, so only the path is logged. + log::info!("Forwarding to Lockr API path: {target_path}"); let request_body = if method == Method::POST { let bytes = @@ -227,8 +228,7 @@ impl LockrIntegration { .send(PlatformHttpRequest::new(target_req, backend_name)) .await .change_context(Self::error(format!( - "Failed to forward request to {}", - target_url + "Failed to forward request to Lockr API path {target_path}" )))? .response; @@ -578,6 +578,41 @@ mod tests { ); } + #[test] + fn lockr_proxy_log_omits_query() { + use crate::test_support::logs::{captured_logs, install_capturing_logger}; + + install_capturing_logger(); + let stub = Arc::new(StubHttpClient::new()); + stub.push_response(200, b"ok".to_vec()); + let services = build_services_with_http_client( + Arc::clone(&stub) as Arc + ); + let settings = create_test_settings(); + let integration = LockrIntegration::new(test_config()); + let req = http::Request::builder() + .method(HttpMethod::GET) + .uri("https://publisher.example/integrations/lockr/api/logpath/v1/token?uid=query-secret-4471&email=person-4471%40example.com") + .body(EdgeBody::empty()) + .expect("should build request"); + + futures::executor::block_on(integration.handle(&settings, &services, req)) + .expect("should proxy request"); + + let logs = captured_logs(); + assert!( + logs.iter() + .any(|line| line == "Forwarding to Lockr API path: /logpath/v1/token"), + "should log the upstream path: {logs:?}" + ); + assert!( + !logs + .iter() + .any(|line| line.contains("query-secret-4471") || line.contains("person-4471")), + "should log no part of the query string" + ); + } + #[test] fn lockr_proxy_forwards_body_and_strips_publisher_credentials() { // Regression guard for the upstream-rejection / credential-leak causes: diff --git a/crates/trusted-server-core/src/integrations/registry.rs b/crates/trusted-server-core/src/integrations/registry.rs index af553bdc5f..1ceff0cc41 100644 --- a/crates/trusted-server-core/src/integrations/registry.rs +++ b/crates/trusted-server-core/src/integrations/registry.rs @@ -5,6 +5,7 @@ use std::sync::{Arc, Mutex, MutexGuard, PoisonError}; use async_trait::async_trait; use edgezero_core::body::Body as EdgeBody; use error_stack::Report; +use futures::StreamExt as _; use http::{Method, Request, Response}; use matchit::Router; @@ -12,6 +13,8 @@ use crate::auction::AuctionPlan; use crate::constants::HEADER_X_TS_EC; use crate::ec::EcContext; use crate::ec::finalize::BrowserSeed; +use crate::ec::identity::coordinator::{CaptureCall, decode_body, has_capture_rule, run_capture}; +use crate::ec::identity::input::ThreadRandom; use crate::ec::identity::{IdentityProvider, IdentityProviderError}; use crate::ec::kv::KvIdentityGraph; use crate::error::TrustedServerError; @@ -893,6 +896,135 @@ pub struct ProxyDispatchInput<'a> { pub req: Request, } +/// What capturing a proxied response reads besides the response. +struct CaptureExchange<'a> { + settings: &'a Settings, + services: &'a RuntimeServices, + kv: Option<&'a KvIdentityGraph>, + ec_context: &'a mut EcContext, + method: &'a Method, + path: &'a str, + is_real_browser: bool, +} + +/// A proxied body, buffered for capture or left alone. +enum Buffered { + /// The whole body, as sent (still encoded). + Whole(bytes::Bytes), + /// A body over the cap, rebuilt so the browser still receives every byte. + Passthrough(EdgeBody), +} + +/// Buffers `body` up to `cap` bytes. A body over the cap, or a stream that +/// fails, is handed back whole so the browser receives it unchanged. +async fn buffer_up_to(body: EdgeBody, cap: usize) -> Buffered { + match body { + EdgeBody::Once(bytes) if bytes.len() <= cap => Buffered::Whole(bytes), + EdgeBody::Stream(mut stream) => { + let mut items = Vec::new(); + let mut total = 0_usize; + while let Some(item) = stream.next().await { + let over = match &item { + Ok(chunk) => { + total = total.saturating_add(chunk.len()); + total > cap + } + Err(_) => true, + }; + items.push(item); + if over { + return Buffered::Passthrough(EdgeBody::from_stream( + futures::stream::iter(items).chain(stream), + )); + } + } + let mut whole = Vec::with_capacity(total); + for chunk in items.iter().filter_map(|item| item.as_ref().ok()) { + whole.extend_from_slice(chunk); + } + Buffered::Whole(bytes::Bytes::from(whole)) + } + other => Buffered::Passthrough(other), + } +} + +/// Offers a proxied 2xx response to the identity providers whose capture rule +/// matches, and returns it unchanged (spec 4.7). +/// +/// A body over `max_response_bytes` is passed through uncaptured. The request +/// body is never read here. +async fn capture_proxied_response( + exchange: CaptureExchange<'_>, + response: Response, +) -> Response { + let CaptureExchange { + settings, + services, + kv, + ec_context, + method, + path, + is_real_browser, + } = exchange; + if !response.status().is_success() { + return response; + } + let Some(runtime) = ec_context.identity().runtime().cloned() else { + return response; + }; + if !has_capture_rule(runtime.catalog(), method, path) { + return response; + } + + let cap = settings.ec.identity.max_response_bytes; + let (parts, body) = response.into_parts(); + let declared_over_cap = parts + .headers + .get(http::header::CONTENT_LENGTH) + .and_then(|value| value.to_str().ok()) + .and_then(|value| value.parse::().ok()) + .is_some_and(|length| length > cap); + if declared_over_cap { + log::debug!("Skipping capture on '{path}': the response is over the size cap"); + return Response::from_parts(parts, body); + } + let raw = match buffer_up_to(body, cap).await { + Buffered::Whole(raw) => raw, + Buffered::Passthrough(body) => { + log::debug!("Skipping capture on '{path}': the response is over the size cap"); + return Response::from_parts(parts, body); + } + }; + + let encoding = parts + .headers + .get(http::header::CONTENT_ENCODING) + .and_then(|value| value.to_str().ok()); + match decode_body(encoding, raw.to_vec(), cap) { + Some(decoded) => { + let random = ThreadRandom; + let deps = runtime.deps(settings, services, &random, None); + run_capture( + &deps, + kv, + ec_context, + &CaptureCall { + method, + path, + is_real_browser, + status: parts.status, + headers: &parts.headers, + body: &decoded, + }, + ); + } + None => { + log::debug!("Skipping capture on '{path}': the body is undecodable or over the cap") + } + } + Response::from_parts(parts, EdgeBody::from_bytes(raw)) +} + /// In-memory registry of integrations discovered from settings. #[derive(Clone, Default)] pub struct IntegrationRegistry { @@ -1172,7 +1304,23 @@ impl IntegrationRegistry { // Remove any caller-supplied EC header rather than forwarding it. req.headers_mut().remove(HEADER_X_TS_EC.clone()); - Some(proxy.handle(settings, services, req).await) + let handled = proxy.handle(settings, services, req).await; + Some(match handled { + Ok(response) => Ok(capture_proxied_response( + CaptureExchange { + settings, + services, + kv, + ec_context, + method, + path, + is_real_browser, + }, + response, + ) + .await), + Err(error) => Err(error), + }) } else { None } @@ -2886,4 +3034,428 @@ mod tests { ); assert_eq!(registry.identity_providers().len(), 1, "should hold one"); } + + /// Revoke-only capture around `handle_proxy` (spec 4.7). + mod capture_proxy { + use std::collections::BTreeMap; + use std::io::Write as _; + use std::sync::{Arc, Mutex}; + + use bytes::Bytes; + use serde_json::json; + + use super::*; + use crate::ec::identity::IdentityRuntime; + use crate::ec::identity::revision::set_hashes; + use crate::ec::identity::test_support::hold_fixture::live_row; + use crate::ec::identity::test_support::{ + TEST_EC_ID, WriteCountingEcKv, allowing_consent, answering_capture_provider, + record_for_test, + }; + use crate::ec::kv_types::KvEntry; + use crate::test_support::logs::{captured_logs, install_capturing_logger}; + use crate::test_support::tests::create_test_settings; + + const FIRST: &str = "a.example.com"; + + /// How the upstream answers. + #[derive(Clone)] + struct Upstream { + status: StatusCode, + headers: Vec<(&'static str, &'static str)>, + body: Vec, + streamed: bool, + } + + impl Upstream { + fn json(body: &serde_json::Value) -> Self { + Self { + status: StatusCode::OK, + headers: vec![("content-type", "application/json")], + body: serde_json::to_vec(body).expect("should serialize the body"), + streamed: false, + } + } + } + + struct ScriptedProxy { + upstream: Upstream, + request_body: Arc>>>, + } + + #[async_trait(?Send)] + impl IntegrationProxy for ScriptedProxy { + fn integration_name(&self) -> &'static str { + "lockr" + } + + fn routes(&self) -> Vec { + vec![ + IntegrationEndpoint::post("/integrations/lockr/*"), + IntegrationEndpoint::post("/integrations/examplecapture/*"), + ] + } + + async fn handle( + &self, + _settings: &Settings, + _services: &RuntimeServices, + req: Request, + ) -> Result, Report> { + let seen = req.into_body().into_bytes().unwrap_or_default().to_vec(); + *self.request_body.lock().expect("should lock the body") = Some(seen); + let mut builder = Response::builder().status(self.upstream.status); + for (name, value) in &self.upstream.headers { + builder = builder.header(*name, *value); + } + let body = if self.upstream.streamed { + let chunks: Vec> = self + .upstream + .body + .chunks(7) + .map(|chunk| Ok(Bytes::copy_from_slice(chunk))) + .collect(); + EdgeBody::from_stream(futures::stream::iter(chunks)) + } else { + EdgeBody::from(self.upstream.body.clone()) + }; + Ok(builder.body(body).expect("should build the response")) + } + } + + /// What one proxied call left behind. + struct Exchange { + response: Response, + request_body: Option>, + store: Arc, + context: EcContext, + row: Option, + } + + impl Exchange { + fn response_bytes(self) -> Vec { + futures::executor::block_on(self.response.into_body().into_bytes_bounded(1 << 20)) + .expect("should read the response body") + .to_vec() + } + } + + struct Call<'a> { + provider: Arc, + path: &'a str, + upstream: Upstream, + request_body: &'a [u8], + real_browser: bool, + row: Option<&'a KvEntry>, + max_response_bytes: usize, + } + + fn call<'a>( + provider: Arc, + path: &'a str, + upstream: Upstream, + ) -> Call<'a> { + Call { + provider, + path, + upstream, + request_body: b"{}", + real_browser: true, + row: None, + max_response_bytes: 65_536, + } + } + + fn exchange(call: Call<'_>) -> Exchange { + let mut settings = create_test_settings(); + settings.ec.identity.max_response_bytes = call.max_response_bytes; + let request_body = Arc::new(Mutex::new(None)); + let registry = IntegrationRegistry::from_registrations(vec![ + IntegrationRegistration::builder("lockr") + .with_proxy(Arc::new(ScriptedProxy { + upstream: call.upstream, + request_body: Arc::clone(&request_body), + })) + .with_identity_provider(call.provider) + .build(), + ]) + .expect("should build the registry"); + let runtime = Arc::new( + IdentityRuntime::from_settings_with_providers( + &settings, + registry.identity_providers(), + BTreeMap::new(), + ) + .expect("should build the runtime"), + ); + let (graph, store) = KvIdentityGraph::write_counting("capture-proxy-store"); + if let Some(row) = call.row { + graph.create(TEST_EC_ID, row).expect("should seed the row"); + } + let mut context = EcContext::new_for_test_with_ip( + Some(TEST_EC_ID.to_owned()), + allowing_consent(), + Some("192.0.2.10".to_owned()), + ); + context.set_identity_runtime(runtime); + context.set_kv_snapshot(graph.load_snapshot(TEST_EC_ID)); + store.reset_counts(); + + let req = http::Request::builder() + .method(http::Method::POST) + .uri(format!("https://www.example.com{}", call.path)) + .body(EdgeBody::from(call.request_body.to_vec())) + .expect("should build the request"); + let response = futures::executor::block_on(registry.handle_proxy(ProxyDispatchInput { + method: &http::Method::POST, + path: call.path, + settings: &settings, + kv: Some(&graph), + ec_context: &mut context, + services: &noop_services(), + is_real_browser: call.real_browser, + seed: &BrowserSeed::empty(), + req, + })) + .expect("should match the route") + .expect("the proxy should answer"); + let row = graph.get(TEST_EC_ID).ok().flatten().map(|(row, _)| row); + Exchange { + response, + request_body: request_body.lock().expect("should lock the body").clone(), + store, + context, + row, + } + } + + fn token_body() -> serde_json::Value { + json!({"tokens": [{"source": FIRST, "value": "secret-uid-a"}]}) + } + + fn gzip(bytes: &[u8]) -> Vec { + let mut encoder = + flate2::write::GzEncoder::new(Vec::new(), flate2::Compression::default()); + encoder.write_all(bytes).expect("should compress"); + encoder.finish().expect("should finish the stream") + } + + fn brotli(bytes: &[u8]) -> Vec { + let mut out = Vec::new(); + { + let mut encoder = ::brotli::CompressorWriter::new(&mut out, 4096, 5, 22); + encoder.write_all(bytes).expect("should compress"); + } + out + } + + fn counted(exchange: &Exchange) -> usize { + exchange.context.identity().capture_unpersisted().len() + } + + #[test] + fn capture_runs_only_for_matching_rules() { + let token = json!({"tokens": [{ + "source": "examplecapture.example.com", "value": "v", "expires": 4_102_444_800_u64 + }]}); + let provider = || -> Arc { + Arc::new(ExampleCaptureProvider::new().expect("should build the provider")) + }; + + let matching = exchange(call( + provider(), + "/integrations/examplecapture/api/token", + Upstream::json(&token), + )); + let other_path = exchange(call( + provider(), + "/integrations/examplecapture/api/other", + Upstream::json(&token), + )); + + assert_eq!(counted(&matching), 1, "should capture on the matching rule"); + assert_eq!( + counted(&other_path), + 0, + "should not capture on a path outside the rules" + ); + } + + #[test] + fn capture_runs_only_on_a_2xx_response() { + let provider = || answering_capture_provider("lockr", &[FIRST], None); + let path = "/integrations/lockr/api/token"; + let mut failing = Upstream::json(&token_body()); + failing.status = StatusCode::INTERNAL_SERVER_ERROR; + + let ok = exchange(call(provider(), path, Upstream::json(&token_body()))); + let error = exchange(call(provider(), path, failing)); + + assert_eq!(counted(&ok), 1, "should capture a 200"); + assert_eq!(counted(&error), 0, "should not capture a 500"); + assert_eq!( + error.response.status(), + StatusCode::INTERNAL_SERVER_ERROR, + "should still return the 500" + ); + } + + #[test] + fn capture_decodes_gzip_and_brotli_responses_and_forwards_them_unchanged() { + let plain = serde_json::to_vec(&token_body()).expect("should serialize"); + for (encoding, compressed) in [("gzip", gzip(&plain)), ("br", brotli(&plain))] { + let upstream = Upstream { + status: StatusCode::OK, + headers: vec![ + ("content-type", "application/json"), + ("content-encoding", encoding), + ], + body: compressed.clone(), + streamed: false, + }; + + let result = exchange(call( + answering_capture_provider("lockr", &[FIRST], None), + "/integrations/lockr/api/token", + upstream, + )); + + assert_eq!(counted(&result), 1, "should decode {encoding} for capture"); + assert_eq!( + result.response.headers()["content-encoding"], + encoding, + "should keep the encoding header" + ); + assert_eq!( + result.response_bytes(), + compressed, + "should forward the {encoding} bytes unchanged" + ); + } + } + + #[test] + fn capture_never_reads_the_request_body() { + install_capturing_logger(); + let request_body = serde_json::to_vec( + &json!({"email": "capture-body-7731@example.com", "consent": true}), + ) + .expect("should serialize"); + let mut request = call( + answering_capture_provider("lockr", &[FIRST], None), + "/integrations/lockr/api/token", + Upstream::json(&token_body()), + ); + request.request_body = &request_body; + + let result = exchange(request); + + assert_eq!( + result.request_body.as_deref(), + Some(request_body.as_slice()), + "should hand the upstream the request body unchanged" + ); + assert_eq!(counted(&result), 1, "control: the response is captured"); + assert!( + !captured_logs() + .iter() + .any(|line| line.contains("capture-body-7731")), + "should log nothing from the request body" + ); + } + + #[test] + fn capture_skips_an_oversized_proxied_body() { + let body = json!({"tokens": [{"source": FIRST, "value": "x".repeat(200)}]}); + let provider = || answering_capture_provider("lockr", &[FIRST], None); + let path = "/integrations/lockr/api/token"; + let plain = serde_json::to_vec(&body).expect("should serialize"); + + for streamed in [false, true] { + let mut upstream = Upstream::json(&body); + upstream.streamed = streamed; + let mut oversized = call(provider(), path, upstream.clone()); + oversized.max_response_bytes = 64; + let skipped = exchange(oversized); + assert_eq!( + counted(&skipped), + 0, + "should not capture (streamed={streamed})" + ); + assert_eq!( + skipped.response_bytes(), + plain, + "should pass the body through unchanged (streamed={streamed})" + ); + + let within = exchange(call(provider(), path, upstream)); + assert_eq!(counted(&within), 1, "control: captures within the cap"); + assert_eq!( + within.response_bytes(), + plain, + "control: forwards the buffered body unchanged (streamed={streamed})" + ); + } + } + + #[test] + fn capture_removal_from_a_real_browser_is_written_before_the_response_returns() { + let mut seeded = live_row(); + seeded + .ids + .insert(FIRST.to_owned(), record_for_test("secret-uid-a", None, 500)); + let mut request = call( + answering_capture_provider("lockr", &[FIRST], None), + "/integrations/lockr/api/token", + Upstream::json(&json!({"invalidate": [FIRST]})), + ); + request.row = Some(&seeded); + + let result = exchange(request); + + let row = result.row.as_ref().expect("should hold the row"); + assert!(!row.ids.contains_key(FIRST), "should remove the record"); + assert_eq!( + row.ps["lockr"].iv, + set_hashes(&seeded.ids[FIRST].eids), + "should hash the record into iv" + ); + assert_eq!(result.store.writes().total(), 1, "should write once"); + } + + #[test] + fn capture_with_replayed_ec_cookie_changes_no_kv_row() { + let mut seeded = live_row(); + seeded.ids.insert( + FIRST.to_owned(), + record_for_test("secret-uid-a", Some("lockr"), 500), + ); + for body in [ + json!({"invalidate": [FIRST]}), + json!({"opted_out": true}), + token_body(), + ] { + let mut request = call( + answering_capture_provider("lockr", &[FIRST], None), + "/integrations/lockr/api/token", + Upstream::json(&body), + ); + request.real_browser = false; + request.row = Some(&seeded); + + let result = exchange(request); + + assert_eq!( + result.store.writes().total(), + 0, + "should record zero writes for {body}" + ); + assert_eq!( + result.row.as_ref(), + Some(&seeded), + "should change no row for {body}" + ); + } + } + } } diff --git a/crates/trusted-server-core/src/test_support.rs b/crates/trusted-server-core/src/test_support.rs index 37b8a9e657..a7a1dca7c1 100644 --- a/crates/trusted-server-core/src/test_support.rs +++ b/crates/trusted-server-core/src/test_support.rs @@ -410,3 +410,59 @@ pub mod nextjs_auction { .build() } } + +/// One process-wide capturing logger, shared by every test that asserts on +/// log output. +/// +/// Tests run in parallel, so a test asserts on text unique to it: that a +/// message holds, or that a value appears in none. +#[cfg(test)] +pub mod logs { + use std::sync::{Mutex, Once}; + + static CAPTURED_LOGS: Mutex> = Mutex::new(Vec::new()); + static INSTALL_LOGGER: Once = Once::new(); + + struct CapturingLogger; + + impl log::Log for CapturingLogger { + fn enabled(&self, metadata: &log::Metadata<'_>) -> bool { + metadata.level() <= log::Level::Debug + } + + fn log(&self, record: &log::Record<'_>) { + if self.enabled(record.metadata()) { + CAPTURED_LOGS + .lock() + .expect("should lock captured logs") + .push(record.args().to_string()); + } + } + + fn flush(&self) {} + } + + /// Installs the logger once for the test process. + /// + /// # Panics + /// + /// Panics when another logger is already installed. + pub fn install_capturing_logger() { + INSTALL_LOGGER.call_once(|| { + log::set_logger(&CapturingLogger).expect("should install the capturing test logger"); + log::set_max_level(log::LevelFilter::Debug); + }); + } + + /// Every message logged so far, at debug and above. + /// + /// # Panics + /// + /// Panics when the log is poisoned. + pub fn captured_logs() -> Vec { + CAPTURED_LOGS + .lock() + .expect("should lock captured logs") + .clone() + } +} From a5310063142bedaae0308ca2fb09154f486023ef Mon Sep 17 00:00:00 2001 From: dhruv8sh Date: Tue, 27 Oct 2026 03:56:56 +0530 Subject: [PATCH 131/194] Count unreadable-row capture removals as lost and measure capture KV operations A capture whose row cannot be read now logs and counts its removals lost instead of dropping them with a warning. The capture's view read and removal-write reads and writes join the measurement line's KV counts, and a requested global revoke skips the other providers' removal writes. Adds a Fastly adapter test for a global-scope capture that tombstones the row and expires the EC cookie. Signed-off-by: dhruv8sh --- .../trusted-server-adapter-fastly/src/main.rs | 49 +++- .../src/ec/identity/coordinator.rs | 238 ++++++++++++++++-- .../src/ec/identity/mutation.rs | 4 - .../src/ec/identity/removal.rs | 34 ++- .../src/ec/identity/request.rs | 20 ++ .../src/ec/identity/test_support.rs | 17 +- crates/trusted-server-core/src/ec/mod.rs | 14 +- 7 files changed, 337 insertions(+), 39 deletions(-) diff --git a/crates/trusted-server-adapter-fastly/src/main.rs b/crates/trusted-server-adapter-fastly/src/main.rs index 0baa30ee8b..1f9e5f98c3 100644 --- a/crates/trusted-server-adapter-fastly/src/main.rs +++ b/crates/trusted-server-adapter-fastly/src/main.rs @@ -3842,11 +3842,33 @@ mod tests { /// provider listening on that proxy's path and a row holding a /// browser-written `ssp.example.com` ID for it to invalidate. fn capture_scenario(body: &str) -> Scenario { + capture_scenario_with(body, false) + } + + /// As [`capture_scenario`], with the provider's `revoke_scope` set to + /// `"global"` when `global`. The settings are changed after the app + /// state is built, as config validation knows no capture-only + /// provider. + fn capture_scenario_with(body: &str, global: bool) -> Scenario { let Scenario { - state, + mut state, store, network, } = Scenario::with_settings(DIDOMI_TOML); + if global { + let owned = Arc::get_mut(&mut state).expect("should own the fresh state"); + Arc::make_mut(&mut owned.settings) + .ec + .identity + .providers + .insert( + "didomi".to_owned(), + trusted_server_core::settings::ProviderSettings { + revoke_scope: Some(trusted_server_core::settings::RevokeScope::Global), + ..trusted_server_core::settings::ProviderSettings::default() + }, + ); + } let provider = trusted_server_core::ec::identity::test_support::answering_capture_provider( "didomi", @@ -3921,6 +3943,31 @@ mod tests { ); } + #[test] + fn capture_global_revoke_tombstones_the_row_and_expires_the_ec_cookie() { + let scenario = capture_scenario_with(r#"{"opted_out":true}"#, true); + + let (response, _) = + scenario.drive(didomi_navigation(Some(&returning_with_shared_id()))); + + assert_eq!(response.status(), 200, "should serve the proxied answer"); + let row = scenario.row(); + assert!(!row.consent.ok, "should tombstone the row"); + assert!(row.ids.is_empty(), "should remove every ID"); + assert!( + set_cookies(&response).iter().any(|cookie| { + cookie_name(cookie) == "ts-ec" && cookie.contains("Max-Age=0") + }), + "should expire the EC cookie" + ); + let writes = scenario.store.writes(); + assert_eq!( + (writes.if_generation_match, writes.overwrite), + (1, 0), + "should write one tombstone and no flush" + ); + } + #[test] fn failed_capture_removal_write_runs_once_more_after_send() { let scenario = capture_scenario(CAPTURE_INVALIDATION); diff --git a/crates/trusted-server-core/src/ec/identity/coordinator.rs b/crates/trusted-server-core/src/ec/identity/coordinator.rs index f6811e6954..e34bd1213d 100644 --- a/crates/trusted-server-core/src/ec/identity/coordinator.rs +++ b/crates/trusted-server-core/src/ec/identity/coordinator.rs @@ -26,16 +26,18 @@ use super::mutation::{ }; use super::policy::{FP_US_OPT_OUT_MASK, PolicySignals, fingerprint_bits, policy_allows}; use super::provider::IdentityProvider; -use super::removal::{RemovalWriteOutcome, write_removals}; +use super::removal::{RemovalWriteOutcome, log_removal_lost, write_removals_counted}; use super::resolution::{ EntrySet, EntrySets, IdentityProviderError, Resolution, ResolvedEid, StateChange, entry_sets, }; -use super::revision::{IdentityKeys, StateRevision, record_revision, uid_hash, visitor_key}; +use super::revision::{ + IdentityKeys, Revision, StateRevision, record_revision, uid_hash, visitor_key, +}; use crate::consent::{ConsentContext, effective_tcf}; use crate::ec::kv::KvIdentityGraph; use crate::ec::kv_types::{KvEid, KvEntry, KvOptOut, KvPartnerId}; use crate::ec::rate_limiter::RateLimiter; -use crate::ec::{EcContext, EcKvSnapshot, current_timestamp, log_id}; +use crate::ec::{EcContext, EcKvSnapshot, current_timestamp}; use crate::platform::{ PlatformBackendSpec, PlatformHttpRequest, PlatformPendingRequest, PlatformResponse, RuntimeServices, @@ -1804,32 +1806,84 @@ pub(crate) fn has_capture_rule( }) } +/// What a capture reads of the row. +struct CaptureView { + view: EligibilityView, + /// The row could not be read, so no removal can be built. + unreadable: bool, + /// Reads this view made. + reads: u32, +} + /// The row a capture reads its view from: the request's own snapshot when it -/// holds one for this EC ID, else one read. Empty when the row is missing, -/// unreadable or not a live row. +/// holds one for this EC ID, else one read. Empty when the row is missing or +/// not a live row. fn capture_view( graph: Option<&KvIdentityGraph>, context: &EcContext, ec_id: Option<&str>, -) -> EligibilityView { +) -> CaptureView { + let empty = |unreadable, reads| CaptureView { + view: EligibilityView::empty(), + unreadable, + reads, + }; let (Some(graph), Some(ec_id)) = (graph, ec_id) else { - return EligibilityView::empty(); + return empty(false, 0); }; - let snapshot = if context.kv_snapshot().belongs_to(ec_id) { - context.kv_snapshot().clone() + let (snapshot, reads) = if context.kv_snapshot().belongs_to(ec_id) { + (context.kv_snapshot().clone(), 0) } else { - graph.load_snapshot(ec_id) + (graph.load_snapshot(ec_id), 1) }; match snapshot { - EcKvSnapshot::Present { entry, .. } if entry.consent.ok => EligibilityView::from_kv(&entry), - EcKvSnapshot::Failed { .. } => { - log::warn!( - "Identity capture for '{}' could not read the row, so its removals cannot be built", - log_id(ec_id) - ); - EligibilityView::empty() - } - _ => EligibilityView::empty(), + EcKvSnapshot::Present { entry, .. } if entry.consent.ok => CaptureView { + view: EligibilityView::from_kv(&entry), + unreadable: false, + reads, + }, + EcKvSnapshot::Failed { .. } => empty(true, reads), + _ => empty(false, reads), + } +} + +/// Counts and logs the removals a capture result asked for when the row could +/// not be read, so they cannot be built or retried (spec 5.3). Holds no value. +fn lose_unbuildable_removals( + context: &mut EcContext, + ec_id: Option<&str>, + provider: &IdentityProviderId, + resolution: &Resolution, + catalog: &ProviderCatalog, +) { + let claimed = catalog.sources_of(provider); + let mut lost = IdentityMutation::default(); + for source in resolution + .invalidated() + .iter() + .filter(|source| claimed.contains(source)) + { + lost.add_change(PartnerIdChange::OwnerRemove { + source: source.clone(), + provider: provider.clone(), + expected: Revision::Unset, + }); + } + if resolution.is_opted_out() { + lost.merge_state( + provider.clone(), + ProviderStateChange { + opt_out: Some(KvOptOut { + at: 0, + fp: String::new(), + }), + ..ProviderStateChange::default() + }, + ); + } + log_removal_lost(ec_id.unwrap_or_default(), &lost); + for _ in lost.removal_labels() { + context.identity_mut().record_removal_lost(); } } @@ -1961,11 +2015,20 @@ pub(crate) fn run_capture( return; } let ec_id = context.ec_value().map(str::to_owned); - let view = if call.is_real_browser { + let CaptureView { + view, + unreadable, + reads: view_reads, + } = if call.is_real_browser { capture_view(graph, context, ec_id.as_deref()) } else { - EligibilityView::empty() + CaptureView { + view: EligibilityView::empty(), + unreadable: false, + reads: 0, + } }; + context.identity_mut().record_capture_kv(view_reads, 0); let response = ProviderResponse::new(call.status, call.headers.clone(), call.body.to_vec(), None); let consent = ProviderConsent::from_context(context.consent()); @@ -2030,6 +2093,10 @@ pub(crate) fn run_capture( continue; } + if unreadable { + lose_unbuildable_removals(context, ec_id.as_deref(), id, &resolution, deps.catalog); + continue; + } let ask = capture_ask( deps, context, @@ -2044,7 +2111,8 @@ pub(crate) fn run_capture( mutation.merge_state(id.clone(), ask.state); } - if !mutation.has_removals() { + // The tombstone the finalize step writes supersedes every other removal. + if context.global_revoke_requested() || !mutation.has_removals() { return; } let (Some(graph), Some(ec_id)) = (graph, ec_id.as_deref()) else { @@ -2056,7 +2124,11 @@ pub(crate) fn run_capture( random: deps.random, catalog: deps.catalog, }; - if write_removals(graph, ec_id, &mutation, &apply_context) == RemovalWriteOutcome::Failed { + let (outcome, cost) = write_removals_counted(graph, ec_id, &mutation, &apply_context); + context + .identity_mut() + .record_capture_kv(cost.reads, cost.writes); + if outcome == RemovalWriteOutcome::Failed { if let Some(earlier) = context.identity().pending_removals().cloned() { mutation.absorb(earlier); } @@ -6145,6 +6217,126 @@ mod tests { } } + #[test] + fn capture_removal_lost_when_the_row_cannot_be_read() { + install_capturing_logger(); + let provider = answering_capture_provider("unreadcap", &[FIRST], None); + let seeded = row_with(&[(FIRST, "secret-uid-unread-5521", None)]); + for (body, expected_lost) in [ + (json!({"invalidate": [FIRST]}), 1), + (json!({"opted_out": true}), 1), + (json!({}), 0), + ] { + let mut fixture = CaptureFixture::new(vec![provider.clone()], Some(&seeded)); + fixture.context.set_kv_snapshot(EcKvSnapshot::Failed { + ec_id: TEST_EC_ID.to_owned(), + }); + + fixture.capture(&capture_path("unreadcap"), &body); + + assert_eq!( + fixture.context.identity().removal_lost(), + expected_lost, + "should count the removals it could not build for {body}" + ); + assert!( + fixture.context.identity().pending_removals().is_none(), + "should keep nothing to retry for {body}" + ); + } + let logs = captured_logs(); + assert!( + logs.iter().any(|line| line.contains("removal lost") + && line.contains("unreadcap") + && line.contains(FIRST)), + "should log the provider and source: {logs:?}" + ); + assert!( + !logs + .iter() + .any(|line| line.contains("secret-uid-unread-5521")), + "should log no value" + ); + } + + #[test] + fn capture_counts_its_kv_operations_in_the_measurement() { + use crate::ec::identity::measurement::MeasurementInput; + + let seeded = row_with(&[(FIRST, "secret-uid-a", None)]); + let mut fixture = CaptureFixture::new(vec![lockr()], Some(&seeded)); + fixture.context.set_kv_snapshot(EcKvSnapshot::NotRead); + fixture.store.refuse_conditional_writes(TEST_EC_ID, 2); + + fixture.capture(&lockr_path(), &json!({"invalidate": [FIRST]})); + + let measurement = fixture.context.identity_measurement(&MeasurementInput { + phase_one_read: false, + kv_reads: 0, + kv_writes: 0, + flush: None, + withdrawal_after_send: false, + withdrawal_after_send_failed: false, + }); + assert_eq!( + measurement.kv_reads(), + 4, + "should count the view read and three removal-write reads" + ); + assert_eq!( + measurement.kv_writes(), + 3, + "should count each removal-write attempt" + ); + assert_eq!( + fixture.store.writes().if_generation_match, + 3, + "control: the store saw the same writes" + ); + } + + #[test] + fn capture_global_revoke_skips_the_other_providers_removals() { + let shared = "/integrations/shared/"; + let global = crate::ec::identity::test_support::answering_capture_provider_at( + "lockr", + &[FIRST], + None, + shared, + ); + let local = crate::ec::identity::test_support::answering_capture_provider_at( + "localcap", + &[SECOND], + None, + shared, + ); + let seeded = row_with(&[ + (FIRST, "secret-uid-a", Some("lockr")), + (SECOND, "secret-uid-b", None), + ]); + let mut fixture = + CaptureFixture::with_scopes(vec![global, local], Some(&seeded), &["lockr"]); + + fixture.capture( + "/integrations/shared/api", + &json!({"opted_out": true, "invalidate": [SECOND]}), + ); + + assert!( + fixture.context.global_revoke_requested(), + "should request the global revoke" + ); + assert_eq!( + fixture.store.writes().total(), + 0, + "should leave every removal to the tombstone" + ); + assert!( + fixture.context.identity().pending_removals().is_none(), + "should keep nothing to retry" + ); + } + #[test] fn capture_state_change_is_ignored() { let mut seeded = row_with(&[(FIRST, "secret-uid-a", None)]); diff --git a/crates/trusted-server-core/src/ec/identity/mutation.rs b/crates/trusted-server-core/src/ec/identity/mutation.rs index c67e4b5c40..89849f56b4 100644 --- a/crates/trusted-server-core/src/ec/identity/mutation.rs +++ b/crates/trusted-server-core/src/ec/identity/mutation.rs @@ -251,8 +251,6 @@ impl IdentityMutation { /// Whether any removal, opt-out or clear is present. #[must_use] - // Temporary: first used by a later identity task. - #[allow(dead_code)] pub(crate) fn has_removals(&self) -> bool { self.partner_ids.iter().any(PartnerIdChange::is_removal) || self @@ -266,8 +264,6 @@ impl IdentityMutation { /// de-duplicated. An opt-out or clear has no source. Holds no UID, so it /// is safe to log. #[must_use] - // Temporary: first used by a later identity task. - #[allow(dead_code)] pub(crate) fn removal_labels(&self) -> Vec<(IdentityProviderId, Option)> { let mut labels: Vec<_> = self .partner_ids diff --git a/crates/trusted-server-core/src/ec/identity/removal.rs b/crates/trusted-server-core/src/ec/identity/removal.rs index 9868aa2e60..035f251dc9 100644 --- a/crates/trusted-server-core/src/ec/identity/removal.rs +++ b/crates/trusted-server-core/src/ec/identity/removal.rs @@ -21,6 +21,13 @@ pub enum RemovalWriteOutcome { Failed, } +/// The KV operations one removal write made. +#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)] +pub(crate) struct RemovalCost { + pub(crate) reads: u32, + pub(crate) writes: u32, +} + /// Reads the row, applies only `removals` and writes them with a CAS, /// retrying only a generation mismatch. /// @@ -33,30 +40,45 @@ pub(crate) fn write_removals( removals: &IdentityMutation, context: &ApplyContext<'_>, ) -> RemovalWriteOutcome { + write_removals_counted(graph, ec_id, removals, context).0 +} + +/// As [`write_removals`], also returning the reads and writes it made, for +/// the request's measurement. +#[must_use] +pub(crate) fn write_removals_counted( + graph: &KvIdentityGraph, + ec_id: &str, + removals: &IdentityMutation, + context: &ApplyContext<'_>, +) -> (RemovalWriteOutcome, RemovalCost) { + let mut cost = RemovalCost::default(); for _attempt in 0..REMOVAL_WRITE_ATTEMPTS { + cost.reads = cost.reads.saturating_add(1); let (entry, generation) = match graph.load_snapshot(ec_id) { EcKvSnapshot::Present { entry, generation: Some(generation), .. } => (entry, generation), - EcKvSnapshot::Missing { .. } => return RemovalWriteOutcome::NothingToWrite, - _ => return RemovalWriteOutcome::Failed, + EcKvSnapshot::Missing { .. } => return (RemovalWriteOutcome::NothingToWrite, cost), + _ => return (RemovalWriteOutcome::Failed, cost), }; let Some(applied) = apply_removals_only(removals, &entry, context).filter(|applied| applied.changed) else { - return RemovalWriteOutcome::NothingToWrite; + return (RemovalWriteOutcome::NothingToWrite, cost); }; + cost.writes = cost.writes.saturating_add(1); match graph.write_if_generation(ec_id, &applied.entry, generation) { - Ok(EcKvWriteOutcome::Written) => return RemovalWriteOutcome::Written, + Ok(EcKvWriteOutcome::Written) => return (RemovalWriteOutcome::Written, cost), Ok(EcKvWriteOutcome::PreconditionFailed) => {} Err(error) => { log::warn!( "Identity removal write failed for '{}': {error:?}", log_id(ec_id) ); - return RemovalWriteOutcome::Failed; + return (RemovalWriteOutcome::Failed, cost); } } } @@ -64,7 +86,7 @@ pub(crate) fn write_removals( "Identity removal write for '{}': CAS conflict after {REMOVAL_WRITE_ATTEMPTS} attempts", log_id(ec_id) ); - RemovalWriteOutcome::Failed + (RemovalWriteOutcome::Failed, cost) } /// Logs a removal that every attempt failed to write. Names providers and diff --git a/crates/trusted-server-core/src/ec/identity/request.rs b/crates/trusted-server-core/src/ec/identity/request.rs index dedbff6341..12b5d07448 100644 --- a/crates/trusted-server-core/src/ec/identity/request.rs +++ b/crates/trusted-server-core/src/ec/identity/request.rs @@ -48,6 +48,8 @@ pub(crate) struct RequestIdentity { create_attempts: u32, capture_unpersisted: BTreeMap<(IdentityProviderId, SourceDomain), u32>, global_revoke: bool, + capture_kv_reads: u32, + capture_kv_writes: u32, } impl core::fmt::Debug for RequestIdentity { @@ -213,6 +215,24 @@ impl RequestIdentity { *count = count.saturating_add(1); } + /// KV reads the capture made (its view read and the removal write's). + #[must_use] + pub(crate) fn capture_kv_reads(&self) -> u32 { + self.capture_kv_reads + } + + /// KV writes the capture removal write attempted. + #[must_use] + pub(crate) fn capture_kv_writes(&self) -> u32 { + self.capture_kv_writes + } + + /// Counts KV operations the capture made before send. + pub(crate) fn record_capture_kv(&mut self, reads: u32, writes: u32) { + self.capture_kv_reads = self.capture_kv_reads.saturating_add(reads); + self.capture_kv_writes = self.capture_kv_writes.saturating_add(writes); + } + /// Whether a real browser's capture asked for a global revoke (spec 8). #[must_use] pub(crate) fn global_revoke_requested(&self) -> bool { diff --git a/crates/trusted-server-core/src/ec/identity/test_support.rs b/crates/trusted-server-core/src/ec/identity/test_support.rs index ad7d916126..ae39cd1205 100644 --- a/crates/trusted-server-core/src/ec/identity/test_support.rs +++ b/crates/trusted-server-core/src/ec/identity/test_support.rs @@ -239,6 +239,21 @@ pub fn answering_capture_provider( id: &str, sources: &[&str], gvl_id: Option, +) -> Arc { + answering_capture_provider_at(id, sources, gvl_id, &format!("/integrations/{id}/")) +} + +/// As [`answering_capture_provider`], with the capture rule on `prefix`. +/// +/// # Panics +/// +/// Panics when `id` or a source is not valid. +#[must_use] +pub fn answering_capture_provider_at( + id: &str, + sources: &[&str], + gvl_id: Option, + prefix: &str, ) -> Arc { let policy = match gvl_id { Some(vendor) => ConsentPolicy::strict().with_tcf_vendor(vendor), @@ -246,7 +261,7 @@ pub fn answering_capture_provider( }; let mut builder = ProviderDescriptor::builder(id.parse().expect("should parse the provider ID")) - .with_captures(Captures::new().with_path_prefix(&format!("/integrations/{id}/"))) + .with_captures(Captures::new().with_path_prefix(prefix)) .with_consent_policy(policy); for source in sources { builder = builder.with_source(source.parse().expect("should parse the source")); diff --git a/crates/trusted-server-core/src/ec/mod.rs b/crates/trusted-server-core/src/ec/mod.rs index b717328271..424aa7e7c6 100644 --- a/crates/trusted-server-core/src/ec/mod.rs +++ b/crates/trusted-server-core/src/ec/mod.rs @@ -545,12 +545,18 @@ impl EcContext { route, // A snapshot that came from the new-EC create read nothing, and // each create attempt is one write made before routing. - kv_reads: input.kv_reads.saturating_add(u32::from( - input.phase_one_read && self.identity.create_attempts() == 0, - )), + kv_reads: input + .kv_reads + .saturating_add(u32::from( + input.phase_one_read && self.identity.create_attempts() == 0, + )) + .saturating_add(self.identity.capture_kv_reads()), + // The capture's pre-send reads and writes go through the + // request-state graph, which the adapter's counters do not see. kv_writes: input .kv_writes - .saturating_add(self.identity.create_attempts()), + .saturating_add(self.identity.create_attempts()) + .saturating_add(self.identity.capture_kv_writes()), flush: input.flush.copied(), removal_lost: self.identity.removal_lost(), withdrawal_after_send: input.withdrawal_after_send, From 0ef5ad4eedaca7aa0caf91718821fc387392fa99 Mon Sep 17 00:00:00 2001 From: dhruv8sh Date: Tue, 27 Oct 2026 04:03:06 +0530 Subject: [PATCH 132/194] Write a capture opt-out even when the row view is unreadable An unreadable view now loses only the source invalidations that need the record revision. The opt-out is built without a view and goes through the removal write, with the after-send retry, and counts as lost only if that write and its retry fail. Signed-off-by: dhruv8sh --- .../src/ec/identity/coordinator.rs | 123 +++++++++++------- 1 file changed, 76 insertions(+), 47 deletions(-) diff --git a/crates/trusted-server-core/src/ec/identity/coordinator.rs b/crates/trusted-server-core/src/ec/identity/coordinator.rs index e34bd1213d..b01e4878ae 100644 --- a/crates/trusted-server-core/src/ec/identity/coordinator.rs +++ b/crates/trusted-server-core/src/ec/identity/coordinator.rs @@ -26,18 +26,16 @@ use super::mutation::{ }; use super::policy::{FP_US_OPT_OUT_MASK, PolicySignals, fingerprint_bits, policy_allows}; use super::provider::IdentityProvider; -use super::removal::{RemovalWriteOutcome, log_removal_lost, write_removals_counted}; +use super::removal::{RemovalWriteOutcome, write_removals_counted}; use super::resolution::{ EntrySet, EntrySets, IdentityProviderError, Resolution, ResolvedEid, StateChange, entry_sets, }; -use super::revision::{ - IdentityKeys, Revision, StateRevision, record_revision, uid_hash, visitor_key, -}; +use super::revision::{IdentityKeys, StateRevision, record_revision, uid_hash, visitor_key}; use crate::consent::{ConsentContext, effective_tcf}; use crate::ec::kv::KvIdentityGraph; use crate::ec::kv_types::{KvEid, KvEntry, KvOptOut, KvPartnerId}; use crate::ec::rate_limiter::RateLimiter; -use crate::ec::{EcContext, EcKvSnapshot, current_timestamp}; +use crate::ec::{EcContext, EcKvSnapshot, current_timestamp, log_id}; use crate::platform::{ PlatformBackendSpec, PlatformHttpRequest, PlatformPendingRequest, PlatformResponse, RuntimeServices, @@ -1847,9 +1845,11 @@ fn capture_view( } } -/// Counts and logs the removals a capture result asked for when the row could -/// not be read, so they cannot be built or retried (spec 5.3). Holds no value. -fn lose_unbuildable_removals( +/// Counts and logs the source invalidations a capture result asked for when +/// the row could not be read: each needs the record's revision, so none can be +/// built or retried (spec 5.3). An opt-out needs no view and is not counted +/// here. Holds no value. +fn lose_unbuildable_invalidations( context: &mut EcContext, ec_id: Option<&str>, provider: &IdentityProviderId, @@ -1857,32 +1857,15 @@ fn lose_unbuildable_removals( catalog: &ProviderCatalog, ) { let claimed = catalog.sources_of(provider); - let mut lost = IdentityMutation::default(); for source in resolution .invalidated() .iter() .filter(|source| claimed.contains(source)) { - lost.add_change(PartnerIdChange::OwnerRemove { - source: source.clone(), - provider: provider.clone(), - expected: Revision::Unset, - }); - } - if resolution.is_opted_out() { - lost.merge_state( - provider.clone(), - ProviderStateChange { - opt_out: Some(KvOptOut { - at: 0, - fp: String::new(), - }), - ..ProviderStateChange::default() - }, + log::error!( + "Identity removal lost for '{}': provider '{provider}', source '{source}'", + log_id(ec_id.unwrap_or_default()), ); - } - log_removal_lost(ec_id.unwrap_or_default(), &lost); - for _ in lost.removal_labels() { context.identity_mut().record_removal_lost(); } } @@ -2094,8 +2077,13 @@ pub(crate) fn run_capture( } if unreadable { - lose_unbuildable_removals(context, ec_id.as_deref(), id, &resolution, deps.catalog); - continue; + lose_unbuildable_invalidations( + context, + ec_id.as_deref(), + id, + &resolution, + deps.catalog, + ); } let ask = capture_ask( deps, @@ -6222,28 +6210,69 @@ mod tests { install_capturing_logger(); let provider = answering_capture_provider("unreadcap", &[FIRST], None); let seeded = row_with(&[(FIRST, "secret-uid-unread-5521", None)]); - for (body, expected_lost) in [ - (json!({"invalidate": [FIRST]}), 1), - (json!({"opted_out": true}), 1), - (json!({}), 0), - ] { - let mut fixture = CaptureFixture::new(vec![provider.clone()], Some(&seeded)); + let unreadable = |fixture: &mut CaptureFixture| { fixture.context.set_kv_snapshot(EcKvSnapshot::Failed { ec_id: TEST_EC_ID.to_owned(), }); + }; - fixture.capture(&capture_path("unreadcap"), &body); + // An invalidation needs the row's revision, so it is lost. + let mut invalidating = CaptureFixture::new(vec![provider.clone()], Some(&seeded)); + unreadable(&mut invalidating); + invalidating.capture(&capture_path("unreadcap"), &json!({"invalidate": [FIRST]})); + assert_eq!( + invalidating.context.identity().removal_lost(), + 1, + "should count the invalidation it could not build" + ); + assert_eq!( + invalidating.store.writes().total(), + 0, + "should write nothing" + ); + + // An opt-out needs no view: the removal write re-reads and writes it. + let mut opting_out = CaptureFixture::new(vec![provider.clone()], Some(&seeded)); + unreadable(&mut opting_out); + opting_out.capture(&capture_path("unreadcap"), &json!({"opted_out": true})); + assert_eq!( + opting_out.context.identity().removal_lost(), + 0, + "should not count the opt-out lost" + ); + let row = opting_out.row(); + assert!(row.ps["unreadcap"].o.is_some(), "should store the opt-out"); + assert!( + !row.ids.contains_key(FIRST), + "should remove the provider's source record" + ); + + // A failed opt-out write gets the after-send retry, then is lost. + let mut failing = CaptureFixture::new(vec![provider.clone()], Some(&seeded)); + unreadable(&mut failing); + failing.store.fail_conditional_writes(TEST_EC_ID); + failing.capture(&capture_path("unreadcap"), &json!({"opted_out": true})); + assert!( + failing.context.identity().pending_removals().is_some(), + "should stage the opt-out for the retry" + ); + assert_eq!(failing.context.identity().removal_lost(), 0, "not lost yet"); + failing.retry_after_send(); + assert_eq!( + failing.context.identity().removal_lost(), + 1, + "should count the opt-out lost once its retry fails" + ); + + let mut empty = CaptureFixture::new(vec![provider], Some(&seeded)); + unreadable(&mut empty); + empty.capture(&capture_path("unreadcap"), &json!({})); + assert_eq!( + empty.context.identity().removal_lost(), + 0, + "control: a result without removals loses nothing" + ); - assert_eq!( - fixture.context.identity().removal_lost(), - expected_lost, - "should count the removals it could not build for {body}" - ); - assert!( - fixture.context.identity().pending_removals().is_none(), - "should keep nothing to retry for {body}" - ); - } let logs = captured_logs(); assert!( logs.iter().any(|line| line.contains("removal lost") From 2dc951a97cf49e5f0ef60ff67c217c6f1551bd83 Mon Sep 17 00:00:00 2001 From: dhruv8sh Date: Tue, 27 Oct 2026 04:21:31 +0530 Subject: [PATCH 133/194] Keep batch sync from replacing an owned source's value A batch update to a source an identity provider owns now applies only when the row holds no value for it, and is otherwise unchanged without a write. Unowned sources keep today's replace behaviour. Signed-off-by: dhruv8sh --- .../trusted-server-adapter-fastly/src/app.rs | 2 +- .../trusted-server-core/src/ec/batch_sync.rs | 211 ++++++++++++++++-- crates/trusted-server-core/src/ec/kv.rs | 26 +++ 3 files changed, 217 insertions(+), 22 deletions(-) diff --git a/crates/trusted-server-adapter-fastly/src/app.rs b/crates/trusted-server-adapter-fastly/src/app.rs index 08b7641267..428131d6e2 100644 --- a/crates/trusted-server-adapter-fastly/src/app.rs +++ b/crates/trusted-server-adapter-fastly/src/app.rs @@ -912,7 +912,7 @@ fn run_batch_sync(state: &AppState, services: &RuntimeServices, req: Request) -> let result = crate::require_identity_graph(&state.settings).and_then(|kv| { let partner_registry = PartnerRegistry::from_config(&state.settings.ec.partners)?; let limiter = FastlyRateLimiter::new(RATE_COUNTER_NAME); - handle_batch_sync(&kv, &partner_registry, &limiter, req) + handle_batch_sync(&kv, &partner_registry, &limiter, &state.identity, req) }); let mut response = result.unwrap_or_else(|e| http_error(&e)); diff --git a/crates/trusted-server-core/src/ec/batch_sync.rs b/crates/trusted-server-core/src/ec/batch_sync.rs index 431eb87ca8..fce040af47 100644 --- a/crates/trusted-server-core/src/ec/batch_sync.rs +++ b/crates/trusted-server-core/src/ec/batch_sync.rs @@ -24,6 +24,8 @@ use crate::error::TrustedServerError; use super::auth::authenticate_bearer; use super::generation::{is_valid_ec_id, normalize_ec_id_for_kv}; +use super::identity::ids::SourceDomain; +use super::identity::runtime::IdentityRuntime; use super::kv::{KvIdentityGraph, UpsertResult}; use super::log_id; use super::rate_limiter::RateLimiter; @@ -45,6 +47,7 @@ trait BatchSyncWriter { ec_id: &str, partner_id: &str, uid: &str, + only_when_absent: bool, ) -> Result>; } @@ -54,8 +57,15 @@ impl BatchSyncWriter for KvIdentityGraph { ec_id: &str, partner_id: &str, uid: &str, + only_when_absent: bool, ) -> Result> { - KvIdentityGraph::upsert_partner_id_if_exists(self, ec_id, partner_id, uid) + KvIdentityGraph::upsert_partner_id_if_exists_policy( + self, + ec_id, + partner_id, + uid, + only_when_absent, + ) } } @@ -108,6 +118,10 @@ struct MappingGroup { /// Handles `POST /_ts/api/v1/batch-sync`. /// +/// A source an identity provider owns is written only when the row holds no +/// value for it; a stored value, including a provider's entry set, is never +/// replaced. +/// /// # Errors /// /// Returns [`TrustedServerError`] on serialization or KV store failures. @@ -115,15 +129,17 @@ pub fn handle_batch_sync( kv: &KvIdentityGraph, registry: &PartnerRegistry, rate_limiter: &dyn RateLimiter, + identity: &IdentityRuntime, req: Request, ) -> Result, Report> { - handle_batch_sync_with_writer(kv, registry, rate_limiter, req) + handle_batch_sync_with_writer(kv, registry, rate_limiter, identity, req) } fn handle_batch_sync_with_writer( writer: &dyn BatchSyncWriter, registry: &PartnerRegistry, rate_limiter: &dyn RateLimiter, + identity: &IdentityRuntime, req: Request, ) -> Result, Report> { // 1. Authenticate @@ -167,7 +183,13 @@ fn handle_batch_sync_with_writer( } // 4. Process mappings with per-item validation and rejection reasons. - let (accepted, errors) = process_mappings(writer, &partner.source_domain, &body.mappings); + let owned = partner + .source_domain + .parse::() + .ok() + .is_some_and(|source| identity.catalog().owner_of(&source).is_some()); + let (accepted, errors) = + process_mappings(writer, &partner.source_domain, owned, &body.mappings); let rejected = errors.len(); let status = if rejected > 0 { @@ -202,6 +224,7 @@ fn content_length_exceeds_limit(req: &Request, max_body_size: usize) - fn process_mappings( writer: &dyn BatchSyncWriter, partner_id: &str, + owned: bool, mappings: &[SyncMapping], ) -> (usize, Vec) { let mut errors = Vec::new(); @@ -244,7 +267,12 @@ fn process_mappings( let mut accepted = 0; for (group_index, group) in groups.iter().enumerate() { - match writer.upsert_partner_id_if_exists(&group.ec_id, partner_id, &group.partner_uid) { + match writer.upsert_partner_id_if_exists( + &group.ec_id, + partner_id, + &group.partner_uid, + owned, + ) { Ok(UpsertResult::Written | UpsertResult::Unchanged) => { accepted += group.indexes.len(); } @@ -308,6 +336,10 @@ mod tests { use super::*; use std::collections::VecDeque; + use crate::ec::identity::test_support::{ + example_entry_set, fixed_provider, full_record, pid, source as source_domain, + }; + use crate::ec::kv_types::{KvEntry, KvPartnerId}; use crate::error::TrustedServerError; use crate::redacted::Redacted; use crate::settings::EcPartner; @@ -382,6 +414,7 @@ mod tests { ec_id: &str, partner_id: &str, uid: &str, + _only_when_absent: bool, ) -> Result> { self.calls.borrow_mut().push(WriterCall { ec_id: ec_id.to_owned(), @@ -438,6 +471,136 @@ mod tests { serde_json::from_slice(&body).expect("should deserialize batch-sync response") } + fn identity_with_owner(owned: &[(&str, &str)]) -> IdentityRuntime { + let settings = crate::test_support::tests::create_test_settings(); + let providers = owned + .iter() + .map(|(id, source)| fixed_provider(id, &[source])) + .collect(); + let owners = owned + .iter() + .map(|(id, source)| (source_domain(source), pid(id))) + .collect(); + IdentityRuntime::from_settings_with_providers(&settings, providers, owners) + .expect("should build the identity runtime") + } + + fn empty_identity() -> IdentityRuntime { + identity_with_owner(&[]) + } + + fn batch_body(ec_id: &str, uid: &str) -> String { + serde_json::json!({ + "mappings": [{ "ec_id": ec_id, "partner_uid": uid, "timestamp": 1 }] + }) + .to_string() + } + + fn seeded_graph(ec_id: &str, record: Option) -> KvIdentityGraph { + let graph = KvIdentityGraph::in_memory("batch-store"); + let mut entry = KvEntry::tombstone(1000); + entry.consent.ok = true; + if let Some(record) = record { + entry.ids.insert("ssp.example.com".to_owned(), record); + } + graph.create(ec_id, &entry).expect("should seed the row"); + graph + } + + fn stored_record(graph: &KvIdentityGraph, ec_id: &str) -> Option { + let (entry, _) = graph + .get(ec_id) + .expect("should read the row") + .expect("should hold the row"); + entry.ids.get("ssp.example.com").cloned() + } + + fn run_batch( + graph: &KvIdentityGraph, + identity: &IdentityRuntime, + ec_id: &str, + uid: &str, + ) -> serde_json::Value { + let limiter = MockRateLimiter { + should_exceed: false, + }; + let response = handle_batch_sync( + graph, + &test_registry(), + &limiter, + identity, + authorized_batch_request(&batch_body(ec_id, uid)), + ) + .expect("should handle the batch"); + response_json(response) + } + + #[test] + fn batch_sync_update_to_owned_source_applies_only_when_absent() { + let ec_id = format!("{}.ABC123", "a".repeat(64)); + let owned = identity_with_owner(&[("example-provider", "ssp.example.com")]); + + let absent = seeded_graph(&ec_id, None); + let summary = run_batch(&absent, &owned, &ec_id, "batch-uid"); + assert_eq!(summary["accepted"], 1, "should accept the absent write"); + assert_eq!( + stored_record(&absent, &ec_id), + Some(KvPartnerId::new("batch-uid")), + "should write an owned source that holds no value" + ); + + let present = seeded_graph(&ec_id, Some(KvPartnerId::new("existing-uid"))); + let summary = run_batch(&present, &owned, &ec_id, "batch-uid"); + assert_eq!(summary["accepted"], 1, "should report the skip as accepted"); + assert_eq!( + stored_record(&present, &ec_id), + Some(KvPartnerId::new("existing-uid")), + "should keep the value an owned source already holds" + ); + + let control = seeded_graph(&ec_id, Some(KvPartnerId::new("existing-uid"))); + let summary = run_batch(&control, &empty_identity(), &ec_id, "batch-uid"); + assert_eq!(summary["accepted"], 1, "should accept the unowned update"); + assert_eq!( + stored_record(&control, &ec_id), + Some(KvPartnerId::new("batch-uid")), + "should keep replacing the value of an unowned source" + ); + } + + #[test] + fn batch_sync_never_touches_an_owners_entry_set() { + let ec_id = format!("{}.ABC123", "a".repeat(64)); + let owned = identity_with_owner(&[("example-provider", "ssp.example.com")]); + let owners_record = full_record(Some("example-provider"), example_entry_set()); + let graph = seeded_graph(&ec_id, Some(owners_record.clone())); + + let before = serde_json::to_vec(&owners_record.eids).expect("should serialize"); + let summary = run_batch(&graph, &owned, &ec_id, "batch-uid"); + let after = stored_record(&graph, &ec_id).expect("should keep the record"); + + assert_eq!(summary["accepted"], 1, "should report the skip as accepted"); + assert_eq!( + serde_json::to_vec(&after.eids).expect("should serialize"), + before, + "should leave the owner's eids byte-equal" + ); + assert_eq!(after.eids.len(), 2, "should keep both entries"); + assert_eq!( + after.w.as_deref(), + Some("example-provider"), + "should keep the owner's writer" + ); + + let unowned = seeded_graph(&ec_id, Some(owners_record)); + run_batch(&unowned, &empty_identity(), &ec_id, "batch-uid"); + assert_eq!( + stored_record(&unowned, &ec_id), + Some(KvPartnerId::new("batch-uid")), + "should write a one-entry, one-UID record without w or rv for an unowned source" + ); + } + fn test_registry() -> PartnerRegistry { let partners = vec![make_test_partner( "ssp.example.com", @@ -498,8 +661,9 @@ mod tests { .body(EdgeBody::from("not-json")) .expect("should build test request"); - let response = handle_batch_sync_with_writer(&writer, ®istry, &limiter, req) - .expect("should return oversized response"); + let response = + handle_batch_sync_with_writer(&writer, ®istry, &limiter, &empty_identity(), req) + .expect("should return oversized response"); assert_eq!( response.status(), @@ -524,8 +688,9 @@ mod tests { .body(EdgeBody::from(oversized_body)) .expect("should build test request"); - let response = handle_batch_sync_with_writer(&writer, ®istry, &limiter, req) - .expect("should return oversized response"); + let response = + handle_batch_sync_with_writer(&writer, ®istry, &limiter, &empty_identity(), req) + .expect("should return oversized response"); assert_eq!( response.status(), @@ -543,7 +708,7 @@ mod tests { mapping(&format!("{}.ABC123", "a".repeat(64)), "u3", 1), ]; - let (accepted, errors) = process_mappings(&writer, "partner", &mappings); + let (accepted, errors) = process_mappings(&writer, "partner", false, &mappings); assert_eq!(accepted, 1, "should count successful writes as accepted"); assert_eq!(errors.len(), 2, "should reject invalid mappings only"); @@ -569,7 +734,7 @@ mod tests { mapping(&format!("{}.ABC123", "c".repeat(64)), "u3", 1), ]; - let (accepted, errors) = process_mappings(&writer, "partner", &mappings); + let (accepted, errors) = process_mappings(&writer, "partner", false, &mappings); assert_eq!(accepted, 1, "should keep accepted count before failure"); assert_eq!( @@ -601,8 +766,8 @@ mod tests { .body(EdgeBody::empty()) .expect("should build test request"); - let response = - handle_batch_sync(&kv, ®istry, &limiter, req).expect("should return response"); + let response = handle_batch_sync(&kv, ®istry, &limiter, &empty_identity(), req) + .expect("should return response"); assert_eq!( response.status(), StatusCode::UNAUTHORIZED, @@ -666,7 +831,7 @@ mod tests { mapping(&withdrawn_ec_id, "uid-2", 101), ]; - let (accepted, errors) = process_mappings(&writer, "partner", &mappings); + let (accepted, errors) = process_mappings(&writer, "partner", false, &mappings); assert_eq!(accepted, 0, "should not accept ineligible mappings"); assert_eq!(errors.len(), 2, "should report both errors"); @@ -683,7 +848,7 @@ mod tests { let ec_id = format!("{}.ABC123", "a".repeat(64)); let mappings = vec![mapping(&ec_id, "uid-1", 100), mapping(&ec_id, "uid-1", 101)]; - let (accepted, errors) = process_mappings(&writer, "partner", &mappings); + let (accepted, errors) = process_mappings(&writer, "partner", false, &mappings); assert_eq!(accepted, 2, "should accept every unchanged group member"); assert!( @@ -702,7 +867,7 @@ mod tests { mapping(&ec_id, "uid-old", 100), ]; - let (accepted, errors) = process_mappings(&writer, "partner", &mappings); + let (accepted, errors) = process_mappings(&writer, "partner", false, &mappings); assert_eq!( accepted, 2, @@ -732,7 +897,7 @@ mod tests { mapping(&ec_id_a_upper, "a-last", 3), ]; - let (accepted, errors) = process_mappings(&writer, "partner", &mappings); + let (accepted, errors) = process_mappings(&writer, "partner", false, &mappings); assert_eq!(accepted, 3, "should accept every valid group member"); assert!(errors.is_empty(), "should report no errors"); @@ -762,7 +927,7 @@ mod tests { .map(|index| mapping(&ec_id, &format!("uid-{index}"), index as u64)) .collect::>(); - let (accepted, errors) = process_mappings(&writer, "partner", &mappings); + let (accepted, errors) = process_mappings(&writer, "partner", false, &mappings); assert_eq!( accepted, MAX_BATCH_SIZE, @@ -790,7 +955,7 @@ mod tests { mapping(&mixed_suffix, "mixed", 2), ]; - let (accepted, errors) = process_mappings(&writer, "partner", &mappings); + let (accepted, errors) = process_mappings(&writer, "partner", false, &mappings); assert_eq!(accepted, 2, "should accept both distinct EC IDs"); assert!(errors.is_empty(), "should report no errors"); @@ -822,7 +987,7 @@ mod tests { mapping(&ec_id, " ", 3), ]; - let (accepted, errors) = process_mappings(&writer, "partner", &mappings); + let (accepted, errors) = process_mappings(&writer, "partner", false, &mappings); assert_eq!(accepted, 2, "should accept valid group members"); assert_eq!(errors.len(), 1, "should retain the invalid UID error"); @@ -850,7 +1015,7 @@ mod tests { mapping(&ec_id_b, "b-2", 4), ]; - let (accepted, errors) = process_mappings(&writer, "partner", &mappings); + let (accepted, errors) = process_mappings(&writer, "partner", false, &mappings); assert_eq!(accepted, 0, "should reject all ineligible group members"); assert_eq!( @@ -891,7 +1056,7 @@ mod tests { mapping(&ec_id_c, "", 6), ]; - let (accepted, errors) = process_mappings(&writer, "partner", &mappings); + let (accepted, errors) = process_mappings(&writer, "partner", false, &mappings); assert_eq!( accepted, 2, @@ -938,6 +1103,7 @@ mod tests { &writer, ®istry, &limiter, + &empty_identity(), authorized_batch_request(&body), ) .expect("should return success response"); @@ -968,6 +1134,7 @@ mod tests { &writer, ®istry, &limiter, + &empty_identity(), authorized_batch_request(&body), ) .expect("should return multi-status response"); @@ -1006,6 +1173,7 @@ mod tests { &writer, ®istry, &limiter, + &empty_identity(), authorized_batch_request(&body), ) .expect("should return validation response"); @@ -1051,6 +1219,7 @@ mod tests { &writer, ®istry, &limiter, + &empty_identity(), authorized_batch_request(&body), ) .expect("should return infrastructure failure response"); diff --git a/crates/trusted-server-core/src/ec/kv.rs b/crates/trusted-server-core/src/ec/kv.rs index 4fc200788c..f1ff496542 100644 --- a/crates/trusted-server-core/src/ec/kv.rs +++ b/crates/trusted-server-core/src/ec/kv.rs @@ -800,6 +800,28 @@ impl KvIdentityGraph { ec_id: &str, partner_id: &str, uid: &str, + ) -> Result> { + self.upsert_partner_id_if_exists_policy(ec_id, partner_id, uid, false) + } + + /// Like [`Self::upsert_partner_id_if_exists`], with the replacement rule + /// chosen by the caller. + /// + /// With `only_when_absent`, a source that already holds a value is left + /// untouched and the call returns [`UpsertResult::Unchanged`] without a + /// write. Batch sync sets it for a source an identity provider owns, so a + /// provider's entry set is never replaced. + /// + /// # Errors + /// + /// Returns [`TrustedServerError::KvStore`] on store I/O or CAS + /// exhaustion errors. + pub fn upsert_partner_id_if_exists_policy( + &self, + ec_id: &str, + partner_id: &str, + uid: &str, + only_when_absent: bool, ) -> Result> { for attempt in 0..MAX_CAS_RETRIES { let (mut entry, generation) = match self.get(ec_id)? { @@ -811,6 +833,10 @@ impl KvIdentityGraph { return Ok(UpsertResult::ConsentWithdrawn); } + if only_when_absent && entry.ids.contains_key(partner_id) { + return Ok(UpsertResult::Unchanged); + } + if entry .ids .get(partner_id) From 263402676634d668757fc5361c5c91f214192a5c Mon Sep 17 00:00:00 2001 From: dhruv8sh Date: Tue, 27 Oct 2026 04:27:58 +0530 Subject: [PATCH 134/194] Add the provider harness and conformance checks Add HarnessSetup, TestConsent, CaptureFixture, ConformanceCases and ConformanceFailure, with try_check_conformance and check_conformance in plain and stateful forms, output_eids, capture and check_descriptor_ignores_secrets. Add the gated StoredId and HashedEmail test constructors, and cover the harness with the spec examples and one negative case per conformance check. Signed-off-by: dhruv8sh --- .../src/ec/identity/examples.rs | 794 +++++++++++++- .../src/ec/identity/input.rs | 79 +- .../src/ec/identity/test_support.rs | 966 ++++++++++++++++-- 3 files changed, 1756 insertions(+), 83 deletions(-) diff --git a/crates/trusted-server-core/src/ec/identity/examples.rs b/crates/trusted-server-core/src/ec/identity/examples.rs index b1ce42a572..4a9a2fd339 100644 --- a/crates/trusted-server-core/src/ec/identity/examples.rs +++ b/crates/trusted-server-core/src/ec/identity/examples.rs @@ -13,7 +13,7 @@ use validator::Validate; use super::descriptor::{Captures, ConsentPolicy, Input, ProviderDescriptor, Resolves}; use super::exchange::{ProviderRequest, ProviderResponse}; use super::ids::UnixTime; -use super::input::{CaptureInput, ResolveInput}; +use super::input::{CaptureInput, ResolveInput, StoredId}; use super::provider::{IdentityProvider, IdentityProviderConfig}; use super::resolution::{IdentityProviderError, Resolution, ResolvedEid, ResolvedUid}; use crate::settings::IntegrationConfig; @@ -227,3 +227,795 @@ impl IdentityProvider for ExampleCaptureProvider { ))) } } + +#[cfg(test)] +mod tests { + use http::header::COOKIE; + use http::{HeaderValue, StatusCode}; + + use super::*; + use crate::ec::identity::descriptor::{EidProvenance, ProviderDescriptorBuilder}; + use crate::ec::identity::ids::SourceDomain; + use crate::ec::identity::test_support::{ + CaptureFixture, ConformanceCases, ConformanceFailure, HarnessSetup, ProviderHarness, + TestConsent, check_descriptor_ignores_secrets, + }; + use crate::openrtb::{Eid, Uid}; + + type RequestFn = + fn(&ResolveInput<'_>) -> Result, Report>; + type ResponseFn = fn( + &ProviderResponse, + &ResolveInput<'_>, + ) -> Result>; + type CaptureFn = fn(&CaptureInput<'_>) -> Result>; + + /// A resolver whose behaviour is two plain functions. + struct TestResolver { + descriptor: ProviderDescriptor, + request: RequestFn, + response: ResponseFn, + } + + impl IdentityProvider for TestResolver { + fn descriptor(&self) -> &ProviderDescriptor { + &self.descriptor + } + + fn request( + &self, + input: &ResolveInput<'_>, + ) -> Result, Report> { + (self.request)(input) + } + + fn response( + &self, + response: &ProviderResponse, + input: &ResolveInput<'_>, + ) -> Result> { + (self.response)(response, input) + } + } + + /// A capturer whose behaviour is one plain function. + struct TestCapturer { + descriptor: ProviderDescriptor, + capture: CaptureFn, + } + + impl IdentityProvider for TestCapturer { + fn descriptor(&self) -> &ProviderDescriptor { + &self.descriptor + } + + fn capture( + &self, + input: &CaptureInput<'_>, + ) -> Result> { + (self.capture)(input) + } + } + + /// A provider that declares a capability and overrides nothing. + struct DeclaresOnly(ProviderDescriptor); + + impl IdentityProvider for DeclaresOnly { + fn descriptor(&self) -> &ProviderDescriptor { + &self.0 + } + } + + const CAPTURE_PATH: &str = "/integrations/exampleid/api/token"; + + fn source() -> SourceDomain { + "exampleid.example.com" + .parse() + .expect("should parse the source") + } + + fn builder() -> ProviderDescriptorBuilder { + ProviderDescriptor::builder( + EXAMPLEID_INTEGRATION_ID + .parse() + .expect("should parse the provider id"), + ) + .with_source(source()) + } + + fn resolves() -> Resolves { + Resolves::new().with_endpoint("https://api.exampleid.example.com/v1/fetch") + } + + fn capturer(capture: CaptureFn) -> TestCapturer { + TestCapturer { + descriptor: builder() + .with_captures(Captures::new().with_path(Method::POST, CAPTURE_PATH)) + .build() + .expect("should build the descriptor"), + capture, + } + } + + fn capture_cases() -> ConformanceCases { + ConformanceCases::default() + .with_capture_case(CaptureFixture::new(Method::POST, CAPTURE_PATH)) + } + + fn plain_request( + input: &ResolveInput<'_>, + ) -> Result, Report> { + ProviderRequest::post_json(input.endpoint()?, &json!({})).map(Some) + } + + fn one_uid( + _response: &ProviderResponse, + _input: &ResolveInput<'_>, + ) -> Result> { + Ok(Resolution::found([ResolvedEid::new( + &source(), + ResolvedUid::new("uid-1"), + )])) + } + + fn resolver( + builder: ProviderDescriptorBuilder, + resolves: Resolves, + request: RequestFn, + response: ResponseFn, + ) -> TestResolver { + TestResolver { + descriptor: builder + .with_resolves(resolves) + .build() + .expect("should build the descriptor"), + request, + response, + } + } + + fn resolve_cases() -> ConformanceCases { + ConformanceCases::default().with_resolve_case( + HarnessSetup::default(), + StatusCode::OK, + &json!({}), + ) + } + + fn failure( + provider: &dyn IdentityProvider, + cases: &ConformanceCases, + ) -> Report { + ProviderHarness::new(provider) + .try_check_conformance(cases) + .expect_err("should fail the conformance check") + } + + #[test] + fn example_provider_round_trip() { + let provider = + ExampleIdProvider::new(&example_config()).expect("should build the provider"); + let mut harness = ProviderHarness::new(&provider) + .with_client_ip("192.0.2.10") + .with_user_agent("ExampleBrowser/1.0"); + + let request = harness + .request() + .expect("should build a request") + .expect("should not skip the request"); + assert_eq!( + request.url().host_str(), + Some("api.example.com"), + "should call the declared host" + ); + + let resolution = harness + .respond( + StatusCode::OK, + &json!({ "uid": "example-uid", "signature": "example-signature" }), + ) + .expect("should parse the response"); + assert_eq!( + resolution.eids()[0].uids()[0].id(), + "example-uid", + "should return the uid" + ); + let state: ExampleIdState = harness.state_of(&resolution).expect("should replace state"); + assert_eq!( + state.signature, "example-signature", + "should keep the signature" + ); + } + + #[test] + fn example_provider_conforms() { + let provider = + ExampleIdProvider::new(&example_config()).expect("should build the provider"); + ProviderHarness::new(&provider).check_conformance_with_state::( + &ConformanceCases::default().with_resolve_case( + HarnessSetup::default() + .with_client_ip("192.0.2.10") + .with_user_agent("ExampleBrowser/1.0"), + StatusCode::OK, + &json!({ "uid": "example-uid", "signature": "s" }), + ), + ); + } + + #[test] + fn stateful_provider_is_pointed_at_the_state_check() { + let provider = + ExampleIdProvider::new(&example_config()).expect("should build the provider"); + + let error = failure(&provider, &resolve_cases()); + + assert!( + matches!( + error.current_context(), + ConformanceFailure::StatefulNeedsStateType + ), + "should point at check_conformance_with_state" + ); + } + + #[test] + fn capture_provider_conforms() { + let provider = ExampleCaptureProvider::new().expect("should build the provider"); + + ProviderHarness::new(&provider).check_conformance( + &ConformanceCases::default().with_capture_case( + CaptureFixture::new(Method::POST, "/integrations/examplecapture/api/token") + .with_response( + StatusCode::OK, + &json!({ "tokens": [{ + "source": "examplecapture.example.com", + "value": "token-1", + "expires": 1_760_086_400_u64, + }] }), + ), + ), + ); + } + + #[test] + fn conformance_catches_an_undeclared_read() { + fn reads_user_agent( + input: &ResolveInput<'_>, + ) -> Result, Report> { + let _ignored = input.user_agent().ok(); + plain_request(input) + } + let provider = resolver(builder(), resolves(), reads_user_agent, one_uid); + + let error = failure(&provider, &resolve_cases()); + + assert!( + matches!( + error.current_context(), + ConformanceFailure::UndeclaredInput { .. } + ), + "should report the undeclared input" + ); + } + + #[test] + fn conformance_catches_an_undeclared_state_read() { + fn reads_state( + input: &ResolveInput<'_>, + ) -> Result, Report> { + let _ignored = input.state::().ok(); + plain_request(input) + } + let provider = resolver(builder(), resolves(), reads_state, one_uid); + + let error = failure(&provider, &resolve_cases()); + + assert!( + matches!( + error.current_context(), + ConformanceFailure::UndeclaredInput { .. } + ), + "should report the undeclared state read" + ); + } + + #[test] + fn harness_skips_the_request_when_the_consent_policy_refuses() { + let provider = + ExampleIdProvider::new(&example_config()).expect("should build the provider"); + let allowed = ProviderHarness::new(&provider) + .with_client_ip("192.0.2.10") + .with_user_agent("ExampleBrowser/1.0") + .request() + .expect("should build a request"); + let refused = ProviderHarness::new(&provider) + .with_client_ip("192.0.2.10") + .with_user_agent("ExampleBrowser/1.0") + .with_consent(TestConsent::default().with_gdpr_applies(true)) + .request() + .expect("should not fail the call"); + + assert!(allowed.is_some(), "should build a request without a signal"); + assert!( + refused.is_none(), + "should skip the request when GDPR applies and no TCF consent is present" + ); + } + + #[test] + fn descriptor_ignores_secrets() { + check_descriptor_ignores_secrets(&example_config(), &example_config()); + } + + #[test] + fn output_eids_applies_the_provenance_default_only_to_bare_entries() { + fn two_entries( + _response: &ProviderResponse, + _input: &ResolveInput<'_>, + ) -> Result> { + Ok(Resolution::found([ + ResolvedEid::new(&source(), ResolvedUid::new("uid-a")) + .with_inserter("vendor-inserter"), + ResolvedEid::new(&source(), ResolvedUid::new("uid-b")), + ])) + } + let provider = resolver( + builder().with_provenance( + &source(), + EidProvenance::new() + .with_inserter("default-inserter") + .with_matcher("default-matcher"), + ), + resolves(), + plain_request, + two_entries, + ); + let mut harness = ProviderHarness::new(&provider); + harness + .request() + .expect("should build a request") + .expect("should not skip the request"); + let resolution = harness + .respond(StatusCode::OK, &json!({})) + .expect("should resolve"); + + let eids = harness.output_eids(&resolution); + + assert_eq!(eids.len(), 2, "should output both entries"); + assert_eq!( + eids[0].inserter.as_deref(), + Some("vendor-inserter"), + "should keep the entry's own fields" + ); + assert_eq!( + eids[0].matcher, None, + "should not mix the default into an entry that sets a field" + ); + assert_eq!( + eids[1].inserter.as_deref(), + Some("default-inserter"), + "should apply the default to a bare entry" + ); + assert_eq!( + eids[1].matcher.as_deref(), + Some("default-matcher"), + "should apply the whole default to a bare entry" + ); + } + + #[test] + fn conformance_catches_a_cookie_header() { + fn sets_cookie( + input: &ResolveInput<'_>, + ) -> Result, Report> { + Ok(plain_request(input)?.map(|request| { + request.with_header(COOKIE, HeaderValue::from_static("session=example")) + })) + } + let provider = resolver(builder(), resolves(), sets_cookie, one_uid); + + let error = failure(&provider, &resolve_cases()); + + assert!( + matches!(error.current_context(), ConformanceFailure::CookieHeader), + "should report the Cookie header" + ); + } + + #[test] + fn conformance_catches_a_request_to_an_undeclared_host() { + fn wrong_host( + _input: &ResolveInput<'_>, + ) -> Result, Report> { + let url = Url::parse("https://other.example.org/v1/fetch").expect("should parse"); + Ok(Some(ProviderRequest::get(&url))) + } + let provider = resolver(builder(), resolves(), wrong_host, one_uid); + + let error = failure(&provider, &resolve_cases()); + + assert!( + matches!(error.current_context(), ConformanceFailure::UndeclaredHost), + "should report the undeclared host" + ); + } + + #[test] + fn conformance_catches_a_request_over_plain_http() { + fn plain_http( + _input: &ResolveInput<'_>, + ) -> Result, Report> { + let url = + Url::parse("http://api.exampleid.example.com/v1/fetch").expect("should parse"); + Ok(Some(ProviderRequest::get(&url))) + } + let provider = resolver(builder(), resolves(), plain_http, one_uid); + + let error = failure(&provider, &resolve_cases()); + + assert!( + matches!(error.current_context(), ConformanceFailure::InsecureScheme), + "should report the plain http request" + ); + } + + #[test] + fn conformance_catches_an_entry_for_an_undeclared_source() { + fn other_source( + _response: &ProviderResponse, + _input: &ResolveInput<'_>, + ) -> Result> { + let other: SourceDomain = "other.example.com".parse().expect("should parse"); + Ok(Resolution::found([ResolvedEid::new( + &other, + ResolvedUid::new("uid-1"), + )])) + } + let provider = resolver(builder(), resolves(), plain_request, other_source); + + let error = failure(&provider, &resolve_cases()); + + assert!( + matches!( + error.current_context(), + ConformanceFailure::UndeclaredSource { .. } + ), + "should report the undeclared source" + ); + } + + #[test] + fn conformance_catches_a_33rd_uid() { + fn thirty_three( + _response: &ProviderResponse, + _input: &ResolveInput<'_>, + ) -> Result> { + let mut eid = ResolvedEid::new(&source(), ResolvedUid::new("uid-0")); + for index in 1..33 { + eid = eid.with_uid(ResolvedUid::new(format!("uid-{index}"))); + } + Ok(Resolution::found([eid])) + } + let provider = resolver(builder(), resolves(), plain_request, thirty_three); + + let error = failure(&provider, &resolve_cases()); + + assert!( + matches!(error.current_context(), ConformanceFailure::LimitHit { .. }), + "should report the dropped UID" + ); + } + + #[test] + fn conformance_catches_a_blank_inserter() { + fn blank_inserter( + _response: &ProviderResponse, + _input: &ResolveInput<'_>, + ) -> Result> { + Ok(Resolution::found([ResolvedEid::new( + &source(), + ResolvedUid::new("uid-1"), + ) + .with_inserter(" ")])) + } + let provider = resolver(builder(), resolves(), plain_request, blank_inserter); + + let error = failure(&provider, &resolve_cases()); + + assert!( + matches!(error.current_context(), ConformanceFailure::LimitHit { .. }), + "should report the dropped inserter" + ); + } + + #[test] + fn conformance_catches_an_oversized_set() { + fn eight_large_entries( + _response: &ProviderResponse, + _input: &ResolveInput<'_>, + ) -> Result> { + let entries = (0..8).map(|index| { + ResolvedEid::new( + &source(), + ResolvedUid::new(format!("uid-{index}")) + .with_ext(json!({ "pad": "x".repeat(1_000) })), + ) + }); + Ok(Resolution::found(entries)) + } + let provider = resolver(builder(), resolves(), plain_request, eight_large_entries); + + let error = failure(&provider, &resolve_cases()); + + assert!( + matches!( + error.current_context(), + ConformanceFailure::SetTooLarge { .. } + ), + "should report the set over MAX_ENTRY_SET_BYTES" + ); + } + + #[test] + fn conformance_catches_a_capability_declared_but_not_overridden() { + let resolving = DeclaresOnly( + builder() + .with_resolves(resolves()) + .build() + .expect("should build the descriptor"), + ); + let capturing = DeclaresOnly( + builder() + .with_captures(Captures::new().with_path(Method::POST, CAPTURE_PATH)) + .build() + .expect("should build the descriptor"), + ); + + let resolve_error = failure(&resolving, &resolve_cases()); + let capture_error = failure(&capturing, &capture_cases()); + + assert!( + matches!( + resolve_error.current_context(), + ConformanceFailure::NotImplemented { method: "request" } + ), + "should report the missing request override" + ); + assert!( + matches!( + capture_error.current_context(), + ConformanceFailure::NotImplemented { method: "capture" } + ), + "should report the missing capture override" + ); + } + + #[test] + fn conformance_catches_a_resolver_that_never_builds_a_request() { + fn skips( + _input: &ResolveInput<'_>, + ) -> Result, Report> { + Ok(None) + } + let provider = resolver(builder(), resolves(), skips, one_uid); + + let error = failure(&provider, &resolve_cases()); + + assert!( + matches!( + error.current_context(), + ConformanceFailure::ResolverNeverRequests + ), + "should report a resolver that never requests" + ); + } + + #[test] + fn conformance_catches_a_capturer_that_never_captures() { + fn nothing(_input: &CaptureInput<'_>) -> Result> { + Ok(Resolution::no_match()) + } + + let error = failure(&capturer(nothing), &capture_cases()); + + assert!( + matches!( + error.current_context(), + ConformanceFailure::CapturerNeverMatches + ), + "should report a capturer that captures nothing" + ); + } + + #[test] + fn conformance_catches_state_from_a_stateless_provider() { + fn returns_state( + _response: &ProviderResponse, + _input: &ResolveInput<'_>, + ) -> Result> { + Ok( + Resolution::found([ResolvedEid::new(&source(), ResolvedUid::new("uid-1"))]) + .replacing_state(&json!({ "signature": "s" })), + ) + } + let provider = resolver(builder(), resolves(), plain_request, returns_state); + + let error = failure(&provider, &resolve_cases()); + + assert!( + matches!(error.current_context(), ConformanceFailure::StatelessState), + "should report state from a provider without private state" + ); + } + + #[test] + fn conformance_catches_a_capture_that_changes_state() { + fn clears_state( + _input: &CaptureInput<'_>, + ) -> Result> { + Ok( + Resolution::found([ResolvedEid::new(&source(), ResolvedUid::new("uid-1"))]) + .clearing_state(), + ) + } + + let error = failure(&capturer(clears_state), &capture_cases()); + + assert!( + matches!( + error.current_context(), + ConformanceFailure::CaptureStateChange + ), + "should report a capture that changes state" + ); + } + + #[test] + fn conformance_catches_a_stateful_provider_that_accepts_a_bare_string_state() { + fn reads_any_state( + input: &ResolveInput<'_>, + ) -> Result, Report> { + let _state = input.state::()?; + plain_request(input) + } + fn keeps_state( + _response: &ProviderResponse, + _input: &ResolveInput<'_>, + ) -> Result> { + Ok( + Resolution::found([ResolvedEid::new(&source(), ResolvedUid::new("uid-1"))]) + .replacing_state(&json!({ "signature": "s" })), + ) + } + let provider = resolver( + builder(), + resolves().with_private_state(), + reads_any_state, + keeps_state, + ); + + let error = ProviderHarness::new(&provider) + .try_check_conformance_with_state::(&resolve_cases()) + .expect_err("should fail a state type that decodes a bare string"); + + assert!( + matches!( + error.current_context(), + ConformanceFailure::BareStringStateAccepted + ), + "should report the accepted bare string" + ); + } + + #[test] + fn conformance_catches_an_expiring_source_without_an_expiry() { + let provider = resolver( + builder().with_expiring_source(&source()), + resolves(), + plain_request, + one_uid, + ); + + let error = failure(&provider, &resolve_cases()); + + assert!( + matches!( + error.current_context(), + ConformanceFailure::MissingExpiry { .. } + ), + "should report the missing expiry" + ); + } + + #[test] + fn conformance_accepts_a_complete_set_within_the_limits() { + fn complete_set( + _response: &ProviderResponse, + _input: &ResolveInput<'_>, + ) -> Result> { + Ok(Resolution::found([ + ResolvedEid::new( + &source(), + ResolvedUid::new("uid-a") + .with_atype(1) + .with_ext(json!({ "stype": "ppuid" })), + ) + .with_uid(ResolvedUid::new("uid-b")) + .with_inserter("inserter.example.com") + .with_matcher("matcher.example.com") + .with_mm(2) + .with_ext(json!({ "segments": ["s"] })), + ResolvedEid::new(&source(), ResolvedUid::new("uid-c")) + .with_ext(json!({ "segments": ["t"] })), + ])) + } + let provider = resolver(builder(), resolves(), plain_request, complete_set); + + let result = ProviderHarness::new(&provider).try_check_conformance(&resolve_cases()); + + assert!( + result.is_ok(), + "should accept two entries and three UIDs with every field" + ); + } + + #[test] + fn stored_full_set_reaches_a_refreshing_provider() { + fn echoes_stored_set( + input: &ResolveInput<'_>, + ) -> Result, Report> { + let stored = input + .stored(&source()) + .map(|stored| stored.eids().to_vec()) + .unwrap_or_default(); + ProviderRequest::post_json(input.endpoint()?, &stored).map(Some) + } + let provider = resolver(builder(), resolves(), echoes_stored_set, one_uid); + let set = vec![ + Eid { + source: "exampleid.example.com".to_owned(), + uids: vec![ + Uid { + id: "uid-a".to_owned(), + atype: Some(1), + ext: None, + }, + Uid { + id: "uid-b".to_owned(), + atype: None, + ext: Some(json!({ "stype": "ppuid" })), + }, + ], + inserter: Some("inserter.example.com".to_owned()), + matcher: None, + mm: Some(2), + ext: None, + }, + Eid { + source: "exampleid.example.com".to_owned(), + uids: vec![Uid { + id: "uid-c".to_owned(), + atype: None, + ext: None, + }], + inserter: None, + matcher: None, + mm: None, + ext: Some(json!({ "segments": ["s"] })), + }, + ]; + let mut harness = ProviderHarness::new(&provider) + .with_stored(StoredId::for_test(&source(), "uid-a").with_eids(set.clone())); + + harness + .request() + .expect("should build a request") + .expect("should not skip the refresh"); + + assert_eq!( + harness.request_json::>(), + set, + "should give the provider the stored full set" + ); + } +} diff --git a/crates/trusted-server-core/src/ec/identity/input.rs b/crates/trusted-server-core/src/ec/identity/input.rs index 58237163b0..bcfdc23d27 100644 --- a/crates/trusted-server-core/src/ec/identity/input.rs +++ b/crates/trusted-server-core/src/ec/identity/input.rs @@ -170,6 +170,14 @@ impl HashedEmail { Self(bytes) } + /// From the SHA-256 digest of a normalized address. Test support only, + /// and the one constructor over [`Self::from_bytes`]. + #[cfg(any(test, feature = "test-utils"))] + #[must_use] + pub fn for_test(bytes: [u8; 32]) -> Self { + Self::from_bytes(bytes) + } + /// The digest. #[must_use] pub fn as_bytes(&self) -> &[u8; 32] { @@ -259,6 +267,53 @@ impl StoredId { self.eids.first().and_then(|eid| eid.uids.first()) } + /// A stored one-entry, one-UID record of `source`. Test support only. + /// + /// # Panics + /// + /// Panics when `uid` is not a valid stored UID. + #[cfg(any(test, feature = "test-utils"))] + #[must_use] + pub fn for_test(source: &SourceDomain, uid: &str) -> Self { + Self::from_record(source, &KvPartnerId::new(uid)) + .expect("should build a stored ID from a record with a UID") + } + + /// Replaces the entry set with a full one, and `uid()` becomes its first + /// UID. Test support only, so a test can give a refreshing provider a + /// stored full set. + /// + /// # Panics + /// + /// Panics when `eids` holds no UID. + #[cfg(any(test, feature = "test-utils"))] + #[must_use] + pub fn with_eids(mut self, eids: Vec) -> Self { + self.uid = eids + .first() + .and_then(|eid| eid.uids.first()) + .map(|uid| uid.id.clone()) + .expect("should give the stored set at least one UID"); + self.eids = eids; + self + } + + /// Records `writer` as the provider that wrote it. Test support only. + #[cfg(any(test, feature = "test-utils"))] + #[must_use] + pub fn written_by(mut self, writer: &IdentityProviderId) -> Self { + self.writer = Some(writer.clone()); + self + } + + /// Sets the time it must stop being sent. Test support only. + #[cfg(any(test, feature = "test-utils"))] + #[must_use] + pub fn expiring_at(mut self, at: UnixTime) -> Self { + self.expires_at = Some(at); + self + } + /// Every stored entry for the source, in stored order, with its UIDs and /// EID-level fields. A record that browser ingest, generic pull, batch /// sync or a version 1 row wrote is one entry with one UID. @@ -389,6 +444,7 @@ pub struct ResolveInput<'a> { state_json: Option, undeclared: RefCell>, state_failed: RefCell, + state_decoded: RefCell, } impl fmt::Debug for ResolveInput<'_> { @@ -432,6 +488,7 @@ impl<'a> ResolveInput<'a> { state_json: None, undeclared: RefCell::new(Vec::new()), state_failed: RefCell::new(false), + state_decoded: RefCell::new(false), } } @@ -478,6 +535,15 @@ impl<'a> ResolveInput<'a> { *self.state_failed.borrow() } + /// Whether a `state()` call decoded the stored state. The conformance + /// check of the test harness uses it to see a provider accept a state + /// that no valid state looks like. + #[cfg(any(test, feature = "test-utils"))] + #[must_use] + pub(crate) fn state_was_decoded(&self) -> bool { + *self.state_decoded.borrow() + } + /// The sources to resolve this time, each with its own kind. pub fn sources(&self) -> impl Iterator + '_ { self.kinds.iter().map(|(source, kind)| (source, *kind)) @@ -525,10 +591,15 @@ impl<'a> ResolveInput<'a> { let Some(raw) = self.state_json.as_deref() else { return Ok(None); }; - serde_json::from_str(raw).map(Some).map_err(|_| { - *self.state_failed.borrow_mut() = true; - Report::new(IdentityProviderError::InvalidState) - }) + serde_json::from_str(raw) + .map(|state| { + *self.state_decoded.borrow_mut() = true; + Some(state) + }) + .map_err(|_| { + *self.state_failed.borrow_mut() = true; + Report::new(IdentityProviderError::InvalidState) + }) } /// The endpoint declared with `Resolves::with_endpoint`. diff --git a/crates/trusted-server-core/src/ec/identity/test_support.rs b/crates/trusted-server-core/src/ec/identity/test_support.rs index ae39cd1205..ef1f1669f1 100644 --- a/crates/trusted-server-core/src/ec/identity/test_support.rs +++ b/crates/trusted-server-core/src/ec/identity/test_support.rs @@ -14,19 +14,21 @@ use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering}; use std::sync::{Arc, Mutex}; use error_stack::Report; -use http::{HeaderMap, HeaderName, HeaderValue, StatusCode}; +use http::{HeaderMap, HeaderName, HeaderValue, Method, StatusCode}; use serde::Serialize; use serde::de::DeserializeOwned; use serde_json::json; use url::Url; -use crate::consent::ConsentContext; use crate::consent::jurisdiction::Jurisdiction; +use crate::consent::types::RawConsentSignals; use crate::consent::types::TcfConsent; +use crate::consent::{ConsentContext, build_context_from_signals}; use crate::ec::identity::catalog::ProviderCatalog; use crate::ec::identity::coordinator::{OutcomeContext, RequestFacts}; use crate::ec::identity::descriptor::{ - Captures, ConsentPolicy, EidProvenance, Input, ProviderDescriptor, ResolveKind, Resolves, + Access, Captures, ConsentPolicy, EidProvenance, Input, ProviderDescriptor, ResolveKind, + Resolves, }; use crate::ec::identity::exchange::{ProviderRequest, ProviderResponse}; use crate::ec::identity::ids::{IdentityProviderId, SourceDomain, UnixTime}; @@ -34,10 +36,10 @@ use crate::ec::identity::input::{ CaptureInput, HashedEmail, ProviderConsent, Region, ResolveInput, ResolveSignals, SeededRandom, StoredId, }; -use crate::ec::identity::policy::PolicySignals; -use crate::ec::identity::provider::IdentityProvider; +use crate::ec::identity::policy::{PolicySignals, policy_allows}; +use crate::ec::identity::provider::{IdentityProvider, IdentityProviderConfig}; use crate::ec::identity::resolution::{ - IdentityProviderError, Resolution, ResolvedEid, ResolvedUid, When, + IdentityProviderError, Resolution, ResolvedEid, ResolvedUid, StateChange, When, entry_sets, }; use crate::ec::identity::revision::{StateRevision, new_revision}; use crate::ec::kv_backend::in_memory::InMemoryEcKv; @@ -45,6 +47,7 @@ use crate::ec::kv_backend::{EcKvLookup, EcKvStore, EcKvWrite, EcKvWriteMode, EcK use crate::ec::kv_types::{KvEid, KvPartnerId, KvUid}; use crate::ec::rate_limiter::RateLimiter; use crate::error::TrustedServerError; +use crate::openrtb::{Eid, Uid}; use crate::redacted::Redacted; use crate::settings::{EcPartner, IdentitySettings, ProviderSettings}; @@ -495,8 +498,7 @@ pub fn ec_partner(source: &str, pull_sync_url: &str, token: &str) -> EcPartner { /// Panics when the record is not valid. #[must_use] pub(crate) fn stored_for_test(source: &SourceDomain, uid: &str) -> StoredId { - StoredId::from_record(source, &KvPartnerId::new(uid)) - .expect("should build a stored ID from a record with a UID") + StoredId::for_test(source, uid) } /// Write and lookup counts of a [`WriteCountingEcKv`]. @@ -663,50 +665,26 @@ impl EcKvStore for WriteCountingEcKv { } } -/// Drives one provider the way the coordinator does, so a vendor's tests -/// stay short. -/// -/// A source with a [`with_stored`](Self::with_stored) record is a `Refresh`, -/// any other an `Initial`. Sources whose kind needs an input the harness -/// does not hold are left out, and with none left -/// [`request`](Self::request) returns `Ok(None)` without calling the -/// provider. -pub struct ProviderHarness<'provider> { - provider: &'provider dyn IdentityProvider, +/// The harness clock when a test does not set one. +const HARNESS_NOW: u64 = 1_760_000_000; + +/// Inputs for one harness run. The harness's own `with_*` methods delegate +/// to an inner `HarnessSetup`. +#[derive(Debug, Clone, Default)] +pub struct HarnessSetup { client_ip: Option, user_agent: Option, page_url: Option, region: Option, hashed_email: Option, - stored: BTreeMap, + consent: TestConsent, + stored: Vec, state_json: Option, - now: UnixTime, - random: SeededRandom, - ec_id: Option, - last_request: Option, + now: Option, + random_seed: Option, } -impl<'provider> ProviderHarness<'provider> { - /// A harness for `provider` with no inputs, the clock at 1,760,000,000 - /// and random seed 1. - #[must_use] - pub fn new(provider: &'provider dyn IdentityProvider) -> Self { - Self { - provider, - client_ip: None, - user_agent: None, - page_url: None, - region: None, - hashed_email: None, - stored: BTreeMap::new(), - state_json: None, - now: UnixTime::from_secs(1_760_000_000), - random: SeededRandom::new(1), - ec_id: None, - last_request: None, - } - } - +impl HarnessSetup { /// Sets the client IP. /// /// # Panics @@ -755,16 +733,19 @@ impl<'provider> ProviderHarness<'provider> { self } + /// Sets the raw consent strings. Default: GDPR does not apply and there + /// are no opt-outs. + #[must_use] + pub fn with_consent(mut self, consent: TestConsent) -> Self { + self.consent = consent; + self + } + /// Gives the provider a stored record, valid or expired. Its source is /// the one its first entry names, or the provider's primary source. #[must_use] pub fn with_stored(mut self, stored: StoredId) -> Self { - let source = stored - .eids() - .first() - .and_then(|eid| eid.source.parse().ok()) - .unwrap_or_else(|| self.provider.descriptor().primary_source().clone()); - self.stored.insert(source, stored); + self.stored.push(stored); self } @@ -779,17 +760,419 @@ impl<'provider> ProviderHarness<'provider> { self } - /// Sets the fixed clock. + /// Sets the fixed clock. Default: 1,760,000,000. #[must_use] pub fn with_now(mut self, now: UnixTime) -> Self { - self.now = now; + self.now = Some(now); self } + /// Seeds the random source. Default: 1. + #[must_use] + pub fn with_random_seed(mut self, seed: u64) -> Self { + self.random_seed = Some(seed); + self + } + + fn now(&self) -> UnixTime { + self.now.unwrap_or(UnixTime::from_secs(HARNESS_NOW)) + } + + fn random(&self) -> SeededRandom { + SeededRandom::new(self.random_seed.unwrap_or(1)) + } +} + +/// Raw TCF, GPP and US Privacy strings, as a request would carry them. +#[derive(Debug, Clone, Default)] +pub struct TestConsent { + gdpr_applies: Option, + tc_string: Option, + gpp: Option<(String, Vec)>, + us_privacy: Option, + gpc: bool, + dnt: bool, +} + +impl TestConsent { + /// Sets whether GDPR applies, whatever the strings say. + #[must_use] + pub fn with_gdpr_applies(mut self, applies: bool) -> Self { + self.gdpr_applies = Some(applies); + self + } + + /// Sets the raw TCF string. + #[must_use] + pub fn with_tc_string(mut self, tc: &str) -> Self { + self.tc_string = Some(tc.to_owned()); + self + } + + /// Sets the raw GPP string and its section IDs. + #[must_use] + pub fn with_gpp_string(mut self, gpp: &str, section_ids: &[u16]) -> Self { + self.gpp = Some((gpp.to_owned(), section_ids.to_vec())); + self + } + + /// Sets the raw US Privacy string. + #[must_use] + pub fn with_us_privacy(mut self, us_privacy: &str) -> Self { + self.us_privacy = Some(us_privacy.to_owned()); + self + } + + /// Sets the Global Privacy Control signal. + #[must_use] + pub fn with_gpc(mut self) -> Self { + self.gpc = true; + self + } + + /// Sets the `DNT: 1` signal. + #[must_use] + pub fn with_dnt(mut self) -> Self { + self.dnt = true; + self + } + + /// The consent context a request carrying these signals would build. + fn context(&self) -> ConsentContext { + let signals = RawConsentSignals { + raw_tc_string: self.tc_string.clone(), + raw_gpp_string: self.gpp.as_ref().map(|(gpp, _)| gpp.clone()), + raw_gpp_sid: self + .gpp + .as_ref() + .map(|(_, ids)| ids.iter().map(u16::to_string).collect::>().join(",")), + raw_us_privacy: self.us_privacy.clone(), + gpc: self.gpc, + }; + let mut context = build_context_from_signals(&signals); + // A known non-regulated region, so only the strings given matter. + context.jurisdiction = Jurisdiction::NonRegulated; + if let Some(applies) = self.gdpr_applies { + context.gdpr_applies = applies; + } + context + } +} + +/// One proxied call: the request a capture rule matches, and its response. +#[derive(Debug, Clone)] +pub struct CaptureFixture { + method: Method, + path: String, + status: StatusCode, + body: Vec, +} + +impl CaptureFixture { + /// A proxied call with `method` and `path`, answered 200 with no body. + #[must_use] + pub fn new(method: Method, path: &str) -> Self { + Self { + method, + path: path.to_owned(), + status: StatusCode::OK, + body: Vec::new(), + } + } + + /// Sets the response status and JSON body. + /// + /// # Panics + /// + /// Panics when `body` does not serialize. + #[must_use] + pub fn with_response(mut self, status: StatusCode, body: &serde_json::Value) -> Self { + self.status = status; + self.body = serde_json::to_vec(body).expect("should serialize the response body"); + self + } +} + +#[derive(Debug, Clone)] +struct ResolveCase { + setup: HarnessSetup, + status: StatusCode, + headers: Vec<(String, String)>, + body: Vec, +} + +/// The setups to run `request` under, the responses to answer with, and the +/// proxied calls to capture from, for one conformance run. +#[derive(Debug, Clone, Default)] +pub struct ConformanceCases { + resolve: Vec, + capture: Vec, +} + +impl ConformanceCases { + /// A setup to run `request` under, and the response to answer it with. + /// + /// # Panics + /// + /// Panics when `response` does not serialize. + #[must_use] + pub fn with_resolve_case( + self, + setup: HarnessSetup, + status: StatusCode, + response: &serde_json::Value, + ) -> Self { + let body = serde_json::to_vec(response).expect("should serialize the response body"); + self.with_resolve_case_raw(setup, status, &[], &body) + } + + /// As [`Self::with_resolve_case`], for text, binary or empty replies, + /// such as a base64 body or a 204. + #[must_use] + pub fn with_resolve_case_raw( + mut self, + setup: HarnessSetup, + status: StatusCode, + headers: &[(&str, &str)], + body: &[u8], + ) -> Self { + self.resolve.push(ResolveCase { + setup, + status, + headers: headers + .iter() + .map(|(name, value)| ((*name).to_owned(), (*value).to_owned())) + .collect(), + body: body.to_vec(), + }); + self + } + + /// A proxied call to run `capture` on. + #[must_use] + pub fn with_capture_case(mut self, fixture: CaptureFixture) -> Self { + self.capture.push(fixture); + self + } +} + +/// The first conformance check a provider failed. Each variant names what +/// failed and never a value. +#[derive(Debug, derive_more::Display)] +#[non_exhaustive] +pub enum ConformanceFailure { + /// A read of something the descriptor does not declare. + #[display("the provider read `{access}` without declaring it")] + UndeclaredInput { + /// What was read. + access: Access, + }, + /// A request to a host the descriptor does not declare. + #[display("a request went to a host the descriptor does not declare")] + UndeclaredHost, + /// A request that is not `https`. + #[display("a request did not use https")] + InsecureScheme, + /// A request that sets a `Cookie` header. + #[display("a request set a Cookie header")] + CookieHeader, + /// A request header whose value is not visible ASCII. + #[display("a request header value is not valid")] + InvalidHeader, + /// An entry for a source the descriptor does not declare. + #[display("a result holds an entry for undeclared source `{source}`")] + UndeclaredSource { + /// The source. + source: SourceDomain, + }, + /// A fixture result that hits a limit of spec 4.5. + #[display("a result for `{source}` hit a limit: {part} was dropped")] + LimitHit { + /// The source. + source: SourceDomain, + /// What the limit dropped. + part: String, + }, + /// A fixture set over `MAX_ENTRY_SET_BYTES`. + #[display("the entry set for `{source}` is over the size limit")] + SetTooLarge { + /// The source. + source: SourceDomain, + }, + /// A capability the descriptor declares and the provider does not + /// override. + #[display("`{method}` is declared but not implemented")] + NotImplemented { + /// The missing method. + method: &'static str, + }, + /// A provider that declares `resolves` and builds no request in any case. + #[display("the provider built no request for any listed setup")] + ResolverNeverRequests, + /// A provider that declares `captures` and captures nothing in any case. + #[display("the provider captured nothing for any listed fixture")] + CapturerNeverMatches, + /// A provider without `with_private_state()` that returns state. + #[display("a provider without private state returned a state change")] + StatelessState, + /// A capture that returns a state change. + #[display("a capture returned a state change")] + CaptureStateChange, + /// A fixture set of an expiring source without an expiry. + #[display("the entry set for expiring source `{source}` carries no expiry")] + MissingExpiry { + /// The source. + source: SourceDomain, + }, + /// A stateful provider that accepts a bare-JSON-string stored state. + #[display("the provider accepted a bare-string stored state as valid")] + BareStringStateAccepted, + /// A returned state that does not round-trip through the state type. + #[display("a returned state does not round-trip through the state type")] + StateRoundTrip, + /// A stateful provider checked without a state type. + #[display( + "the provider keeps private state: use `check_conformance_with_state` with its state type" + )] + StatefulNeedsStateType, + /// A provider bug that is not one of the checks above. + #[display("the provider failed in `{stage}` with a configuration error")] + ProviderFailed { + /// The failing method. + stage: &'static str, + }, +} + +impl core::error::Error for ConformanceFailure {} + +/// Checks that a returned state round-trips through the provider's state type. +type StateRoundTrip<'a, 'provider> = dyn Fn(&ProviderHarness<'provider>, &str) -> bool + 'a; + +/// Drives one provider the way the coordinator does, so a vendor's tests +/// stay short. +/// +/// A source with a [`with_stored`](Self::with_stored) record is a `Refresh`, +/// any other an `Initial`. Sources whose kind needs an input the harness +/// does not hold are left out, and with none left +/// [`request`](Self::request) returns `Ok(None)` without calling the +/// provider. So does a consent the provider's policy refuses. +pub struct ProviderHarness<'provider> { + provider: &'provider dyn IdentityProvider, + setup: HarnessSetup, + random: SeededRandom, + ec_id: Option, + last_request: Option, + recorded: Vec, + state_decoded: bool, +} + +impl<'provider> ProviderHarness<'provider> { + /// A harness for `provider` with no inputs, the clock at 1,760,000,000 + /// and random seed 1. + #[must_use] + pub fn new(provider: &'provider dyn IdentityProvider) -> Self { + let setup = HarnessSetup::default(); + Self { + provider, + random: setup.random(), + setup, + ec_id: None, + last_request: None, + recorded: Vec::new(), + state_decoded: false, + } + } + + /// Replaces every input at once, for conformance cases. + #[must_use] + pub fn with_setup(mut self, setup: HarnessSetup) -> Self { + self.random = setup.random(); + self.setup = setup; + self + } + + fn map_setup(mut self, change: impl FnOnce(HarnessSetup) -> HarnessSetup) -> Self { + self.setup = change(core::mem::take(&mut self.setup)); + self + } + + /// Sets the client IP. + /// + /// # Panics + /// + /// Panics when `ip` is not an IP address. + #[must_use] + pub fn with_client_ip(self, ip: &str) -> Self { + self.map_setup(|setup| setup.with_client_ip(ip)) + } + + /// Sets the `User-Agent`. + #[must_use] + pub fn with_user_agent(self, user_agent: &str) -> Self { + self.map_setup(|setup| setup.with_user_agent(user_agent)) + } + + /// Sets the page URL. + /// + /// # Panics + /// + /// Panics when `url` is not a URL. + #[must_use] + pub fn with_page_url(self, url: &str) -> Self { + self.map_setup(|setup| setup.with_page_url(url)) + } + + /// Sets the region. + /// + /// # Panics + /// + /// Panics when `country` is not a country code. + #[must_use] + pub fn with_region(self, country: &str, subdivision: Option<&str>) -> Self { + self.map_setup(|setup| setup.with_region(country, subdivision)) + } + + /// Sets the hashed email. + #[must_use] + pub fn with_hashed_email(self, email: HashedEmail) -> Self { + self.map_setup(|setup| setup.with_hashed_email(email)) + } + + /// Sets the raw consent strings. Default: GDPR does not apply and no + /// opt-outs. + #[must_use] + pub fn with_consent(self, consent: TestConsent) -> Self { + self.map_setup(|setup| setup.with_consent(consent)) + } + + /// Gives the provider a stored record, valid or expired. Its source is + /// the one its first entry names, or the provider's primary source. + #[must_use] + pub fn with_stored(self, stored: StoredId) -> Self { + self.map_setup(|setup| setup.with_stored(stored)) + } + + /// Gives the provider its private state, as stored. + /// + /// # Panics + /// + /// Panics when `state` does not serialize. + #[must_use] + pub fn with_state(self, state: &impl Serialize) -> Self { + self.map_setup(|setup| setup.with_state(state)) + } + + /// Sets the fixed clock. + #[must_use] + pub fn with_now(self, now: UnixTime) -> Self { + self.map_setup(|setup| setup.with_now(now)) + } + /// Reseeds the random source. #[must_use] pub fn with_random_seed(mut self, seed: u64) -> Self { - self.random = SeededRandom::new(seed); + self.setup = core::mem::take(&mut self.setup).with_random_seed(seed); + self.random = self.setup.random(); self } @@ -807,14 +1190,14 @@ impl<'provider> ProviderHarness<'provider> { /// Returns what the provider returns. pub fn request(&mut self) -> Result, Report> { let kinds = self.kinds(); - let request = if kinds.is_empty() { - None + let (result, recorded, decoded) = if kinds.is_empty() || !self.consent_allows() { + (Ok(None), Vec::new(), false) } else { - let signals = self.signals(); - let input = ResolveInput::new(self.provider.descriptor(), kinds, &signals) - .with_state_json(self.state_json.clone()); - self.provider.request(&input)? + self.call(kinds, |input| self.provider.request(input)) }; + self.recorded.extend(recorded); + self.state_decoded |= decoded; + let request = result?; self.last_request.clone_from(&request); Ok(request) } @@ -878,26 +1261,375 @@ impl<'provider> ProviderHarness<'provider> { .as_ref() .and_then(|request| request.call_context_json().map(str::to_owned)); let response = ProviderResponse::new(status, header_map, body.to_vec(), call_context); - let signals = self.signals(); - let input = ResolveInput::new(self.provider.descriptor(), self.kinds(), &signals) - .with_state_json(self.state_json.clone()); - self.provider.response(&response, &input) + let (result, recorded, decoded) = self.call(self.kinds(), |input| { + self.provider.response(&response, input) + }); + self.recorded.extend(recorded); + self.state_decoded |= decoded; + result + } + + /// Runs a capture as the coordinator does: the fixture's response is + /// already decoded. A call no capture rule matches is `no_match`, as + /// the coordinator never runs the provider for it. + /// + /// # Errors + /// + /// Returns what the provider returns. + pub fn capture( + &self, + fixture: &CaptureFixture, + ) -> Result> { + let descriptor = self.provider.descriptor(); + if !descriptor + .captures() + .is_some_and(|captures| captures.matches(&fixture.method, &fixture.path)) + { + return Ok(Resolution::no_match()); + } + let response = + ProviderResponse::new(fixture.status, HeaderMap::new(), fixture.body.clone(), None); + let stored = self.stored_by_source(); + let consent = ProviderConsent::from_context(&self.setup.consent.context()); + let input = CaptureInput::new( + descriptor, + &fixture.path, + &response, + &stored, + &consent, + "www.example.com", + self.setup.now(), + ); + self.provider.capture(&input) } /// Resolves an expiry or refresh against the fixed clock. #[must_use] pub fn resolve(&self, when: When) -> UnixTime { - when.resolve(self.now) + when.resolve(self.setup.now()) + } + + /// The replacement state in a result, decoded, for assertions. + #[must_use] + pub fn state_of(&self, resolution: &Resolution) -> Option { + match resolution.state_change() { + StateChange::Replace(json) => serde_json::from_str(json).ok(), + StateChange::Keep | StateChange::Clear => None, + } + } + + /// The EIDs bidders would see for a result: the section 4.5 rules, then + /// the descriptor's provenance default for an entry that sets none of + /// `inserter`, `matcher` and `mm` (section 7.1). + #[must_use] + pub fn output_eids(&self, resolution: &Resolution) -> Vec { + let sets = entry_sets(resolution.eids(), self.setup.now()); + let mut order: Vec<&SourceDomain> = Vec::new(); + for eid in resolution.eids() { + if !order.contains(&eid.source()) { + order.push(eid.source()); + } + } + let descriptor = self.provider.descriptor(); + let mut output = Vec::new(); + for source in order { + let Some(set) = sets.sets.get(source) else { + continue; + }; + for entry in &set.entries { + let mut eid = Eid { + source: source.as_str().to_owned(), + uids: entry + .uids + .iter() + .map(|uid| Uid { + id: uid.id.clone(), + atype: uid.atype, + ext: uid.ext.clone(), + }) + .collect(), + inserter: entry.inserter.clone(), + matcher: entry.matcher.clone(), + mm: entry.mm, + ext: entry.ext.clone(), + }; + if let Some(default) = descriptor.provenance(source) + && eid.inserter.is_none() + && eid.matcher.is_none() + && eid.mm.is_none() + { + eid.inserter = default.inserter().map(str::to_owned); + eid.matcher = default.matcher().map(str::to_owned); + eid.mm = default.mm(); + } + output.push(eid); + } + } + output + } + + /// Runs only the cases passed in, never the harness's own inputs, and + /// fails the test on the first failed check. A provider declared + /// `with_private_state()` fails here with a pointer to + /// [`check_conformance_with_state`](Self::check_conformance_with_state). + /// + /// # Panics + /// + /// Panics when the provider does not conform. + pub fn check_conformance(&self, cases: &ConformanceCases) { + self.try_check_conformance(cases) + .expect("should conform: the provider failed a conformance check"); + } + + /// As [`check_conformance`](Self::check_conformance), for a provider with + /// private state of type `S`. Also checks that `S` round-trips through + /// `replacing_state` and `input.state()`. + /// + /// # Panics + /// + /// Panics when the provider does not conform. + pub fn check_conformance_with_state( + &self, + cases: &ConformanceCases, + ) { + self.try_check_conformance_with_state::(cases) + .expect("should conform: the provider failed a conformance check"); + } + + /// The same checks as [`check_conformance`](Self::check_conformance), + /// returning the first failure instead of failing the test. A panic + /// aborts a `wasm32-wasip1` test run, so the harness's own negative + /// tests use this. + /// + /// # Errors + /// + /// The first failed check. + pub fn try_check_conformance( + &self, + cases: &ConformanceCases, + ) -> Result<(), Report> { + self.conform(cases, None) + } + + /// As [`try_check_conformance`](Self::try_check_conformance), for a + /// provider with private state of type `S`. + /// + /// # Errors + /// + /// The first failed check. + pub fn try_check_conformance_with_state( + &self, + cases: &ConformanceCases, + ) -> Result<(), Report> { + self.conform( + cases, + Some(&|harness, json| harness.state_round_trips::(json)), + ) + } + + fn conform( + &self, + cases: &ConformanceCases, + round_trip: Option<&StateRoundTrip<'_, 'provider>>, + ) -> Result<(), Report> { + let descriptor = self.provider.descriptor(); + let stateful = descriptor + .resolves() + .is_some_and(Resolves::has_private_state); + if stateful && round_trip.is_none() { + return Err(Report::new(ConformanceFailure::StatefulNeedsStateType)); + } + + let mut requested = false; + for case in &cases.resolve { + let mut harness = Self::new(self.provider).with_setup(case.setup.clone()); + let request = harness.request(); + harness.check_recorded()?; + let request = match request { + Ok(Some(request)) => request, + Ok(None) => continue, + Err(error) => { + classify(error.current_context(), "request")?; + continue; + } + }; + requested = true; + check_request(descriptor, &request)?; + let headers: Vec<(&str, &str)> = case + .headers + .iter() + .map(|(name, value)| (name.as_str(), value.as_str())) + .collect(); + let resolution = harness.respond_raw(case.status, &headers, &case.body); + harness.check_recorded()?; + match resolution { + Ok(resolution) => { + if !stateful && resolution.changes_state() { + return Err(Report::new(ConformanceFailure::StatelessState)); + } + self.check_resolution(&resolution)?; + if let (StateChange::Replace(json), Some(round_trip)) = + (resolution.state_change(), round_trip) + && !round_trip(&harness, json) + { + return Err(Report::new(ConformanceFailure::StateRoundTrip)); + } + } + Err(error) => classify(error.current_context(), "response")?, + } + if stateful { + self.check_bare_string_state(case)?; + } + } + if descriptor.resolves().is_some() && !requested { + return Err(Report::new(ConformanceFailure::ResolverNeverRequests)); + } + + let mut captured = false; + for fixture in &cases.capture { + match self.capture(fixture) { + Ok(resolution) => { + if resolution.changes_state() { + return Err(Report::new(ConformanceFailure::CaptureStateChange)); + } + self.check_resolution(&resolution)?; + captured |= !resolution.is_no_match(); + } + Err(error) => classify(error.current_context(), "capture")?, + } + } + if descriptor.captures().is_some() && !captured { + return Err(Report::new(ConformanceFailure::CapturerNeverMatches)); + } + Ok(()) + } + + fn check_recorded(&self) -> Result<(), Report> { + match self.recorded.first() { + Some(access) => Err(Report::new(ConformanceFailure::UndeclaredInput { + access: *access, + })), + None => Ok(()), + } + } + + fn check_resolution(&self, resolution: &Resolution) -> Result<(), Report> { + let descriptor = self.provider.descriptor(); + for eid in resolution.eids() { + if !descriptor.sources().contains(eid.source()) { + return Err(Report::new(ConformanceFailure::UndeclaredSource { + source: eid.source().clone(), + })); + } + } + let sets = entry_sets(resolution.eids(), self.setup.now()); + if let Some(dropped) = sets.drops.first() { + return Err(Report::new(ConformanceFailure::LimitHit { + source: dropped.source.clone(), + part: dropped.part.to_string(), + })); + } + if let Some(source) = sets.oversized.first() { + return Err(Report::new(ConformanceFailure::SetTooLarge { + source: source.clone(), + })); + } + for (source, set) in &sets.sets { + if descriptor.is_expiring(source) && set.expires_at.is_none() { + return Err(Report::new(ConformanceFailure::MissingExpiry { + source: source.clone(), + })); + } + } + Ok(()) + } + + /// Injects a stored state that is a bare JSON string, which no struct + /// decodes, and fails when the provider decodes it as valid state. + fn check_bare_string_state( + &self, + case: &ResolveCase, + ) -> Result<(), Report> { + let mut setup = case.setup.clone(); + setup.state_json = Some("\"bare\"".to_owned()); + let mut probe = Self::new(self.provider).with_setup(setup); + if matches!(probe.request(), Ok(Some(_))) { + let _ignored = probe.respond_raw(case.status, &[], &case.body); + } + if probe.state_decoded { + return Err(Report::new(ConformanceFailure::BareStringStateAccepted)); + } + Ok(()) + } + + /// Whether `json` decodes as `S`, serializes back to the same value and + /// decodes again through `input.state()`. + fn state_round_trips(&self, json: &str) -> bool { + let Ok(state) = serde_json::from_str::(json) else { + return false; + }; + let Ok(again) = serde_json::to_string(&state) else { + return false; + }; + let same = serde_json::from_str::(json).ok() + == serde_json::from_str::(&again).ok(); + let mut setup = self.setup.clone(); + setup.state_json = Some(again); + let probe = Self::new(self.provider).with_setup(setup); + #[allow( + clippy::redundant_closure_for_method_calls, + reason = "the method reference does not satisfy the higher-ranked closure bound" + )] + let (decoded, _, _) = + probe.call(probe.kinds(), |input: &ResolveInput<'_>| input.state::()); + same && matches!(decoded, Ok(Some(_))) + } + + fn call( + &self, + kinds: BTreeMap, + run: impl FnOnce(&ResolveInput<'_>) -> R, + ) -> (R, Vec, bool) { + let signals = self.signals(); + let input = ResolveInput::new(self.provider.descriptor(), kinds, &signals) + .with_state_json(self.setup.state_json.clone()); + let result = run(&input); + (result, input.recorded_errors(), input.state_was_decoded()) + } + + fn consent_allows(&self) -> bool { + policy_allows( + self.provider.descriptor().consent_policy(), + &self.setup.consent.context(), + PolicySignals { + dnt: self.setup.consent.dnt, + }, + ) + } + + fn stored_by_source(&self) -> BTreeMap { + self.setup + .stored + .iter() + .map(|stored| { + let source = stored + .eids() + .first() + .and_then(|eid| eid.source.parse().ok()) + .unwrap_or_else(|| self.provider.descriptor().primary_source().clone()); + (source, stored.clone()) + }) + .collect() } fn available_inputs(&self) -> BTreeSet { let mut available = BTreeSet::new(); for (input, present) in [ - (Input::ClientIp, self.client_ip.is_some()), - (Input::UserAgent, self.user_agent.is_some()), - (Input::PageUrl, self.page_url.is_some()), - (Input::Region, self.region.is_some()), - (Input::HashedEmail, self.hashed_email.is_some()), + (Input::ClientIp, self.setup.client_ip.is_some()), + (Input::UserAgent, self.setup.user_agent.is_some()), + (Input::PageUrl, self.setup.page_url.is_some()), + (Input::Region, self.setup.region.is_some()), + (Input::HashedEmail, self.setup.hashed_email.is_some()), ] { if present { available.insert(input); @@ -912,11 +1644,12 @@ impl<'provider> ProviderHarness<'provider> { return BTreeMap::new(); }; let available = self.available_inputs(); + let stored = self.stored_by_source(); descriptor .sources() .iter() .filter_map(|source| { - let kind = if self.stored.contains_key(source) { + let kind = if stored.contains_key(source) { ResolveKind::Refresh } else { ResolveKind::Initial @@ -929,21 +1662,98 @@ impl<'provider> ProviderHarness<'provider> { fn signals(&self) -> ResolveSignals<'_> { ResolveSignals { - client_ip: self.client_ip, - user_agent: self.user_agent.as_deref(), - page_url: self.page_url.clone(), - region: self.region.clone(), - hashed_email: self.hashed_email.clone(), - consent: ProviderConsent::default(), + client_ip: self.setup.client_ip, + user_agent: self.setup.user_agent.as_deref(), + page_url: self.setup.page_url.clone(), + region: self.setup.region.clone(), + hashed_email: self.setup.hashed_email.clone(), + consent: ProviderConsent::from_context(&self.setup.consent.context()), publisher_domain: "www.example.com", - now: self.now, + now: self.setup.now(), random: &self.random, ec_id: self.ec_id.clone(), - stored: self.stored.clone(), + stored: self.stored_by_source(), + } + } +} + +/// Maps a provider error to the conformance check it breaks. Failures the +/// coordinator treats as a vendor outcome, such as a refusal or a body that +/// does not parse, are not bugs. +fn classify( + error: &IdentityProviderError, + stage: &'static str, +) -> Result<(), Report> { + match error { + IdentityProviderError::UndeclaredInput { access } => { + Err(Report::new(ConformanceFailure::UndeclaredInput { + access: *access, + })) + } + error if error.is_not_implemented() => { + Err(Report::new(ConformanceFailure::NotImplemented { + method: stage, + })) + } + IdentityProviderError::Config { .. } => { + Err(Report::new(ConformanceFailure::ProviderFailed { stage })) } + _ => Ok(()), } } +/// The request rules of the coordinator, each as its own failure. +fn check_request( + descriptor: &ProviderDescriptor, + request: &ProviderRequest, +) -> Result<(), Report> { + if request.url().scheme() != "https" { + return Err(Report::new(ConformanceFailure::InsecureScheme)); + } + let host = request.url().host_str().unwrap_or_default(); + let declared = descriptor.resolves().is_some_and(|resolves| { + resolves + .hosts() + .iter() + .any(|allowed| allowed.eq_ignore_ascii_case(host)) + }); + if !declared { + return Err(Report::new(ConformanceFailure::UndeclaredHost)); + } + if request.has_cookie_header() { + return Err(Report::new(ConformanceFailure::CookieHeader)); + } + if request + .headers() + .values() + .any(|value| value.to_str().is_err()) + { + return Err(Report::new(ConformanceFailure::InvalidHeader)); + } + Ok(()) +} + +/// Builds the descriptor from a config with placeholder secret names and +/// from one with resolved secrets, and fails the test when the two differ +/// (spec 10.2). +/// +/// # Panics +/// +/// Panics when either config cannot describe a provider, or when the +/// descriptors differ. +pub fn check_descriptor_ignores_secrets(placeholder: &C, resolved: &C) { + let from_placeholder = placeholder + .build_descriptor() + .expect("should build the descriptor from the placeholder config"); + let from_resolved = resolved + .build_descriptor() + .expect("should build the descriptor from the resolved config"); + assert_eq!( + from_placeholder, from_resolved, + "should build the same descriptor whatever the secrets" + ); +} + struct DescriptorOnly(ProviderDescriptor); impl IdentityProvider for DescriptorOnly { From 34d286ea0a80fa95d25770c07921ae1ee860cb20 Mon Sep 17 00:00:00 2001 From: dhruv8sh Date: Tue, 27 Oct 2026 04:44:14 +0530 Subject: [PATCH 135/194] Prove complete provider EIDs end to end through a registered provider Add MultiEntryTestProvider to the core test support, and request-level tests that register it through an integration registration and follow its two-entry result to KV, the mirror payload, the bidstream and identify, with refresh, oversized, rollback, capture and 404 cases. Run the Part 1 golden tests again with the provider registered. Fix the Fastly adapter test driver from Task 29: it drove the router with futures::executor::block_on, which panicked when the hold dispatch of an auction navigation blocked on its own futures. It now polls the future directly. No production code changed. Signed-off-by: dhruv8sh --- .../trusted-server-adapter-fastly/src/main.rs | 921 +++++++++++++++++- .../src/ec/identity/test_support.rs | 155 +++ 2 files changed, 1072 insertions(+), 4 deletions(-) diff --git a/crates/trusted-server-adapter-fastly/src/main.rs b/crates/trusted-server-adapter-fastly/src/main.rs index 1f9e5f98c3..4c6166faa4 100644 --- a/crates/trusted-server-adapter-fastly/src/main.rs +++ b/crates/trusted-server-adapter-fastly/src/main.rs @@ -1042,6 +1042,23 @@ mod tests { request } + /// Polls `future` to completion on this thread without entering an + /// executor, so code under test that blocks on its own futures, such as + /// the hold dispatch of an auction navigation, can run inside it. The + /// scripted network is ready at once, so a future that stays pending is a + /// test bug. + fn poll_ready(future: F) -> F::Output { + let waker = futures::task::noop_waker(); + let mut context = core::task::Context::from_waker(&waker); + let mut future = core::pin::pin!(future); + for _ in 0..10_000 { + if let core::task::Poll::Ready(output) = future.as_mut().poll(&mut context) { + return output; + } + } + unreachable!("the future should complete against the scripted network") + } + /// Routes `request`, then finalizes, sends and flushes it as /// `edgezero_main` does, keeping the sent response. fn drive_request_for_test( @@ -1060,8 +1077,7 @@ mod tests { prepare: impl FnOnce(&mut EcFinalizeState), ) -> (HttpResponse, Option) { let router = TrustedServerApp::routes_for_state(state); - let mut response = - futures::executor::block_on(router.oneshot(request)).expect("should route the request"); + let mut response = poll_ready(router.oneshot(request)).expect("should route the request"); let Some(mut ec_state) = response.extensions_mut().remove::() else { return (response, None); }; @@ -1761,15 +1777,20 @@ mod tests { use std::collections::BTreeMap; use std::sync::Mutex; + use trusted_server_core::ec::identity::IdentityProviderError; use trusted_server_core::ec::identity::test_support::{ - WriteCountingEcKv, allowing_consent, + MULTI_ENTRY_SOURCE, MultiEntryTestProvider, WriteCountingEcKv, allowing_consent, }; use trusted_server_core::ec::kv_backend::EcKvStore as _; - use trusted_server_core::ec::kv_types::{KvEntry, KvNetwork, KvPartnerId}; + use trusted_server_core::ec::kv_types::{ + KvEid, KvEntry, KvNetwork, KvOptOut, KvPartnerId, KvProviderState, KvUid, + }; + use trusted_server_core::integrations::IntegrationRegistration; use trusted_server_core::platform::{ GeoInfo, PlatformError, PlatformGeo, PlatformHttpClient, PlatformHttpRequest, PlatformPendingRequest, PlatformResponse, PlatformSelectResult, }; + use trusted_server_core::settings::{EcPartner, ProviderMode, ProviderSettings}; /// The host the publisher origin answers on. const ORIGIN_HOST: &str = "origin.test-publisher.com"; @@ -1841,6 +1862,14 @@ mod tests { ); } + /// Forgets the answer for `host`, so a call to it fails in transport. + fn clear_reply(&self, host: &str) { + self.replies + .lock() + .expect("should lock the replies") + .remove(host); + } + /// Every request that went to `host`, in order. fn requests_to(&self, host: &str) -> Vec { self.seen @@ -4367,5 +4396,889 @@ mod tests { ); assert_no_writes(&scenario.store, "/auction"); } + + // ---- A registered provider's complete set, end to end (Task 39b) ---- + + /// The provider `MultiEntryTestProvider` registers as. + const MULTI_PROVIDER: &str = "exampleid"; + + /// The host `MultiEntryTestProvider` calls. + const MULTI_HOST: &str = "api.exampleid.example.com"; + + /// The registration `exampleid` makes for `provider`. + fn multi_registration(provider: MultiEntryTestProvider) -> IntegrationRegistration { + IntegrationRegistration::builder("exampleid") + .try_with_identity_provider(Ok::<_, Report>(provider)) + .expect("should register the provider") + .build() + } + + /// A scenario with the auction, the `exampleid` source registered for + /// the bidstream, and `registrations` contributing providers. + /// `exampleid` owns [`MULTI_ENTRY_SOURCE`] and runs in `mode`. + fn registered_scenario( + registrations: Vec, + mode: ProviderMode, + ) -> Scenario { + let Scenario { + mut state, + store, + network, + } = Scenario::with_settings(&format!( + "{AUCTION_TOML}{}", + partner_toml(MULTI_ENTRY_SOURCE) + )); + let owned = Arc::get_mut(&mut state).expect("should own the fresh state"); + Arc::make_mut(&mut owned.settings) + .ec + .identity + .providers + .insert( + MULTI_PROVIDER.to_owned(), + ProviderSettings { + mode, + ..ProviderSettings::default() + }, + ); + let providers = registrations + .into_iter() + .flat_map(|registration| registration.identity_providers) + .collect(); + let runtime = IdentityRuntime::from_settings_with_providers( + &state.settings, + providers, + BTreeMap::from([( + MULTI_ENTRY_SOURCE.parse().expect("should parse the source"), + MULTI_PROVIDER + .parse() + .expect("should parse the provider ID"), + )]), + ) + .expect("should build the identity runtime"); + let scenario = Scenario { + state: crate::app::app_state_with_identity_for_test(state, runtime), + store, + network, + }; + scenario.network.respond( + BIDDER_HOST, + 200, + "application/json", + r#"{"id":"response","seatbid":[]}"#, + ); + answer_with(&scenario, "full"); + scenario + } + + /// A scenario whose one provider is `MultiEntryTestProvider`. + fn multi_scenario(provider: MultiEntryTestProvider, mode: ProviderMode) -> Scenario { + registered_scenario(vec![multi_registration(provider)], mode) + } + + /// Makes the provider's endpoint answer with the `entries` shape. + fn answer_with(scenario: &Scenario, entries: &str) { + scenario.network.respond( + MULTI_HOST, + 200, + "application/json", + &serde_json::json!({ "entries": entries }).to_string(), + ); + } + + fn kv_uid(id: &str, atype: Option, ext: Option) -> KvUid { + KvUid { + id: id.to_owned(), + atype, + ext, + } + } + + /// What the provider's `full` answer stores. + fn multi_full_set() -> Vec { + vec![ + KvEid { + uids: vec![ + kv_uid( + "uid-a", + Some(1), + Some(serde_json::json!({ "rtiPartner": "example" })), + ), + kv_uid("uid-b", None, None), + ], + inserter: Some("inserter-a.example.com".to_owned()), + matcher: Some("matcher-a.example.com".to_owned()), + mm: Some(2), + ext: None, + }, + KvEid { + uids: vec![kv_uid("uid-c", None, None)], + inserter: None, + matcher: None, + mm: None, + ext: Some(serde_json::json!({ "segments": ["example"] })), + }, + ] + } + + /// The first entry of the `full` answer, which the `first` answer + /// returns alone. + fn multi_first_set() -> Vec { + multi_full_set().into_iter().take(1).collect() + } + + /// The first entry as bidders see it. + fn multi_first_eid() -> serde_json::Value { + serde_json::json!({ + "source": MULTI_ENTRY_SOURCE, + "inserter": "inserter-a.example.com", + "matcher": "matcher-a.example.com", + "mm": 2, + "uids": [ + { "id": "uid-a", "atype": 1, "ext": { "rtiPartner": "example" } }, + { "id": "uid-b", "atype": EcPartner::default_openrtb_atype() } + ] + }) + } + + /// The second entry as bidders see it, with the provenance default. + fn multi_second_eid() -> serde_json::Value { + serde_json::json!({ + "source": MULTI_ENTRY_SOURCE, + "inserter": "default-inserter.example.com", + "matcher": "default-matcher.example.com", + "ext": { "segments": ["example"] }, + "uids": [{ "id": "uid-c", "atype": EcPartner::default_openrtb_atype() }] + }) + } + + /// A row holding `eids` for [`MULTI_ENTRY_SOURCE`], written by the + /// provider and due for a refresh. + fn multi_row(eids: Vec) -> KvEntry { + let mut entry = live_entry(); + entry.ids.insert( + MULTI_ENTRY_SOURCE.to_owned(), + KvPartnerId { + w: Some(MULTI_PROVIDER.to_owned()), + rv: Some("0123456789abcdef".to_owned()), + at: Some(1_000), + ra: Some(1_500), + eids, + ..KvPartnerId::default() + }, + ); + entry + } + + /// The record under [`MULTI_ENTRY_SOURCE`], with its fresh `rv` and `at` + /// checked and copied into the expectation of the caller. + fn fresh_record(row: &KvEntry) -> KvPartnerId { + let record = row + .ids + .get(MULTI_ENTRY_SOURCE) + .expect("should hold the provider's record") + .clone(); + let revision = record.rv.as_deref().expect("should carry a revision"); + assert!( + revision.len() == 16 + && revision + .bytes() + .all(|byte| byte.is_ascii_digit() || (b'a'..=b'f').contains(&byte)), + "should carry a 16 digit lowercase hex rv" + ); + assert_ne!( + revision, "0123456789abcdef", + "should carry a fresh rv, not the seeded one" + ); + assert!(record.at.is_some(), "should record when it was acquired"); + record + } + + #[test] + fn registered_provider_full_set_reaches_kv_mirror_and_bidstream_in_order() { + let scenario = multi_scenario(MultiEntryTestProvider::new(), ProviderMode::Hold); + scenario.seed_live(); + + let (_, outcome) = scenario.drive(navigation(Some(&ec_cookie()))); + + assert_eq!(outcome, Some(FlushOutcome::Written), "should flush"); + assert_eq!( + scenario.network.requests_to(MULTI_HOST).len(), + 1, + "should call the registered provider once" + ); + assert_writes(&scenario.store, (0, 1, 0), "one flush holds the whole set"); + let record = fresh_record(&scenario.row()); + assert_eq!( + record, + KvPartnerId { + w: Some(MULTI_PROVIDER.to_owned()), + rv: record.rv.clone(), + at: record.at, + exp: None, + ra: None, + ru: None, + eids: multi_full_set(), + }, + "should store both entries as returned, written by the provider" + ); + + scenario.store.reset_counts(); + let body = auction_body(Some(&serde_json::json!([browser_entry()]))); + let (response, _) = scenario.drive(auction_request(Some(&ec_cookie()), &body)); + + let expected = + serde_json::json!([browser_entry(), multi_first_eid(), multi_second_eid()]); + assert_eq!( + bidder_eids(&bidder_body(&scenario)), + expected, + "user.ext.eids should carry the browser entry, then both entries in order" + ); + assert_eq!( + header_eids(&response), + expected, + "x-ts-eids should carry the same" + ); + assert_eq!( + mirror_of(&json_of(response)), + serde_json::json!({ "v": 1, "eids": [multi_first_eid(), multi_second_eid()] }), + "the mirror should decode to the same two entries, defaults included" + ); + assert_eq!( + scenario.network.requests_to(MULTI_HOST).len(), + 1, + "control: the /auction should serve from KV without calling the provider" + ); + assert_no_writes(&scenario.store, "/auction"); + } + + #[test] + fn one_entry_one_uid_result_is_written_as_one_entry() { + let scenario = multi_scenario(MultiEntryTestProvider::new(), ProviderMode::Hold); + scenario.seed_live(); + answer_with(&scenario, "single"); + + scenario.drive(navigation(Some(&ec_cookie()))); + + assert_writes(&scenario.store, (0, 1, 0), "one flush"); + let record = fresh_record(&scenario.row()); + assert_eq!( + record, + KvPartnerId { + w: Some(MULTI_PROVIDER.to_owned()), + rv: record.rv.clone(), + at: record.at, + exp: None, + ra: None, + ru: None, + eids: vec![KvEid { + uids: vec![kv_uid( + "uid-a", + Some(1), + Some(serde_json::json!({ "rtiPartner": "example" })) + )], + inserter: None, + matcher: None, + mm: None, + ext: None, + }], + }, + "should store one entry holding one UID" + ); + let stored = stored_body(&scenario); + assert_eq!( + stored["ids"][MULTI_ENTRY_SOURCE], + serde_json::json!({ + "w": MULTI_PROVIDER, + "rv": record.rv, + "at": record.at, + "eids": [{ + "uids": [{ + "id": "uid-a", + "atype": 1, + "ext": { "rtiPartner": "example" } + }] + }] + }), + "should write the one shape of spec 6.1" + ); + } + + #[test] + fn later_result_without_an_entry_removes_it() { + let scenario = multi_scenario(MultiEntryTestProvider::new(), ProviderMode::Hold); + let mut row = multi_row(multi_full_set()); + row.ids.insert( + "id5-sync.com".to_owned(), + KvPartnerId::browser("id5-uid", Some(1_500)), + ); + scenario.seed(&row); + answer_with(&scenario, "first"); + + let (_, outcome) = scenario.drive(navigation(Some(&ec_cookie()))); + + assert_eq!(outcome, Some(FlushOutcome::Written), "should flush"); + assert_eq!( + scenario.network.requests_to(MULTI_HOST).len(), + 1, + "control: the due record should have been refreshed" + ); + let after = scenario.row(); + let record = fresh_record(&after); + assert_eq!( + record, + KvPartnerId { + w: Some(MULTI_PROVIDER.to_owned()), + rv: record.rv.clone(), + at: record.at, + exp: None, + ra: None, + ru: None, + eids: multi_first_set(), + }, + "should leave exactly the returned entry in KV" + ); + assert_eq!( + after.ids.get("id5-sync.com"), + row.ids.get("id5-sync.com"), + "should keep the record of a source the result does not name" + ); + + let (_, _) = scenario.drive(auction_request(Some(&ec_cookie()), &auction_body(None))); + assert_eq!( + bidder_eids(&bidder_body(&scenario)), + serde_json::json!([ + multi_first_eid(), + { "source": "id5-sync.com", "uids": [{ "id": "id5-uid", "atype": 3 }] } + ]), + "should serve exactly the remaining entry, and the other source's record" + ); + } + + #[test] + fn stateful_result_with_an_oversized_set_keeps_only_its_removals() { + let scenario = multi_scenario(MultiEntryTestProvider::stateful(), ProviderMode::Hold); + let mut row = multi_row(multi_first_set()); + row.ps.insert( + MULTI_PROVIDER.to_owned(), + KvProviderState { + st: Some(serde_json::json!({ "seen": false })), + ..KvProviderState::default() + }, + ); + scenario.seed(&row); + answer_with(&scenario, "oversized"); + + scenario.drive(navigation(Some(&ec_cookie()))); + + assert_eq!( + scenario.network.requests_to(MULTI_HOST).len(), + 1, + "control: the provider should have been called" + ); + let after = scenario.row(); + assert!( + !after.ids.contains_key(MULTI_ENTRY_SOURCE), + "should apply the also_invalidate removal and store none of the oversized set" + ); + let state = after + .ps + .get(MULTI_PROVIDER) + .expect("should keep the provider's row state"); + assert!( + state.n.is_some_and(|wait_until| wait_until > 1_700_000_000), + "should back off the rejected answer" + ); + assert_eq!( + state, + &KvProviderState { + n: state.n, + f: 1, + o: None, + oc: None, + iv: state.iv.clone(), + ov: Vec::new(), + st: Some(serde_json::json!({ "seen": false })), + }, + "should keep the old state, not the rejected replacement, and count one failure" + ); + assert_eq!( + state.iv.len(), + 2, + "should record the two invalidated UIDs, so the removal is not undone" + ); + } + + #[test] + fn identify_returns_the_first_uid_of_a_full_record() { + let scenario = multi_scenario(MultiEntryTestProvider::new(), ProviderMode::Hold); + let mut row = multi_row(multi_full_set()); + row.ids + .get_mut(MULTI_ENTRY_SOURCE) + .expect("should hold the record") + .ra = None; + row.ids.insert( + "id5-sync.com".to_owned(), + KvPartnerId::browser("id5-uid", Some(1_500)), + ); + scenario.seed(&row); + let atype = EcPartner::default_openrtb_atype(); + + let full = json_of( + scenario + .drive(identify_request(&ec_cookie(), MULTI_ENTRY_SOURCE)) + .0, + ); + let one_uid = json_of( + scenario + .drive(identify_request(&ec_cookie(), "id5-sync.com")) + .0, + ); + + assert_eq!(full["uid"], "uid-a", "should return the first UID"); + assert_eq!( + full["eid"], + serde_json::json!({ + "source": MULTI_ENTRY_SOURCE, + "uids": [{ "id": "uid-a", "atype": atype }] + }), + "should return the first UID with the partner's openrtb_atype" + ); + assert_eq!( + one_uid["eid"], + serde_json::json!({ + "source": "id5-sync.com", + "uids": [{ "id": "id5-uid", "atype": atype }] + }), + "control: a one-UID record should have the same shape" + ); + let keys = |body: &serde_json::Value| { + body.as_object() + .expect("should be an object") + .keys() + .cloned() + .collect::>() + }; + assert_eq!( + keys(&full), + keys(&one_uid), + "should answer in the same shape" + ); + } + + #[test] + fn rollback_to_part_1_config_keeps_the_owners_opt_out() { + let scenario = Scenario::with_settings(&format!( + "{AUCTION_TOML}{}", + partner_toml(MULTI_ENTRY_SOURCE) + )); + scenario.network.respond( + BIDDER_HOST, + 200, + "application/json", + r#"{"id":"response","seatbid":[]}"#, + ); + let mut row = multi_row(multi_full_set()); + row.ps.insert( + MULTI_PROVIDER.to_owned(), + KvProviderState { + n: Some(FAR_FUTURE), + f: 2, + o: Some(KvOptOut { + at: 1_200, + fp: "0".to_owned(), + }), + oc: Some(1_100), + iv: vec!["00112233445566ff".to_owned()], + ov: vec!["8899aabbccddeeff".to_owned()], + st: Some(serde_json::json!({ "seen": true })), + }, + ); + scenario.seed(&row); + + let (_, outcome) = scenario.drive(navigation(Some(&returning_with_shared_id()))); + + assert_eq!(outcome, Some(FlushOutcome::Written), "should flush"); + let after = scenario.row(); + assert_eq!( + after.ps.get(MULTI_PROVIDER), + Some(&KvProviderState { + n: None, + f: 0, + o: Some(KvOptOut { + at: 1_200, + fp: "0".to_owned(), + }), + oc: Some(1_100), + iv: Vec::new(), + ov: vec!["8899aabbccddeeff".to_owned()], + st: None, + }), + "should keep o, oc and ov and lose n, f, iv and st" + ); + assert_eq!( + after.ids.get(MULTI_ENTRY_SOURCE), + row.ids.get(MULTI_ENTRY_SOURCE), + "should keep the multi-entry record readable" + ); + assert_eq!( + stored_uid(&after, "sharedid.org").as_deref(), + Some(SHARED_ID), + "control: the flush should have written the ingest" + ); + + scenario.drive(auction_request(Some(&ec_cookie()), &auction_body(None))); + let sent = bidder_eids(&bidder_body(&scenario)); + assert!( + !sent.to_string().contains(MULTI_ENTRY_SOURCE), + "should not serve a record whose provider the catalog no longer has: {sent}" + ); + assert!( + sent.to_string().contains("sharedid.org"), + "control: should still serve the browser record: {sent}" + ); + } + + /// A scenario whose Didomi proxy answers `body`, with a capture-only + /// provider registered through an `IntegrationRegistration`, and a + /// row holding a browser-written `ssp.example.com` ID. + fn registered_capture_scenario(body: &str, global: bool) -> Scenario { + let Scenario { + mut state, + store, + network, + } = Scenario::with_settings(DIDOMI_TOML); + if global { + let owned = Arc::get_mut(&mut state).expect("should own the fresh state"); + Arc::make_mut(&mut owned.settings) + .ec + .identity + .providers + .insert( + "didomi".to_owned(), + ProviderSettings { + revoke_scope: Some(trusted_server_core::settings::RevokeScope::Global), + ..ProviderSettings::default() + }, + ); + } + let registration = IntegrationRegistration::builder("didomi") + .with_identity_provider( + trusted_server_core::ec::identity::test_support::answering_capture_provider( + "didomi", + &["ssp.example.com"], + None, + ), + ) + .build(); + let runtime = IdentityRuntime::from_settings_with_providers( + &state.settings, + registration.identity_providers, + BTreeMap::new(), + ) + .expect("should build the identity runtime"); + let scenario = Scenario { + state: crate::app::app_state_with_identity_for_test(state, runtime), + store, + network, + }; + scenario + .network + .respond("sdk.example.com", 200, "application/json", body); + let mut entry = live_entry(); + entry.ids.insert( + "ssp.example.com".to_owned(), + KvPartnerId { + rv: Some("0123456789abcdef".to_owned()), + at: Some(1_000), + ..KvPartnerId::new("probe-uid") + }, + ); + scenario.seed(&entry); + scenario + } + + #[test] + fn captured_revocation_writes_with_cas() { + let clean = registered_capture_scenario(CAPTURE_INVALIDATION, false); + clean.drive(didomi_navigation(Some(&returning_with_shared_id()))); + assert_writes( + &clean.store, + (0, 2, 0), + "one CAS write for the removal before send, then the writer's flush", + ); + assert_eq!( + stored_uid(&clean.row(), "ssp.example.com"), + None, + "should remove the captured ID" + ); + + let fifth = registered_capture_scenario(CAPTURE_INVALIDATION, false); + fifth.store.refuse_conditional_writes(EC_ID, 4); + fifth.drive(didomi_navigation(Some(&returning_with_shared_id()))); + assert_writes( + &fifth.store, + (0, 6, 0), + "four refused attempts and the fifth before send, no retry after send, the flush", + ); + assert_eq!( + stored_uid(&fifth.row(), "ssp.example.com"), + None, + "should remove the ID on the fifth attempt" + ); + + let retried = registered_capture_scenario(CAPTURE_INVALIDATION, false); + retried.store.refuse_conditional_writes(EC_ID, 5); + retried.drive(didomi_navigation(Some(&returning_with_shared_id()))); + assert_writes( + &retried.store, + (0, 7, 0), + "five refused attempts before send, one more after send, the flush", + ); + assert_eq!( + stored_uid(&retried.row(), "ssp.example.com"), + None, + "should remove the ID on the attempt after send" + ); + } + + #[test] + fn global_revoke_from_real_browser_writes_as_withdrawal() { + let withdrawn = Scenario::new(); + withdrawn.seed_live(); + drive_with_runtime( + &withdrawn.state, + navigation(Some(&ec_cookie())), + Some(&withdrawn.state.identity), + withdraw, + ); + let withdrawal_writes = withdrawn.store.writes(); + + let scenario = registered_capture_scenario(r#"{"opted_out":true}"#, true); + + let (response, _) = + scenario.drive(didomi_navigation(Some(&returning_with_shared_id()))); + + let row = scenario.row(); + assert!(!row.consent.ok, "should tombstone the row"); + assert!(row.ids.is_empty(), "should remove every ID"); + assert_eq!( + scenario.row().consent.ok, + withdrawn.row().consent.ok, + "should end as a consent withdrawal does" + ); + let writes = scenario.store.writes(); + assert_eq!( + (writes.add, writes.if_generation_match, writes.overwrite), + ( + withdrawal_writes.add, + withdrawal_writes.if_generation_match, + withdrawal_writes.overwrite + ), + "should write the tombstone and its marker as a withdrawal does" + ); + assert!( + set_cookies(&response).iter().any(|cookie| { + cookie_name(cookie) == "ts-ec" && cookie.contains("Max-Age=0") + }), + "should expire the EC cookie" + ); + } + + #[test] + fn non_browser_capture_writes_nothing() { + for body in [CAPTURE_INVALIDATION, r#"{"opted_out":true}"#] { + let scenario = registered_capture_scenario(body, true); + let before = scenario.row(); + let mut request = didomi_navigation(Some(&returning_with_shared_id())); + request.extensions_mut().clear(); + request.headers_mut().remove("sec-fetch-dest"); + request + .headers_mut() + .insert("user-agent", HeaderValue::from_static("curl/8.0")); + + let (response, _) = scenario.drive(request); + + assert_eq!(response.status(), 200, "should serve the proxied answer"); + assert_no_writes( + &scenario.store, + "a capture from a client that is no browser", + ); + assert_eq!( + scenario.row(), + before, + "should leave the row as it was, whatever the body asks: {body}" + ); + assert!( + !set_cookies(&response).iter().any( + |cookie| cookie_name(cookie) == "ts-ec" && cookie.contains("Max-Age=0") + ), + "should not expire the EC cookie" + ); + } + } + + #[test] + fn returning_navigation_answered_404_flushes_only_hold_removals_and_backoff() { + let removal_row = || multi_row(multi_first_set()); + let drive_home = |scenario: &Scenario, status: u16| { + scenario.network.respond( + ORIGIN_HOST, + status, + "text/html", + "page", + ); + let mut request = navigation(Some(&ec_cookie())); + *request.uri_mut() = "https://test-publisher.com/" + .parse() + .expect("should parse the home page URI"); + scenario.drive(request) + }; + + let ok = multi_scenario(MultiEntryTestProvider::stateful(), ProviderMode::Hold); + ok.seed(&removal_row()); + answer_with(&ok, "oversized"); + drive_home(&ok, 200); + assert!( + !ok.row().ids.contains_key(MULTI_ENTRY_SOURCE), + "control: a 200 should flush the removal" + ); + + let not_found = multi_scenario(MultiEntryTestProvider::stateful(), ProviderMode::Hold); + not_found.seed(&removal_row()); + answer_with(¬_found, "oversized"); + drive_home(¬_found, 404); + + assert_eq!( + not_found.network.requests_to(MULTI_HOST).len(), + 1, + "should still have run the hold call" + ); + let after = not_found.row(); + assert!( + !after.ids.contains_key(MULTI_ENTRY_SOURCE), + "should flush the hold answer's removal on a 404" + ); + assert_eq!( + after.ps.get(MULTI_PROVIDER).map(|state| state.iv.len()), + Some(2), + "should flush the removal's record of the two invalidated UIDs" + ); + + let backoff = multi_scenario(MultiEntryTestProvider::new(), ProviderMode::Hold); + backoff.seed_live(); + answer_with(&backoff, "none"); + drive_home(&backoff, 404); + assert!( + backoff + .row() + .ps + .get(MULTI_PROVIDER) + .and_then(|state| state.n) + .is_some_and(|wait_until| wait_until > 1_700_000_000), + "should flush the no_match backoff on a 404" + ); + + let positive = multi_scenario(MultiEntryTestProvider::new(), ProviderMode::Hold); + positive.seed_live(); + answer_with(&positive, "full"); + drive_home(&positive, 404); + assert!( + !positive.row().ids.contains_key(MULTI_ENTRY_SOURCE), + "should store no positive hold answer on a 404" + ); + } + + /// A scenario like [`Scenario::new`] with `MultiEntryTestProvider` + /// registered on its own source, and its endpoint left unanswered so + /// the provider writes nothing the golden tests would see. + fn golden_registered_scenario() -> Scenario { + let scenario = registered_scenario( + vec![multi_registration(MultiEntryTestProvider::new())], + ProviderMode::AfterSend, + ); + scenario.network.clear_reply(MULTI_HOST); + scenario + } + + #[test] + fn golden_browser_ingest_with_a_registered_provider_writes_the_first_uid() { + let cookie = format!("ts-eids={GOLDEN_INGEST_COOKIE}; sharedId=shared-1"); + + let returning = golden_registered_scenario(); + returning.seed_live(); + let (_, outcome) = + returning.drive(navigation(Some(&format!("ts-ec={EC_ID}; {cookie}")))); + assert_eq!(outcome, Some(FlushOutcome::Written), "should flush"); + assert_writes(&returning.store, (0, 1, 0), "one flush"); + assert_eq!( + assert_one_uid_records(&returning.row()), + golden_expected_ingest(), + "the flushed row should hold the UIDs the code before this spec stored" + ); + + let new_visitor = golden_registered_scenario(); + let (response, _) = new_visitor.drive(navigation(Some(&cookie))); + let minted = set_cookie_value(&response, "ts-ec").expect("should mint an EC"); + assert_writes(&new_visitor.store, (1, 0, 0), "one create"); + assert_eq!( + assert_one_uid_records(&new_visitor.row_of(&minted)), + golden_expected_ingest(), + "the created row should hold the same UIDs" + ); + } + + #[test] + fn golden_one_uid_row_with_a_registered_provider_goes_out_unchanged() { + let scenario = golden_registered_scenario(); + let mut entry = live_entry(); + for (source, uid) in [("id5-sync.com", "id5-uid"), ("sharedid.org", "shared-uid")] { + entry + .ids + .insert(source.to_owned(), KvPartnerId::browser(uid, Some(1_500))); + } + scenario.seed(&entry); + + let (response, _) = + scenario.drive(auction_request(Some(&ec_cookie()), &auction_body(None))); + + assert_eq!( + bidder_eids(&bidder_body(&scenario)), + serde_json::json!([ + { "source": "id5-sync.com", "uids": [{ "id": "id5-uid", "atype": 3 }] }, + { "source": "sharedid.org", "uids": [{ "id": "shared-uid", "atype": 3 }] } + ]), + "user.ext.eids should equal what the code before this spec sent" + ); + assert_eq!( + response + .headers() + .get("x-ts-eids") + .and_then(|value| value.to_str().ok()), + Some(GOLDEN_ROW_HEADER), + "x-ts-eids should equal what the code before this spec built, byte for byte" + ); + } + + #[test] + fn golden_auction_body_forwarding_with_a_registered_provider_is_unchanged() { + let scenario = golden_registered_scenario(); + scenario.seed_live(); + let eids: serde_json::Value = + serde_json::from_str(GOLDEN_BODY_EIDS).expect("should parse the body EIDs"); + let forwarded: serde_json::Value = serde_json::from_str(GOLDEN_BODY_FORWARDED) + .expect("should parse the forwarded EIDs"); + + scenario.drive(auction_request( + Some(&ec_cookie()), + &auction_body(Some(&eids)), + )); + + assert_eq!( + bidder_eids(&bidder_body(&scenario)), + forwarded, + "user.ext.eids should equal what the code before this spec forwarded" + ); + assert_no_writes(&scenario.store, "/auction"); + } } } diff --git a/crates/trusted-server-core/src/ec/identity/test_support.rs b/crates/trusted-server-core/src/ec/identity/test_support.rs index ef1f1669f1..3b1bbe5553 100644 --- a/crates/trusted-server-core/src/ec/identity/test_support.rs +++ b/crates/trusted-server-core/src/ec/identity/test_support.rs @@ -307,6 +307,161 @@ pub fn provider_from(descriptor: ProviderDescriptor) -> Arc Self { + Self::build(false) + } + + /// The same provider, declared `with_private_state()`. + /// + /// # Panics + /// + /// Panics when the fixed descriptor does not build. + #[must_use] + pub fn stateful() -> Self { + Self::build(true) + } + + fn build(stateful: bool) -> Self { + let mut resolves = Resolves::new().with_endpoint(MULTI_ENTRY_ENDPOINT); + if stateful { + resolves = resolves.with_private_state(); + } + let descriptor = + ProviderDescriptor::builder("exampleid".parse().expect("should parse the provider ID")) + .with_source(source(MULTI_ENTRY_SOURCE)) + .with_resolves(resolves) + .with_consent_policy(ConsentPolicy::strict()) + .with_provenance( + &source(MULTI_ENTRY_SOURCE), + EidProvenance::new() + .with_inserter("default-inserter.example.com") + .with_matcher("default-matcher.example.com"), + ) + .build() + .expect("should build the descriptor"); + Self { + descriptor, + stateful, + } + } + + fn first_entry(source: &SourceDomain) -> ResolvedEid { + ResolvedEid::new( + source, + ResolvedUid::new("uid-a") + .with_atype(1) + .with_ext(json!({ "rtiPartner": "example" })), + ) + .with_uid(ResolvedUid::new("uid-b")) + .with_inserter("inserter-a.example.com") + .with_matcher("matcher-a.example.com") + .with_mm(2) + } +} + +impl Default for MultiEntryTestProvider { + fn default() -> Self { + Self::new() + } +} + +impl IdentityProvider for MultiEntryTestProvider { + fn descriptor(&self) -> &ProviderDescriptor { + &self.descriptor + } + + fn request( + &self, + input: &ResolveInput<'_>, + ) -> Result, Report> { + Ok(Some(ProviderRequest::get(input.endpoint()?))) + } + + fn response( + &self, + response: &ProviderResponse, + _input: &ResolveInput<'_>, + ) -> Result> { + let body: serde_json::Value = response.success_json()?; + let source = source(MULTI_ENTRY_SOURCE); + let resolution = match body["entries"].as_str() { + Some("none") => Resolution::no_match(), + Some("first") => Resolution::found([Self::first_entry(&source)]), + Some("single") => Resolution::found([ResolvedEid::new( + &source, + ResolvedUid::new("uid-a") + .with_atype(1) + .with_ext(json!({ "rtiPartner": "example" })), + )]), + Some("oversized") => { + let entries = (0..8).map(|index| { + ResolvedEid::new( + &source, + ResolvedUid::new(format!("uid-{index}")) + .with_ext(json!({ "pad": "x".repeat(1_000) })), + ) + }); + return Ok(self.with_state(Resolution::found(entries).also_invalidate(&source))); + } + _ => Resolution::found([ + Self::first_entry(&source), + ResolvedEid::new(&source, ResolvedUid::new("uid-c")) + .with_ext(json!({ "segments": ["example"] })), + ]), + }; + Ok(self.with_state(resolution)) + } +} + +impl MultiEntryTestProvider { + fn with_state(&self, resolution: Resolution) -> Resolution { + if self.stateful { + resolution.replacing_state(&json!({ "seen": true })) + } else { + resolution + } + } +} + /// A `fixed_provider` declared `with_private_state()`. /// /// # Panics From 38f7728b9b38ae04bc3f57e8d51896e2008028f8 Mon Sep 17 00:00:00 2001 From: dhruv8sh Date: Tue, 27 Oct 2026 04:03:27 +0530 Subject: [PATCH 136/194] Document the identity provider framework for operators and authors Add the provider author guide, the owner and per-provider configuration keys with their validation errors, the capture provider checklist, revoke scope, complete provider EIDs and rollback steps, the commented capture keys in the example config, and the changelog entries. Signed-off-by: dhruv8sh --- CHANGELOG.md | 9 + docs/.vitepress/config.mts | 4 + docs/guide/configuration.md | 59 +++++- docs/guide/edge-cookies.md | 65 +++++++ docs/guide/identity-providers.md | 310 +++++++++++++++++++++++++++++++ trusted-server.example.toml | 22 ++- 6 files changed, 460 insertions(+), 9 deletions(-) create mode 100644 docs/guide/identity-providers.md diff --git a/CHANGELOG.md b/CHANGELOG.md index 00af3f5699..098beb0b8c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Changed +- Consent decoding for identity providers is stricter: a raw consent string that cannot be decoded blocks a provider whose policy is strict, and the GPP sharing, targeted-advertising and GPC fields of the US sections now block it too. A header-only GPP string with no applicable section does not block where TCF does not apply. +- The Lockr proxy now logs only the path of the upstream URL, never its query string. - Identity KV is written at most once per EC-bearing request. Only a full page navigation from a returning visitor (the writer request) writes KV, once, after the response is sent: browser EIDs, pull partner results and the cluster size all go in that one write, with a generation check and no retry of the whole write. Every other route reads KV and stages but writes nothing. A consent-withdrawal tombstone and the removal-only retry of a failed write are the only exceptions. Mid-branch builds drop browser ingest until the whole change ships. - `POST /auction`, `GET /_ts/page-bids` and `GET /_ts/api/v1/identify` no longer write KV, and `/_ts/api/v1/identify` omits expired IDs. The `/identify` response shape is unchanged: one UID per source. - Pull partners now run from a visitor's second page load, from the writer request after send, and back off on errors. They are paced by `[ec.identity.providers.pull-]` (`refresh_after_sec`, `max_calls_per_minute`), so the old default of 10 calls per hour becomes an effective 60 per hour. That 60 per hour is a ceiling, not a rate: `refresh_after_sec` and error backoff gate each partner first. Runtime startup now builds each pull partner's descriptor: a partner whose allowed host fails the descriptor host check stops startup. See [Identity Providers](docs/guide/configuration.md#identity-providers-ec-identity). @@ -53,6 +55,13 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Added +- Server-side identity provider framework. An integration registers an `IdentityProvider` with `try_with_identity_provider`, and the coordinator handles consent, transport limits, backoff and the single KV write. Provider authors get a test harness (`ProviderHarness`, `check_conformance`, `output_eids`). See [Writing an Identity Provider](docs/guide/identity-providers.md). No vendor ships with the framework. +- Source ownership: `owner` on an `[[ec.partners]]` entry names the provider that owns a source. An owner's result replaces the source's whole entry set, and batch sync never replaces an owned value. Validation rejects an owner that is not enabled, is capture-only, does not claim the source, or collides with another provider. See [Source ownership](docs/guide/configuration.md#source-ownership-ec-partners-owner). +- Hold mode: `mode = "hold"` in `[ec.identity.providers.]` runs a provider before the navigation's auction, within the shared `hold_budget_ms`, with `timeout_ms` and `after_send_timeout_ms` as call timeouts. +- Revoke-only passive capture for providers that proxy a vendor's API: a captured removal or opt-out is written to KV, a positive result clears an opt-out, and positive IDs are only counted. `revoke_scope` (`provider` by default, or `global` to run the consent-withdrawal tombstone) sets how far an opt-out reaches. Only a real browser's capture writes, and a capture-only provider must list its `sources`. See the [capture provider checklist](docs/guide/edge-cookies.md#capture-provider-checklist). +- Complete provider EIDs: a provider can return several entries and UIDs per source, with UID-level `atype` and `ext` and EID-level `inserter`, `matcher`, `mm` and `ext`. KV, the bidstream, `x-ts-eids` and the `ts-eids-s` mirror carry the whole set in order; `/identify` still returns one UID per source. See [Complete provider EIDs](docs/guide/edge-cookies.md#complete-provider-eids). +- The `capture_unpersisted` and `removal_lost` counters in the `identity request measurement: ...` log line, and a per-provider circuit breaker (`breaker_failures_per_minute`) that skips a provider for every visitor after repeated transport failures, timeouts, 429s or 5xx responses. + - Added the `[auction].rewrite_creatives` (default `true`) and `[auction].sanitize_creatives` (default `false`) options. `rewrite_creatives` rewrites winning-bid adm to first-party endpoints across `POST /auction` and publisher SSAT/page-bids delivery (proxy/click URL conversion, bidder `` removal; creative TSJS injection on `POST /auction` only). Enabling `sanitize_creatives` strips executable markup from winning-bid adm before delivery. - `creative_opportunities.slot.gam_unit_path` is now a template supporting `{network_id}`, `{slot_id}`, and `{section}`, so a publisher whose ad unit varies by site section expresses it in one slot rule instead of one per (slot × section). `{section}` derives from the request path: `[creative_opportunities].section_segment` selects which path segment names the section (0-based, default `0`; set `1` for locale-prefixed URLs), and `section_root` supplies the value for paths with no such segment. `section_root` is required when a template uses `{section}`. Existing static and absent `gam_unit_path` configs are unchanged. Startup rejects a blank `gam_network_id` only when an absent/default path or `{network_id}` template consumes it. Trusted Server conservatively caps whole rendered dynamic paths at 100 UTF-8 bytes, informed by Google's 100-character per-ad-unit-code limit; an over-limit request-specific path omits that slot without failing the response. During typed/startup finalization, every placeholder-bearing template that omits `section_segment` materializes `section_segment = 0`, so an older binary rejects the blob loudly. Static and absent paths remain legacy-schema compatible only when both `section_root` and `section_segment` are omitted. Before rolling back below this feature, replace or remove dynamic paths, remove both keys, re-push and finalize the config, then roll back the binary. - Added opt-in APS HTTP debug metadata for controlled test sites, exposing the direct request and response under `/auction` provider metadata using the Prebid Server `debug.httpcalls` shape. diff --git a/docs/.vitepress/config.mts b/docs/.vitepress/config.mts index 4220851797..d4a41b2800 100644 --- a/docs/.vitepress/config.mts +++ b/docs/.vitepress/config.mts @@ -111,6 +111,10 @@ export default withMermaid( items: [ { text: 'Edge Cookies', link: '/guide/edge-cookies' }, { text: 'EC Setup', link: '/guide/ec-setup-guide' }, + { + text: 'Identity Providers', + link: '/guide/identity-providers', + }, { text: 'GDPR Compliance', link: '/guide/gdpr-compliance' }, { text: 'Ad Serving', link: '/guide/ad-serving' }, { diff --git a/docs/guide/configuration.md b/docs/guide/configuration.md index e141f1d133..51409e2d7c 100644 --- a/docs/guide/configuration.md +++ b/docs/guide/configuration.md @@ -645,7 +645,9 @@ Policy for server-side identity providers. The table is optional; every key has | `hold_retry_sec` | Integer | `300` | Seconds to wait before calling a provider again after the hold budget cut its call off | | `max_response_bytes` | Integer | `65536` | Largest provider response body accepted, before and after decoding | -Each provider can have an `[ec.identity.providers.]` table. A provider with no table uses the defaults. +#### Per-provider keys + +Each provider can have an `[ec.identity.providers.]` table. A provider with no table uses the defaults. The table name is the provider ID: the integration ID of a vendor provider, or the generated `pull-` ID of a pull partner. | Key | Type | Default | Meaning | | ----------------------------- | ------- | ---------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- | @@ -657,7 +659,54 @@ Each provider can have an `[ec.identity.providers.]` table. A provider with | `max_calls_per_minute` | Integer | `1` | Per-visitor burst cap, enforced in 60-second windows | | `breaker_failures_per_minute` | Integer | `50` | Transport failures, timeouts, 429s and 5xx responses in one 60-second window after which the provider is skipped for every visitor | | `cookie_priority` | Integer | `0` | Lower values are left out of the `__Host-ts-eids-s` mirror first when it exceeds its size cap | -| `revoke_scope` | String | `provider` (capture only) | `provider` or `global`. Applies only to a provider that captures IDs, from Part 2 of the identity rollout. | +| `revoke_scope` | String | `provider` (capture only) | `provider` or `global`. Applies only to a provider that captures IDs; setting it on any other provider is a validation error. See below. | + +**`mode`, `timeout_ms` and `after_send_timeout_ms`.** A provider runs only on a returning visitor's full page navigation. In `hold` mode the call runs before the navigation's auction, within the shared `hold_budget_ms`, and its answer joins that auction. A call the budget cuts off is not a failure: the provider waits `hold_retry_sec` before the next call. When the navigation runs no auction, a hold-mode provider runs after send instead and waits `after_send_timeout_ms`. In `after_send` mode (the default) the provider always runs after the response is sent. + +**`sources`.** Narrows the source domains the provider may write, before ownership is checked. It must name only sources the provider declares and keep at least one. A capture-only provider has no usable default and must list its `sources`: its opt-out removes browser-ingested records of every source it lists, so list only the sources it is the delivery path for. + +**`refresh_after_sec`, `max_calls_per_minute`, `breaker_failures_per_minute`.** `refresh_after_sec` is how long a stored ID lives before it is refreshed when the provider gives no hint, and how long to wait after a skipped call. `max_calls_per_minute` caps bursts per visitor in 60-second windows. `breaker_failures_per_minute` counts transport failures, timeouts, 429s and 5xx responses across all visitors at one edge location. Past it the provider is skipped for everyone until the window clears. Input-caused 4xx responses and unparseable replies back off only that visitor and never open the breaker. + +**`cookie_priority`.** When the `__Host-ts-eids-s` mirror exceeds its size cap, whole sources are left out, lowest `cookie_priority` first. + +**`revoke_scope`.** For a provider that captures IDs from proxied responses. With `provider`, a captured opt-out removes that provider's records and, for a capture-only provider, the records of its listed sources that no provider wrote. With `global`, a captured opt-out from a real browser runs the consent-withdrawal tombstone: the row is tombstoned, the `ts-ec` cookie is cleared, and everything is removed, provider opt-outs included. A client that is not a real browser never changes the row either way. + +#### Source ownership (`[[ec.partners]].owner`) + +`owner` is an optional key on an `[[ec.partners]]` entry. It names the identity provider that owns that source. An owner refreshes the source and wins over browser and other writers: its result replaces the source's whole entry set, and batch sync never replaces its value. A source with no `owner` is unowned, and a provider can only fill it when absent or expired. + +```toml +[[ec.partners]] +name = "Example ID" +source_domain = "exampleid.example.com" +openrtb_atype = 1 +bidstream_enabled = true +owner = "exampleid" + +[ec.identity.providers.exampleid] +mode = "hold" +timeout_ms = 80 +``` + +**Validation.** Each of these stops startup and fails `ts config validate` and `ts config push`. + +| Mistake | Error | +| ---------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `owner` names a provider that is not enabled | ``ec.partners source `exampleid.example.com` names owner `exampleid`, which is not an enabled identity provider`` | +| `owner` names a capture-only provider | ``... names owner `examplecapture`, which is capture-only; a capture-only provider stores no positive ID and cannot own a source`` | +| `owner` names a provider that does not claim the source | ``... names owner `exampleid`, which does not claim that source`` | +| Two providers claim an owned source | ``identity provider `other` claims source `exampleid.example.com`, which `exampleid` owns; use ec.identity.providers.other.sources to narrow one of them`` | +| A provider claims a source with no `[[ec.partners]]` entry | ``identity provider `exampleid` claims source `exampleid.example.com`, which has no [[ec.partners]] entry`` | +| A capture-only provider has no `sources` | ``provider `examplecapture` is capture-only, so ec.identity.providers.examplecapture.sources must list the sources it is the delivery path for`` | +| `sources` names a source the provider does not declare | ``ec.identity.providers.exampleid.sources names `other.example.com`, which provider `exampleid` does not declare`` | +| A provider table names no enabled provider | `ec.identity.providers.missing names no enabled identity provider; configured providers: exampleid` | +| `revoke_scope` on a provider that declares no captures | ``ec.identity.providers.exampleid.revoke_scope is set, but provider `exampleid` declares no captures`` | +| `timeout_ms` above the hold budget in hold mode | `ec.identity.providers.exampleid.timeout_ms must not exceed hold_budget_ms (100)` | +| `after_send_timeout_ms` outside 1 to 10000 | `ec.identity.providers.exampleid.after_send_timeout_ms must be between 1 and 10000, got 0` | +| `refresh_after_sec = 0` | `ec.identity.providers.exampleid.refresh_after_sec must be greater than 0` | +| `hold_budget_ms` above 1000, `max_concurrency = 0` | `ec.identity.hold_budget_ms must be at most 1000, got 1001` and `ec.identity.max_concurrency must be at least 1` | + +Also an error: a duplicate provider ID, more than 32 enabled providers (generic pull partners included), a `mode` other than `hold` or `after_send`, and a `revoke_scope` other than `provider` or `global`. Before enabling a capture provider on a live service, work through the [capture provider checklist](/guide/edge-cookies#capture-provider-checklist). **Pull partner provider IDs.** Each `[[ec.partners]]` entry with `pull_sync_enabled = true` registers a provider whose ID is `pull-` plus the source domain with dots replaced by `-`. For example, `ssp.example.com` is `pull-ssp-example-com`. A domain that contains `-`, or an ID that would be longer than 63 characters, also gets `-` and the first 8 hex digits of the SHA-256 of the full domain, so `a-b.example.com` and `a.b.example.com` get different IDs. Configure the partner's pacing in `[ec.identity.providers.pull-ssp-example-com]`. @@ -675,11 +724,7 @@ The old rate limit counted calls per hour; the new one counts them per minute an This release moves the KV identity row to schema version 2. A release before it cannot read a version 2 row, so rolling back past this release is unsupported once it has written one; going back needs a KV migration to version 1, which is not provided. -A later release that adds vendor providers can be rolled back to this one with these steps: - -- Remove `owner` and every vendor `[ec.identity.providers.*]` table from the config first. This release rejects an `owner` or a provider table it does not know. -- Complete entry sets that a vendor wrote stay readable, but they are not served while the provider that wrote them is not configured. -- A write prunes an unconfigured provider's state down to its opt-out fields, so opt-outs survive. Its invalidated-value list and its provider state do not, so invalidated values can be seeded once more and stateful providers start over when rolled forward again. +To roll back from a release that adds vendor providers to the identity framework without vendors, follow the steps in [Rolling back from vendor providers](/guide/edge-cookies#rolling-back-from-vendor-providers): remove every `owner` and vendor `[ec.identity.providers.*]` table first, because the earlier release rejects both. ### `[ec]` diff --git a/docs/guide/edge-cookies.md b/docs/guide/edge-cookies.md index 31e1085e8b..7e8a28e7cb 100644 --- a/docs/guide/edge-cookies.md +++ b/docs/guide/edge-cookies.md @@ -294,6 +294,70 @@ Each request also logs one `identity request measurement: ...` line after the re - Size: at most 2,048 bytes. When the IDs do not fit, whole sources are left out, lowest `cookie_priority` first, never part of a source. - The shim's `ts-eids` trim now keeps registered sources first, so a size-capped `ts-eids` loses unregistered sources before registered ones. +## Server-Side Identity Providers + +Identity providers fetch or capture IDs on the server. Engineers who write one should read [Writing an Identity Provider](/guide/identity-providers). The sections below are for operators. Per-key reference is in the [Configuration Reference](/guide/configuration#identity-providers-ec-identity). + +### Capture Provider Checklist + +A capture provider reads responses that Trusted Server proxies for a vendor. It is revoke-only: a captured revocation, opt-out or clear is written to KV, and captured positive IDs are only counted in `capture_unpersisted`, never stored. Work through these steps before enabling one on a live service: + +1. **Register every captured source.** Each source the provider declares has an `[[ec.partners]]` entry with `bidstream_enabled = true` and the right `openrtb_atype`. +2. **Move a source off pull sync first.** A source that is going to have an `owner` must not also have `pull_sync_enabled = true`. A pull-sync entry registers a generic pull provider that claims the source, and two providers cannot claim an owned source. +3. **Keep `sources` minimal.** The provider's `sources` list in `[ec.identity.providers.]` holds only the sources it is the delivery path for. Its opt-out removes browser-ingested records of every source on the list, whoever produced them. +4. **Run the live acceptance check.** After a revocation is captured on a live page, look up the visitor's row with `GET /_ts/admin/ec` and confirm it shows the provider's `ps.o` and no record for the provider's sources. After the vendor issues again, confirm the row shows `ps.oc`, and that the next returning-visitor navigation ingests the source again. Use `GET /_ts/admin/eids` with the visitor's `ts-eids` cookie to preview what the browser would ingest. Read the `identity request measurement: ...` log line of the revocation request: `removal_lost` must stay at `0`, and `capture_unpersisted` shows the positive IDs a durable path would add. + +### Revoke Scope + +`revoke_scope` in `[ec.identity.providers.]` decides how far a captured opt-out reaches. The default, `provider`, removes only that provider's records and, for a capture-only provider, the records of its listed sources that no provider wrote. + +With `revoke_scope = "global"`, a captured opt-out from a real browser runs the consent-withdrawal tombstone: the row is tombstoned, the `ts-ec` cookie is cleared, and everything is removed, provider opt-outs included. The visitor then starts again with a new EC. A client that is not a real browser, such as a script replaying a copied `ts-ec` cookie, never changes the KV row: its capture is dropped and logged by provider and source only. + +### Complete Provider EIDs + +A provider can return more than one EID entry for a source, several UIDs per entry, and EID-level `inserter`, `matcher`, `mm` and `ext`. KV stores the whole set in the source's `eids` array, and the admin lookup (`GET /_ts/admin/ec`) shows it in the row's `ids` map, as stored (other row fields omitted): + +```json +{ + "entry": { + "ids": { + "exampleid.example.com": { + "w": "exampleid", + "rv": "0123456789abcdef", + "at": 1790000000, + "eids": [ + { + "uids": [ + { "id": "uid-a", "atype": 1, "ext": { "kind": "a" } }, + { "id": "uid-b" } + ], + "inserter": "exampleid.example.com", + "matcher": "exampleid.example.com", + "mm": 2 + }, + { "uids": [{ "id": "uid-c" }], "ext": { "segment": "x" } } + ] + } + } + } +} +``` + +Bidders receive the entries in that order, after the browser's entries. An entry that sets none of `inserter`, `matcher` and `mm` gets the provider's configured defaults at output. The `ext` data is stored only in KV and reaches bidders, `x-ts-eids` and the `__Host-ts-eids-s` mirror under the same consent gates as the UIDs. The mirror can be read by every script on the page. + +`GET /_ts/api/v1/identify` is unchanged: it still returns one UID per source, the first UID of the first entry, with the partner's `openrtb_atype`. + +### Rolling Back from Vendor Providers + +To go back to the identity release without vendor providers: + +1. Remove every `owner` key from `[[ec.partners]]` and every vendor `[ec.identity.providers.*]` table first, then push the config. The earlier release's validation rejects both. Pull-partner tables (`pull-...`) stay. +2. Deploy the earlier release. + +After the rollback, multi-entry records stay readable but are not served while the provider that wrote them is not configured, and browser ingest may replace them. Each provider's `ps.o`, `ps.oc` and `ps.ov` (the opt-out, its clear and the removed-ID hashes) are kept, so opt-outs survive; the rest of its `ps` is pruned. Invalidated values can therefore be seeded once more, and stateful providers start over when rolled forward again. + +Rolling back past the identity release is unsupported, because a release before it cannot read KV schema version 2. + ## Configuration Configure EC settings in the `[ec]` section of `trusted-server.toml`. See the [Configuration Reference](/guide/configuration) for the full surface and environment variable overrides. @@ -347,6 +411,7 @@ receives any Trusted Server identity cookie (`ts-ec`, `ts-eids`, or - Follow the [EC Setup Guide](/guide/ec-setup-guide) - [Configuration Reference](/guide/configuration) +- [Writing an Identity Provider](/guide/identity-providers) - [GDPR Compliance](/guide/gdpr-compliance) for consent signal handling - Configure [Ad Serving](/guide/ad-serving) - [Collective Sync](/guide/collective-sync) for cross-publisher data sharing details and diagrams diff --git a/docs/guide/identity-providers.md b/docs/guide/identity-providers.md new file mode 100644 index 0000000000..c91f6fd603 --- /dev/null +++ b/docs/guide/identity-providers.md @@ -0,0 +1,310 @@ +# Writing an Identity Provider + +This guide is for engineers who add a vendor to Trusted Server's server-side identity framework. Operators configuring a provider that already exists want the [Configuration Reference](/guide/configuration#identity-providers-ec-identity) and the [Edge Cookies](/guide/edge-cookies) guide instead. + +Every example uses a fictional provider, `exampleid`, that calls `api.exampleid.example.com`. No vendor ships with the framework. Vendors arrive in their own specs. + +For the protocol, the storage layout and the reasoning behind each rule, see the [design spec](https://github.com/IABTechLab/trusted-server/blob/main/docs/superpowers/specs/2026-10-20-server-side-identity-providers-design.md). Section numbers below point into it. + +## What a provider is + +A provider turns request signals into OpenRTB EIDs. It does that in one or both of two ways: + +- **Active resolution.** It builds an HTTPS request to the vendor from the visitor's signals, then parses the vendor's reply into IDs. +- **Passive capture.** It reads a response that Trusted Server already proxies for the vendor, with no outbound call. Capture is revoke-only (see [Capture](#capture)). + +A provider does no I/O. It builds requests and parses bytes that are already buffered. The coordinator does the rest: + +| The coordinator handles | You write | +| ------------------------------------------------------------------------- | ----------------------------------------- | +| Consent checks against your policy before any byte leaves | A `ConsentPolicy` in the descriptor | +| Real-browser gate, eligibility, refresh timing, per-visitor burst cap | Nothing | +| HTTPS, host allow-list, no `Cookie` header, redirects, timeouts, body cap | Declared hosts | +| Backoff on failure and the circuit breaker | Nothing | +| The single KV write, ownership, expiry and opt-outs | A `Resolution` that says what you learned | +| Hold mode versus after-send dispatch | Nothing (the operator picks the mode) | + +## The pieces + +Implement `IdentityProvider`. Only `descriptor` is required. You add a capability by declaring it in the descriptor and overriding the matching method (`request` and `response` for resolving, `capture` for capturing). Every default body returns a `not_implemented` marker, so a capability that is declared but not implemented fails in the harness. + +Implement `IdentityProviderConfig` on your integration's config type. Its `build_descriptor` builds the descriptor from config alone, with no secrets, so `ts config validate` and the running provider see the same descriptor. Your constructor calls it and stores the result. + +## The descriptor + +```rust +ProviderDescriptor::builder("exampleid".parse()?) + .with_source("exampleid.example.com".parse()?) + .with_resolves( + Resolves::new() + .with_endpoint("https://api.exampleid.example.com/v1/fetch") + .with_needs([Input::ClientIp, Input::UserAgent]) + .with_wants([Input::PageUrl]) + .with_private_state(), + ) + .with_consent_policy(ConsentPolicy::strict().with_tcf_vendor(9999)) + .build()? +``` + +- `with_source` declares each source domain you may write. A source also needs an `[[ec.partners]]` entry. +- `Resolves::with_endpoint` must be an `https` URL. Its host becomes the only host you may call. `with_host` adds more. +- `with_needs` inputs are required: without them the provider is skipped. `with_wants` inputs are passed when available. +- `with_private_state` lets you keep a small JSON blob between calls (see [Private state](#private-state)). +- `with_expiring_ids` or `with_expiring_source` marks IDs as expiring. A set for an expiring source must then carry an expiry. +- `ConsentPolicy::strict()` is the default. It blocks when any consent signal cannot be decoded, requires TCF Purposes 1 and 4 where TCF applies, and blocks on every US opt-out, `Sec-GPC: 1` and `DNT: 1`. Relaxations such as `allowing_dnt` need a documented legal basis in the vendor spec. +- `build()` fails with `Config` for a descriptor with no source or no capability, a missing or non-`https` endpoint, or invalid provenance. + +The descriptor must never read a secret field. Deploy validation runs before secrets resolve and would see key names. `check_descriptor_ignores_secrets` enforces this in tests. + +### Provenance defaults + +`with_provenance(source, EidProvenance)` sets default `inserter`, `matcher` and `mm` for a source. The default is applied at output only, and only to a stored entry that sets none of the three. An entry that sets any of them is the vendor's complete statement, and no default mixes in. Defaults are not stored, so a config change applies at once. A blank `inserter` or `matcher`, one over 255 bytes, or an `mm` below 0 fails `build()`. + +## Reading inputs + +You get only what the descriptor declared. Read a required input with `?` and a wanted one with `.ok()`: + +```rust +let ip = input.client_ip()?; +let page = input.page_url().ok(); +``` + +An accessor fails in one of two ways: + +- `UndeclaredInput`: you read something the descriptor never declared. This is a provider bug. Even if you hide the error with `.ok()`, the read is recorded, the coordinator logs it as `Config`, and the harness fails your test. +- `InputUnavailable`: the input was declared but is missing this time. This is a normal skip, handled like returning `None` from `request`. + +Always available: `consent()`, `publisher_domain()`, `now()`, `random_bytes()`, `stored(source)` and the declared `endpoint()`. When the sources in one request have different kinds, use `input.is_initial(&source)` or `input.is_refresh(&source)` rather than matching on the kind. + +## Building a request + +`request` returns `Ok(Some(request))`, or `Ok(None)` to skip this time. A skip is not a failure: the coordinator waits `refresh_after_sec` before asking again. + +```rust +fn request( + &self, + input: &ResolveInput<'_>, +) -> Result, Report> { + let consent = input.consent(); + let body = FetchBody { + partner: self.partner_id, + ip: input.client_ip()?.to_string(), + ua: input.user_agent()?, + page: input.page_url().ok().map(Url::as_str), + gdpr: consent.gdpr_applies(), + consent: consent.raw_tc_string(), + signature: input.state::()?.map(|state| state.signature), + }; + ProviderRequest::post_json(input.endpoint()?, &body).map(Some) +} +``` + +The coordinator refuses to send a request that is not `https`, that targets an undeclared host, that carries a `Cookie` header, or whose headers are invalid. Use `try_with_call_context` to pass data from `request` to `response`, such as a nonce. It stays in memory, is never sent, and is limited to 1,024 bytes of JSON. + +## Parsing a response + +`response` receives the buffered, already decoded body and returns a `Resolution`: + +```rust +fn response( + &self, + response: &ProviderResponse, + input: &ResolveInput<'_>, +) -> Result> { + let reply: FetchReply = response.success_json()?; + let Some(uid) = reply.uid else { + return Ok(Resolution::no_match()); + }; + + let mut uid = ResolvedUid::new(uid).with_atype(1); + if let Some(link_type) = reply.link_type { + uid = uid.with_ext(json!({ "linkType": link_type })); + } + let mut eid = ResolvedEid::new(input.primary_source(), uid); + if let Some(max_age) = reply.max_age_sec { + eid = eid.with_refresh_in(Duration::from_secs(max_age)); + } + let resolution = Resolution::found([eid]); + Ok(match reply.signature { + Some(signature) => resolution.replacing_state(&ExampleIdState { signature }), + None => resolution, + }) +} +``` + +Return `Rejected` (through `success_json` or `IdentityProviderError::rejected`) for a refusal and `Malformed` for a body you cannot parse. Both back the visitor off. Do not return an error for "this visitor is unknown": that is `Resolution::no_match()`. + +| Constructor | Meaning | +| ----------------------------------- | ------------------------------------------------------------------- | +| `Resolution::found(entries)` | One or more `ResolvedEid` entries | +| `Resolution::no_match()` | Nobody recognized. `with_retry_in` sets the wait (default 24 hours) | +| `Resolution::invalidate(source)` | The stored ID for `source` is no longer valid | +| `found(..).also_invalidate(source)` | Return some sources and drop others in one response | +| `Resolution::opted_out()` | The visitor opted out with the vendor. Sticky until a re-grant | + +## Complete provider EIDs + +A `ResolvedEid` is one OpenRTB EID entry: a source, one or more `ResolvedUid`s, and EID-level fields. A result may hold several entries for one source, in the order bidders should see them, and several UIDs per entry. + +```rust +let first = ResolvedEid::new(&source, ResolvedUid::new("uid-a").with_atype(1).with_ext(json!({ "kind": "a" }))) + .with_uid(ResolvedUid::new("uid-b")) + .with_inserter("exampleid.example.com") + .with_matcher("exampleid.example.com") + .with_mm(2); +let second = ResolvedEid::new(&source, ResolvedUid::new("uid-c")).with_ext(json!({ "segment": "x" })); +let resolution = Resolution::found([first, second]); +``` + +- UID-level fields: `atype` and `ext`. +- EID-level fields: `inserter`, `matcher`, `mm` and `ext`. +- Timing (`with_expires_in`, `with_expires_at`, `with_refresh_in`, `with_refresh_at`) applies to the source's whole set. When several entries of a source set it, the earliest time wins. +- `with_refresh_until` is for a provider with private state. It marks when an ID stops being refreshable, so an expired record stays as a refresh anchor until then. + +### Limits + +The coordinator groups a result's entries by source and checks each group with the browser parser's field rules, plus per-source caps the browser parser does not have. It applies them before the result joins the write. + +- An entry for a source you do not claim, or that the operator narrowed away, is discarded. +- A UID that is empty, blank or over 512 bytes is dropped. An entry left with no UID is discarded. +- A blank or over-255-byte `inserter` or `matcher`, an `ext` that is not an object or exceeds 2,048 bytes as compact JSON, and an `mm` or `atype` below 0 are dropped as fields. The entry is kept. +- An entry equal to an earlier kept entry of the same source is dropped. +- A source keeps at most **8 entries** and **32 UIDs** across them. The rest are dropped. +- A source's entry set over **4,096 bytes** (`MAX_ENTRY_SET_BYTES`, measured as stored, without provenance defaults) is discarded whole, never trimmed. + +Each `ext` may hold 2,048 bytes and each `inserter` or `matcher` 255 bytes, but those field caps do not guarantee the set cap. A set whose legal fields add up to more than 4,096 bytes is discarded whole. + +A discarded set leaves the stored set as it is, as if the result had not named the source. A result all of whose sets are discarded counts as `Malformed`: the visitor backs off, and the result's removals still apply. The harness fails a fixture that hits any limit, so keep your test responses at realistic sizes. + +### Replace or fill + +What a result does depends on who owns the source: + +- For a source the provider **owns** (the `owner` key of its `[[ec.partners]]` entry), a result replaces the source's whole entry set. An entry the next result omits is removed. +- For an **unowned** source, a result only fills an absent or expired record. It is never replaced by its writer, and browser values and other writers win. + +A source your result does not name keeps its record either way. + +### Vendor `ext` data and privacy + +Vendor `ext` data is stored only in KV, never in the `ts-eids` cookie. It reaches bidders, the `x-ts-eids` response header and the `__Host-ts-eids-s` mirror under the same consent gates as the UIDs. The mirror is not `HttpOnly`, so every script on the page can read it. If you store segments or match metadata in `ext`, say exactly what they hold in your vendor spec. If a vendor's IDs must not reach page scripts, the operator sets the source's `bidstream_enabled = false` or keeps the provider off. + +Never log UIDs, provider state, consent strings or URLs with a query string. Log errors with `current_context()` only. + +## Private state + +A provider declared `with_private_state()` can keep up to 1,024 bytes of JSON in the visitor's row, for things like a refresh token or a signature. Read it with `input.state::()` and replace it with `Resolution::replacing_state(&value)` or `clearing_state()`. + +- State lives in KV only. It is never sent to bidders, put in a cookie, or returned by the admin lookup (which shows `[redacted]`). +- A state that does not decode as `T` returns `InvalidState`, and the coordinator clears it so the next call starts fresh. Use a strict type: a `#[serde(default)]` type that decodes anything fails the harness. +- An oversized or unserializable replacement rejects the result's positive part whole (no ID, no state), while its removals still apply. +- A stateful result applies whole. If another writer stored a newer ID while your call was in flight, your ID, state and scheduling are all dropped together. + +## Capture + +A capture-only provider declares `with_captures(Captures::new().with_path(Method::POST, "/integrations/examplecapture/api/token"))` and no `resolves`, so it is never dispatched. Every capture path must lie under the registering integration's own proxy routes, or startup fails. It overrides one method: + +```rust +fn capture( + &self, + input: &CaptureInput<'_>, +) -> Result> { + let Ok(reply) = input.response().json::() else { + return Ok(Resolution::no_match()); + }; + Ok(Resolution::found(reply.tokens.into_iter().filter_map(|token| { + let source = input.source_named(&token.source)?; + Some(ResolvedEid::new(source, ResolvedUid::new(token.value))) + }))) +} +``` + +Capture is **revoke-only**. A capture result can do three things, and no more: + +- Its removals (`invalidate`, `also_invalidate`) and an opt-out (`opted_out`) are written to KV. +- A positive result clears the provider's opt-out (the vendor issuing again after the visitor re-consented), provided consent allows it. +- Its positive IDs are only counted in the `capture_unpersisted` counter, by provider and source and never by value. They are not stored. The browser's own path carries them through ordinary browser ingest. + +More rules: + +- Capture never changes private state, and never touches backoff or the circuit breaker. A proxied call that held nothing is not a failure. A state change in a capture result is ignored with a warning, and the harness fails it. +- Return `Ok(Resolution::no_match())` for any body you do not recognize. +- `CaptureInput` never exposes request bodies or request headers. +- **Real-browser gate.** A capture from a client that fails the real-browser check writes nothing: its removals, opt-outs and global revokes are dropped and logged by provider and source only. A script replaying a copied `ts-ec` cookie through the proxy cannot strip or tombstone a row. +- **A capture-only provider must list its `sources`** in `[ec.identity.providers.]`, and it **cannot own a source**. Its opt-out removes browser-ingested records of every listed source, so the operator names only the sources it is the delivery path for. +- `revoke_scope` controls how far a captured opt-out reaches (see [Configuration Reference](/guide/configuration#per-provider-keys)). With `"provider"` (the default) it removes only that provider's records and, for a capture-only provider, the records of its listed sources that no provider wrote. With `"global"`, a captured opt-out from a real browser runs the consent-withdrawal tombstone. + +## Registering a provider + +An integration's `register` function registers a provider through the existing builder. `try_with_identity_provider` wraps a fallible constructor and turns a failure into `TrustedServerError::Integration`: + +```rust +Ok(Some( + IntegrationRegistration::builder(EXAMPLEID_INTEGRATION_ID) + .try_with_identity_provider(ExampleIdProvider::new(&config))? + .build(), +)) +``` + +`with_identity_provider(Arc)` takes an already built provider. An integration can register more than one. Add a `validate_identity_integration` line for your config type next to the existing per-type list in the deploy validator, so `ts config validate` builds your descriptor. + +Provider IDs match `^[a-z][a-z0-9_-]{0,62}$`, and two providers cannot share an ID. At most 32 providers can be enabled, generic pull partners included. + +## Testing with the harness + +`ProviderHarness` drives a provider the way the coordinator does, so vendor tests stay short. It lives in `trusted_server_core::ec::identity::test_support`, behind `cfg(any(test, feature = "test-utils"))`. A provider in its own crate enables `test-utils` as a dev-dependency feature of `trusted-server-core`. + +```rust +let provider = ExampleIdProvider::new(&example_config()).expect("should build the provider"); +let mut harness = ProviderHarness::new(&provider) + .with_client_ip("192.0.2.10") + .with_user_agent("ExampleBrowser/1.0"); + +let request = harness + .request() + .expect("should build a request") + .expect("should not skip the request"); +assert_eq!( + request.url().host_str(), + Some("api.exampleid.example.com"), + "should call the declared host" +); + +let resolution = harness + .respond(StatusCode::OK, &json!({ "uid": "example-uid", "signature": "example-signature" })) + .expect("should parse the response"); +assert_eq!( + resolution.eids()[0].uids()[0].id(), + "example-uid", + "should return the uid" +); +``` + +- `respond_raw(status, headers, bytes)` covers empty and non-JSON bodies. `request_json::()` decodes the built body. `capture(&CaptureFixture)` runs a capture. `state_of::(&resolution)` decodes the replacement state. +- `with_stored(StoredId::for_test(..))` makes a source a refresh. `StoredId::with_eids` gives a refreshing provider a stored full set. +- The `Resolution` getters show what the provider set. Provenance defaults are applied only by `output_eids(&resolution)`, which returns the EIDs bidders would see after the section 4.5 rules and your descriptor's defaults. +- `check_conformance(&cases)` runs the checks every provider must pass; `check_conformance_with_state::` also round-trips your state type. Both panic with a message naming the failure. `try_check_conformance` and `try_check_conformance_with_state::` return a `Result` with a `ConformanceFailure` instead. Use them to assert that a deliberately broken provider fails, because a panic aborts the whole `wasm32-wasip1` test run. +- `check_descriptor_ignores_secrets(&placeholder_config, &resolved_config)` fails if your descriptor depends on a secret. + +`check_conformance` verifies that inputs hold only what the descriptor declares, that a resolver returns a request for at least one listed setup, that requests go only to declared hosts over HTTPS with valid headers and no `Cookie`, that entries are for declared sources and no fixture hits a limit, that expiries are reported and present for expiring sources, that nothing returns state without `with_private_state()`, that `capture` never changes state, and that a bare-JSON-string stored state is never accepted. + +```rust +#[test] +fn example_provider_conforms() { + let provider = ExampleIdProvider::new(&example_config()).expect("should build the provider"); + ProviderHarness::new(&provider).check_conformance_with_state::( + &ConformanceCases::default().with_resolve_case( + HarnessSetup::default().with_client_ip("192.0.2.10").with_user_agent("ExampleBrowser/1.0"), + StatusCode::OK, + &json!({ "uid": "example-uid", "signature": "s" }), + ), + ); +} +``` + +Add a fixture response for each outcome the vendor can return, and assert it through the `Resolution` getters. + +## Next steps + +- [Edge Cookies](/guide/edge-cookies) for the operator checklist and rollback steps +- [Configuration Reference](/guide/configuration#identity-providers-ec-identity) for every operator key diff --git a/trusted-server.example.toml b/trusted-server.example.toml index c362b5e22b..25f52bfdc0 100644 --- a/trusted-server.example.toml +++ b/trusted-server.example.toml @@ -110,8 +110,26 @@ ec_store = "ec_identity_store" # max_calls_per_minute = 1 # breaker_failures_per_minute = 50 # cookie_priority = 0 -# `revoke_scope = "provider"` (or "global") applies only to a provider that -# captures IDs, from Part 2 of the identity rollout; do not set it here. + +# Capture provider (a vendor whose responses Trusted Server proxies; the +# provider declares `captures`). A capture-only provider must list its +# `sources`: its opt-out removes browser-ingested records of every source +# listed here, so list only the sources it is the delivery path for. Each +# source also needs an [[ec.partners]] entry with bidstream_enabled = true. +# [ec.identity.providers.examplecapture] +# sources = ["capture.example.com"] +# `provider` (default) removes only this provider's records. `global` runs +# the consent-withdrawal tombstone for a captured opt-out from a real browser. +# revoke_scope = "provider" + +# A source can name the identity provider that owns it. The owner refreshes the +# source and its result replaces the source's whole entry set. Absent: unowned. +# [[ec.partners]] +# name = "Example ID" +# source_domain = "exampleid.example.com" +# openrtb_atype = 1 +# bidstream_enabled = true +# owner = "exampleid" # Example partner configuration. Provision referenced keys in # trusted_server_secrets before validating/pushing. From 7074ec1c56becff04458cb1a087a84b8cb562111 Mon Sep 17 00:00:00 2001 From: dhruv8sh Date: Tue, 27 Oct 2026 21:46:12 +0530 Subject: [PATCH 137/194] Cancel superseded PR runs and queue each job per author A workflow-level group per pull request now cancels a run when a newer push arrives, so stale runs no longer hold an author's queue or count toward GitHub's 100-pending cap. The per-author queue moves to each job's own concurrency block with queue: max, keyed on the matrix for matrix jobs so their legs do not queue behind each other. Bot-authored PRs and non-PR events still key on the run ID. The shared comment now states the FIFO ordering and the 100-pending limit. Signed-off-by: dhruv8sh --- .github/workflows/codeql.yml | 20 +++++++----- .github/workflows/format.yml | 32 ++++++++++++++----- .github/workflows/integration-tests.yml | 29 ++++++++++++----- .github/workflows/pbs-example.yml | 20 +++++++----- .github/workflows/test.yml | 41 ++++++++++++++++++++----- 5 files changed, 102 insertions(+), 40 deletions(-) diff --git a/.github/workflows/codeql.yml b/.github/workflows/codeql.yml index 09f03e0e0a..1d3f825ffe 100644 --- a/.github/workflows/codeql.yml +++ b/.github/workflows/codeql.yml @@ -19,15 +19,16 @@ on: schedule: - cron: "42 6 * * 0" -# Each pull request author runs one PR at a time per workflow, so one person -# opening many PRs cannot fill the runner queue ahead of everyone else. Their -# other runs wait in order instead of being cancelled, and every job has a -# timeout so a hung run cannot hold that queue for hours. Bot-authored PRs and -# other events get their own group from the run ID and are never held back. +# A newer push to a pull request cancels that PR's superseded run. Each job +# then joins a per-author queue for that job (its own `concurrency` block), so +# one person opening many PRs runs one copy of each job at a time and cannot +# fill the runner pool ahead of everyone else. Queued jobs start in the order they began waiting; GitHub +# keeps up to 100 pending per queue and cancels arrivals past that. Every job +# has a timeout so a hung job cannot hold its queue for hours. Bot-authored +# PRs and non-PR events key on the run ID and never wait on a queue. concurrency: - group: ${{ github.workflow }}-${{ github.event_name == 'pull_request' && github.event.pull_request.user.type != 'Bot' && github.event.pull_request.user.login || github.run_id }} - cancel-in-progress: false - queue: max + group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.run_id }} + cancel-in-progress: true jobs: analyze: @@ -39,6 +40,9 @@ jobs: # Consider using larger runners or machines with greater resources for possible analysis time improvements. runs-on: ${{ (matrix.language == 'swift' && 'macos-latest') || 'ubuntu-latest' }} timeout-minutes: 45 + concurrency: + group: ${{ github.workflow }}-analyze-${{ matrix.language }}-${{ github.event_name == 'pull_request' && github.event.pull_request.user.type != 'Bot' && github.event.pull_request.user.login || github.run_id }} + queue: max permissions: # required for all workflows security-events: write diff --git a/.github/workflows/format.yml b/.github/workflows/format.yml index 7de102c2de..1d03f47364 100644 --- a/.github/workflows/format.yml +++ b/.github/workflows/format.yml @@ -8,21 +8,25 @@ on: permissions: contents: read -# Each pull request author runs one PR at a time per workflow, so one person -# opening many PRs cannot fill the runner queue ahead of everyone else. Their -# other runs wait in order instead of being cancelled, and every job has a -# timeout so a hung run cannot hold that queue for hours. Bot-authored PRs and -# other events get their own group from the run ID and are never held back. +# A newer push to a pull request cancels that PR's superseded run. Each job +# then joins a per-author queue for that job (its own `concurrency` block), so +# one person opening many PRs runs one copy of each job at a time and cannot +# fill the runner pool ahead of everyone else. Queued jobs start in the order they began waiting; GitHub +# keeps up to 100 pending per queue and cancels arrivals past that. Every job +# has a timeout so a hung job cannot hold its queue for hours. Bot-authored +# PRs and non-PR events key on the run ID and never wait on a queue. concurrency: - group: ${{ github.workflow }}-${{ github.event_name == 'pull_request' && github.event.pull_request.user.type != 'Bot' && github.event.pull_request.user.login || github.run_id }} - cancel-in-progress: false - queue: max + group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.run_id }} + cancel-in-progress: true jobs: check-claude-md-symlink: name: CLAUDE.md symlink guard runs-on: ubuntu-latest timeout-minutes: 5 + concurrency: + group: ${{ github.workflow }}-check-claude-md-symlink-${{ github.event_name == 'pull_request' && github.event.pull_request.user.type != 'Bot' && github.event.pull_request.user.login || github.run_id }} + queue: max steps: - uses: actions/checkout@v4 @@ -54,6 +58,9 @@ jobs: name: cargo fmt runs-on: ubuntu-latest timeout-minutes: 25 + concurrency: + group: ${{ github.workflow }}-format-rust-${{ github.event_name == 'pull_request' && github.event.pull_request.user.type != 'Bot' && github.event.pull_request.user.login || github.run_id }} + queue: max steps: - uses: actions/checkout@v4 @@ -116,6 +123,9 @@ jobs: format-typescript: runs-on: ubuntu-latest timeout-minutes: 10 + concurrency: + group: ${{ github.workflow }}-format-typescript-${{ github.event_name == 'pull_request' && github.event.pull_request.user.type != 'Bot' && github.event.pull_request.user.login || github.run_id }} + queue: max defaults: run: working-directory: crates/trusted-server-js/lib @@ -148,6 +158,9 @@ jobs: format-docs: runs-on: ubuntu-latest timeout-minutes: 10 + concurrency: + group: ${{ github.workflow }}-format-docs-${{ github.event_name == 'pull_request' && github.event.pull_request.user.type != 'Bot' && github.event.pull_request.user.login || github.run_id }} + queue: max defaults: run: working-directory: docs @@ -190,6 +203,9 @@ jobs: name: docs-proposal scripts runs-on: ubuntu-latest timeout-minutes: 10 + concurrency: + group: ${{ github.workflow }}-docs-proposal-scripts-${{ github.event_name == 'pull_request' && github.event.pull_request.user.type != 'Bot' && github.event.pull_request.user.login || github.run_id }} + queue: max steps: - uses: actions/checkout@v4 diff --git a/.github/workflows/integration-tests.yml b/.github/workflows/integration-tests.yml index 3867d79224..a276d23ecc 100644 --- a/.github/workflows/integration-tests.yml +++ b/.github/workflows/integration-tests.yml @@ -10,15 +10,16 @@ on: types: [opened, synchronize, reopened] workflow_dispatch: -# Each pull request author runs one PR at a time per workflow, so one person -# opening many PRs cannot fill the runner queue ahead of everyone else. Their -# other runs wait in order instead of being cancelled, and every job has a -# timeout so a hung run cannot hold that queue for hours. Bot-authored PRs and -# other events get their own group from the run ID and are never held back. +# A newer push to a pull request cancels that PR's superseded run. Each job +# then joins a per-author queue for that job (its own `concurrency` block), so +# one person opening many PRs runs one copy of each job at a time and cannot +# fill the runner pool ahead of everyone else. Queued jobs start in the order they began waiting; GitHub +# keeps up to 100 pending per queue and cancels arrivals past that. Every job +# has a timeout so a hung job cannot hold its queue for hours. Bot-authored +# PRs and non-PR events key on the run ID and never wait on a queue. concurrency: - group: ${{ github.workflow }}-${{ github.event_name == 'pull_request' && github.event.pull_request.user.type != 'Bot' && github.event.pull_request.user.login || github.run_id }} - cancel-in-progress: false - queue: max + group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.run_id }} + cancel-in-progress: true env: ORIGIN_PORT: 8888 @@ -33,6 +34,9 @@ jobs: name: prepare integration artifacts runs-on: ubuntu-latest timeout-minutes: 30 + concurrency: + group: ${{ github.workflow }}-prepare-artifacts-${{ github.event_name == 'pull_request' && github.event.pull_request.user.type != 'Bot' && github.event.pull_request.user.login || github.run_id }} + queue: max steps: - uses: actions/checkout@v4 @@ -70,6 +74,9 @@ jobs: needs: prepare-artifacts runs-on: ubuntu-latest timeout-minutes: 20 + concurrency: + group: ${{ github.workflow }}-integration-tests-${{ github.event_name == 'pull_request' && github.event.pull_request.user.type != 'Bot' && github.event.pull_request.user.login || github.run_id }} + queue: max steps: - uses: actions/checkout@v4 @@ -129,6 +136,9 @@ jobs: needs: prepare-artifacts runs-on: ubuntu-latest timeout-minutes: 15 + concurrency: + group: ${{ github.workflow }}-integration-tests-fastly-ec-${{ github.event_name == 'pull_request' && github.event.pull_request.user.type != 'Bot' && github.event.pull_request.user.login || github.run_id }} + queue: max steps: - uses: actions/checkout@v4 @@ -173,6 +183,9 @@ jobs: # once the pinned Playwright version supports the newer image. runs-on: ubuntu-24.04 timeout-minutes: 15 + concurrency: + group: ${{ github.workflow }}-browser-tests-${{ github.event_name == 'pull_request' && github.event.pull_request.user.type != 'Bot' && github.event.pull_request.user.login || github.run_id }} + queue: max steps: - uses: actions/checkout@v4 diff --git a/.github/workflows/pbs-example.yml b/.github/workflows/pbs-example.yml index 2c1fb07303..d61072867a 100644 --- a/.github/workflows/pbs-example.yml +++ b/.github/workflows/pbs-example.yml @@ -17,21 +17,25 @@ on: permissions: contents: read -# Each pull request author runs one PR at a time per workflow, so one person -# opening many PRs cannot fill the runner queue ahead of everyone else. Their -# other runs wait in order instead of being cancelled, and every job has a -# timeout so a hung run cannot hold that queue for hours. Bot-authored PRs and -# other events get their own group from the run ID and are never held back. +# A newer push to a pull request cancels that PR's superseded run. Each job +# then joins a per-author queue for that job (its own `concurrency` block), so +# one person opening many PRs runs one copy of each job at a time and cannot +# fill the runner pool ahead of everyone else. Queued jobs start in the order they began waiting; GitHub +# keeps up to 100 pending per queue and cancels arrivals past that. Every job +# has a timeout so a hung job cannot hold its queue for hours. Bot-authored +# PRs and non-PR events key on the run ID and never wait on a queue. concurrency: - group: ${{ github.workflow }}-${{ github.event_name == 'pull_request' && github.event.pull_request.user.type != 'Bot' && github.event.pull_request.user.login || github.run_id }} - cancel-in-progress: false - queue: max + group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.run_id }} + cancel-in-progress: true jobs: local-checks: name: PBS Terraform and Compose checks runs-on: ubuntu-latest timeout-minutes: 20 + concurrency: + group: ${{ github.workflow }}-local-checks-${{ github.event_name == 'pull_request' && github.event.pull_request.user.type != 'Bot' && github.event.pull_request.user.login || github.run_id }} + queue: max env: TF_IN_AUTOMATION: 'true' AWS_EC2_METADATA_DISABLED: 'true' diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index a2e90e4b19..a5c669f83e 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -8,21 +8,25 @@ on: branches: [main] pull_request: -# Each pull request author runs one PR at a time per workflow, so one person -# opening many PRs cannot fill the runner queue ahead of everyone else. Their -# other runs wait in order instead of being cancelled, and every job has a -# timeout so a hung run cannot hold that queue for hours. Bot-authored PRs and -# other events get their own group from the run ID and are never held back. +# A newer push to a pull request cancels that PR's superseded run. Each job +# then joins a per-author queue for that job (its own `concurrency` block), so +# one person opening many PRs runs one copy of each job at a time and cannot +# fill the runner pool ahead of everyone else. Queued jobs start in the order they began waiting; GitHub +# keeps up to 100 pending per queue and cancels arrivals past that. Every job +# has a timeout so a hung job cannot hold its queue for hours. Bot-authored +# PRs and non-PR events key on the run ID and never wait on a queue. concurrency: - group: ${{ github.workflow }}-${{ github.event_name == 'pull_request' && github.event.pull_request.user.type != 'Bot' && github.event.pull_request.user.login || github.run_id }} - cancel-in-progress: false - queue: max + group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.run_id }} + cancel-in-progress: true jobs: test-rust: name: cargo test runs-on: ubuntu-latest timeout-minutes: 45 + concurrency: + group: ${{ github.workflow }}-test-rust-${{ github.event_name == 'pull_request' && github.event.pull_request.user.type != 'Bot' && github.event.pull_request.user.login || github.run_id }} + queue: max steps: - uses: actions/checkout@v4 @@ -79,6 +83,9 @@ jobs: name: HTML EOF runtime (Fastly) runs-on: ubuntu-latest timeout-minutes: 25 + concurrency: + group: ${{ github.workflow }}-test-fastly-eof-${{ github.event_name == 'pull_request' && github.event.pull_request.user.type != 'Bot' && github.event.pull_request.user.login || github.run_id }} + queue: max steps: - uses: actions/checkout@v4 @@ -128,6 +135,9 @@ jobs: name: cargo test (axum native) runs-on: ubuntu-latest timeout-minutes: 30 + concurrency: + group: ${{ github.workflow }}-test-axum-${{ github.event_name == 'pull_request' && github.event.pull_request.user.type != 'Bot' && github.event.pull_request.user.login || github.run_id }} + queue: max steps: - uses: actions/checkout@v4 @@ -199,6 +209,9 @@ jobs: name: cargo check (cloudflare native + wasm32-unknown-unknown) runs-on: ubuntu-latest timeout-minutes: 15 + concurrency: + group: ${{ github.workflow }}-test-cloudflare-${{ github.event_name == 'pull_request' && github.event.pull_request.user.type != 'Bot' && github.event.pull_request.user.login || github.run_id }} + queue: max steps: - uses: actions/checkout@v4 @@ -225,6 +238,9 @@ jobs: name: cargo check/build/test (spin native + wasm32-wasip1) runs-on: ubuntu-latest timeout-minutes: 20 + concurrency: + group: ${{ github.workflow }}-test-spin-${{ github.event_name == 'pull_request' && github.event.pull_request.user.type != 'Bot' && github.event.pull_request.user.login || github.run_id }} + queue: max steps: - uses: actions/checkout@v4 @@ -263,6 +279,9 @@ jobs: name: cargo test (cross-adapter parity) runs-on: ubuntu-latest timeout-minutes: 15 + concurrency: + group: ${{ github.workflow }}-test-parity-${{ github.event_name == 'pull_request' && github.event.pull_request.user.type != 'Bot' && github.event.pull_request.user.login || github.run_id }} + queue: max steps: - uses: actions/checkout@v4 @@ -289,6 +308,9 @@ jobs: os: [ubuntu-latest, macos-latest] runs-on: ${{ matrix.os }} timeout-minutes: 45 + concurrency: + group: ${{ github.workflow }}-test-cli-${{ matrix.os }}-${{ github.event_name == 'pull_request' && github.event.pull_request.user.type != 'Bot' && github.event.pull_request.user.login || github.run_id }} + queue: max steps: - uses: actions/checkout@v4 @@ -354,6 +376,9 @@ jobs: name: vitest runs-on: ubuntu-latest timeout-minutes: 10 + concurrency: + group: ${{ github.workflow }}-test-typescript-${{ github.event_name == 'pull_request' && github.event.pull_request.user.type != 'Bot' && github.event.pull_request.user.login || github.run_id }} + queue: max defaults: run: working-directory: crates/trusted-server-js/lib From 600f6a9c19e4e2a9efbe59eab240290646aa0282 Mon Sep 17 00:00:00 2001 From: dhruv8sh Date: Tue, 27 Oct 2026 21:46:40 +0530 Subject: [PATCH 138/194] Guard the CI concurrency blocks and job timeouts The required cargo fmt job now fails when a PR-triggered workflow lacks the per-PR cancellation block from test.yml, or when a job lacks timeout-minutes or a per-author queue keyed on pull_request.user.login with queue: max. Matrix jobs must also include the matrix in their group. Jobs that call a reusable workflow are skipped. The step uses the runner image's yq. Signed-off-by: dhruv8sh --- .github/workflows/format.yml | 42 ++++++++++++++++++++++++++++++++++++ 1 file changed, 42 insertions(+) diff --git a/.github/workflows/format.yml b/.github/workflows/format.yml index 1d03f47364..0379093897 100644 --- a/.github/workflows/format.yml +++ b/.github/workflows/format.yml @@ -64,6 +64,48 @@ jobs: steps: - uses: actions/checkout@v4 + # A hung job, a PR-triggered workflow without per-PR cancellation, or a + # job outside its per-author queue brings back the runner pile-up that + # the concurrency blocks prevent. Every PR-triggered workflow must carry + # test.yml's workflow-level block, and every job must set timeout-minutes + # and a per-author queue (keyed on the matrix too, for matrix jobs). + # Jobs that call a reusable workflow are skipped: they cannot set these. + - name: Require per-PR cancellation, per-author job queues and timeouts + shell: bash + run: | + reference=.github/workflows/test.yml + if [ "$(yq '.concurrency."cancel-in-progress" == true and (.concurrency.group // "" | test("pull_request\.number"))' "$reference")" != true ]; then + echo "::error file=$reference::test.yml must cancel superseded runs per pull request." + exit 1 + fi + expected=$(yq -o=json -I=0 '.concurrency' "$reference") + status=0 + for f in .github/workflows/*.yml .github/workflows/*.yaml; do + [ -f "$f" ] || continue + on_pr=$(yq '.on | ((tag == "!!map" and has("pull_request")) or (tag == "!!str" and . == "pull_request") or (tag == "!!seq" and ([.[] | select(. == "pull_request")] | length > 0)))' "$f") + [ "$on_pr" = true ] || continue + if [ "$(yq -o=json -I=0 '.concurrency' "$f")" != "$expected" ]; then + echo "::error file=$f::PR-triggered workflows must use the workflow-level concurrency block from test.yml." + status=1 + fi + while read -r job; do + [ -n "$job" ] || continue + echo "::error file=$f::Job '$job' must set timeout-minutes." + status=1 + done < <(yq '.jobs | to_entries[] | select(.value.uses == null and .value["timeout-minutes"] == null) | .key' "$f") + while read -r job; do + [ -n "$job" ] || continue + echo "::error file=$f::Job '$job' must join a per-author queue: concurrency.group keyed on pull_request.user.login and queue: max." + status=1 + done < <(yq '.jobs | to_entries[] | select(.value.uses == null and (.value.concurrency.queue != "max" or (.value.concurrency.group // "" | test("pull_request\.user\.login") | not))) | .key' "$f") + while read -r job; do + [ -n "$job" ] || continue + echo "::error file=$f::Matrix job '$job' must include the matrix in its concurrency group so its legs do not queue behind each other." + status=1 + done < <(yq '.jobs | to_entries[] | select(.value.strategy.matrix != null and (.value.concurrency.group // "" | test("matrix\.") | not)) | .key' "$f") + done + exit "$status" + # Rust jobs must set up the toolchain through ./.github/actions/setup-rust, # which removes the image's stable and only saves caches from main. A # direct setup-rust-toolchain or rust-cache step skips both silently. From ab58ced95c20731f722f0daa02cdd992f2063180 Mon Sep 17 00:00:00 2001 From: Christian Date: Tue, 27 Oct 2026 13:27:37 -0500 Subject: [PATCH 139/194] Test zero-provider page-bids diagnostics omission --- crates/trusted-server-core/src/publisher.rs | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/crates/trusted-server-core/src/publisher.rs b/crates/trusted-server-core/src/publisher.rs index b556e81688..21ae41cf75 100644 --- a/crates/trusted-server-core/src/publisher.rs +++ b/crates/trusted-server-core/src/publisher.rs @@ -25147,6 +25147,16 @@ mod tests { .expect("should produce a no-bid response without providers"); assert_eq!(response.status(), StatusCode::OK); + let body_bytes = response + .into_body() + .into_bytes() + .expect("should read zero-provider page-bids body"); + let body: serde_json::Value = serde_json::from_slice(&body_bytes) + .expect("should parse zero-provider page-bids body"); + assert!( + body.get("auctionDiagnostics").is_none(), + "should omit diagnostics when no provider launched" + ); let snapshot = timings.snapshot(); assert!(snapshot.auction_dispatched_ms.is_none()); assert!(snapshot.auction_resolved_ms.is_none()); From 39568b1729498573f2149bb66475080a6b575fd0 Mon Sep 17 00:00:00 2001 From: Christian Date: Tue, 27 Oct 2026 13:27:37 -0500 Subject: [PATCH 140/194] Keep immediate test outcomes unlaunched --- .../src/auction/orchestrator.rs | 22 ++++++++++--------- crates/trusted-server-core/src/publisher.rs | 8 +++---- 2 files changed, 16 insertions(+), 14 deletions(-) diff --git a/crates/trusted-server-core/src/auction/orchestrator.rs b/crates/trusted-server-core/src/auction/orchestrator.rs index fc97a585c4..e6b7655c02 100644 --- a/crates/trusted-server-core/src/auction/orchestrator.rs +++ b/crates/trusted-server-core/src/auction/orchestrator.rs @@ -133,8 +133,8 @@ impl DispatchedAuction { #[cfg(test)] impl DispatchedAuction { - /// Creates a token with an immediate no-bid outcome for collector tests. - pub(crate) fn immediate_no_bid_for_test(request: AuctionRequest, timeout_ms: u32) -> Self { + /// Creates a token modeling a started provider request that resolved to no bid. + pub(crate) fn launched_no_bid_for_test(request: AuctionRequest, timeout_ms: u32) -> Self { let mut dispatched = Self::empty_for_test(request, timeout_ms); dispatched.provider_launched = true; dispatched @@ -1871,10 +1871,12 @@ impl AuctionOrchestrator { /// [`DispatchedAuction`] token. The Fastly host begins the SSP round-trips /// while WASM continues to `pending_origin.wait()`. /// - /// Returns [`DispatchAuctionOutcome::NotStarted`] when no providers are configured or - /// all providers are disabled / over budget. Returns - /// [`DispatchAuctionOutcome::DispatchFailed`] when provider launch attempts - /// happened but none could be started. + /// Returns [`DispatchAuctionOutcome::NotStarted`] when the auction is disabled + /// or has no providers to execute. Routing-only skips can instead yield a + /// [`DispatchAuctionOutcome::Dispatched`] token with immediate responses but + /// no provider launch. Admission or launch failures can yield + /// [`DispatchAuctionOutcome::DispatchFailed`]; its `provider_launched` flag + /// preserves evidence of requests that started before a later failure. #[must_use] pub async fn dispatch_auction( &self, @@ -2066,10 +2068,6 @@ impl AuctionOrchestrator { } Ok(ProviderRequestOutcome::Immediate(response)) => { immediate_response_count += 1; - #[cfg(test)] - { - provider_launch_count += 1; - } completed_responses.push(response); } Err(e) => { @@ -4505,6 +4503,10 @@ mod tests { else { panic!("enabled immediate provider should dispatch"); }; + assert!( + !dispatched.has_provider_launch(), + "should not count an immediate no-bid response as a provider request" + ); let result = orchestrator .collect_dispatched_auction(dispatched, &services, &context) .await; diff --git a/crates/trusted-server-core/src/publisher.rs b/crates/trusted-server-core/src/publisher.rs index 21ae41cf75..437f9815db 100644 --- a/crates/trusted-server-core/src/publisher.rs +++ b/crates/trusted-server-core/src/publisher.rs @@ -22253,7 +22253,7 @@ mod tests { let mut params = make_stream_params(&settings, ""); params.content_type = "text/html; charset=utf-8".to_string(); - params.dispatched_auction = Some(DispatchedAuction::immediate_no_bid_for_test( + params.dispatched_auction = Some(DispatchedAuction::launched_no_bid_for_test( test_auction_request(), 500, )); @@ -22328,7 +22328,7 @@ mod tests { let orchestrator = Arc::new(AuctionOrchestrator::new(settings.auction.clone())); let timings = RequestTimings::new(); let mut params = make_stream_params(&settings, ""); - params.dispatched_auction = Some(DispatchedAuction::immediate_no_bid_for_test( + params.dispatched_auction = Some(DispatchedAuction::launched_no_bid_for_test( test_auction_request(), 500, )); @@ -22382,7 +22382,7 @@ mod tests { let orchestrator = Arc::new(AuctionOrchestrator::new(settings.auction.clone())); let timings = RequestTimings::new(); let mut params = make_stream_params(&settings, ""); - params.dispatched_auction = Some(DispatchedAuction::immediate_no_bid_for_test( + params.dispatched_auction = Some(DispatchedAuction::launched_no_bid_for_test( test_auction_request(), 500, )); @@ -22516,7 +22516,7 @@ mod tests { let mut params = make_stream_params(&settings, ""); params.content_type = "text/html; charset=utf-8".to_string(); - params.dispatched_auction = Some(DispatchedAuction::immediate_no_bid_for_test( + params.dispatched_auction = Some(DispatchedAuction::launched_no_bid_for_test( test_auction_request(), 500, )); From ba18e09a2a60559e1d52e124857a1081e8323c28 Mon Sep 17 00:00:00 2001 From: Christian Date: Tue, 27 Oct 2026 13:27:37 -0500 Subject: [PATCH 141/194] Reject legacy access datasets when emission is on --- crates/trusted-server-core/src/settings.rs | 49 +++++++++++++++++++--- 1 file changed, 44 insertions(+), 5 deletions(-) diff --git a/crates/trusted-server-core/src/settings.rs b/crates/trusted-server-core/src/settings.rs index cbbb7f313a..75d9335946 100644 --- a/crates/trusted-server-core/src/settings.rs +++ b/crates/trusted-server-core/src/settings.rs @@ -2028,7 +2028,7 @@ impl TinybirdSettings { let token = self.auction_token_secret.as_ref().ok_or_else(|| { Report::new(TrustedServerError::Configuration { message: - "tinybird.auction_token_secret is required when Tinybird telemetry is enabled" + "tinybird.auction_token_secret is required when tinybird.auction_enabled is true" .to_owned(), }) })?; @@ -2036,6 +2036,11 @@ impl TinybirdSettings { } if self.access_enabled { validate_tinybird_dataset(&self.access_dataset, "tinybird.access_dataset")?; + if self.access_dataset == "access_logs_raw" { + return Err(Report::new(TrustedServerError::Configuration { + message: "tinybird.access_dataset must not be access_logs_raw when access telemetry is enabled; use access_logs_v2 or a datasource with the v2 schema".to_owned(), + })); + } let token = self.access_token_secret.as_ref().ok_or_else(|| { Report::new(TrustedServerError::Configuration { message: @@ -2863,10 +2868,10 @@ pub enum AuctionDebugCommentFormat { /// Request-observability toggles exposed to operators. /// -/// The default table must stay omitted from serialized config blobs: this -/// struct denies unknown fields, so an older binary loading a config blob -/// carrying an `[observability]` table it does not know would reject it, -/// breaking rollback. See [`Settings::observability`]. +/// The default table must stay omitted from serialized config blobs. Older +/// binaries deny unknown fields on [`Settings`], so a blob carrying the new +/// `[observability]` table would fail to load and break rollback. See +/// [`Settings::observability`]. #[derive(Debug, Clone, Default, PartialEq, Deserialize, Serialize)] #[serde(deny_unknown_fields)] pub struct ObservabilitySettings { @@ -4971,6 +4976,40 @@ mod tests { ); } + #[test] + fn access_enabled_rejects_legacy_dataset_after_normalization() { + for dataset in ["access_logs_raw", " access_logs_raw "] { + let config = format!( + "[tinybird]\nenabled = true\napi_host = \"api.example.com\"\nauction_enabled = false\naccess_enabled = true\naccess_dataset = \"{dataset}\"\naccess_token_secret = \"test-access-token\"\naccess_sample_rate = 1.0\n" + ); + let error = settings_from_toml_with(&config) + .expect_err("should reject v2 access rows targeting the legacy datasource"); + assert!( + error.to_string().contains("tinybird.access_dataset"), + "should identify the invalid access dataset: {error:?}" + ); + } + } + + #[test] + fn disabled_access_accepts_legacy_dataset() { + let settings = settings_from_toml_with( + "[tinybird]\nenabled = true\napi_host = \"api.example.com\"\nauction_token_secret = \"test-auction-token\"\naccess_dataset = \"access_logs_raw\"\n", + ) + .expect("should retain legacy dataset settings while access emission is disabled"); + assert_eq!(settings.tinybird.access_dataset, "access_logs_raw"); + assert!(!settings.tinybird.access_enabled); + } + + #[test] + fn access_enabled_accepts_custom_versioned_dataset() { + let settings = settings_from_toml_with( + "[tinybird]\nenabled = true\napi_host = \"api.example.com\"\nauction_enabled = false\naccess_enabled = true\naccess_dataset = \"custom_access_v2\"\naccess_token_secret = \"test-access-token\"\naccess_sample_rate = 1.0\n", + ) + .expect("should allow custom access datasource names"); + assert_eq!(settings.tinybird.access_dataset, "custom_access_v2"); + } + #[test] fn access_enabled_requires_tinybird_enabled() { // access_enabled = true with tinybird.enabled omitted (defaults From b8ccff4f9e859fd22bbb7634b2682a734cf58003 Mon Sep 17 00:00:00 2001 From: Christian Date: Tue, 27 Oct 2026 13:27:37 -0500 Subject: [PATCH 142/194] Retry failed Prebid diagnostics registration --- .../lib/src/integrations/prebid/index.ts | 2 +- .../test/integrations/prebid/index.test.ts | 53 +++++++++++++++++++ 2 files changed, 54 insertions(+), 1 deletion(-) diff --git a/crates/trusted-server-js/lib/src/integrations/prebid/index.ts b/crates/trusted-server-js/lib/src/integrations/prebid/index.ts index 4bcdd58b63..a2f4d23aa5 100644 --- a/crates/trusted-server-js/lib/src/integrations/prebid/index.ts +++ b/crates/trusted-server-js/lib/src/integrations/prebid/index.ts @@ -1120,7 +1120,6 @@ function recordCompletedPrebidAuction( function installPrebidWinDiagnostics(): void { const diagnosticPbjs = pbjs as PbjsGlobal & { __tsDiagnosticsBidWonInstalled?: boolean }; if (diagnosticPbjs.__tsDiagnosticsBidWonInstalled || typeof pbjs.onEvent !== 'function') return; - diagnosticPbjs.__tsDiagnosticsBidWonInstalled = true; pbjs.onEvent('bidWon', (rawBid: unknown) => { if (typeof rawBid !== 'object' || rawBid === null) return; const bid = rawBid as Record; @@ -1159,6 +1158,7 @@ function installPrebidWinDiagnostics(): void { // Diagnostics must not alter delivery. } }); + diagnosticPbjs.__tsDiagnosticsBidWonInstalled = true; } function dispatchPrebidRefresh( diff --git a/crates/trusted-server-js/lib/test/integrations/prebid/index.test.ts b/crates/trusted-server-js/lib/test/integrations/prebid/index.test.ts index 73e414c93a..20fba6325b 100644 --- a/crates/trusted-server-js/lib/test/integrations/prebid/index.test.ts +++ b/crates/trusted-server-js/lib/test/integrations/prebid/index.test.ts @@ -5844,6 +5844,59 @@ describe('prebid publisher snapshots and delivery refreshes', () => { }); }); + it('retries failed bidWon registration without duplicating a successful listener', () => { + const slot = { + getSlotElementId: () => 'example-retry-slot', + getTargeting: () => [], + clearTargeting: vi.fn(), + }; + const recordPrebidWin = vi.fn(); + testWindow.tsjs = { + gptDiagnosticsRecorder: { + recordPrebidRefresh: vi.fn(), + recordPrebidAuction: vi.fn(), + recordPrebidWin, + }, + }; + const { originalRefresh, pubads } = installGpt([slot]); + installPrebidNpm(); + mockPbjs.setTargetingForGPTAsync = vi.fn(); + let auctionSequence = 0; + mockRequestBids.mockImplementation((opts) => { + opts.bidsBackHandler?.({}, false, `example-retry-auction-${++auctionSequence}`); + }); + mockOnEvent.mockClear(); + mockOnEvent.mockImplementationOnce(() => { + throw new Error('publisher onEvent failed once'); + }); + + try { + pubads.refresh([slot]); + expect(mockOnEvent).toHaveBeenCalledExactlyOnceWith('bidWon', expect.any(Function)); + expect(mockOnEvent.mock.results[0].type).toBe('throw'); + + pubads.refresh([slot]); + expect(mockOnEvent).toHaveBeenCalledTimes(2); + expect(mockOnEvent.mock.results[1].type).toBe('return'); + const bidWon = mockOnEvent.mock.calls[1][1]; + bidWon({ + auctionId: 'example-retry-auction-2', + adUnitCode: 'example-retry-slot', + adserverTargeting: { hb_bidder: 'example-client', hb_pb: '2.40' }, + }); + expect(recordPrebidWin).toHaveBeenCalledExactlyOnceWith(slot, 'example-retry-auction-2', { + bidder: 'example-client', + priceBucket: '2.40', + }); + + pubads.refresh([slot]); + expect(mockOnEvent).toHaveBeenCalledTimes(2); + expect(originalRefresh).toHaveBeenCalledTimes(3); + } finally { + mockOnEvent.mockReset(); + } + }); + it('does not record a completed auction for duplicate ad-unit element IDs', () => { const code = 'duplicate-client-slot'; const firstSlot = { From 0305f9f7a41ab1ed13fb0ea2c087cc5b05449dc1 Mon Sep 17 00:00:00 2001 From: Christian Date: Tue, 27 Oct 2026 13:27:37 -0500 Subject: [PATCH 143/194] Suppress referrers on diagnostics dictionary links --- .../lib/src/integrations/gpt_diagnostics/overlay.ts | 2 +- .../lib/test/integrations/gpt_diagnostics/overlay.test.ts | 5 ++++- 2 files changed, 5 insertions(+), 2 deletions(-) diff --git a/crates/trusted-server-js/lib/src/integrations/gpt_diagnostics/overlay.ts b/crates/trusted-server-js/lib/src/integrations/gpt_diagnostics/overlay.ts index fcc666fcc9..b0011594bc 100644 --- a/crates/trusted-server-js/lib/src/integrations/gpt_diagnostics/overlay.ts +++ b/crates/trusted-server-js/lib/src/integrations/gpt_diagnostics/overlay.ts @@ -721,7 +721,7 @@ export class GptDiagnosticsOverlay { dictionaryLink.href = 'https://iabtechlab.github.io/trusted-server/guide/integrations/gpt-diagnostics-dictionary'; dictionaryLink.target = '_blank'; - dictionaryLink.rel = 'noopener'; + dictionaryLink.rel = 'noopener noreferrer'; dictionaryLink.textContent = 'Label dictionary'; dictionaryLink.setAttribute('aria-label', 'Open GPT diagnostics label dictionary'); filterLabel.append(select); diff --git a/crates/trusted-server-js/lib/test/integrations/gpt_diagnostics/overlay.test.ts b/crates/trusted-server-js/lib/test/integrations/gpt_diagnostics/overlay.test.ts index ecea57a2c5..8343f7e743 100644 --- a/crates/trusted-server-js/lib/test/integrations/gpt_diagnostics/overlay.test.ts +++ b/crates/trusted-server-js/lib/test/integrations/gpt_diagnostics/overlay.test.ts @@ -532,9 +532,12 @@ describe('GptDiagnosticsOverlay', () => { expect(emptySummary?.textContent).not.toContain('Served bidder not confirmed'); expect(root!.textContent).toContain('How to read this evidence'); - expect(root!.querySelector('a')?.href).toBe( + const dictionaryLink = root!.querySelector('a'); + expect(dictionaryLink?.href).toBe( 'https://iabtechlab.github.io/trusted-server/guide/integrations/gpt-diagnostics-dictionary' ); + expect(dictionaryLink?.relList.contains('noopener')).toBe(true); + expect(dictionaryLink?.relList.contains('noreferrer')).toBe(true); button(root!, 'Export JSON').click(); expect(exportSnapshot).toHaveBeenCalledTimes(1); From 2e0722f841abab6d015621a780b3f31172ad83c2 Mon Sep 17 00:00:00 2001 From: Christian Date: Tue, 27 Oct 2026 13:27:37 -0500 Subject: [PATCH 144/194] Honor readiness in the Axum dev-server bridge --- .../trusted-server-adapter-axum/src/main.rs | 135 ++++++++++++++++-- .../trusted-server-adapter-axum/src/timing.rs | 12 +- 2 files changed, 129 insertions(+), 18 deletions(-) diff --git a/crates/trusted-server-adapter-axum/src/main.rs b/crates/trusted-server-adapter-axum/src/main.rs index b012f1fdef..db21ab3d86 100644 --- a/crates/trusted-server-adapter-axum/src/main.rs +++ b/crates/trusted-server-adapter-axum/src/main.rs @@ -1,14 +1,16 @@ +use std::convert::Infallible; use std::net::SocketAddr; use axum::Router; +use axum::body::Body; +use axum::http::{Request, Response}; use edgezero_adapter_axum::dev_server::AxumDevServerConfig; use edgezero_adapter_axum::service::EdgeZeroAxumService; use log::LevelFilter; use tokio::net::TcpListener; use tokio::runtime::Builder as RuntimeBuilder; use tokio::signal; -use tower::Service as _; -use tower::service_fn; +use tower::{Service, ServiceExt as _, service_fn}; use trusted_server_adapter_axum::app::TrustedServerApp; use trusted_server_adapter_axum::timing::TimingService; @@ -46,7 +48,6 @@ fn main() { None => AxumDevServerConfig::default(), }; - log::info!("Listening on http://{}", config.addr); let service = TrustedServerApp::dev_server_service(); if let Err(err) = run(service, config) { log::error!("trusted-server-adapter-axum failed: {err}"); @@ -59,7 +60,7 @@ fn main() { /// `EdgeZeroAxumService`, ahead of `axum::serve`. /// /// This does not use `edgezero_adapter_axum::dev_server::AxumDevServer::run`: -/// that helper only accepts a bare [`RouterService`] and builds its own +/// that helper only accepts a bare [`RouterService`](edgezero_core::router::RouterService) and builds its own /// `EdgeZeroAxumService` and `axum::Router` internally, with no seam for an /// outer service wrapper. Router-generated 404/405 responses bypass /// `RouterBuilder::middleware` (see `trusted_server_adapter_axum::timing`), @@ -76,7 +77,15 @@ fn run( service: TimingService, config: AxumDevServerConfig, ) -> std::io::Result<()> { - let runtime = RuntimeBuilder::new_multi_thread().enable_all().build()?; + let runtime = RuntimeBuilder::new_multi_thread() + .enable_all() + .build() + .map_err(|error| { + std::io::Error::new( + error.kind(), + format!("failed to build tokio runtime: {error}"), + ) + })?; runtime.block_on(serve(service, config)) } @@ -91,11 +100,10 @@ async fn serve( ) })?; - let axum_router = Router::new().fallback_service(service_fn(move |req| { - let mut svc = service.clone(); - async move { svc.call(req).await } - })); - let make_service = axum_router.into_make_service_with_connect_info::(); + log::info!("Listening on http://{}", listener.local_addr()?); + + let make_service = + dev_server_router(service).into_make_service_with_connect_info::(); let server = axum::serve(listener, make_service); if config.enable_ctrl_c { @@ -107,6 +115,19 @@ async fn serve( } else { server.await } + .map_err(|error| std::io::Error::new(error.kind(), format!("axum server error: {error}"))) +} + +fn dev_server_router(service: S) -> Router +where + S: Service, Response = Response, Error = Infallible> + + Clone + + Send + + Sync + + 'static, + S::Future: Send + 'static, +{ + Router::new().fallback_service(service_fn(move |req| service.clone().oneshot(req))) } /// Parses an optional `RUST_LOG` value into the logger's maximum level. @@ -148,10 +169,100 @@ fn port_from_env() -> Option { #[cfg(test)] mod tests { + use std::future::{Ready, ready}; + use std::sync::Arc; + use std::sync::atomic::{AtomicUsize, Ordering}; + use std::task::{Context, Poll}; + use super::*; - #[test] - fn crate_compiles() {} + struct ReadinessSensitiveService { + polled: bool, + readied: bool, + pending_polls: Arc, + calls: Arc, + } + + impl Clone for ReadinessSensitiveService { + fn clone(&self) -> Self { + Self { + polled: false, + readied: false, + pending_polls: Arc::clone(&self.pending_polls), + calls: Arc::clone(&self.calls), + } + } + } + + impl Service> for ReadinessSensitiveService { + type Response = Response; + type Error = Infallible; + type Future = Ready>; + + fn poll_ready(&mut self, cx: &mut Context<'_>) -> Poll> { + if !self.polled { + self.polled = true; + self.pending_polls.fetch_add(1, Ordering::Relaxed); + cx.waker().wake_by_ref(); + return Poll::Pending; + } + self.readied = true; + Poll::Ready(Ok(())) + } + + fn call(&mut self, _request: Request) -> Self::Future { + assert!( + self.readied, + "should wait for this instance to become ready" + ); + self.readied = false; + self.calls.fetch_add(1, Ordering::Relaxed); + ready(Ok(Response::builder() + .header("cache-control", "private") + .body(Body::empty()) + .expect("should build a private response"))) + } + } + + #[tokio::test] + async fn dev_server_bridge_waits_for_readiness_on_timed_and_health_paths() { + for path in ["/private", "/health"] { + for enabled in [true, false] { + let pending_polls = Arc::new(AtomicUsize::new(0)); + let calls = Arc::new(AtomicUsize::new(0)); + let router = dev_server_router(TimingService::new( + ReadinessSensitiveService { + polled: false, + readied: false, + pending_polls: Arc::clone(&pending_polls), + calls: Arc::clone(&calls), + }, + enabled, + )); + + for expected_calls in 1..=2 { + let response = router + .clone() + .oneshot( + Request::builder() + .uri(path) + .body(Body::empty()) + .expect("should build a bridge request"), + ) + .await + .expect("should serve the request after readiness"); + + assert_eq!(calls.load(Ordering::Relaxed), expected_calls); + assert_eq!(pending_polls.load(Ordering::Relaxed), expected_calls); + assert_eq!( + response.headers().contains_key("server-timing"), + enabled && path != "/health", + "should preserve timing configuration and health exclusion" + ); + } + } + } + } #[test] fn parse_log_level_defaults_to_info_when_unset_or_blank() { diff --git a/crates/trusted-server-adapter-axum/src/timing.rs b/crates/trusted-server-adapter-axum/src/timing.rs index 3fbe623357..4e97a27525 100644 --- a/crates/trusted-server-adapter-axum/src/timing.rs +++ b/crates/trusted-server-adapter-axum/src/timing.rs @@ -10,9 +10,9 @@ //! //! This wraps *outside* `RouterService` rather than registering as //! `RouterBuilder::middleware` because the tower boundary is the terminal -//! freeze point: by the time a response reaches this layer -- after +//! freeze point. Before a response reaches this layer, //! `RouterService::oneshot` inside `EdgeZeroAxumService::call` has -//! converted any dispatch error into a plain response -- every response is +//! converted any dispatch error into a plain response. Every response is //! covered uniformly regardless of how routing produced it, and the //! position survives future routing changes. (In this application's router //! a catch-all fallback spans every path and publisher method, so @@ -24,10 +24,10 @@ //! collector is even created: this wrapper bypasses timing for every method //! on that path, leaving any preinstalled handle unchanged. //! -//! Unlike the Fastly adapter (state built per request, adding -//! `Phase::AppBuild` to the rendered header), the Axum dev server builds its -//! application state once at startup. There is no per-request app-build -//! interval to measure, so `ts-appbuild` never appears in the header here. +//! The Axum dev server builds its application state once at startup, so +//! `ts-appbuild` never appears in its per-request header. Fastly records +//! per-request setup and, on cold requests, application construction in +//! `Phase::AppBuild`. use std::convert::Infallible; use std::future::Future; From 5cf32ab5f55e71ead4353b6b36d776aa6250ef53 Mon Sep 17 00:00:00 2001 From: Christian Date: Tue, 27 Oct 2026 13:27:37 -0500 Subject: [PATCH 145/194] Borrow settings when building access targets --- .../trusted-server-adapter-fastly/Cargo.toml | 2 +- .../trusted-server-adapter-fastly/src/main.rs | 5 +- .../src/tinybird.rs | 53 +++++++++++++------ 3 files changed, 41 insertions(+), 19 deletions(-) diff --git a/crates/trusted-server-adapter-fastly/Cargo.toml b/crates/trusted-server-adapter-fastly/Cargo.toml index a8bfff87d3..ed66130f72 100644 --- a/crates/trusted-server-adapter-fastly/Cargo.toml +++ b/crates/trusted-server-adapter-fastly/Cargo.toml @@ -32,10 +32,10 @@ fern = { workspace = true } futures = { workspace = true } log = { workspace = true } log-fastly = { workspace = true } +rand = { workspace = true } serde = { workspace = true } serde_json = { workspace = true } trusted-server-core = { workspace = true } -rand = { workspace = true } url = { workspace = true } urlencoding = { workspace = true } diff --git a/crates/trusted-server-adapter-fastly/src/main.rs b/crates/trusted-server-adapter-fastly/src/main.rs index 0b7c845d9f..5dcfbe7246 100644 --- a/crates/trusted-server-adapter-fastly/src/main.rs +++ b/crates/trusted-server-adapter-fastly/src/main.rs @@ -1,6 +1,4 @@ use std::sync::Arc; - -use rand::Rng as _; use std::time::{Instant, SystemTime, UNIX_EPOCH}; use edgezero_adapter_fastly::config_store::FastlyConfigStore as EdgeZeroFastlyConfigStore; @@ -15,6 +13,7 @@ use edgezero_core::response::IntoResponse; use error_stack::Report; use fastly::http::Method as FastlyMethod; use fastly::{Request as FastlyRequest, Response as FastlyResponse}; +use rand::Rng as _; use trusted_server_core::access_telemetry::{ AccessTelemetrySnapshot, RouteClass, RouteMetadata, access_event_row, @@ -833,7 +832,7 @@ fn emit_access_telemetry_after_send( .unwrap_or_default(); let epoch_ms = u64::try_from(since_epoch.as_millis()).unwrap_or(u64::MAX); let row = access_event_row(snapshot, &timings.snapshot(), epoch_ms); - let target = tinybird::TinybirdEventsTarget::from_access_config(settings.tinybird.clone()); + let target = tinybird::TinybirdEventsTarget::from_access_config(&settings.tinybird); let result = futures::executor::block_on(tinybird::emit_access_event( &platform::FastlyPlatformHttpClient, &target, diff --git a/crates/trusted-server-adapter-fastly/src/tinybird.rs b/crates/trusted-server-adapter-fastly/src/tinybird.rs index 69d5e168e8..4350c290cb 100644 --- a/crates/trusted-server-adapter-fastly/src/tinybird.rs +++ b/crates/trusted-server-adapter-fastly/src/tinybird.rs @@ -81,13 +81,13 @@ impl TinybirdEventsTarget { /// `access_token_secret` instead of the auction pair, so access-log /// emission never shares a datasource or token with auction telemetry /// even though both configs come from the same [`TinybirdSettings`]. - pub(crate) fn from_access_config(config: TinybirdSettings) -> Self { + pub(crate) fn from_access_config(config: &TinybirdSettings) -> Self { let uri = tinybird_events_uri(&config.api_host, &config.access_dataset); let backend_spec = tinybird_backend_spec(&config.api_host); Self { - api_host: config.api_host, - dataset: config.access_dataset, - append_token: config.access_token_secret.expect( + api_host: config.api_host.clone(), + dataset: config.access_dataset.clone(), + append_token: config.access_token_secret.clone().expect( "should contain a resolved Tinybird access token when access telemetry is enabled", ), uri, @@ -215,7 +215,7 @@ impl AuctionTelemetrySink for FastlyTinybirdAuctionTelemetrySink { } // --------------------------------------------------------------------------- -// Access telemetry: confirmed-delivery emitter +// Access telemetry: post-send emitter // --------------------------------------------------------------------------- /// Decides whether one request's access-telemetry row should be emitted. @@ -257,14 +257,15 @@ fn build_access_events_request( }) } -/// Sends one confirmed access-log row to the Tinybird Events API and waits +/// Sends one post-send access-log row to the Tinybird Events API and waits /// for the response. /// -/// Unlike [`FastlyTinybirdAuctionTelemetrySink::emit_auction_events`] (fire- -/// and-forget, dispatched mid-request so it never adds latency to the -/// response), this runs post-delivery: the response has already reached the -/// client, so there is no latency budget left to protect, and the send can -/// afford to wait for — and validate — the reply. `client` is the adapter's +/// Unlike [`FastlyTinybirdAuctionTelemetrySink::emit_auction_events`], which +/// dispatches without awaiting the ingest reply, this waits after the client +/// response has been handed to the transport. It does not extend that response's +/// measured timing, but it occupies a reused sandbox and can delay its next +/// request. A completed transport handoff does not confirm client receipt. +/// `client` is the adapter's /// stateless platform HTTP client in production /// ([`crate::platform::FastlyPlatformHttpClient`]); accepting it as `&dyn /// PlatformHttpClient` here (rather than that concrete type) is what lets @@ -845,11 +846,33 @@ mod tests { ); } + #[test] + fn access_target_borrows_settings_and_uses_only_access_fields() { + let mut config = enabled_config(); + config.auction_token_secret = None; + config.max_body_bytes = 2048; + + let target = TinybirdEventsTarget::from_access_config(&config); + + assert_eq!(target.api_host, config.api_host); + assert_eq!(target.dataset, config.access_dataset); + assert_eq!(target.append_token.expose(), "access-append-token"); + assert_eq!(target.max_body_bytes, config.max_body_bytes); + assert_eq!( + config + .access_token_secret + .as_ref() + .expect("should retain the access token in settings") + .expose(), + "access-append-token" + ); + } + #[test] fn access_emitter_rejects_oversized_row_before_sending() { let mut config = enabled_config(); config.max_body_bytes = 1024; - let target = TinybirdEventsTarget::from_access_config(config); + let target = TinybirdEventsTarget::from_access_config(&config); let http_client = RecordingHttpClient::respond_with(202); let row = "x".repeat(1025); @@ -874,7 +897,7 @@ mod tests { fn access_emitter_accepts_row_exactly_at_body_limit_including_newline() { let mut config = enabled_config(); config.max_body_bytes = 1024; - let target = TinybirdEventsTarget::from_access_config(config); + let target = TinybirdEventsTarget::from_access_config(&config); let http_client = RecordingHttpClient::respond_with(202); let row = "x".repeat(1023); @@ -892,7 +915,7 @@ mod tests { #[test] fn access_emitter_posts_ndjson_and_validates_2xx() { - let target = TinybirdEventsTarget::from_access_config(enabled_config()); + let target = TinybirdEventsTarget::from_access_config(&enabled_config()); let http_client = RecordingHttpClient::respond_with(202); let row = r#"{"status":200}"#.to_owned(); @@ -927,7 +950,7 @@ mod tests { #[test] fn access_emitter_warns_and_drops_on_non_2xx() { - let target = TinybirdEventsTarget::from_access_config(enabled_config()); + let target = TinybirdEventsTarget::from_access_config(&enabled_config()); let http_client = RecordingHttpClient::respond_with(422); let result = futures::executor::block_on(emit_access_event( From c4cec089c313b5274f8fd41121c82124ff975dd3 Mon Sep 17 00:00:00 2001 From: Christian Date: Tue, 27 Oct 2026 13:27:37 -0500 Subject: [PATCH 146/194] Correct timing review documentation and fixtures --- .gitignore | 2 +- CHANGELOG.md | 4 ++ .../src/access_telemetry.rs | 5 +- .../src/auction/endpoints.rs | 10 ++- crates/trusted-server-core/src/publisher.rs | 54 +++++++------- .../trusted-server-core/src/request_timing.rs | 13 ++-- docs/guide/configuration.md | 70 +++++++++---------- .../gpt-diagnostics-dictionary.md | 29 ++++---- docs/guide/integrations/gpt-diagnostics.md | 5 +- docs/guide/telemetry.md | 36 +++++++--- .../2026-08-24-request-phase-timing-design.md | 52 +++++++------- tinybird/README.md | 19 +++-- tinybird/fixtures/access_logs_v2.ndjson | 4 +- 13 files changed, 166 insertions(+), 137 deletions(-) diff --git a/.gitignore b/.gitignore index 238e6b1333..4ca57e92f8 100644 --- a/.gitignore +++ b/.gitignore @@ -65,5 +65,5 @@ src/*.html /crates/integration-tests/ # Wrangler config generated by the Cloudflare integration-test harness from -# wrangler.toml at run time; regenerated on every run. +# wrangler.ci.toml at run time; regenerated on every run. wrangler.integration.generated.toml diff --git a/CHANGELOG.md b/CHANGELOG.md index ddae0a689c..7a10115271 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Changed +- Auction telemetry failure batches for `/auction` and page-bids now include provider launch-failure rows alongside the summary, so a failed dispatch can emit more than one row. `/auction` summary `elapsed_ms` now uses the orchestrator's elapsed time. +- GPT diagnostics JSON exports now include bounded server winner and Prebid candidate/win bidder codes and bucketed `hb_pb` values. These observations do not establish which creative GAM ultimately served; raw CPM and creative markup remain excluded. +- The Axum dev server now runs its own serve loop to apply terminal request-timing handling. +- The default `tinybird.access_dataset` is now `access_logs_v2`. Since this field is serialized, an upgrade from TOML that omits it may produce a one-time config-envelope diff; review and push once to settle it. Explicit legacy dataset values must be changed before enabling access telemetry. - **Breaking:** `x-ts-version` now reports the deployed git version — the release tag, else the branch, else the first 6 characters of the commit — compiled in from the build-time `TRUSTED_SERVER__GIT_VERSION` (set by the deploy pipeline) or local git, and is sent by every adapter, including on the Fastly `GET /health` probe. The Fastly service version it previously carried is now `x-ts-fastly-version`; update dashboards, monitors, and scripts that read `x-ts-version` as the Fastly version number. Builds with neither the override nor git omit the header. - **Breaking:** `ts prebid bundle` is now `ts prebid client`, alongside the new `ts prebid server` namespace. Update scripts and runbooks to use `ts prebid client` with the same arguments. The old `bundle` spelling is no longer accepted and has no compatibility alias. - The S2S `/_ts/api/v1/batch-sync` endpoint now validates the full batch and calls the CAS-protected update path once per distinct normalized EC ID. The last valid UID wins within a group, and infrastructure failures reject the failing and each unprocessed group, so accepted and `kv_unavailable` input indexes may interleave. diff --git a/crates/trusted-server-core/src/access_telemetry.rs b/crates/trusted-server-core/src/access_telemetry.rs index c029a2db33..de7334d842 100644 --- a/crates/trusted-server-core/src/access_telemetry.rs +++ b/crates/trusted-server-core/src/access_telemetry.rs @@ -20,8 +20,9 @@ use crate::request_timing::{AuctionWaitPlacement, TimingSnapshot}; /// entirely client controlled. Capturing one verbatim into a 30-day /// `LowCardinality(String)` column would let a single caller inflate that /// column's cardinality without bound and would violate this dataset's -/// bounded-dimension privacy rule (see the module doc). Every standard -/// method maps to its uppercase form; anything else maps to `"other"`. Runs +/// bounded-dimension privacy rule (see the module doc). GET, HEAD, POST, PUT, +/// DELETE, PATCH and OPTIONS map to their uppercase form; all other methods, +/// including CONNECT and TRACE, map to `"other"`. Runs /// inside [`access_event_row`] rather than at each capture site, so every /// row-building path is covered regardless of how `method` was populated. /// diff --git a/crates/trusted-server-core/src/auction/endpoints.rs b/crates/trusted-server-core/src/auction/endpoints.rs index 3de4e3daba..552721487b 100644 --- a/crates/trusted-server-core/src/auction/endpoints.rs +++ b/crates/trusted-server-core/src/auction/endpoints.rs @@ -1513,9 +1513,17 @@ mod tests { assert_eq!(rows[0].terminal_status.as_deref(), Some("skipped")); assert_eq!(rows[0].terminal_reason.as_deref(), Some("consent_denied")); let snapshot = timings.snapshot(); - assert_eq!(rows[0].auction_id, snapshot.auction_id.unwrap().to_string()); + assert_eq!( + rows[0].auction_id, + snapshot + .auction_id + .expect("should record the consent-denied auction UUID") + .to_string(), + "should retain the skipped-summary join key" + ); assert!(snapshot.auction_dispatched_ms.is_none()); assert!(snapshot.auction_resolved_ms.is_none()); + assert!(snapshot.auction_committed_ms.is_none()); let ndjson = batches[0] .to_ndjson(16 * 1024) .expect("should serialize telemetry"); diff --git a/crates/trusted-server-core/src/publisher.rs b/crates/trusted-server-core/src/publisher.rs index 437f9815db..a079a5aad8 100644 --- a/crates/trusted-server-core/src/publisher.rs +++ b/crates/trusted-server-core/src/publisher.rs @@ -3278,20 +3278,7 @@ fn request_origin(scheme: &str, host: &str) -> String { } } -/// This request's auction result, in both forms the seams need. -/// -/// The inline `` seam injects a rendered ``; const script = template.content.querySelector('script')!; - const attribute = raw.replaceAll('\0', '\ufffd'); + const attribute = raw.split('\0').join('\ufffd'); expect(script.getAttribute('src')).toBe(attribute); container.appendChild(script); diff --git a/docs/superpowers/specs/2026-10-16-1199-sdk-url-ownership-design.md b/docs/superpowers/specs/2026-10-16-1199-sdk-url-ownership-design.md index adb4e22930..acef88eedd 100644 --- a/docs/superpowers/specs/2026-10-16-1199-sdk-url-ownership-design.md +++ b/docs/superpowers/specs/2026-10-16-1199-sdk-url-ownership-design.md @@ -258,25 +258,24 @@ focused JS tests. It reported no findings. The original regression tests failed against the substring matchers before the ownership fixes. The normalization regressions also failed before expanding the -common preprocessing policy. The shared corpus now contains 105 cases, including -22 normalization positives and negatives. The all-guard test expands them into -1,376 ownership and insertion checks across both handler orders and DOM paths. - -| Check | Result | -| ------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------- | -| Full JS suite, `npx vitest run --maxWorkers=4` | Passed, 2,605 tests across 52 files; no reported type errors | -| JS `npm run lint`, `npm run format`, `npm run build` | Passed | -| Native Lockr and Permutive inline test modules | Passed, 11 and 9 tests, including the shared corpus and attribute rewriting | -| `cargo test-fastly`, `cargo test-fastly-reuse`, `cargo test-axum`, `cargo test-cloudflare`, `cargo test-spin` | Passed; existing ignored tests remain ignored | -| All eight target-specific clippy aliases listed in `AGENTS.md` | Passed | -| Integration parity test | Passed | -| Native template build-digest tests and clippy | Passed | -| `cargo fmt --all -- --check`, wasm clock lint | Passed | -| Docs and repository Markdown formatting | Passed | -| URL-host lint on added lines | Passed | - -Default-worker JS runs hit timeouts in core, creative, and Prebid tests locally. -The full four-worker run passed without changing test timeouts or assertions. +common preprocessing policy. The shared corpus now contains 106 cases, including +23 normalization positives and negatives. The all-guard test expands them into +1,390 ownership and insertion checks across both handler orders and DOM paths. + +| Check | Result | +| -------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | +| Full JS suite, `npx vitest run --maxWorkers=4` | Passed, 2,640 tests across 52 files; scoped Vitest type assertions pass | +| JS `npm run lint`, `npm run format`, `npm run build` | Passed | +| Native Lockr and Permutive inline test modules | Passed, 11 and 9 tests, including the shared corpus and attribute rewriting | +| `cargo test-axum`, `cargo test-cloudflare`, `cargo test-spin` | Passed; existing ignored tests remain ignored | +| `cargo test-fastly`, `cargo test-fastly-reuse` | Local reruns blocked before test execution: installed Viceroy 0.17.0 lacks `fastly_http_req::pending_req_header_insert`; the repository pins 0.21.0 | +| Package-wide `npx tsc --noEmit -p .` | Still fails on pre-existing errors; comparison with the unchanged branch confirms the test-only `replaceAll` error was removed | +| All eight target-specific clippy aliases listed in `AGENTS.md` | Passed | +| Integration parity test | Passed | +| Native template build-digest tests and clippy | Passed | +| `cargo fmt --all -- --check`, wasm clock lint | Passed | +| Docs and repository Markdown formatting | Passed | +| URL-host lint on added lines | Passed | No live vendor SDK fetch or production deployment smoke test was performed. From 32523f1848a62ff2c8f8d06c6ca2bcb31cdb938e Mon Sep 17 00:00:00 2001 From: dhruv8sh Date: Fri, 30 Oct 2026 12:48:24 +0530 Subject: [PATCH 184/194] Strengthen and document CI concurrency policy checks --- .github/workflows/codeql.yml | 13 +- .github/workflows/format.yml | 76 +++----- .github/workflows/integration-tests.yml | 13 +- .github/workflows/pbs-example.yml | 13 +- .github/workflows/test.yml | 17 +- AGENTS.md | 1 + scripts/README.md | 1 + scripts/check-ci-concurrency.sh | 164 +++++++++++++++++ scripts/check-ci-concurrency.test.mjs | 233 ++++++++++++++++++++++++ 9 files changed, 453 insertions(+), 78 deletions(-) create mode 100755 scripts/check-ci-concurrency.sh create mode 100644 scripts/check-ci-concurrency.test.mjs diff --git a/.github/workflows/codeql.yml b/.github/workflows/codeql.yml index 1d3f825ffe..45ea7f8a5e 100644 --- a/.github/workflows/codeql.yml +++ b/.github/workflows/codeql.yml @@ -19,13 +19,12 @@ on: schedule: - cron: "42 6 * * 0" -# A newer push to a pull request cancels that PR's superseded run. Each job -# then joins a per-author queue for that job (its own `concurrency` block), so -# one person opening many PRs runs one copy of each job at a time and cannot -# fill the runner pool ahead of everyone else. Queued jobs start in the order they began waiting; GitHub -# keeps up to 100 pending per queue and cancels arrivals past that. Every job -# has a timeout so a hung job cannot hold its queue for hours. Bot-authored -# PRs and non-PR events key on the run ID and never wait on a queue. +# New pushes cancel older runs for the same pull request. Each job queues by +# workflow, job, and author, so one author runs one copy of each job at a time. +# GitHub orders jobs by when they start waiting, but does not guarantee FIFO. +# Each queue holds up to 100 pending jobs; later arrivals are canceled. +# Timeouts release queues after hung jobs. Bots and non-PR events use unique +# run IDs and do not queue together. concurrency: group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.run_id }} cancel-in-progress: true diff --git a/.github/workflows/format.yml b/.github/workflows/format.yml index 5e80456a57..2398eb8a91 100644 --- a/.github/workflows/format.yml +++ b/.github/workflows/format.yml @@ -8,13 +8,12 @@ on: permissions: contents: read -# A newer push to a pull request cancels that PR's superseded run. Each job -# then joins a per-author queue for that job (its own `concurrency` block), so -# one person opening many PRs runs one copy of each job at a time and cannot -# fill the runner pool ahead of everyone else. Queued jobs start in the order they began waiting; GitHub -# keeps up to 100 pending per queue and cancels arrivals past that. Every job -# has a timeout so a hung job cannot hold its queue for hours. Bot-authored -# PRs and non-PR events key on the run ID and never wait on a queue. +# New pushes cancel older runs for the same pull request. Each job queues by +# workflow, job, and author, so one author runs one copy of each job at a time. +# GitHub orders jobs by when they start waiting, but does not guarantee FIFO. +# Each queue holds up to 100 pending jobs; later arrivals are canceled. +# Timeouts release queues after hung jobs. Bots and non-PR events use unique +# run IDs and do not queue together. concurrency: group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.run_id }} cancel-in-progress: true @@ -57,9 +56,22 @@ jobs: check-ci-node-pin: name: CI Node pin guard runs-on: ubuntu-latest + timeout-minutes: 5 + concurrency: + group: ${{ github.workflow }}-check-ci-node-pin-${{ github.event_name == 'pull_request' && github.event.pull_request.user.type != 'Bot' && github.event.pull_request.user.login || github.run_id }} + queue: max steps: - uses: actions/checkout@v4 + - name: Install yq + shell: bash + run: | + curl --fail --location --silent --show-error \ + https://github.com/mikefarah/yq/releases/download/v4.54.1/yq_linux_amd64 \ + --output "$RUNNER_TEMP/yq" + chmod +x "$RUNNER_TEMP/yq" + echo "$RUNNER_TEMP" >> "$GITHUB_PATH" + # The guard's tests check the tree before the guard landed; fetch only # that commit instead of the full history. - name: Fetch the pre-guard commit @@ -70,7 +82,7 @@ jobs: with: node-version-file: .tool-versions - - name: Test the guard + - name: Test the Node pin guard run: node --test scripts/check-ci-node-pin.test.mjs # Every job that compiles trusted-server-core also runs the TSJS build, so @@ -78,6 +90,12 @@ jobs: - name: Verify jobs pin Node before compiling Rust run: node scripts/check-ci-node-pin.mjs + - name: Test the CI concurrency policy guard + run: node --test scripts/check-ci-concurrency.test.mjs + + - name: Require per-PR cancellation, per-author job queues and timeouts + run: scripts/check-ci-concurrency.sh + format-rust: name: cargo fmt runs-on: ubuntu-latest @@ -88,48 +106,6 @@ jobs: steps: - uses: actions/checkout@v4 - # A hung job, a PR-triggered workflow without per-PR cancellation, or a - # job outside its per-author queue brings back the runner pile-up that - # the concurrency blocks prevent. Every PR-triggered workflow must carry - # test.yml's workflow-level block, and every job must set timeout-minutes - # and a per-author queue (keyed on the matrix too, for matrix jobs). - # Jobs that call a reusable workflow are skipped: they cannot set these. - - name: Require per-PR cancellation, per-author job queues and timeouts - shell: bash - run: | - reference=.github/workflows/test.yml - if [ "$(yq '.concurrency."cancel-in-progress" == true and (.concurrency.group // "" | test("pull_request\.number"))' "$reference")" != true ]; then - echo "::error file=$reference::test.yml must cancel superseded runs per pull request." - exit 1 - fi - expected=$(yq -o=json -I=0 '.concurrency' "$reference") - status=0 - for f in .github/workflows/*.yml .github/workflows/*.yaml; do - [ -f "$f" ] || continue - on_pr=$(yq '.on | ((tag == "!!map" and has("pull_request")) or (tag == "!!str" and . == "pull_request") or (tag == "!!seq" and ([.[] | select(. == "pull_request")] | length > 0)))' "$f") - [ "$on_pr" = true ] || continue - if [ "$(yq -o=json -I=0 '.concurrency' "$f")" != "$expected" ]; then - echo "::error file=$f::PR-triggered workflows must use the workflow-level concurrency block from test.yml." - status=1 - fi - while read -r job; do - [ -n "$job" ] || continue - echo "::error file=$f::Job '$job' must set timeout-minutes." - status=1 - done < <(yq '.jobs | to_entries[] | select(.value.uses == null and .value["timeout-minutes"] == null) | .key' "$f") - while read -r job; do - [ -n "$job" ] || continue - echo "::error file=$f::Job '$job' must join a per-author queue: concurrency.group keyed on pull_request.user.login and queue: max." - status=1 - done < <(yq '.jobs | to_entries[] | select(.value.uses == null and (.value.concurrency.queue != "max" or (.value.concurrency.group // "" | test("pull_request\.user\.login") | not))) | .key' "$f") - while read -r job; do - [ -n "$job" ] || continue - echo "::error file=$f::Matrix job '$job' must include the matrix in its concurrency group so its legs do not queue behind each other." - status=1 - done < <(yq '.jobs | to_entries[] | select(.value.strategy.matrix != null and (.value.concurrency.group // "" | test("matrix\.") | not)) | .key' "$f") - done - exit "$status" - # Rust jobs must set up the toolchain through ./.github/actions/setup-rust, # which removes the image's stable and only saves caches from main. A # direct setup-rust-toolchain or rust-cache step skips both silently. diff --git a/.github/workflows/integration-tests.yml b/.github/workflows/integration-tests.yml index 2956834fbb..b1d1262750 100644 --- a/.github/workflows/integration-tests.yml +++ b/.github/workflows/integration-tests.yml @@ -10,13 +10,12 @@ on: types: [opened, synchronize, reopened] workflow_dispatch: -# A newer push to a pull request cancels that PR's superseded run. Each job -# then joins a per-author queue for that job (its own `concurrency` block), so -# one person opening many PRs runs one copy of each job at a time and cannot -# fill the runner pool ahead of everyone else. Queued jobs start in the order they began waiting; GitHub -# keeps up to 100 pending per queue and cancels arrivals past that. Every job -# has a timeout so a hung job cannot hold its queue for hours. Bot-authored -# PRs and non-PR events key on the run ID and never wait on a queue. +# New pushes cancel older runs for the same pull request. Each job queues by +# workflow, job, and author, so one author runs one copy of each job at a time. +# GitHub orders jobs by when they start waiting, but does not guarantee FIFO. +# Each queue holds up to 100 pending jobs; later arrivals are canceled. +# Timeouts release queues after hung jobs. Bots and non-PR events use unique +# run IDs and do not queue together. concurrency: group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.run_id }} cancel-in-progress: true diff --git a/.github/workflows/pbs-example.yml b/.github/workflows/pbs-example.yml index d61072867a..0282e5a066 100644 --- a/.github/workflows/pbs-example.yml +++ b/.github/workflows/pbs-example.yml @@ -17,13 +17,12 @@ on: permissions: contents: read -# A newer push to a pull request cancels that PR's superseded run. Each job -# then joins a per-author queue for that job (its own `concurrency` block), so -# one person opening many PRs runs one copy of each job at a time and cannot -# fill the runner pool ahead of everyone else. Queued jobs start in the order they began waiting; GitHub -# keeps up to 100 pending per queue and cancels arrivals past that. Every job -# has a timeout so a hung job cannot hold its queue for hours. Bot-authored -# PRs and non-PR events key on the run ID and never wait on a queue. +# New pushes cancel older runs for the same pull request. Each job queues by +# workflow, job, and author, so one author runs one copy of each job at a time. +# GitHub orders jobs by when they start waiting, but does not guarantee FIFO. +# Each queue holds up to 100 pending jobs; later arrivals are canceled. +# Timeouts release queues after hung jobs. Bots and non-PR events use unique +# run IDs and do not queue together. concurrency: group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.run_id }} cancel-in-progress: true diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index 173b83d05b..0642b6a959 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -8,13 +8,12 @@ on: branches: [main] pull_request: -# A newer push to a pull request cancels that PR's superseded run. Each job -# then joins a per-author queue for that job (its own `concurrency` block), so -# one person opening many PRs runs one copy of each job at a time and cannot -# fill the runner pool ahead of everyone else. Queued jobs start in the order they began waiting; GitHub -# keeps up to 100 pending per queue and cancels arrivals past that. Every job -# has a timeout so a hung job cannot hold its queue for hours. Bot-authored -# PRs and non-PR events key on the run ID and never wait on a queue. +# New pushes cancel older runs for the same pull request. Each job queues by +# workflow, job, and author, so one author runs one copy of each job at a time. +# GitHub orders jobs by when they start waiting, but does not guarantee FIFO. +# Each queue holds up to 100 pending jobs; later arrivals are canceled. +# Timeouts release queues after hung jobs. Bots and non-PR events use unique +# run IDs and do not queue together. concurrency: group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.run_id }} cancel-in-progress: true @@ -415,6 +414,10 @@ jobs: name: ts dev lint domains (changed lines) if: github.event_name == 'pull_request' runs-on: ubuntu-latest + timeout-minutes: 10 + concurrency: + group: ${{ github.workflow }}-lint-domains-${{ github.event_name == 'pull_request' && github.event.pull_request.user.type != 'Bot' && github.event.pull_request.user.login || github.run_id }} + queue: max steps: # A PR checkout is GitHub's merge commit; depth 2 also fetches its # parents, so `HEAD^1` is the exact base commit the PR merges into. diff --git a/AGENTS.md b/AGENTS.md index 7688c1829f..28f815c938 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -375,6 +375,7 @@ Every PR must pass: 11. Documentation proposal scripts: `shellcheck scripts/docs-proposal/*.sh && node --test scripts/docs-proposal/hunks.test.mjs && scripts/docs-proposal/test.sh` 12. URL-host linter on lines the PR adds (`lint-domains` job, PRs only); run it locally with `cargo run --manifest-path crates/trusted-server-cli/Cargo.toml --target --bin ts -- dev lint domains --changed-vs origin/main`, using the PR's base branch in place of `main` when it targets another branch 13. CI Node pin guard: `node --test scripts/check-ci-node-pin.test.mjs && node scripts/check-ci-node-pin.mjs` with the `.tool-versions` Node (every CI job that compiles Rust sets up the `.tool-versions` Node before its first cargo step) +14. CI concurrency policy guard: `node --test scripts/check-ci-concurrency.test.mjs && scripts/check-ci-concurrency.sh` (requires yq v4); every PR-triggered workflow must use `test.yml`'s per-PR cancellation block, and every non-reusable job must set a timeout and a per-author `queue: max` group. Matrix jobs must also include their matrix values in the group. CI also runs these HTML EOF checks. Run them locally only when changing EOF finalization, publisher or HTML-processor code: diff --git a/scripts/README.md b/scripts/README.md index 21fee51f52..c86cad849a 100644 --- a/scripts/README.md +++ b/scripts/README.md @@ -8,6 +8,7 @@ their documented orchestration handles a probe deliberately. | `batch-sync.sh` | Endpoint, bearer token, EC ID, partner UID; `curl` and Python | Sends one authenticated batch-sync request. Its temporary response file is removed on exit. | | `benchmark.sh` | A running server; `curl`, `bc`, and optionally `hey` | May install `hey` through Homebrew. `--save` writes under `benchmark-results/`; it does not stop the server. | | `check-ci-node-pin.mjs` | Pinned Node; tests in `check-ci-node-pin.test.mjs` | Read-only. Fails when a CI job compiles Rust before an unconditional `actions/setup-node` step pinned to `.tool-versions`; `--root` checks another checkout. | +| `check-ci-concurrency.sh` | yq v4; tests in `check-ci-concurrency.test.mjs` | Read-only. Checks per-PR workflow cancellation, per-job author queues, matrix groups, and timeouts; `--root` checks another checkout and `--yq` selects yq. | | `profile.sh` | Fastly CLI, Rust WASM target, `curl`; endpoint and request options | Builds and starts the Fastly app, stops the owned process, and retains a profile under `benchmark-results/profiles/`. `--open` launches the local viewer. | | `generate-integration-viceroy-configs.sh` | Rust toolchain; optional origin port and artifact directory | Builds the native generator and writes Viceroy config under `target/integration-test-artifacts/`; generated files persist. | | `integration-tests.sh` | Docker, Viceroy, Rust WASM target, pinned Node | Builds WASM/native artifacts and two Docker images, generates Viceroy config, and runs native integration tests serially. Build products and images persist. | diff --git a/scripts/check-ci-concurrency.sh b/scripts/check-ci-concurrency.sh new file mode 100755 index 0000000000..36d3a06b89 --- /dev/null +++ b/scripts/check-ci-concurrency.sh @@ -0,0 +1,164 @@ +#!/usr/bin/env bash +set -euo pipefail + +script_dir=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) +repo_root=$(cd "$script_dir/.." && pwd) +root=$repo_root +yq_bin=${YQ_BIN:-yq} + +usage() { + printf 'usage: check-ci-concurrency.sh [--root ROOT] [--yq PATH]\n' >&2 +} + +while (($#)); do + case "$1" in + --root) + [[ $# -ge 2 ]] || { usage; exit 2; } + root=$2 + shift 2 + ;; + --yq) + [[ $# -ge 2 ]] || { usage; exit 2; } + yq_bin=$2 + shift 2 + ;; + *) + usage + exit 2 + ;; + esac +done + +workflow_dir=$root/.github/workflows +reference=$workflow_dir/test.yml +if [[ ! -d "$workflow_dir" || ! -f "$reference" ]]; then + printf 'error: expected workflow directory and test.yml under %s\n' "$root" >&2 + exit 1 +fi +if ! command -v "$yq_bin" >/dev/null 2>&1; then + printf 'error: yq v4 is required to check workflow policies\n' >&2 + exit 1 +fi + +workflow_prefix='${{ github.workflow }}' +expected_workflow_group="${workflow_prefix}-\${{ github.event.pull_request.number || github.run_id }}" +expected_author_suffix="\${{ github.event_name == 'pull_request' && github.event.pull_request.user.type != 'Bot' && github.event.pull_request.user.login || github.run_id }}" +matrix_prefix="\${{ matrix." +matrix_suffix=' }}-' +matrix_key_pattern='^[[:alnum:]_.-]+$' + +status=0 + +report_error() { + printf '::error file=%s::%s\n' "$1" "$2" + status=1 +} + +has_pull_request_trigger() { + "$yq_bin" -r ' + .on + | ((tag == "!!map" and has("pull_request")) + or (tag == "!!str" and . == "pull_request") + or (tag == "!!seq" and ([.[] | select(. == "pull_request")] | length > 0))) + ' "$1" +} + +job_value() { + local file=$1 + local job=$2 + local expression=$3 + YQ_JOB=$job "$yq_bin" -r ".jobs[strenv(YQ_JOB)].${expression}" "$file" +} + +validate_matrix_segment() { + local remaining=$1 + local matrix_key + + while [[ "$remaining" == "$matrix_prefix"* ]]; do + remaining=${remaining#"$matrix_prefix"} + [[ "$remaining" == *"$matrix_suffix"* ]] || return 1 + matrix_key=${remaining%%"$matrix_suffix"*} + [[ "$matrix_key" =~ $matrix_key_pattern ]] || return 1 + remaining=${remaining#*"$matrix_suffix"} + done + + [[ -z "$remaining" ]] +} + +validate_job_group() { + local file=$1 + local job=$2 + local group=$3 + local has_matrix=$4 + local prefix="${workflow_prefix}-${job}-" + local remainder + local matrix_segment + + if [[ "$group" != "$prefix"* ]]; then + report_error "$file" "Job '$job' concurrency group must start with the workflow name and its own job ID." + return + fi + remainder=${group#"$prefix"} + if [[ "$remainder" != *"$expected_author_suffix" ]]; then + report_error "$file" "Job '$job' concurrency group must use the pull-request author for human PRs and the run ID otherwise." + return + fi + matrix_segment=${remainder%"$expected_author_suffix"} + + if [[ "$has_matrix" == true ]]; then + if [[ -z "$matrix_segment" ]] || ! validate_matrix_segment "$matrix_segment"; then + report_error "$file" "Matrix job '$job' concurrency group must include each matrix leg before the per-author suffix." + fi + elif [[ -n "$matrix_segment" ]]; then + report_error "$file" "Non-matrix job '$job' concurrency group must not add a shared queue segment." + fi +} + +check_workflow() { + local file=$1 + local is_pr + local group + local cancel + local queue + local job + local timeout + local job_group + local job_queue + local has_matrix + + is_pr=$(has_pull_request_trigger "$file") + [[ "$is_pr" == true ]] || return 0 + + group=$("$yq_bin" -r '.concurrency.group // ""' "$file") + cancel=$("$yq_bin" -r '.concurrency."cancel-in-progress" // false' "$file") + queue=$("$yq_bin" -r '.concurrency.queue // ""' "$file") + if [[ "$group" != "$expected_workflow_group" || "$cancel" != true || "$queue" == max ]]; then + report_error "$file" "PR workflow concurrency group must be per-PR cancellation from test.yml, without queue: max." + fi + + while IFS= read -r job; do + [[ -n "$job" ]] || continue + if [[ "$(job_value "$file" "$job" "uses")" != null ]]; then + continue + fi + + timeout=$(job_value "$file" "$job" '"timeout-minutes" // 0') + if [[ ! "$timeout" =~ ^[1-9][0-9]*$ ]]; then + report_error "$file" "Job '$job' must set a positive timeout-minutes value." + fi + + job_group=$(job_value "$file" "$job" "concurrency.group") + job_queue=$(job_value "$file" "$job" "concurrency.queue") + if [[ "$job_queue" != max ]]; then + report_error "$file" "Job '$job' must set queue: max on its per-author concurrency group." + fi + has_matrix=$(job_value "$file" "$job" "strategy.matrix != null") + validate_job_group "$file" "$job" "$job_group" "$has_matrix" + done < <("$yq_bin" -r '.jobs | to_entries[] | select(.value.uses == null) | .key' "$file") +} + +while IFS= read -r -d '' workflow_file; do + check_workflow "$workflow_file" +done < <(find "$workflow_dir" -type f \( -name '*.yml' -o -name '*.yaml' \) -print0) + +exit "$status" diff --git a/scripts/check-ci-concurrency.test.mjs b/scripts/check-ci-concurrency.test.mjs new file mode 100644 index 0000000000..e69d0c8b25 --- /dev/null +++ b/scripts/check-ci-concurrency.test.mjs @@ -0,0 +1,233 @@ +import assert from 'node:assert/strict' +import { mkdtempSync, mkdirSync, rmSync, writeFileSync } from 'node:fs' +import os from 'node:os' +import path from 'node:path' +import { spawnSync } from 'node:child_process' +import { fileURLToPath } from 'node:url' +import test from 'node:test' + +const repositoryRoot = path.resolve( + path.dirname(fileURLToPath(import.meta.url)), + '..' +) +const guardPath = path.join(repositoryRoot, 'scripts/check-ci-concurrency.sh') +const workflowPrefix = '$' + '{{ github.workflow }}' +const workflowGroup = + workflowPrefix + + '-' + + '$' + + '{{ github.event.pull_request.number || github.run_id }}' +const authorGroupSuffix = + '$' + + "{{ github.event_name == 'pull_request' && github.event.pull_request.user.type != 'Bot' && github.event.pull_request.user.login || github.run_id }}" + +function authorGroup(job, matrix = '') { + return [workflowPrefix, job, matrix, authorGroupSuffix] + .filter(Boolean) + .join('-') +} + +function workflow(job, group, options = {}) { + const lines = [ + 'name: ' + job, + 'on:', + ' pull_request:', + 'concurrency:', + ' group: ' + workflowGroup, + ' cancel-in-progress: true', + 'jobs:', + ' ' + job + ':', + ' runs-on: ubuntu-latest', + ] + if (options.timeout !== false) lines.push(' timeout-minutes: 5') + if (options.matrix) { + lines.push(' strategy:', ' matrix:', ' os: [ubuntu-latest]') + } + lines.push(' concurrency:', ' group: ' + group) + if (options.queue !== false) lines.push(' queue: max') + lines.push(' steps:', ' - run: echo test') + return lines.join('\n') + '\n' +} + +function fixtureRoot() { + const root = mkdtempSync(path.join(os.tmpdir(), 'check-ci-concurrency-')) + const workflowsPath = path.join(root, '.github/workflows') + mkdirSync(workflowsPath, { recursive: true }) + writeFileSync( + path.join(workflowsPath, 'test.yml'), + workflow('test-rust', authorGroup('test-rust')) + ) + writeFileSync( + path.join(workflowsPath, 'format.yml'), + workflow('format-rust', authorGroup('format-rust')) + ) + writeFileSync( + path.join(workflowsPath, 'push-only.yml'), + [ + 'name: Push only', + 'on:', + ' push:', + 'jobs:', + ' build:', + ' runs-on: ubuntu-latest', + ' steps:', + ' - run: echo test', + '', + ].join('\n') + ) + writeFileSync( + path.join(workflowsPath, 'reusable.yml'), + [ + 'name: Reusable caller', + 'on:', + ' pull_request:', + 'concurrency:', + ' group: ' + workflowGroup, + ' cancel-in-progress: true', + 'jobs:', + ' call:', + ' uses: example/reusable/.github/workflows/test.yml@main', + '', + ].join('\n') + ) + return { root, workflowsPath } +} + +function runGuard(root) { + return spawnSync('bash', [guardPath, '--root', root], { encoding: 'utf8' }) +} + +function withFixture(callback) { + const fixture = fixtureRoot() + try { + callback(fixture) + } finally { + rmSync(fixture.root, { recursive: true, force: true }) + } +} + +function assertRejected(root, expectedMessage) { + const result = runGuard(root) + assert.equal( + result.error, + undefined, + 'should start the guard: ' + result.error + ) + assert.notEqual( + result.status, + 0, + 'should reject the policy:\n' + result.stdout + '\n' + result.stderr + ) + assert.match(result.stdout + result.stderr, expectedMessage) +} + +test('accepts per-PR cancellation and distinct per-author job queues', () => { + withFixture(({ root }) => { + const result = runGuard(root) + assert.equal( + result.error, + undefined, + 'should start the guard: ' + result.error + ) + assert.equal(result.status, 0, result.stdout + '\n' + result.stderr) + }) +}) + +test('rejects author groups that use the actor or queue bot PRs', () => { + withFixture(({ root, workflowsPath }) => { + const formatPath = path.join(workflowsPath, 'format.yml') + const source = workflow('format-rust', authorGroup('format-rust')) + writeFileSync( + formatPath, + source.replace('pull_request.user.login', 'github.actor') + ) + assertRejected(root, /concurrency group/i) + writeFileSync(formatPath, source.replace("user.type != 'Bot' && ", '')) + assertRejected(root, /concurrency group/i) + }) +}) + +test('rejects a per-author group without the non-PR run-ID fallback', () => { + withFixture(({ root, workflowsPath }) => { + const formatPath = path.join(workflowsPath, 'format.yml') + const group = authorGroup('format-rust').replace(' || github.run_id', '') + writeFileSync(formatPath, workflow('format-rust', group)) + assertRejected(root, /concurrency group/i) + }) +}) + +test('rejects copied job groups and groups without the workflow prefix', () => { + withFixture(({ root, workflowsPath }) => { + const formatPath = path.join(workflowsPath, 'format.yml') + writeFileSync(formatPath, workflow('format-rust', authorGroup('test-rust'))) + assertRejected(root, /concurrency group/i) + writeFileSync( + formatPath, + workflow( + 'format-rust', + authorGroup('format-rust').replace(workflowPrefix + '-', '') + ) + ) + assertRejected(root, /concurrency group/i) + }) +}) + +test('requires matrix jobs to separate their legs in the author queue', () => { + withFixture(({ root, workflowsPath }) => { + const formatPath = path.join(workflowsPath, 'format.yml') + writeFileSync( + formatPath, + workflow( + 'format-rust', + authorGroup('format-rust', '$' + '{{ matrix.os }}'), + { + matrix: true, + } + ) + ) + const passing = runGuard(root) + assert.equal( + passing.error, + undefined, + 'should start the guard: ' + passing.error + ) + assert.equal(passing.status, 0, passing.stdout + '\n' + passing.stderr) + + writeFileSync( + formatPath, + workflow('format-rust', authorGroup('format-rust'), { matrix: true }) + ) + assertRejected(root, /matrix job/i) + }) +}) + +test('requires every non-reusable PR job to set a timeout and queue max', () => { + withFixture(({ root, workflowsPath }) => { + const formatPath = path.join(workflowsPath, 'format.yml') + writeFileSync( + formatPath, + workflow('format-rust', authorGroup('format-rust'), { timeout: false }) + ) + assertRejected(root, /timeout-minutes/) + writeFileSync( + formatPath, + workflow('format-rust', authorGroup('format-rust'), { queue: false }) + ) + assertRejected(root, /queue: max/) + }) +}) + +test('requires the exact per-PR cancellation group on every PR workflow', () => { + withFixture(({ root, workflowsPath }) => { + const formatPath = path.join(workflowsPath, 'format.yml') + const wrongGroup = workflow( + 'format-rust', + authorGroup('format-rust') + ).replace( + 'github.event.pull_request.number || github.run_id', + 'github.actor' + ) + writeFileSync(formatPath, wrongGroup) + assertRejected(root, /workflow concurrency group/) + }) +}) From bb8a85e0fe23653d6b2a1283268303f64947f5a2 Mon Sep 17 00:00:00 2001 From: Christian Date: Fri, 30 Oct 2026 10:51:36 -0500 Subject: [PATCH 185/194] Record served HTML SDK ownership and recognition limits --- CHANGELOG.md | 3 +- .../src/integrations/lockr.rs | 12 ++-- .../src/integrations/permutive.rs | 12 ++-- .../src/integrations/sdk_url_test_support.rs | 31 ++++++---- .../lib/test/fixtures/sdk-url-ownership.json | 3 +- ...026-10-16-1199-sdk-url-ownership-design.md | 56 ++++++++++++------- 6 files changed, 68 insertions(+), 49 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 23cffe0c63..95bf2e4d77 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -37,8 +37,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Fixed -- Lockr and Permutive SDK recognition now uses parsed vendor hostnames and SDK pathnames instead of searching the whole URL (#1199). The browser Lockr guard no longer replaces unrelated scripts that mention a Lockr host in their path or query, use a lookalike host, or serve a non-SDK resource from a Lockr host. Query strings and fragments no longer prevent SDK recognition in the server matchers, the browser Permutive guard, or the browser Lockr guard on `identity.loc.kr`. Matching URLs use the existing configured-SDK routes; their original query and fragment are not forwarded upstream. Server matching trims leading and trailing C0 controls and spaces, removes tab, CR, and LF, and accepts backslashes as authority separators. Literal NULs remain rejected. - +- Lockr and Permutive SDK recognition now uses parsed vendor hostnames and SDK pathnames instead of searching the whole URL (#1199). The browser Lockr guard no longer replaces unrelated scripts that mention a Lockr host in their path or query, use a lookalike host, or serve a non-SDK resource from a Lockr host. Query strings and fragments no longer prevent SDK recognition in the server matchers, the browser Permutive guard, or the browser Lockr guard on `identity.loc.kr`. Matching URLs use the existing configured-SDK routes; their original query and fragment are not forwarded upstream. Server matching trims leading and trailing C0 controls and spaces, removes tab, CR, and LF, and accepts backslashes as authority separators. It no longer recognizes attribute values that contain a NUL or that write slashes as character references such as `/`, so markup like that can load the SDK from the vendor host. - `rewrite.exclude_domains` matching is now case-insensitive and ignores surrounding whitespace. Entries written with uppercase letters previously never matched and now take effect, so those hosts stop being proxied and click-wrapped, and `/first-party/sign` now rejects them with `502`; it used to sign them, or return `403` when the host was also outside `proxy.allowed_domains`. Empty and bare `"*"` `exclude_domains` entries, which never matched a host, are dropped at load with a warning, and other entries that can never match a host are logged with a warning. Absolute `http(s)` creative URLs that cannot be parsed (for example, a host containing a space) are now left untouched; previously the trimmed value was written back re-quoted (with `https:` prepended to `//` input, so `src=" //exa mple.example/x "` became `https://exa mple.example/x`), an unparseable quoted URL in `