Skip to content

perf(audit-logs): read a listing narrowed only by time through per-workspace keyset scans - #9450

Merged
waleedlatif1 merged 2 commits into
stagingfrom
perf/audit-log-organization-scope
Oct 29, 2026
Merged

waleedlatif1 merged 2 commits into
stagingfrom
perf/audit-log-organization-scope

Conversation

@waleedlatif1

Copy link
Copy Markdown
Collaborator

Summary

An organization's audit listing scopes rows with one OR predicate (the org's workspaces, or workspace-less rows naming the org). With no filter, or only a time range (the audit page's default "Past 30 days" view), no index returns that OR in listing order, so each page collects every in-scope row inside the range and sorts it. For a busy organization that becomes a scan over a large part of audit_log, long enough to hit replica recovery conflicts.

This PR changes only those listings, the ones narrowed by nothing but a time range:

  • The page ids come from a disjoint union. Each organization workspace gets one keyset scan of audit_log_workspace_created_at_id_idx, through unnest(...) cross join lateral (... limit page). The metadata-attributed org-level rows are read through audit_log_org_level_created_at_id_idx (0444). The organization-resource rows that don't name the org in metadata get a small resource_idx read. Every part is ordered and limited to one page, so the first page of the union is exactly the first page of the scope, and union all can't produce duplicates because the parts are disjoint. Workspace ids are deduplicated.
  • The time range is kept as written (created_at >= / <=). It is also stated as implied bounds on the sort key (date_trunc('milliseconds', created_at) >= / <= date_trunc(bound)). Because date_trunc never decreases, these bounds drop no row the exact predicate keeps, and each scan starts and stops at the range.
  • The cursor is the same keyset position, written as a row comparison so the scan can seek to it.

Any other filter (action, resource type/id, actor, actor email, workspace, search, alone or combined with time) keeps the single predicate unchanged, SQL and parameters byte for byte. Under such a filter a per-workspace scan cannot stop after a page; that is what made #9419's fan-out slow, and #9438 reverted it. The single predicate lets the filter's own index drive.

queryAuditLogs now takes the scope and the filters instead of prebuilt SQL so it can make this choice. Callers updated: list and export use cases, /api/v1/audit-logs. buildOrgScopeCondition output is unchanged, so the detail read and the data drain are untouched.

No schema change. audit_log_workspace_created_at_id_idx stays: this path depends on it, and workspace-filtered listings already use it today.

Type of Change

  • Other: performance

Testing

  • list-scope.integration.ts pages through 13 filter shapes (none, start, end inside a millisecond, range, action, action+time, resource type(s), resource, actor, workspace, search). Each runs with current members only and with departed members included, at page sizes 1/3/7/100, with and without the byte budget. Every run must match the single predicate's order exactly. The fixture covers workspace rows, org-level rows of both kinds, another org's rows, departed and null actors, four rows per PostgreSQL millisecond (ties broken by id), a duplicated workspace id, and a workspace the organization no longer holds.
  • list-plans.integration.ts checks the plans. Unfiltered, departed-included, after-cursor, start-date and date-range listings must read each workspace with an Index Scan Backward on the keyset index directly under the limit that stops it, with the range and cursor in its Index Cond. These checks fail on the previous single-predicate listing. Every narrowed shape must keep the single predicate (no fan-out).
  • Mutation checks: dropping the workspace-id dedupe, dropping the exact time predicate, or routing everything back to the single predicate each turns these tests red.
  • A one-off compile of every narrowed filter combination, with and without a cursor, across three scopes produced SQL text and parameters identical to staging.
  • On the production replica, the busiest organization's unfiltered and 30-day listings go from a scan of the whole scope to a few thousand buffer hits. A one-hour window costs about the same as today. Organizations with many workspaces stay bounded at one short scan per workspace.
  • Audit-log and data-drain integration suites, related unit suites, lint, check:audits, type-check, test:scripts.

Checklist

  • Code follows project style guidelines
  • Self-reviewed my changes
  • Tests added/updated and passing
  • No new warnings introduced
  • I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

…rkspace keyset scans

An organization's audit listing is one OR predicate over the organization's workspaces and its
organization-level rows. With no filter, or only a time range (the audit page's default view),
PostgreSQL has no index that returns that OR in listing order, so it collects every row of the
scope inside the range and sorts it for each page; for a busy organization that is a scan of a
large part of the table.

Such a listing now reads its page ids as a disjoint union: one keyset scan of
audit_log_workspace_created_at_id_idx per organization workspace, the organization-level
metadata rows through audit_log_org_level_created_at_id_idx, and the organization resource rows
that do not name it in metadata. Each part stops after a page, and the first page of the union is
exactly the first page of the scope. The time range is kept verbatim and also stated as implied
bounds on the sort key, so each scan starts and stops at the range; the cursor is the same keyset
position written as a row comparison so the scan seeks to it.

A listing with any other filter (action, resource, actor, workspace, search) keeps the single
predicate unchanged, byte for byte: per-workspace scans cannot stop early under such a filter,
which is what made #9419 slow, while the single predicate lets the filter's own index drive.
queryAuditLogs now takes the scope and the filters rather than prebuilt SQL so it can make that
choice.

list-scope.integration.ts pages every filter shape, with and without departed members, across
page sizes and the byte budget, and requires exactly the single predicate's order, including
sub-millisecond ties, bounds inside a millisecond and a workspace the organization no longer
holds. list-plans.integration.ts pins the keyset plan for the unfiltered and time-bounded shapes
(failing on the previous single-predicate listing) and the single predicate for every other one.
@vercel

vercel Bot commented Oct 29, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated
docs Skipped Skipped Oct 29, 2026 6:40pm UTC

Request Review

@greptile-apps

greptile-apps Bot commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerView in GreptileConfidence Score: 5/5

[Medium impact] The PR appears safe to merge; no new actionable issue was found.

Summary

The PR uses per-workspace keyset scans for audit listings filtered only by time. Other filters keep the existing query.

  • Time-only audit-log pages load with short scans across each workspace.

The earlier unnumbered finding is addressed by the changed actor mix. waleedlatif1 explicitly declined a rows-read or buffer assertion, accepting plan-only checks because the intended contract is the scan's cost bound and citing production measurements.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Audit listing] --> B{Filter beyond time?}
  B -->|Yes| C[Existing single-scope query]
  B -->|No| D[Limited scan per workspace]
  B -->|No| E[Limited organization-level scans]
  D --> F[Combine and order page IDs]
  E --> F
  F --> G[Read page rows]
  C --> H[Return page and cursor]
  G --> H
Loading

Reviews (2) · Last reviewed commit: "test(audit-logs): plan the keyset listin..." · Reviewed by Greptile

Comment thread apps/sim/lib/audit-logs/list-plans.integration.ts Outdated
@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@greptile

@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@cubic-dev-ai review this PR

@cubic-dev-ai

cubic-dev-ai Bot commented Oct 29, 2026

Copy link
Copy Markdown
Contributor

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 9 files

Confidence score: 5/5

  • Automated review surfaced no issues in the provided summaries.
  • No files require special attention.

Turn on auto-fix | Re-trigger cubic

@waleedlatif1
waleedlatif1 merged commit d59a275 into staging Oct 29, 2026
68 checks passed
@waleedlatif1
waleedlatif1 deleted the perf/audit-log-organization-scope branch October 29, 2026 19:55

This branch was previously deployed

1 inactive deployment
Preview — 0c1f3e88 Deployed Oct 29, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant