Skip to content

perf(admin): read dashboard ledger usage one bounded scan per entity - #9448

Merged
waleedlatif1 merged 2 commits into
stagingfrom
perf/admin-dashboard-usage-reads
Oct 29, 2026
Merged

waleedlatif1 merged 2 commits into
stagingfrom
perf/admin-dashboard-usage-reads

Conversation

@waleedlatif1

@waleedlatif1 waleedlatif1 commented Oct 29, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

The admin dashboard computed each page's per-entity spend and workflow runs in one statement that OR'd every entity's period together. The planner answers that with a BitmapOr and a heap fetch for every matched ledger row, or with a scan over every organization entry in the period index, filtered row by row. Both run on the primary.

  • Each read is now a UNION ALL with one arm per entity. Every arm is bounded by that entity's period on the existing indexes: usage_log_billing_entity_period_cost_idx for stamped periods and usage_log_billing_entity_created_at_cost_idx for reporting periods.
  • Spend is index-only.
  • Workflow runs are a separate statement on the same indexes. No index carries execution_id, so this statement fetches each matched workflow row from the heap. Keeping it separate leaves the spend scans index-only. A dedicated index was rejected: it would add a write to every workflow ledger insert to speed up a staff-only page.
  • The four copies of the period predicate are consolidated. The org list, org summary/detail, historical actors and personal users views now use buildUsageAnalyticsScope, which already defines the reporting-vs-stamped period predicate for usage analytics.
  • The reads stay on the primary. The replica serves these index pages from disk rather than cache, and it cancels statements held past its replay delay (recovery conflicts). The largest entities' periods can reach that delay.

What the dashboard shows is unchanged. A workflow run is still a distinct execution_id among the entity's source = 'workflow' ledger rows in the period. Execution logs are not a substitute: they are subject to retention, they include runs with no billed lines, and they don't record which entity a run was billed to.

No schema change.

Type of Change

  • Bug fix
  • New feature
  • Breaking change
  • Documentation
  • Other: performance

Testing

  • dashboard-usage.integration.ts runs against real Postgres. It checks spend and run counts per entity, per user, filtered by user, and for historical actors. The seed covers:
    • multi-row executions
    • zero-cost runs
    • a run whose rows have two different actors
    • non-workflow sources and workflow-test
    • workflow rows with a null execution_id
    • rows stamped with the previous period
    • a resumed run stamped with a period that has since ended
    • rows billed to a user entity with the same id, or to no entity
    • reporting-period start/end boundaries
    • a run billed partly to another entity
    • duplicate windows
  • A local differential run (not committed) compared the new readers with the previous OR'd queries on 400k randomized ledger rows across 25 organizations and 25 user entities, mixing stamped and reporting periods. Results were identical for all four reads. Spend arms planned as Index Only Scans with zero heap fetches; run-count arms planned as Index Scans bounded per entity.
  • Against the production planner (plain EXPLAIN on the primary), for a large yearly reporting period plus a stamped period:
    • The previous statement scanned every organization entry in the period index and filtered it.
    • The spend arms are now Index Only Scans bounded by entity and period.
    • The run-count arms are Index Scans with entity, period and source in the index condition. There is no sequential scan anywhere.
  • check:migrations (no new migrations), drizzle-kit generate (no changes), check:audits, type-check, lint, and the lib/admin unit tests all pass.

Checklist

  • Code follows project style guidelines
  • Self-reviewed my changes
  • Tests added/updated and passing
  • No new warnings introduced
  • I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

The dashboard's per-entity usage was one statement OR-ing every entity's
period, which the planner can only answer with a BitmapOr and a heap fetch
per matched ledger row, and its count(distinct execution_id) needed the heap
anyway because no index carries execution_id.

Each read is now a UNION ALL with one arm per entity, so spend is an
index-only scan on the existing covering cost indexes, and workflow runs are
a separate statement on a new partial index over workflow rows that carries
execution_id. The four copies of the period predicate now come from
buildUsageAnalyticsScope. What the dashboard shows is unchanged: a run is
still a distinct execution_id among the entity's workflow rows in the period.

The reads stay on the primary: the replica serves these scans from disk and
cancels statements held past its replay delay.
@vercel

vercel Bot commented Oct 29, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated
docs Ignored Ignored Preview Oct 29, 2026 6:48pm UTC

Request Review

@greptile-apps

greptile-apps Bot commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerView in GreptileConfidence Score: 5/5

[Medium impact] The changes since the previous review appear safe to merge.

Summary

The dashboard now reads spend and workflow counts separately, with one UNION ALL arm per billing entity.

  • Dashboard usage reads split each entity's spend from its workflow-run count.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Admin dashboard] --> B[Shared ledger readers]
  B --> C[Spend query: one arm per entity]
  B --> D[Workflow count query: one arm per entity]
  C --> E[(Primary database)]
  D --> E
  E --> F[Combine spend and run counts]
  F --> A
Loading

Reviews (2) · Last reviewed commit: "perf(admin): count dashboard workflow ru..." · Reviewed by Greptile

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 8 files

Reply with feedback, questions, or to request a fix.

Turn on auto-fix | Re-trigger cubic

Comment thread packages/db/schema.ts Outdated
…exes

Drops the workflow-runs index and its migration: an extra write on every
workflow ledger insert is not worth paying for a staff-only page. Spend stays
index-only per entity; the separate run-count statement fetches the matched
workflow rows from the heap on the existing period and created_at indexes.
@waleedlatif1 waleedlatif1 changed the title perf(admin): read dashboard spend and workflow runs index-only perf(admin): read dashboard ledger usage one bounded scan per entity Oct 29, 2026
@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@greptile

@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@cubic-dev-ai review this PR

@cubic-dev-ai

cubic-dev-ai Bot commented Oct 29, 2026

Copy link
Copy Markdown
Contributor

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 4 files

Reply with feedback, questions, or to request a fix.

Turn on auto-fix | Re-trigger cubic

Comment thread apps/sim/lib/admin/dashboard-usage.ts
@waleedlatif1
waleedlatif1 merged commit 6a269f9 into staging Oct 29, 2026
69 checks passed
@waleedlatif1
waleedlatif1 deleted the perf/admin-dashboard-usage-reads branch October 29, 2026 19:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant