Repository navigation
perf(db): serve workspace usage drill-downs index-only on usage_log - #9447
Conversation
Every Workspaces drill-down read filters on the billing entity, one workspace and a created_at window, and reads cost with user_id, source or workflow_id. usage_log_workspace_created_at_idx held only (workspace_id, created_at), so the planner matched the window there or on usage_log_billing_entity_created_at_cost_idx and fetched a heap row per match to check the rest: the workspace's rows on one, the whole organization's on the other. 0441 dropped the compact entity/period index that used to narrow the second through a BitmapAnd, so a large organization's workflow ranking went from about 4.6s to 13.7s, reading thousands of rows per row it returned. usage_log_workspace_created_at_cost_idx replaces it: the same leading (workspace_id, created_at), then the entity and every column the drill-down reads. Each drill-down read is now an index-only scan bounded by the workspace's rows in the window, and the readers that name only the workspace (the foreign key's ON DELETE SET NULL, the ledger list filtered to a workspace) keep the order they had. The new index is built concurrently before the old one is dropped, so inserts maintain the same number of indexes. The plan integration test explains the exact SQL of the totals, series and rankings for a workspace holding most and a small share of its organization's usage, and fails on the old schema.
|
The latest updates on your projects. Learn more about Vercel for GitHub. |
[Medium impact] The PR appears safe to merge; no actionable issues were found. SummaryReplaces the workspace usage index with a wider index that covers totals, series, and member, source, and workflow breakdowns.
Reviews (2) · Last reviewed commit: "perf(db): scope the workspace covering i..." · Reviewed by Greptile |
There was a problem hiding this comment.
All reported issues were addressed across 7 files
This PR changes a database schema or migrates data. Ultrareviews find 2.4x more serious bugs than standard reviews. Comment @cubic-dev-ai ultrareview to run one.
Turn on auto-fix | Re-trigger cubic
|
@cubic-dev-ai review this PR |
@waleedlatif1 I have started the AI code review. It will take a few minutes to complete. |
Summary
usage_log_workspace_created_at_idx(workspace_id, created_at)withusage_log_workspace_created_at_cost_idx(workspace_id, created_at, billing_entity_type, billing_entity_id, user_id, source, cost, workflow_id), built concurrently before the old one is dropped (0446, replay-safe in the 0441/0442 pattern)created_atwindow. With only(workspace_id, created_at)to work with, the planner either fetched a heap row per workspace row in the window, or scannedusage_log_billing_entity_created_at_cost_idxand fetched a heap row per organization row to checkworkspace_id. 0441 dropped the compact entity/period index that used to narrow the second through a BitmapAnd, so a large organization's workflow ranking went from ~4.6s to ~13.7s, reading thousands of rows per row returned. Each of these reads is now an index-only scan bounded by the workspace's rows in the windowworkspace_id, created_atstill lead, so the readers that name only the workspace keep their plans and order: theworkspace_idFK'sON DELETE SET NULL, and the ledger list filtered to a workspace (which now also checksuser_idin the index instead of on the heap)usage-analytics.ts/usage-analytics-queries.tsthat said a narrowed read heap-fetches per row are updatedWhy this shape
created_at, not before.(workspace_id, billing_entity_type, billing_entity_id, created_at)would bound the same drill-downs, but the readers without an entity predicate (FK cascade, workspace ledger list) would lose thecreated_atorder and bound. A workspace's rows billed to another entity (personal usage before it joined the org) are rare, and the index drops them without a heap fetch either wayusage_log_billing_entity_created_at_cost_idxinstead. Addingworkspace_id/workflow_idthere would make the drill-down scan the whole organization's window and slow the org-level totals and series, which are by far the heaviest readers of that indexINCLUDE. drizzle cannot declare one, anddrizzle generatemust stay in sync with the schema; trailing key columns serve the same scans (same as 0441)Left alone
user_id+ entity + window): rare, and already index-only onusage_log_billing_entity_created_at_cost_idx. The analogous change would widenusage_log_user_created_at_idx, the most-scanned index on the table, for little gaindescription/metadatafrom the heap, now only for the workspace's rowsType of Change
Testing
apps/sim/lib/billing/core/usage-analytics-plans.integration.ts: EXPLAINs the exact SQL of the totals, time series, source series, member series and workflow ranking narrowed to a workspace holding most and a small share of its organization's usage (with an unrelated organization and rows billed to another entity in the same workspace) and asserts an Index Only Scan of the new index bounded onworkspace_id+created_at. All 10 cases fail on the old schema; droppinguser_idandworkflow_idfrom the key fails exactly the member series and workflow ranking cases. Passes with bothpushandINTEGRATION_DB_PROVISION=migrateEXPLAIN(read-only) of the drill-down fingerprints: today they pick the narrow workspace index plus heap, a BitmapAnd withusage_log_workflow_id_idx, or the entity index with a heap filter onworkspace_id(row estimates are skewed for large organizations). The competing index-only scan bounded onworkspace_id+created_atcosts a fraction of each of thosebun run check:migrations,drizzle-kit generate(no changes),bun run lint,bun run check:audits,type-check(apps/sim, packages/db),usage-analytics.test.tsChecklist
test-auditauthoring gate)