chore(search): retire legacy search knowledge base projection rows in place by waleedlatif1 · Pull Request #9511 · simstudioai/sim

waleedlatif1 · 2026-10-31T08:25:14Z

Summary

The shadow-copy Search retirement tool (retire-indexed-search.ts) can't practically finish:
- While its change log is past 10k rows, each page only reconciles, so the copy stops advancing.
- At 100-row pages, chunk churn outpaces it.
- This replaces it with an in-place retirement.
Migration 0449: CREATE OR REPLACE FUNCTION sync_embedding_search().
- A chunk write in a retired Search knowledge base (is_search_index) now deletes that chunk's candidate row instead of projecting it.
- Ordinary knowledge bases project exactly as before: same single KB lookup, INTO STRICT, 512-prefix rule and upsert.
- Takes no table lock. The embedding_search_sync trigger, its WHEN guard, the source/ACL trigger and the FK cascade are unchanged.
- So a dormant Search knowledge base named explicitly by id has no vector candidates: a vector search of it finds nothing, while hybrid search still finds its rows by keyword and decides access on the document. workspace-kb-document-access.integration.ts now asserts this.
Single source for the body: SYNC_EMBEDDING_SEARCH_FUNCTION in knowledge-projection.ts. Script migration 0016, and so fresh databases and db:push, install the same body. An integration test keeps the provisioned and 0449-installed bodies identical to it, through the same whitespace-insensitive matcher (isCurrentSyncEmbeddingSearchSource) the purge preflight uses.
- A dev db:push on a database that already recorded 0016 skips the backfill but still reinstalls the function, because push runs no versioned migration.
New operator script db:purge-search-projection:
- Dry run by default: a planner estimate plus an exact "any remaining" probe, with no counting scan.
- --execute deletes Search-KB candidate rows in bounded batches through embedding_search_kb_idx, deleting by ctid. Each batch locks its rows (FOR UPDATE) in one statement and deletes them in the next, so a concurrent update cannot move them away and leave a falsely empty batch. Each batch is its own short transaction under statement and lock timeouts, with a FOR SHARE re-check of is_search_index.
- Stops on any error. It's stateless, so a rerun resumes.
- Single-run advisory lock, which every batch re-checks: the client reconnects silently after a dropped connection, and the new session does not inherit the lock, so the run stops instead.
- Pins search_path to public, pg_temp. Requires sslmode=verify-full&sslrootcert=system in the URL, but enforces them as ssl: 'verify-full' and removes them before connecting. postgres.js otherwise forwards sslrootcert as a startup parameter, and the server refuses the connection (42704), which kept the old CLI from ever connecting.
- Refuses non-primary endpoints, and refuses to start while the old retirement job is still installed.
- Refuses --execute until migration 0449 is live: the live public.sync_embedding_search body must match the constant, and the embedding_search_sync trigger must fire for ordinary writes (not disabled or replica-only) and call that function.
- Never touches embedding or document.
Removes the old tool: maintenance module, health reader, CLI, package script and its tests. db:push keeps its migration fence.
contract-pending: the Search-KB delete branch in the function can go once those rows are purged and Search knowledge bases are unwritable.

Merge order: merge only after the stalled shadow-copy job has been aborted with the current release's tool. This PR deletes that tool.

Operator runbook

Abort any shadow-copy retirement job with the release that shipped it, then deploy this release. Wait for the deploy's migrate job to finish: --execute refuses until migration 0449 is live, and the dry run reports projectionRetired.
db:purge-search-projection dry run: confirm projectionRetired: true and that the listed knowledge bases are the retired Search ones.
--execute --max-batches <n> first, watching database latency, CPU, replica lag and WAL. Then rerun until the dry run reports none remaining.
Before the rebuild, check pg_stat_activity for an anti-wraparound autovacuum of the table (query LIKE '%embedding_search%(to prevent wraparound)%'), and wait while one runs.
From a direct primary session, off-peak:
```
SET maintenance_work_mem = '2GB';
SET statement_timeout = 0;
REINDEX INDEX CONCURRENTLY embedding_search_512_cosine_hnsw_idx;
VACUUM (ANALYZE) embedding_search;
```
If the REINDEX fails, run DROP INDEX CONCURRENTLY embedding_search_512_cosine_hnsw_idx_ccnew; and retry. Only the 512 HNSW index needs rebuilding: the retired Search knowledge bases use text-embedding-3-small, whose chunks project only vector_512. In rehearsal a larger maintenance_work_mem was OOM-killed, which is why the setting above is lower.

Rehearsal timings on a production-sized copy: purge ~27 min (batch p50 0.65s), REINDEX 30+ min, VACUUM ~5–6 min.

Type of Change

Other: maintenance / data retirement

Testing

New packages/db/scripts/purge-search-projection.integration.ts (9 tests, green locally on Postgres 17 + pgvector):
- Migration, provisioned and TS bodies are identical, and the triggers are unchanged.
- A Search-KB insert projects nothing, and an update deletes the existing row while canonical rows stay.
- Ordinary knowledge bases at every width, including the 512 prefix, binaries and ACL.
- The purge deletes only Search rows; rerun and batch-limit resume; it stops when a knowledge base flips mid-run; CLI behaviour.
- It stops when its connection is terminated between batches and the client reconnects without the lock.
- It deletes rows that a concurrent, then-committed update moved after the batch chose them (previously such a batch deleted 0 rows and reported the knowledge base finished).
- --execute refuses, deleting nothing, while the old function body is installed or the trigger is disabled, replica-only or calling another function, and proceeds once 0449 is applied. The dry run reports the same check.
- The CLI purges the public tables even with a database-level search_path pointing at a shadow schema.
New unit tests:
- Operator URL refusals.
- The client built from a verified primary URL carries no sslrootcert/sslmode startup parameter and uses ssl: 'verify-full'.
Reverting the Search branch in the constant or in the migration turns the parity and behaviour tests red. Reverting each review-round guard on its own (per-batch lock check, search_path pin, sslrootcert removal, dev-push function refresh) turns its test red.
0016 integration (11, including a dev push with 0016 already recorded reinstalling the function), push.integration, knowledge-projection.integration and search-index.integration pass locally.
check:migrations, drizzle-kit generate (no changes), bun run lint, check:audits, block registry check and type-check pass. The full test suite runs in CI.

Checklist

Code follows project style guidelines
Self-reviewed my changes
Tests added/updated and passing (new tests pass the test-audit authoring gate)
No new warnings introduced
I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

vercel · 2026-10-31T08:25:16Z

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated
docs	Skipped		Oct 31, 2026 6:35pm UTC

greptile-apps · 2026-10-31T08:28:45Z

Confidence Score: 5/5

[Medium impact] The reviewed changes appear safe to merge after the old shadow-copy job is aborted as required by the runbook.

Summary

Replaces shadow-copy retirement with an in-place purge of retired Search candidate rows.

Search knowledge bases stop adding vector candidates when chunks change.
Operators inspect and remove retired Search candidates in resumable batches.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Operator command] --> B{Execute requested?}
  B -->|No| C[Report estimates and remaining rows]
  B -->|Yes| D[Check primary, function, trigger, and old job]
  D --> E[Acquire purge lock]
  E --> F[Check lock and Search marker]
  F --> G[Lock a bounded batch]
  G --> H[Delete selected candidate rows and commit]
  H --> I{Rows deleted?}
  I -->|Yes| F
  I -->|No| J[Continue to next knowledge base]

_{Reviews (8) · Last reviewed commit: "fix(search): require the purge preflight..." · Reviewed by Greptile}

cubic-dev-ai

All reported issues were addressed across 22 files

_{This PR changes a database schema or migrates data. Ultrareviews find 2.4x more serious bugs than standard reviews. Comment @cubic-dev-ai ultrareview to run one.

Turn on auto-fix | Re-trigger cubic}

waleedlatif1 · 2026-10-31T08:41:13Z

@greptile

waleedlatif1 · 2026-10-31T08:41:14Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-10-31T08:41:18Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

No issues found across 23 files

Confidence score: 5/5

Automated review surfaced no issues in the provided summaries.
No files require special attention.

_{Turn on auto-fix | Re-trigger cubic}

waleedlatif1 · 2026-10-31T08:54:56Z

@greptile

waleedlatif1 · 2026-10-31T08:54:57Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-10-31T08:55:02Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

No issues found across 24 files

Confidence score: 5/5

Automated review surfaced no issues in the provided summaries.
No files require special attention.

_{Turn on auto-fix | Re-trigger cubic}

… place

… projection function - Re-check the session advisory lock inside every purge batch: the client reconnects silently after a dropped connection, and the new backend does not hold the old one's lock - Build the purge client without the sslmode/sslrootcert URL parameters, enforcing them as ssl verify-full; postgres.js forwards sslrootcert as a startup parameter the server rejects - Pin the purge connection's search_path to public, pg_temp - Reinstall sync_embedding_search() on a dev push that already recorded 0016, since push runs no versioned migration

A Search knowledge base's chunk writes now delete its candidate rows, so an explicit vector search of one finds nothing; hybrid search still decides access on the document

…fter staging's 0448

waleedlatif1 · 2026-10-31T17:37:05Z

@greptile

waleedlatif1 · 2026-10-31T17:37:06Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-10-31T17:37:13Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

No issues found across 24 files

Confidence score: 5/5

Automated review surfaced no issues in the provided summaries.
No files require special attention.

_{Turn on auto-fix | Re-trigger cubic}

…le id A concurrent update could move every chosen row to a new tuple id before the delete, so the batch deleted nothing and the purge reported the knowledge base finished with rows left. The batch now locks its rows in one statement and deletes them in the next, whose snapshot sees the current versions

waleedlatif1 · 2026-10-31T17:44:56Z

@greptile

waleedlatif1 · 2026-10-31T17:44:57Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-10-31T17:45:02Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

All reported issues were addressed across 24 files

_{You've manually re-run cubic several times on this PR. Each manual re-review checks the full PR again and counts toward your usage quota. To preserve your usage limits, we recommend letting cubic automatically review new commits.

Turn on auto-fix | Re-trigger cubic}

waleedlatif1 · 2026-10-31T18:00:13Z

@greptile

waleedlatif1 · 2026-10-31T18:00:14Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-10-31T18:00:21Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

No issues found across 24 files

Confidence score: 5/5

Automated review surfaced no issues in the provided summaries.
No files require special attention.

_{You've manually re-run cubic several times on this PR. Each manual re-review checks the full PR again and counts toward your usage quota. To preserve your usage limits, we recommend letting cubic automatically review new commits.

Turn on auto-fix | Re-trigger cubic}

- --execute checks that public.sync_embedding_search has the retiring body and its embedding_search_sync trigger is enabled; before that, chunk writes keep projecting the rows a purge deletes. The dry run reports the same check - One whitespace-insensitive matcher for the live function source, shared by the preflight and the integration parity tests; drop the unit test that compared the migration file text - --help adds the rebuild precautions: wait out an anti-wraparound autovacuum, bound maintenance_work_mem, clean up a failed concurrent reindex

waleedlatif1 · 2026-10-31T18:08:23Z

@greptile

waleedlatif1 · 2026-10-31T18:08:24Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-10-31T18:08:28Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

All reported issues were addressed across 23 files

_{You've manually re-run cubic several times on this PR. Each manual re-review checks the full PR again and counts toward your usage quota. To preserve your usage limits, we recommend letting cubic automatically review new commits.

Turn on auto-fix | Re-trigger cubic}

…he retiring function - embedding_search_sync must be enabled for ordinary writes (origin or always, not replica-only) and call public.sync_embedding_search - Drop the redundant client port option; postgres.js reads the URL port

waleedlatif1 · 2026-10-31T18:35:26Z

@greptile

waleedlatif1 · 2026-10-31T18:35:27Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-10-31T18:35:32Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

No issues found across 23 files

Confidence score: 5/5

Automated review surfaced no issues in the provided summaries.
No files require special attention.

_{You've manually re-run cubic several times on this PR. Each manual re-review checks the full PR again and counts toward your usage quota. To preserve your usage limits, we recommend letting cubic automatically review new commits.

Turn on auto-fix | Re-trigger cubic}

waleedlatif1 requested a review from a team as a code owner October 31, 2026 08:25

greptile-apps Bot reviewed Oct 31, 2026

View reviewed changes

Comment thread packages/db/scripts/purge-search-projection.ts Resolved

Comment thread packages/db/script-migrations/0016_backfill_search_vectors.ts Resolved

vercel Bot deployed to Preview October 31, 2026 08:30

View deployment

cubic-dev-ai Bot reviewed Oct 31, 2026

View reviewed changes

Comment thread packages/db/scripts/purge-search-projection.ts Resolved

Comment thread packages/db/scripts/purge-search-projection.ts Resolved

Comment thread packages/db/script-migrations/0016_backfill_search_vectors.ts Resolved

vercel Bot temporarily deployed to Preview October 31, 2026 08:40 Inactive

cubic-dev-ai Bot reviewed Oct 31, 2026

View reviewed changes

vercel Bot temporarily deployed to Preview October 31, 2026 08:54 Inactive

cubic-dev-ai Bot reviewed Oct 31, 2026

View reviewed changes

waleedlatif1 added 4 commits October 31, 2026 10:35

chore(search): retire legacy search knowledge base projection rows in…

293513f

… place

test(search): expect no vector candidates for a dormant search index

e404119

A Search knowledge base's chunk writes now delete its candidate rows, so an explicit vector search of one finds nothing; hybrid search still decides access on the document

chore(search): renumber the projection retirement migration to 0449 a…

ae18bda

…fter staging's 0448

waleedlatif1 force-pushed the chore/retire-search-projection-rows branch from f386125 to ae18bda Compare October 31, 2026 17:37

greptile-apps Bot reviewed Oct 31, 2026

View reviewed changes

Comment thread packages/db/scripts/purge-search-projection.ts Resolved

cubic-dev-ai Bot reviewed Oct 31, 2026

View reviewed changes

vercel Bot deployed to Preview October 31, 2026 17:42

View deployment

vercel Bot temporarily deployed to Preview October 31, 2026 17:44 Inactive

cubic-dev-ai Bot reviewed Oct 31, 2026

View reviewed changes

Comment thread packages/db/scripts/purge-search-projection.ts Resolved

Comment thread packages/db/scripts/purge-search-projection.integration.ts Resolved

test(search): bound the wait for the purge to block on the updated rows

30f50b4

vercel Bot temporarily deployed to Preview October 31, 2026 18:00 Inactive

cubic-dev-ai Bot reviewed Oct 31, 2026

View reviewed changes

Comment thread packages/db/scripts/purge-search-projection.ts Resolved

vercel Bot temporarily deployed to Preview October 31, 2026 18:35 Inactive

cubic-dev-ai Bot reviewed Oct 31, 2026

View reviewed changes

waleedlatif1 merged commit 20fb744 into staging Oct 31, 2026
59 checks passed

waleedlatif1 deleted the chore/retire-search-projection-rows branch October 31, 2026 18:48

Conversation

waleedlatif1 commented Oct 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Operator runbook

Type of Change

Testing

Checklist

Uh oh!

vercel Bot commented Oct 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

greptile-apps Bot commented Oct 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Confidence Score: 5/5

Summary

Diagram

Uh oh!

Uh oh!

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

waleedlatif1 commented Oct 31, 2026

Uh oh!

waleedlatif1 commented Oct 31, 2026

Uh oh!

cubic-dev-ai Bot commented Oct 31, 2026

Uh oh!

cubic-dev-ai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

waleedlatif1 commented Oct 31, 2026

Uh oh!

waleedlatif1 commented Oct 31, 2026

Uh oh!

cubic-dev-ai Bot commented Oct 31, 2026

Uh oh!

cubic-dev-ai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

waleedlatif1 commented Oct 31, 2026

Uh oh!

waleedlatif1 commented Oct 31, 2026

Uh oh!

cubic-dev-ai Bot commented Oct 31, 2026

Uh oh!

Uh oh!

cubic-dev-ai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

waleedlatif1 commented Oct 31, 2026

Uh oh!

waleedlatif1 commented Oct 31, 2026

Uh oh!

cubic-dev-ai Bot commented Oct 31, 2026

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

waleedlatif1 commented Oct 31, 2026

Uh oh!

waleedlatif1 commented Oct 31, 2026

Uh oh!

cubic-dev-ai Bot commented Oct 31, 2026

Uh oh!

cubic-dev-ai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

waleedlatif1 commented Oct 31, 2026

Uh oh!

waleedlatif1 commented Oct 31, 2026

Uh oh!

waleedlatif1 commented Oct 31, 2026 •

edited

Loading

vercel Bot commented Oct 31, 2026 •

edited

Loading

greptile-apps Bot commented Oct 31, 2026 •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading