[jpr5]

What failed in prod

On Oct 28 at 12:12:44 PM PDT, the prod worker (deployment 7947134d, commit 00dc276) dead-lettered job cmvshmzp10002lq011ki9p02j (GITHUB_REACTION_POLL) after 5 of 5 attempts. The error was Job timed out after 30000ms.

The scheduler created this job when the worker booted at 12:09:01 PM PDT. This is not a leftover from the redeploys. The "fenced" lines come from the job itself: the timeout is a Promise.race, so the handler keeps running after the timeout, and its late progress write is then fenced out. Each attempt logged [GithubReactionPoll] Checked 256 messages after it had already timed out. The same dead-letter shows up on every earlier worker deployment we have logs for (1 to 7 per deployment, with the pending count growing from 225 to 256).

Cause

handleGithubReactionPoll made one GitHub request per pending AI comment, one at a time. The pending list only grows, because comments deleted on GitHub return 404 and stay pending. At about 240 ms per request, 256 requests take more than 30s, which is the default job timeout. This job type has no override.

Fix

The handler now keeps up to 8 reaction lookups in flight (GITHUB_REACTION_POLL_CONCURRENCY). Each message is still handled the same way: skips, 404 logging, POSITIVE/NEGATIVE updates and the ESCALATION job are unchanged.

Local red/green proof (real surface)

Setup: the real Worker (prod default 30s timeout, concurrency 1 for this type) and the real handler, loaded through @copilotkit/outpost/queue dist, against a local Postgres (pgvector pg16, all migrations applied). I seeded 256 pending AI-generated GITHUB_ISSUE messages. Their comment IDs are the 12 deleted-comment IDs from the prod log, plus 244 real CopilotKit/CopilotKit issue comment IDs. Reporter logins are placeholders. Reaction reads go to the real api.github.com. Only the GitHub App token exchange is stubbed locally.

RED (base 00dc276), 2:01 PM PDT:

seeded 256 pending AI messages
[Queue Worker] Job cmvslndgc00ld2l43scc9dow9 (GITHUB_REACTION_POLL) moved to dead letter queue after 1 attempt(s): Job timed out after 30000ms
RESULT status=DEAD_LETTER attempts=1/1 error=Job timed out after 30000ms elapsed=30.1s githubCalls=123

GREEN (this branch, same seed and same script), 2:02 PM PDT:

seeded 256 pending AI messages
[GithubReactionPoll] Checked 256 messages, updated 0 with feedback
RESULT status=COMPLETED attempts=1/1 error=- elapsed=8.0s githubCalls=256

The 12 deleted comments were logged as 404s and skipped in both runs.

Tests

There are 3 new unit tests in github-reaction-poll.test.ts. On the old code, 2 of them fail:

× reads reactions for several messages in parallel instead of one at a time
  AssertionError: expected 1 to be greater than 1
× caps the number of GitHub reads in flight
  AssertionError: expected 1 to be undefined

On this branch, all 12 tests in the file pass. The third new test checks that +1, -1 and 404 outcomes are all still handled when the lookups run in parallel.

Local gates: lint and typecheck pass. Prettier passes on the changed files. Repo-wide format:check already fails on base (276 files). pnpm test passes except starts the probe child cleanly, which fails on base 00dc276 too. That is local-only (Node 26 on this machine; the repo targets Node 24).