Skip to content

fix(queue): stop polling reactions on deleted GitHub comments - #293

Merged
jpr5 merged 1 commit into
mainfrom
fix/reaction-poll-deleted-comments
Oct 28, 2026
Merged

jpr5 merged 1 commit into
mainfrom
fix/reaction-poll-deleted-comments

Conversation

@jpr5

@jpr5 jpr5 commented Oct 28, 2026

Copy link
Copy Markdown
Contributor

Summary

The GITHUB_REACTION_POLL job logged [GithubReactionPoll] Failed to list reactions for message …: Not Found for the same 12 messages on every attempt. This PR stops those messages from being polled again. #292 fixed the timeout in the same job.

Root cause

The poll selects every AI GitHub message with feedback = null. When the comment behind a message is deleted on GitHub, the reactions endpoint returns 404. The handler logged an error, skipped the message, and left feedback null, so the next run polled it again.

Prod evidence (outpost-worker, Oct 28, 12:10 to 12:13 PM PDT): the same 12 comment IDs returned 404, 4 times each, across the job's attempts. All 12 are on CopilotKit/CopilotKit. A read-only gh api call with a user token that has full access to the repo also returns 404 for both the comment and its reactions. The comments are gone. This is not an App permission problem.

Fix

When the reactions call returns 404, the handler checks once per run whether the App can see the repo:

  • Repo visible, so the comment was deleted: the message gets the terminal feedback value COMMENT_DELETED. The handler logs one info line and does not poll the message again.
  • Repo not visible (404), so the problem is access: nothing is marked. The handler logs one error for the repo per run, and the messages stay pending. A lost installation therefore cannot mark a whole repo's messages as deleted.
  • Any other failure: unchanged. The handler logs an error and the message stays pending.

feedback is a free-form string column. feedback-calibration counts only POSITIVE and NEGATIVE, and the Discord feedback buttons apply only to Discord tickets, so the new value does not affect either one. The schema comment now documents the new value. This is a comment-only change, so no migration is needed. The job's result data now includes deleted.

Files: queue/src/handlers/github-reaction-poll.ts, the isGithubNotFound and isRepoVisible helpers in shared/src/integrations/github-client.ts, the tests for both, and the schema comment.

Local red-green (real GitHub API, real handler)

Harness: a local Postgres seeded with 256 pending GitHub messages to match prod's count. 12 of them use the real comment IDs that return 404 in prod, and 244 use a live CopilotKit/CopilotKit comment. The harness runs the real handleGithubReactionPoll twice against the real GitHub API with the prod App credentials, under the worker's 30s default timeout. Raw GET … 404 lines from Octokit are omitted.

RED (before #292 and this PR):

--- run 1
elapsed=59577ms jobTimeout=30000ms -> TIMED OUT (worker would fail this attempt)
error lines=12 info lines=0 result={"success":true,"data":{"checked":256,"updated":0}}
  ERROR [GithubReactionPoll] Failed to list reactions for message cmvslmj9700042l4y32pgnfit: Not Found - https://docs.github.com/rest/reactions/reactions#list-reactions-for-an-issue-comment
deleted-comment rows by feedback: {"null":12}
--- run 2 (next scheduled poll)
elapsed=56972ms jobTimeout=30000ms -> TIMED OUT (worker would fail this attempt)
error lines=12 info lines=0 result={"success":true,"data":{"checked":256,"updated":0}}
deleted-comment rows by feedback: {"null":12}

GREEN (this branch, on top of #292). A rerun after rebasing onto main with #292 merged gave the same result: 8.1s, then 7.8s, with 0 errors and the same 12 rows marked.

--- run 1
elapsed=9112ms jobTimeout=30000ms -> within timeout
error lines=0 info lines=0 result={"success":true,"data":{"checked":256,"updated":0,"deleted":12}}
  LOG   [GithubReactionPoll] Comment 5560982384 for message cmvslwluq000k2lnv7n1dgkou no longer exists on CopilotKit/CopilotKit; marked COMMENT_DELETED, it will not be polled again
deleted-comment rows by feedback: {"COMMENT_DELETED":12}
--- run 2 (next scheduled poll)
elapsed=7277ms jobTimeout=30000ms -> within timeout
error lines=0 info lines=0 result={"success":true,"data":{"checked":244,"updated":0,"deleted":0}}
  LOG   [GithubReactionPoll] Checked 244 messages, updated 0 with feedback, marked 0 deleted
deleted-comment rows by feedback: {"COMMENT_DELETED":12}

Run 2 polls 244 messages instead of 256 and logs no errors.

Unit tests

New tests fail on the old code and pass on this branch:

  • handler: a deleted comment is marked COMMENT_DELETED and logs one info line and no error.
  • handler: if the repo is not visible, nothing is marked, isRepoVisible is called once, and one error is logged.
  • handler: if the repo check fails, the message stays pending and one error is logged.
  • isGithubNotFound and isRepoVisible (true, false on 404, rethrow on other failures).

A non-404 test confirms that transient failures still log and stay pending, and that they do not trigger a repo check.

pnpm turbo run typecheck lint: 21/21 passed. The two changed test files pass (21 tests). Prettier is clean on the changed files.

Deploy note

After deploy, the first run marks the 12 known messages and logs 12 info lines. Later runs log no 404 errors for them.

The reaction poll selects every AI GitHub message with feedback = null.
When the comment behind a message is deleted on GitHub, the reactions
endpoint answers 404, the handler logs "Failed to list reactions ... Not
Found" and moves on, and the message stays pending. Prod has 12 such
comments on CopilotKit/CopilotKit, and each poll attempt logged all 12 as
errors again.

On a 404, the handler now checks once per run whether the App can see the
repo. If it can, the comment is gone: the message gets the terminal
feedback value COMMENT_DELETED, one info line is logged, and the message
leaves the polled set. If the repo itself 404s, it is an access problem,
not a deletion: nothing is marked and the repo is reported once as an
error. Other failures are logged as before and stay pending.

Feedback calibration counts only POSITIVE and NEGATIVE, so the new value
does not change it.
@jpr5
jpr5 merged commit e7f73ee into main Oct 28, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant