[epipav]

Summary

Since #5028, issue_analysis_copy_pipe takes 76 to 80 minutes per run (it was about 28) and it is scheduled every hour. Runs overlap and hold one or two of the workspace's four copy workers all day, so the nightly batch reaches the 12-job copy limit and scheduled copies fail with "You have reached the maximum number of concurrent copy jobs (12)" every night around 01:00 to 02:45 and 04:00 to 05:45 UTC.

The cost is the read pattern. Each run reads the unfiltered bucket union three times: issues_opened is referenced twice (a node referenced twice is computed twice) and issues_closed reads the buckets again. Issue opened and closed events are 3.7% of rows but sit in about 93% of granules, because the buckets sort by segmentId, timestamp, type, so each read is close to a full read of every needed column.

This PR reads the buckets once per run and keeps the output identical.

Changes

• issue_events reads opened and closed events from the buckets in one pass, plus the MV delta. An opened event is still replaced by its MV version through the NOT IN; closes keep both versions as before, since only the earliest close is used.
• issues groups events by issue key (sourceId for opened, sourceParentId for closes) to get the first close without a join, and counts the distinct non-empty openers per issue. This replaces issues_opened and issues_closed.
• issue_first_commenters keeps the first two distinct commenters per issue from the cleaned buckets. The response rule reproduces the current join: the first commenter, or the second when the first is the issue's single opener. When an issue has two or more distinct openers, the first comment counts, which is what the current join does (today that is 1,908 GitHub issues recorded under two member records).
• issues is referenced only by the final node, so the response lookup no longer reads the opened events a second time.

Verification

Ran the current pipe and this one side by side, read-only, on the same inputs: every source restricted to the sampled segments, and the realtime MV pinned to a fixed updatedAt cutoff. Every output column was compared row by row.

Sample	Rows	Differences
All segments containing the 1,908 multi-opener issues	815,405	0
Random 5% of segments (593)	873,903	0

As a negative control, a variant that excludes only each row's own opener showed differences on the multi-opener issues, so the comparison catches logic changes.

On these samples, the new single read of the buckets takes about half the time of the current three. The full-run time will show in the first runs after deploy; issues_analyzed keeps the same schema.

Type of change

• Bug fix
• New feature
• Refactor / cleanup
• Performance improvement
• Chore / dependency update
• Documentation

JIRA ticket

None

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[cursor]

PR Summary

Medium Risk
Analytics pipe logic for first response and close timing was rewritten in non-trivial SQL; behavior is claimed identical via large row-level comparisons, but metric regressions would only show up in product dashboards if an edge case was missed.

Overview
issue_analysis_copy_pipe was spending most of each run scanning the daily unfiltered activity buckets three times (opened events computed twice plus a separate close pass), which pushed hourly copies to ~76–80 minutes and caused nightly copy-job limit failures. This refactor keeps issues_analyzed output the same while cutting that I/O.

Opened and closed events are merged into a single issue_events node (buckets + MV delta in one read). issues replaces separate opened/close nodes by grouping on an issue key so first close and enabled-repo opens land in one step, including openerCount / soleOpener for response rules. issue_first_commenters replaces the old non-author comment join by taking the first two distinct commenters and issue_analysis_results_merged uses multiIf to pick the first real response (including multi-opener cases). issues is only consumed in the final node so opened events are not recomputed for the comment lookup.

^{Reviewed by Cursor Bugbot for commit 73c74b5. Bugbot is set up for automated code reviews on this repo. Configure here.}

[Copilot]

🔵 Needs a closer look

Segment-restricted benchmarks do not establish that the hourly copy’s full-dataset issue/commenter aggregation will stay within schedule and workspace memory limits.

No open issues 🎉

What changed in this PR

Optimizes the hourly Tinybird issue-analysis copy pipeline to reduce overlapping runs and copy-worker contention following #5028, while preserving the existing output schema and response rules.

Changes:

• Combines opened and closed events into one unfiltered bucket read, retaining MV delta handling.
• Groups lifecycle events by issue key to calculate the first close without a join.
• Uses the first two distinct commenters to determine non-author responses, including multi-opener issues.

File	Description
services/​libs/​tinybird/​pipes/​issue_analysis_copy_pipe.pipe	Reworks lifecycle aggregation and response lookup to eliminate repeated unfiltered bucket reads.

_{Review effort: Balanced}