Repository navigation
Conversation
An image background whose file is a video (mp4, mov, m4v, webm, mkv) or a GIF now plays and loops instead of showing its first frame. Each one is decoded on its own thread with FFmpeg (VideoToolbox on macOS, D3D11VA on Windows, software elsewhere), scaled once to RGBA no larger than the output needs (max 1920px), and read ahead between frames. The start of the clip stays decoded so a loop restarts without a seek, and short clips and most GIFs stay entirely in memory. The GPUI editor's image pane now also accepts videos and shows their first frame as the thumbnail.
|
@AatmanAJ is attempting to deploy a commit to the Cap Software Inc Team on Vercel. A member of the Team first needs to authorize it. |
…ound request lands on the newest frame
…mes that are never shown
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| let interval = self.request_interval; | ||
| let next = self.next_shown(|frame_time| match interval { | ||
| Some(interval) if frame_time > anchor => { | ||
| anchor + ((frame_time - anchor) / interval - TIME_EPSILON).ceil() * interval | ||
| } | ||
| _ => frame_time.max(anchor), | ||
| }); |
There was a problem hiding this comment.
prefetch drops frames based on the previous request interval, but covering_frame treats the remaining frames as continuous coverage. For a streaming 10 fps clip, requests spaced 0.5 seconds apart can leave cached frames at 1.5 and 2.0 seconds. A later scrub to 1.7 seconds returns the 1.5-second frame without decoding again.
Keep each cached frame's actual end time and decode again when a request lands in a skipped gap. Add a test that changes request spacing after read-ahead.
Knowledge Base Used: Roll Back render-farm padding and opening-frame replay
Prompt To Fix With AI
This is a comment left during a code review.
Path: crates/rendering/src/layers/video_background.rs
Line: 497-503
Comment:
**Scrubbing shows an earlier frame**
`prefetch` drops frames based on the previous request interval, but `covering_frame` treats the remaining frames as continuous coverage. For a streaming 10 fps clip, requests spaced 0.5 seconds apart can leave cached frames at 1.5 and 2.0 seconds. A later scrub to 1.7 seconds returns the 1.5-second frame without decoding again.
Keep each cached frame's actual end time and decode again when a request lands in a skipped gap. Add a test that changes request spacing after read-ahead.
**Knowledge Base Used:** [Roll Back render-farm padding and opening-frame replay](https://app.greptile.com/cap/-/custom-context/knowledge-base/capsoftware/cap/-/reverts/rollback_2393-20260930-zero-padded-exports-and-held-opening-frames-9228121.md)
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.| fn reached_end<T>(&mut self) -> Option<T> { | ||
| if let Some(last) = self.last_time { | ||
| let end = last + self.frame_interval; | ||
| if end > 0.0 { | ||
| self.duration = end; | ||
| } |
There was a problem hiding this comment.
reached_end replaces the file's duration with the last frame's start time plus the previous frame interval. That interval does not include the last frame's own delay. A GIF with frames starting at 0, 0.1 and 0.2 seconds, with the last frame held for two seconds, loops after 0.3 seconds instead of 2.2 seconds.
Preserve a valid file duration or use the last frame's actual duration before falling back to the previous interval.
Prompt To Fix With AI
This is a comment left during a code review.
Path: crates/rendering/src/layers/video_background.rs
Line: 615-620
Comment:
**GIFs restart too early**
`reached_end` replaces the file's duration with the last frame's start time plus the previous frame interval. That interval does not include the last frame's own delay. A GIF with frames starting at 0, 0.1 and 0.2 seconds, with the last frame held for two seconds, loops after 0.3 seconds instead of 2.2 seconds.
Preserve a valid file duration or use the last frame's actual duration before falling back to the previous interval.
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.| fn scale(&mut self, decoded: &frame::Video) -> Option<(Vec<u8>, u32, u32)> { | ||
| let input = (decoded.format(), decoded.width(), decoded.height()); | ||
| if input.1 == 0 || input.2 == 0 { | ||
| return None; | ||
| } | ||
|
|
||
| if self | ||
| .scaler | ||
| .as_ref() | ||
| .is_none_or(|scaler| scaler.input != input) | ||
| { | ||
| let size = self.output_size(input.1, input.2); | ||
| let bt709 = decoded.color_space() == ffmpeg::util::color::Space::BT709; | ||
| self.scaler = Some(Scaler::new(input, size, bt709)?); |
There was a problem hiding this comment.
The new video path scales decoded pixels without applying the file's display rotation. Portrait phone videos commonly store landscape pixels with a 90-degree rotation tag, so these files appear sideways in both the background and its poster.
Read and apply the display rotation before choosing the displayed dimensions and crop.
Prompt To Fix With AI
This is a comment left during a code review.
Path: crates/rendering/src/layers/video_background.rs
Line: 632-645
Comment:
**Portrait videos appear sideways**
The new video path scales decoded pixels without applying the file's display rotation. Portrait phone videos commonly store landscape pixels with a 90-degree rotation tag, so these files appear sideways in both the background and its poster.
Read and apply the display rotation before choosing the displayed dimensions and crop.
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.| fn output_size(&self, width: u32, height: u32) -> (u32, u32) { | ||
| let longest = width.max(height).max(1); | ||
| let scale = (f64::from(self.max_dimension) / f64::from(longest)).min(1.0); | ||
| let even = |value: u32| ((f64::from(value) * scale).round() as u32).max(2) & !1; | ||
| (even(width), even(height)) |
There was a problem hiding this comment.
output_size preserves the stored pixel dimensions but ignores the video's sample aspect ratio. Non-square-pixel videos, such as 720×576 footage meant to display at 16:9, are stretched into the wrong shape before the background crop is applied.
Include the sample aspect ratio when choosing the output dimensions.
Prompt To Fix With AI
This is a comment left during a code review.
Path: crates/rendering/src/layers/video_background.rs
Line: 625-629
Comment:
**Videos look squeezed**
`output_size` preserves the stored pixel dimensions but ignores the video's sample aspect ratio. Non-square-pixel videos, such as 720×576 footage meant to display at 16:9, are stretched into the wrong shape before the background crop is applied.
Include the sample aspect ratio when choosing the output dimensions.
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.| } | ||
| } | ||
|
|
||
| /// The next frame, decoded and scaled. |
There was a problem hiding this comment.
The added decode_next comment only repeats what the function does. The repository guide requires comments to explain non-obvious context and explicitly bans comments that narrate the code. Remove this comment and the equivalent poster description at lines 54–55 before merging. Keep the comments that explain seek behavior and scaling trade-offs.
Context Used: CLAUDE.md (source)
Prompt To Fix With AI
This is a comment left during a code review.
Path: crates/rendering/src/layers/video_background.rs
Line: 531
Comment:
**Comments repeat the code**
The added `decode_next` comment only repeats what the function does. The repository guide requires comments to explain non-obvious context and explicitly bans comments that narrate the code. Remove this comment and the equivalent poster description at lines 54–55 before merging. Keep the comments that explain seek behavior and scaling trade-offs.
**Context Used:** CLAUDE.md ([source](https://github.com/capsoftware/cap/blob/main/CLAUDE.md))
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
Lets you pick a video or GIF as the background in the editor. It loops behind the recording, in the preview and in exports. Richie OK'd the idea on Discord, on the condition that it's performant on every platform, so this PR is built around that.
What changes
cap-rendering): when an image background's file is a video (mp4,mov,m4v,webm,mkv) or a GIF, it's played and looped instead of showing only its first frame. There's no project-config change: it's stillBackgroundSource::Image { path }, so older builds that open the project just show the first frame (GIF) or fall back to white (video), the same as any image they can't read.mp4,mov,m4vandwebm, and a video's thumbnail is its first frame. The screenshot editor still accepts images only.How it stays cheap
layers/video_background.rs:cap_video_decode::FFmpegDecoder, with VideoToolbox on macOS, D3D11VA on Windows and software decoding on Linux. GIFs always decode in software.threadsset before init;software::scaling::Context::getcan't do that). The output is pixel-identical to the single-threaded scaler, and a test checks it.write_texturewhen the frame actually changes, so a 30 fps clip in a 60 fps export uploads every other frame. A paused preview uploads nothing.Benchmark
background-benchmark(existing example) renders 1920×1080 frames back to back, with no idle time for read-ahead, so these numbers are the worst case. Apple M5 Max, 300 frames after 20 warm-up frames, mean of 2 runs:For a 4K60 source, the cost is the decode (~3.3 ms including the hardware frame transfer) plus the 4K → 1920 conversion (~3.8 ms with AREA; it was 5.4 ms with bilinear). Snapshots at output frames 45 and 330 show the clip's burned-in counter at 0.733 s (source frame 22) and 5.500 s (frame 165), so playback stays in sync.
Optimisation pass
Measured on the M5 Max with VideoToolbox, before and after the third commit, timing
frame_aton its own:I also tried software decoding on macOS. It's about 3x faster back to back (4K export 1.6 ms), but it uses 8.5 cores to get there, against 1.1 for VideoToolbox. A decorative layer shouldn't take CPU from the recording's own decode and encode, especially on smaller machines, so hardware decoding stays. Jumping to a random time costs whatever it takes to decode forward from the previous keyframe, so clips with a short keyframe interval (
-g 30) scrub faster.Windows and Linux
Measured on GitHub's hosted x86 runners (4 vCPUs, AMD EPYC, no GPU) with a throwaway workflow on my fork (run). Neither runner has a GPU, so D3D11VA isn't available and both decode in software, which is the worst case.
Decode thread on its own (time for
frame_atto return):At playback pace, read-ahead hides the decode completely, even for 4K HEVC in software. Back to back (export with nothing else to do) is the raw software decode-and-scale cost.
Full render, extra time per frame vs. a PNG background. Frames are drawn by WARP on Windows and lavapipe on Linux, so absolute times (~105 ms and ~35 ms per frame) aren't meaningful, only the difference:
This run found a bug, fixed in the second commit. When a request landed exactly on the newest decoded frame, the reader seeked to the previous keyframe instead of decoding one more frame. On the Mac, read-ahead always got there first, but on these runners it happened on almost every frame of an export: 4K averaged 223 ms with 2.4 s spikes, and now averages 15 ms with a 19 ms worst case.
Testing
video_background.rs:cargo test -p cap-rendering: all pass (424 lib tests).cargo +1.95.0 test --bin cap-gpui: 1522 passed, 0 failed.Not done yet:
To reproduce the benchmark:
where
cases.jsonis[{"name":"video","source":{"type":"image","path":"/abs/path/clip.mp4"}}].Fix incorrect frames after scrubbing, shortened GIF loops, and sideways portrait videos before merging.
Findings
Fix with agent prompt
Summary
Adds looping video and GIF backgrounds to the shared renderer and video selection to the GPUI editor.
Reviews (1) · Last reviewed commit: "perf(rendering): scale video backgrounds..." · Reviewed by Greptile