Time-Aligned Visual Diffs

Abstract

When a pull request changes a UI, the usual visual proof is two screen recordings of the same interaction, one on main and one on the branch, and a reviewer has to spot the difference. Screenshot tools can't help much here, because they freeze the animation and compare a single frame. I looked into comparing the whole recordings instead, using a technique from my short film: to rebuild its trailer, I matched every frame by its structure rather than its exact pixels. Applied to UI recordings, the same approach lines up main and the branch frame by frame with dynamic time warping, then says what kind of change each difference is: the same picture moved, made darker, changed in place, or shown later. Each finding comes with the two frames that show it, so the proof can be checked at a glance. I tested this on a real pull request in the Sonner toast library and on three changes I planted, recording each version three times. It described the real change correctly in 3 of 3 recording pairs: the toast stack sits 8 px lower while hovered, and nothing else changed. It also caught a 75 ms flash and a slower animation, both of which a screenshot test misses.

Figure 1.Main and PR #644, lined up frame by frame and played along the matched path. The box marks frames that differ. Below, one bar per recording spans its full 5.3 s; each cursor shows that side's current frame, and a cursor that pauses means that side is holding a frame while the other catches up. Click a bar to jump.

1Origin: rebuilding a film trailer

The trailer for my short film Beijing Rodeo existed only as one flattened export. It carried the old colour grade, a letterbox, burned-in titles, speed changes and punch-ins. Rebuilding it from the newly graded clips meant finding each of its 1,427 frames again among 56,820 graded frames. A conform editor quoted $5,000 for the job. I built the tool with Claude instead [1]. Four ideas did the work:

Fingerprints that ignore the grade. Each frame becomes a small brightness pattern with its average brightness and contrast removed, so two grades of the same frame still agree. In standard terms, comparing two fingerprints is a normalised cross-correlation.
A path through a similarity matrix. Comparing every trailer frame with every frame of a candidate clip gives a matrix. If the shot came from that clip, the best path is a straight line: its start is the in-point and its slope is the speed (Figure 2a). Letting the path bend, so the speed can change along a shot, is dynamic time warping, which the UI version uses.
Proof images, judged blind. Each match is drawn as old frame, new frame and an edge overlay: old edges magenta, new edges green, white where they coincide. Two AI reviewers judged each image without seeing the scores. One compared; the other tried to disprove (Figure 2b).
Reporting what failed. 48.9% of the trailer was rebuilt. The rest is split into four causes instead of being reported as one miss.

Reviewing a UI change from two recordings has the same shape: two versions of the same footage that differ in timing and look, and a person who has to decide whether they show the same thing.

Film similarity matrix with a bright diagonal ridge and a fitted white path — Figure 2.From the film conform [1]. (a) 54 trailer frames (rows) against 131 frames of one graded clip (columns); bright means similar. The path's slope of 0.84 clip frames per trailer frame means the old editor slowed the shot to 84%. (b) Old trailer, graded source and edge overlay at an early, middle and late frame. The contours coincide in all three rows.

Film proof image: old trailer frame, graded frame and edge overlay at three frames — Figure 2.From the film conform [1]. (a) 54 trailer frames (rows) against 131 frames of one graded clip (columns); bright means similar. The path's slope of 0.84 clip frames per trailer frame means the old editor slowed the shot to 84%. (b) Old trailer, graded source and edge overlay at an early, middle and late frame. The contours coincide in all three rows.

2The problem: two recordings, one reviewer

Coding agents increasingly attach evidence to their pull requests, and for a UI change that evidence is often a screen recording. Some tools now record the same interaction on main and on the PR so the two can be compared. Blacksmith's Proof, in private beta, describes doing this [2]. Factory's Droid Control records both branches side by side [25], and Meticulous replays one session on both commits and, since September 2026, shows the two replays side by side [27]. In each case a person, or the agent's own summary, decides what changed. Blacksmith's framing is that writing code is cheap and trusting it is not, and that evidence is what lets a reviewer trust work they did not watch [3]. With two recordings side by side, the reviewer still has to find the difference.

2.1Existing tools

Visual-testing tools solve a narrower problem well, mostly on still frames. Playwright's toHaveScreenshot disables CSS animations and transitions by default and waits until two consecutive screenshots match [4]. Chromatic pauses CSS animations, videos and GIFs before it captures [5]. Applitools compares stills with match levels that tolerate rendering noise [6]. Meticulous replays recorded sessions on the base and head commits and compares screenshots taken during the replay [7]. Percy's Visual Review Agent labels each detected change as matching the PR summary or not [8]. Two tools do compare motion: Happo's animate mode (September 2026) and Paparazzi on Android render an animation's frames at fixed times under a controlled clock and compare them frame by frame [23] [24]. That needs control of the renderer's clock, and it cannot see real pacing.

Research and open source come closer to recordings. AMOS GUI Frame Diff, a 2023–24 student project, aligns two GUI test-run videos with gap-based sequence alignment and renders a diff video, without labelling what kind of change each difference is [22]. RippleGUItester runs scenarios on the builds before and after a change and judges screenshot differences against the change's intent; it attributes 78 false positives to screenshot timing and unstable rendering [26]. Janus aligns recordings to detect duplicate video bug reports [17], building on Tango [16]; GIFdroid maps a recording's keyframes to app states to replay a bug [18]; and Seenomaly and GUIWatcher find animation and lag problems within a single recording [19] [20]. The alignment technique itself, dynamic time warping, is standard [12] and has been used to score animations [9]. As of October 2026 I found nothing that aligns real-time main-vs-PR recordings and labels each difference, with "same picture, later" as a category of its own. That is the open part.

2.2What carries over

I asked Claude to map each part of the film method onto reviewing two UI recordings and rate the fit (Table 1). The strong fits all deal with time: the path through the matrix, the motion matrix for still stretches, and checking every frame instead of samples. Those are what the rest of this note tests.

Table 1.Parts of the film method, their counterparts in UI review, and the strength of the fit.
Film technique	UI-review counterpart	Fit	Reason
Path through a similarity matrix (dynamic time warping)	Two recordings with different timing	●●●strong	Frames with a partner are "same, perhaps later"; frames without one are a state only one side shows. No tool found that does this.
Motion matrix for near-static shots (frame differencing)	A UI idling between clicks	●●●strong	UIs are mostly still, which is where matching by picture alone is ambiguous.
Check every frame, not three samples	A one- or two-frame flicker	●●●strong	Screenshot tests wait for a stable picture, which hides flicker by design.
Framing search (image registration)	Content pushed down by a new element	●●○partial	Reports "moved 8 px" instead of "everything changed". Applitools' layout level is close.
Median over 32 blocks (survives burned-in titles)	Clocks, avatars, ads	●●○partial	Masking dynamic regions already exists (Playwright `mask`, Applitools).
Magenta/green edge proof image (Canny edges, false-colour overlay)	Evidence a reviewer trusts at a glance	●●○partial	Diff overlays are common. Choosing which frames to show from a video is new.
Fingerprints that ignore the colour grade (normalised cross-correlation)	Rendering noise: anti-aliasing, fonts, GPU	●○○weak	Main and PR record on the same machine, so noise is small. A theme change is a change of "grade" that must be seen: use the invariance to align, never to judge.
Blind reviewers: compare, then disprove (adversarial review)	Does the change match the PR's claim?	○○○exists for stills	Percy labels changes "Matches PR summary" or not [8].

3Method

The implementation follows the film pipeline and changes it where a UI differs from film. It is about 400 lines of Python using NumPy and OpenCV, plus a Playwright recorder. Every step is a standard technique (Box 1); what is new is using them together to diff two recordings of a pull request.

Box 1. Standard techniques used

Step	Standard technique	Origin
Frame fingerprints	Zero-mean normalised cross-correlation of downscaled greyscale frames	Template matching [13]
Motion matrix	Frame differencing	Video analysis
Lining up the recordings	Dynamic time warping, with a slope penalty and a Sakoe–Chiba-style band that follows the logged actions	Speech recognition [12]
Judging each frame	Block-wise change detection, run in both directions (bidirectional matching)	Change detection
"Moved"	Block matching: an exhaustive translation search, the simplest form of image registration	Image registration [14]
"Darker"	Gain compensation (photometric normalisation), fitted by least squares	Image registration [14]
Proof images	Canny edge detection, shown as a false-colour overlay	Edge detection [15]
Thresholds	A/A comparisons (`main` against `main`) set the noise floor; other same-build pairs are held out	Controlled experiments [21]
Planted builds	Fault injection: seeded changes with a known answer	Software testing
Screenshot control	Golden-image test with pixelmatch, as in Playwright's `toHaveScreenshot`	Visual testing [4]
Review	Blind review, plus an adversarial reviewer who tries to disprove each finding	Peer review

3.1 Fingerprints (normalised cross-correlation). Each frame is reduced to a 96 × 60 greyscale thumbnail. The first frame, the page at rest, is subtracted, and mean and contrast are removed, so comparing two fingerprints is a zero-mean normalised cross-correlation [13]. As in the film, this invariance is used only to line frames up. A theme change is a change of grade that a reviewer needs to see, so the invariance is never used to judge.

3.2 Similarity and motion (frame differencing). Every frame of one recording is compared with every frame of the other. Where both sides are moving, a motion matrix built by frame differencing (next frame minus previous frame) is averaged in; it is sharp exactly where the page is nearly still. Whether a frame is at rest or moving is decided per 12 × 12 block, because a small toast barely changes a whole-frame average.

3.3 Path (dynamic time warping). The path is found with dynamic time warping [12]: the cheapest forward-only route through one minus similarity, with a small cost for steps where one side holds a frame. The recorder logs when each click and hover happened, so the path stays within a band of six frames (240 ms) around a line through those moments: a Sakoe–Chiba band that follows the clicks instead of the diagonal. The film had no such log; an agent recording a PR would have one.

3.4 Judging (block-wise change detection). Each frame is compared at 480 × 300 with its partner on the path and one frame either side, so 40 ms of jitter is not reported. The frame is cut into 8 × 5 blocks and scored by the share of differing pixels in its worst block. Both directions are checked (bidirectional matching): a main frame with no match in the PR, and a PR frame with no match in main.

3.5 Explaining. Each run of differing frames gets one label by majority vote over its frames: moved (block matching: a translation search of up to ±20 px, the simplest form of image registration [14], makes the region match), darker or lighter (gain compensation: one gain, fitted by least squares to the changed pixels only, makes it match), or different content. Where something moves and the path sits more than the timing threshold from the click-anchored line, the change is reported as timing.

Similarity matrix, PR against the slower build: the white path bends right of the dotted line during each animation — Figure 3.The film's matrix, drawn for a UI. Rows are frames of the first recording, columns frames of the second; bright means similar. The dotted blue line is where the logged clicks place the path; the white line is the dynamic-time-warping path found. (a) During each animation the path bends right: the slower build shows the same pictures later. (b) Two recordings of one build stay on the line. Their residual differences set the thresholds (Section 4.4).

Similarity matrix, main against main: the white path follows the dotted line — Figure 3.The film's matrix, drawn for a UI. Rows are frames of the first recording, columns frames of the second; bright means similar. The dotted blue line is where the logged clicks place the path; the white line is the dynamic-time-warping path found. (a) During each animation the path bends right: the slower build shows the same pictures later. (b) Two recordings of one build stay on the line. Their residual differences set the thresholds (Section 4.4).

4Experimental setup

4.1The pull request

Sonner is a widely used React toast library. PR #644, "chore: remove lift interaction", was merged on May 30, 2025 [10]. Its description reads:

While it looks better with lift, it was causing some hover state bugs.

It deletes an 8 px upward translation of the toast stack on hover, animated over 400 ms. The change is small, and it is motion.

4.2Builds

Table 2.The four builds. The two planted builds are fault injection: seeded changes with a known answer, made only for this test.
Build	Contents	Question it answers
main	Sonner immediately before #644	Baseline
PR #644	The merged change	Does the method describe a real PR correctly?
PR + planted	#644 plus a ~75 ms dark flash when a toast appears, and title weight 500 → 600	Does it catch changes the PR does not claim, including one too short to see?
PR + slower	#644 with toast motion at 700 ms instead of 400 ms; same end state	Does it catch a change that is only timing?

4.3Recording

Each build ran Sonner's own Next.js test app. One Playwright script loaded the page, clicked "Render Toast" three times 350 ms apart, hovered the stack for 1.5 s and moved away. Playwright recorded video at 25 fps and 960 × 600 and logged the time of each action. Each build was recorded three times on one laptop, twelve recordings in all. The challenge at the top of this note uses run 1 of main and of the PR, cropped to the toasts; a reviewer today receives the full page.

4.4Calibration and controls

Thresholds come from A/A comparisons [21]: the three main recordings compared with each other. The worst differences there were 1.03% of a block on still frames, 6.3% on moving frames and 70 ms of timing drift. Thresholds were set at 2×, 1.25× and 2× those values: 2.06%, 7.88% and 140 ms. The margins were chosen after a first look at the data. The nine same-build pairs of the other three builds were not used for calibration and are held out as a false-flag check.

For comparison, each build also got a golden-image test: two still screenshots taken the way toHaveScreenshot takes them (animations disabled, wait for two identical shots), compared with pixelmatch at Playwright's default threshold of 0.2.

5Results

5.1The real pull request

On main against PR #644 the method reported one change, seen from both sides: the same picture, moved, with the PR's toasts 8 px lower than main's. The window opens between 2.76 and 2.88 s and closes between 3.84 and 3.96 s depending on the pair; that is the hover. No timing difference was reported. All three recording pairs agree. This is the change the PR describes (Figures 1 and 4).

main: hovered toast stack — Figure 4.Proof image for the real PR, in the film's format. Every magenta outline sits directly above its green twin: the lift, and nothing else.

PR: hovered toast stack, 8 px lower — Figure 4.Proof image for the real PR, in the film's format. Every magenta outline sits directly above its green twin: the lift, and nothing else.

5.2Changes the PR does not claim

Against the planted build, the method reported the same 8 px move plus a darkening at 0.84, 1.16 and 1.52 s, one for each new toast, with a fitted gain between 0.61 and 0.74, in all three pairs (Figure 5). The flash lasts about two frames.

The bolder titles were reported only when the planted build was compared with the PR build: as different content during the hover, roughly 2.9 to 3.9 s, in three of three pairs, and only while all three titles were visible. With one title showing, the difference scored 1.5% against a 2.06% threshold. Compared with main, the heavier text sat inside the region that had moved 8 px, the move explained that region within threshold, and the method reported only the move (Figure 6). Section 5.7 describes the fix.

Figure 5.Main against PR + planted, at quarter speed. darker: the flash as each toast arrives. moved: the lift the PR does claim.

main: new toast, white — Figure 6.The two unclaimed changes. (a) The flash. The tool also called this "the same picture"; the disprove reviewer showed that the toasts in these frames sit 3–6 px apart (Section 5.6). (b) Title weight 500 against 600, found when the planted build is compared with the PR build.

planted: new toast, grey — Figure 6.The two unclaimed changes. (a) The flash. The tool also called this "the same picture"; the disprove reviewer showed that the toasts in these frames sit 3–6 px apart (Section 5.6). (b) Title weight 500 against 600, found when the planted build is compared with the PR build.

5.3A timing-only change

The slower build ends each animation where the PR build does, so its still frames match. Aligned in time it showed no picture difference at all. During each animation it ran 150–233 ms behind the PR build, against at most 70 ms between recordings of the same build (Figures 7 and 8). Figure 9 shows the evidence at one moment: at the same clock time the outlines do not coincide; at the frame the path chose, they do. The lag is measured against the click times, which landed 40 ms apart in these two recordings, so the frame times shown differ by 240 ms while the lag is 200 ms.

Figure 7.PR #644 against PR + slower, at half speed. timing: the slower build is running behind. No frame is boxed, because no picture differs.

Figure 8.How far the second recording runs behind the first, frame by frame. PR against slower, three pairs. main against main, three pairs. Dashed: the 140 ms threshold. Only frames where something moves are plotted; on a still stretch every alignment is equally right.

Edge overlay at the same clock time: outlines do not coincide — Figure 9.Edge overlays, PR in magenta and slower in green. (a) At the same clock time the outlines do not coincide. (b) At the frame the path chose they are white: the same picture, later.

Edge overlay at the matched frame: outlines coincide — Figure 9.Edge overlays, PR in magenta and slower in green. (a) At the same clock time the outlines do not coincide. (b) At the frame the path chose they are white: the same picture, later.

5.4Comparison with a still-screenshot (golden-image) test

Table 3.Each change against each check. Frame-matching results are over three recording pairs.
Change	Still-screenshot test	Frame matching	Blind reviewer
8 px lift removedthe PR's real change	Caught. 804 px differ while hovered; no description of the change	Caught 3/3. "Same picture, moved 8 px lower"	Seen. "About 7–8 px lower"
75 ms flashplanted	Missed. Over before a still is taken	Caught 3/3. Darker, gain 0.61–0.74, at each toast	Seen. "Solid grey fill as each toast arrives"
Bolder titlesplanted	Caught. 48 px differ with one toast showing	Caught 3/3 against the PR build. Against main: missed at first, inside the move; caught 3/3 after the fix (5.7)	Seen in all four planted cards
Slower motionplanted	Missed. 0 px differ	Caught 3/3. Same picture, 150–233 ms later	Seen. "Settles about 0.2 s later"

The two checks are complementary. The screenshot test caught the font-weight change that the method missed against main. The method caught the flash and the slowdown, which a screenshot test cannot see because it waits for a stable picture with animation off.

5.5Noise and false flags (A/A comparisons)

Table 4.A/A comparisons: recordings of the same build compared with each other.
Pairs	Count	Flags
main against main	3	Used to set the thresholds
PR against PR	3	0
Slower against slower	3	0
Planted against planted	3	2, 4 and 3 single-frame flags, all at a flash

The planted-against-planted flags are real flashes recorded unevenly. At 25 fps a frame lasts 40 ms, so a 75 ms flash lands on different frames in each recording, and one recording can show a frame of flash where the other does not. Detecting flicker reliably needs a higher frame rate.

5.6Blind review

As in the film, two AI agents judged the proof images without seeing any scores. The first was not shown the tool's sentences and described what differs. The second was given the sentences and tried to break each one. Both saw only the images and the PR's description; neither was told which builds were planted.

The describe reviewer matched the tool's label on all six cards and also saw the bolder titles in all four planted cards, including the one the tool labelled only "moved". The disprove reviewer confirmed one sentence and found five partly right (Table 5).

Table 5.The disprove reviewer's verdict on each of the tool's sentences.
Card	Tool's sentence	Verdict	What the reviewer found
A1	Moved: PR toasts 8 px lower (real PR)	Confirmed	All three toasts 8 px lower; spacing, width, text and shadows match.
B1–B3	Same picture, darker, at each new toast	Partly	The grey flash is real, but the toasts also sit 3–6 px apart and the grey toast is partly transparent, so "same picture" is overstated. Bold labels not mentioned.
B4	Moved: toasts 8 px lower (planted build)	Partly	The move is real; every label is also bolder. The sentence is identical to A1's.
C1	Same picture, about 200 ms later; motion is slower	Partly	The matched frame lines up. A single matched frame cannot tell slower motion from a later start, and subtracting the frame times gives 240 ms, not 202.

The reviews corrected the tool's sentences without changing any detection. That matches the film, where the reviewers, not the scores, made the final call.

5.7Fixing the miss

The B4 finding pointed at a gap: once a region was explained as moved, the method stopped looking. It now undoes the move, judges the frame again on the same blocks against the same calibrated threshold, and adds a second label when most frames of the run still differ.

Table 6.Frames that still differ after the move is undone, per finding, over three recording pairs.
Pair	Frames still differing	Second label
main against PR #644	0 of 25–30	None (correct: nothing else changed)
main against PR + planted	24–29 of 24–29	Different content, 3 of 3 pairs (correct: bolder titles)
main against PR + slower	0–1 of 19–21	None (correct)

The false-flag checks in Table 4 are unchanged. A second blind review of the corrected sentences confirmed A1 ("moved 8 px", no second label). It rated B4 partly right: the titles do differ after the move, but the words are the same and only the weight changed, so "different content" overstates it. The method can say that something changed in place; it cannot say what.

5.8Run time and cost

Lining up and judging one main/PR pair took a median of 4.4 s (longest 6.2 s) in one process on an Apple M5 Pro laptop, for 5.2 s of video per side, after decoding. The code is unoptimised Python and was not timed on a CI runner. For scale, Blacksmith lists Linux x64 runners from $0.004 a minute [11]; even at ten times the laptop's time, one comparison would cost about $0.003.

6Limitations

No recordings from a real agent pipeline were tested. Proof, for example, is in private beta and its recordings are not public [2], so how much such recordings drift between runs is unknown.
One app, one interaction, one machine, 25 fps. Theme changes, long flows, scrolling and live network data were not tested.
Click times came from the Playwright script. A production pipeline would need its agent's action log, or would have to align by picture alone.
Thresholds come from three calibration pairs, with margins chosen after a first look at the data, and they are not stable. A second, fresh recording session on the same laptop (the code kit's test, Appendix A) found the same three things in 3 of 3 pairs each: the 8 px move, the flash and the slowdown. But its main recordings had no still-frame noise, so the still threshold fell from 2.06% to its 0.5% floor. It then caught the bolder titles in more frames, and it also raised false flags in two of three slower-against-slower pairs, all from one recording. Pooling more same-build pairs, or a fixed floor, is the next fix.
The second-label check (Section 5.7) was added after seeing these results and has only been run on these builds. It says that something changed in place, not what: it cannot tell a heavier font from different words.
At 25 fps a 75 ms flash is recorded unevenly (Section 5.5).
No person has reviewed the findings; the reviewers are AI agents.

7Where this could fit

For any tool that posts recordings as PR evidence, the results suggest a different shape for that evidence: one synced video in place of two, a timeline that marks where the recordings differ, and each sentence of the summary tied to the frame pair that supports it. Timing-only changes, which no screenshot tool reports, become a finding of their own. Recording main against itself once per repository would give the noise floor that every threshold depends on. A still-screenshot check is worth keeping alongside: it caught the font-weight change before this method did.

Visual proof· comment on the pull requestMock-up

Hover lift removed; nothing else changed

Recorded the same interaction on main and on this branch, three times each: load the page, add three toasts, hover the stack, move away. Lined up frame by frame.

2.84–3.96 sWhile the stack is hovered, the toasts sit 8 px lower on this branch. Same picture otherwise. In the description
whole runNo timing difference: the recordings never drift more than 70 ms apart, which is run-to-run noise.
3 runsSame result in all three recording pairs.

Thresholds come from main recorded against itself three times. A still-screenshot check is run alongside and reported separately.

Figure 10.A mock-up of a pull-request comment carrying visual proof, assembled from the tool's output for the first recording pair: one synced video in place of two, and a frame link on every sentence. It is a mock-up only, not any vendor's design or output. Whether a finding is in the PR's description was judged by the blind describe reviewer, not by the tool.

Three questions would decide whether this is worth building:

Do the recordings carry the agent's action timestamps?
Do reviewers watch both videos today, or read the summary sentence?
How different does main look from itself between two runs of the same pipeline?

References

K. Geiler. "The first time AI has done something meaningfully helpful." Substack, October 12, 2026. kennygeiler.substack.com/p/the-first-time-ai-has-done-something
Blacksmith. "Proof." Documentation, private beta. docs.blacksmith.sh/proof/overview
Blacksmith. "Why [code]smith." Documentation. docs.blacksmith.sh/codesmith/why
Playwright. "PageAssertions: toHaveScreenshot." playwright.dev/docs/api/class-pageassertions
Chromatic. "Snapshots." chromatic.com/docs/snapshots
Applitools. "Working with match levels." applitools.com/tutorials/guides/advanced-use-cases/choosing-match-level
Meticulous. "Architecture overview." app.meticulous.ai/docs/concepts/architecture-overview
BrowserStack. "Percy Visual Review Agent." browserstack.com/docs/percy/ai-agents/visual-review-agent/overview
"Animation2Code: Evaluating Temporal Visual Reasoning in Video-to-Code Generation." arXiv:2606.28593. arxiv.org/abs/2606.28593
E. Kowalski. "chore: remove lift interaction." sonner pull request #644, May 30, 2025. github.com/emilkowalski/sonner/pull/644
Blacksmith. "Pricing." blacksmith.sh/pricing, accessed October 20, 2026.
H. Sakoe and S. Chiba. "Dynamic programming algorithm optimization for spoken word recognition." IEEE Transactions on Acoustics, Speech, and Signal Processing 26(1):43–49, 1978. doi.org/10.1109/TASSP.1978.1163055
J. P. Lewis. "Fast normalized cross-correlation." Vision Interface, 1995.
B. Zitová and J. Flusser. "Image registration methods: a survey." Image and Vision Computing 21(11):977–1000, 2003. doi.org/10.1016/S0262-8856(03)00137-9
J. Canny. "A computational approach to edge detection." IEEE Transactions on Pattern Analysis and Machine Intelligence 8(6):679–698, 1986. doi.org/10.1109/TPAMI.1986.4767851
N. Cooper, C. Bernal-Cárdenas, O. Chaparro, K. Moran and D. Poshyvanyk. "It takes two to Tango: combining visual and textual information for detecting duplicate video-based bug reports." ICSE 2021. arxiv.org/abs/2101.09194
Y. Yan, N. Cooper, O. Chaparro, K. Moran and D. Poshyvanyk. "Semantic GUI scene learning and video alignment for detecting duplicate video-based bug reports" (Janus). ICSE 2024. arxiv.org/abs/2407.08610
S. Feng and C. Chen. "GIFdroid: automated replay of visual bug reports for Android apps." ICSE 2022. arxiv.org/abs/2112.04128
D. Zhao, Z. Xing, C. Chen, X. Xu, L. Zhu, G. Li and J. Wang. "Seenomaly: vision-based linting of GUI animation effects against design-don't guidelines." ICSE 2020, 1286–1297.
W. Liu, F. Lin, L. Guo, T.-H. Chen and A. E. Hassan. "GUIWatcher: automatically detecting GUI lags by analyzing mobile application screencasts." ICSE-SEIP 2025. arxiv.org/abs/2502.04202
R. Kohavi, D. Tang and Y. Xu. Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing. Cambridge University Press, 2020. (A/A tests.)
AMOS project, FAU Erlangen-Nürnberg, with e.solutions GmbH. "GUI Frame Diff." Open source, 2023–24. github.com/amosproj/amos2023ws03-gui-frame-diff
Happo. "Animations" (animate mode, happo 6.15.0, September 2026). happo.io/animations
Cash App. "Paparazzi" (animated snapshots with gif(), since 1.3.4, May 2024). github.com/cashapp/paparazzi
Factory. "Droid Control." Documentation. docs.factory.com/cli/features/droid-control
Su, Pradel and Chen. "RippleGUItester: change-aware exploratory testing." arXiv:2603.03121, March 2026. arxiv.org/abs/2603.03121
Meticulous. "Changelog" (Side-by-Side Replay, 14 September 2026). app.meticulous.ai/changelog

AReproduction

Everything ran locally on one laptop: Node 24, pnpm 9, Playwright 1.60.0 with Chromium, ffmpeg, Python 3.14 with NumPy and OpenCV. No paid API calls were made; the reviewers ran as Claude Code subagents. Sonner is MIT-licensed.

Download the code and recordings (zip, 2.1 MB). It contains the recorder, the method, the twelve recordings used here and this note's results for comparison:

./run.sh                       # analyse the included recordings: reproduces these numbers, about 2 minutes
.venv/bin/python compare.py    # check your numbers against this note's
./run.sh --record              # clone Sonner, build all four, record fresh videos, analyse (about 10 minutes)

From a clean copy, the first command matched every result in this note. The third was also run from a clean copy; that second session's results ship with the kit and are described in Section 6. The blind review needs a vision model and is not scripted; the kit's REVIEW.md has the prompts and this run's verdicts.

Film figures are reproduced from the ConformAI write-up [1]. All other figures are cut from the twelve recordings described in Section 4.