Decisions, not tokens: a shared-prefix VLM engine for live video

01The premise and its corrections

Most VLM demos ask a model to describe an image. Robots and monitoring systems don't need descriptions. They need answers to specific questions, many at once, over and over:

Is a person present? Is the person holding a pen? Is the door behind them open? Is anyone on their phone?

Asking a VLM each of those as a separate prompt means re-encoding the image and decoding text every time. This project builds the opposite: encode the frame once, fork every question off that shared state in a single batch, and read a probability instead of generating tokens.

frame ──► vision encoder ──► LM prefill ──► prefix state
                                              ├─► "Is a person present?"          → P(Yes)
                                              ├─► "Is the person holding a pen?"  → P(Yes)
                                              └─► … N questions, one batched forward

The starting point was an architecture blueprint for an "edge-optimized visual decision engine". Its argument: a VLM that answers structured questions about a camera frame should never decode text. Instead:

1. Causal prefix. Run the vision encoder and the language model over the image once, and keep the resulting KV cache as the frame's state.
2. Tree attention. Pack many questions (<q> Is a human present? <opt> Yes </opt> …<decide>) behind that prefix, with a block-diagonal mask so each branch sees the image and itself, never another branch.
3. Pointer head. Read the hidden state at each <decide> token and compare it against the option tokens to get a probability distribution in one forward pass.

A review of the blueprint corrected several claims before I wrote code. Visual KV reuse is intra-frame, not temporal: a new frame needs a new vision pass and a new prefill. Radix-style prefix caching deduplicates identical tokens, not similar-looking frames. Option-token hidden states are not class prototypes, so a pointer head needs training. And 30 Hz is not implied by any of this for a 2B-parameter model at useful resolution.

So the project became a sequence of questions, each answered with a measurement:

• Can the shared prefix be made exactly equal to running each question alone?
• On what kind of decision does a VLM beat a cheap image classifier?
• How deep into the model does each kind of decision become readable?
• Does the model use time when given video, and can the whole thing run live?

02Making the shared prefix exact

The first implementation (PyTorch, Qwen2-VL-2B on MPS) looked plausible on reading. It had four bugs stacked on top of each other, and a fallback that would have hidden them.

Bug 1: M-RoPE positions are not token indices

Qwen2-VL uses multimodal rotary embeddings: image tokens get 3D (temporal, height, width) positions that span far fewer indices than there are tokens. A prefix of 282 tokens (256 of them visual) ends at text position 41, so the first query token belongs at position 42, not 282. The suffix pass passed no position_ids, so transformers 5.17's compute_3d_position_ids built positions from a 2D mask covering prefix + suffix, for an input that was suffix-only. The result could not line up with the suffix tokens, and it also leaned on rope_deltas state left behind by earlier calls.

Bug 2: failures were caught

A try/except around the suffix pass fell back to the naive path, which repeats the image once per question. Any failure in the shared path would have been timed as the baseline and reported as the "optimized" number. I removed the fallback: a failure should raise.

Bug 3: a trailing space changed the tokens

The prefix ended in "…based on the image: ". Tokenized as one string, ": Is a human…" merges the space into " Is". Tokenized as prefix and suffix separately, it does not. Different token ids meant the shared path could never match a full-sequence run (cosine 0.997, max diff up to 5.5). Ending the prefix with "\n" fixes it, because the Qwen pre-tokenizer always splits newlines into their own token:

'image: '  + 'Is a human present?'  -> ids equal to full string? False
'image:\n' + 'Is a human present?'  -> ids equal to full string? True

Bug 4: the cache API moved

transformers 5.x stores DynamicCache as layers[i].keys / .values. The old code looked for key_cache, silently took a legacy tuple branch, and never exercised the intended path.

The fixed design

I did not build a literal packed sequence with a tree mask. Each question is a row in a batch of N, all reading one copy of the prefix KV cache. Attention never crosses batch rows, so branch isolation is exactly the block-diagonal property, with no mask to materialize:

# engine.py (torch), core of decision_hidden()
text_pos     = torch.arange(prefix.next_pos, prefix.next_pos + S)      # starts at 42, not 282
position_ids = text_pos.view(1, 1, -1).expand(3, N, -1)                # text: t = h = w
attention    = torch.cat([ones(N, prefix_len), suffix.attention_mask], dim=1)  # right padding
out = model.model(input_ids=suffix.input_ids, attention_mask=attention,
                  position_ids=position_ids,
                  past_key_values=expanded_cache(prefix, N))           # fresh cache object per call
decision = out.last_hidden_state[rows, suffix.attention_mask.sum(1) - 1]   # last real token

Explicit positions bypass compute_3d_position_ids, so nothing depends on the model's cached rope_deltas state. A new cache object per call means the shared prefix is never mutated.

Proving it, with a control that must fail

check_equivalence.py compares each question's decision hidden state from the batched path against one ordinary full-sequence forward. An equivalence test that always passes proves nothing, so the script has a --wrong-pos mode that deliberately uses prefix_len as the start position.

What sharing buys

Qwen2-VL-2B, MPS, bf16, 448² budget (256 visual tokens), p50 of 20 runs after 3 warmups, with torch.mps.synchronize() around every measurement.

The frame encode is 329 ms of each total. Sharing makes additional questions almost free, but it does nothing for that fixed cost, and with a single question the extra pass costs 40 ms. The shared prefix also is not "stored once" in memory yet: DynamicLayer.update concatenates, so expanding the cache to N rows materializes N copies of the prefix KV. The compute is shared, the bandwidth is not. The hybrid model in section 5 removes most of that cost.

03An honest baseline

A decision engine is only useful if it beats cheaper models on the decisions you care about, so I built an evaluation set before optimizing anything.

The first corruption draft was too easy: a flat, opaque blob covering 60–70% of the frame, and near-black low-light images. Any model would have scored near 1.0, and the test would have measured nothing. I made both harder before running any model.

The comparison is a linear probe (logistic regression) on frozen features for every model, plus the VLM's zero-shot P(Yes). The VLM's feature is the decision token's hidden state; SigLIP's is its image embedding.

Test split AUROC over six tasks. "Small" means the largest instance covers under 1% of the frame. PyTorch MPS, bf16.

• Resolution matters little here. Qwen3-VL at 224 is within 0.003 of 448, at 70% of the encode time.
• Zero-shot probabilities rank well but are not calibrated. At a 0.5 cutoff, Qwen2-VL @448 missed 42% of vehicles and 75% of obstructions. A probe on the same forward pass cut vehicle misses to 6% and calibration error from 0.13 to 0.04.
• On MLX, 4-bit weights were slower than bf16 (Qwen3-VL-2B: 120 vs 81 ms at 224, 170 vs 148 ms at 448). Prefill over hundreds of tokens is compute-bound, so dequantization is pure overhead. This contradicts the blueprint's bandwidth argument for quantizing.

04The decision ladder

The COCO tasks test what classifiers already do well. To test what a VLM should be good at, I built a graded set of 10,304 yes/no questions over 3,371 images. Splits are by image, so a probe never sees a test image.

The baseline changes too. On question-specific tasks an image-only classifier cannot see the question, so the fast competitor becomes SigLIP-so400m image and text embeddings combined as [i, t, i ⊙ t] into a logistic probe. A text-only probe measures how much the answer can be guessed from the wording alone.

How deep does each decision need to go?

Because the forked batch returns hidden states at every layer, I fitted a probe per layer. The chart puts both models on relative depth, so their 28 and 24 layers line up.

• Reasoning appears abruptly, then plateaus. Qwen3-VL is near chance until layer 12 and jumps between layers 14 and 16 (spatial 0.66 → 0.82). Qwen3.5 reaches its plateau earlier, by layer 14 of 24 (58% of depth). Stopping there keeps essentially all the accuracy.
• Presence does not need the language model at all. The mean of the visual tokens straight out of the vision encoder (layer 0, before any language-model layer) scores 0.97–0.98, against 0.99 for the best full-depth readout.

The architecture this suggests has tiers: camera health on SigLIP, presence on pooled vision features, and the language model only for relational and open-ended questions, stopped early when a trained head allows it.

05Qwen3.5 on MLX: forking a hybrid model

Qwen3.5-2B (February 2026) has the same vision grid as Qwen3-VL but a hybrid language model. That changes what "the prefix state" is:

Forking N question branches from this state can be free. The question is whether the suffix pass ever writes into shared memory. Reading mlx-vlm's cache code settled it: ArraysCache.update_window and the gated-delta kernel replace the conv and recurrent states, and KVCache reallocates when tokens are appended. So a fork can be a set of broadcast views:

# mlx_engine.py
def _fork(cache, n, n_layers):
    forked = []
    for c in cache[:n_layers]:
        if isinstance(c, KVCache):                         # 6 full-attention layers
            keys, values = c.state
            f = KVCache()
            f.keys   = mx.broadcast_to(keys,   (n,) + keys.shape[1:])
            f.values = mx.broadcast_to(values, (n,) + values.shape[1:])
            f.offset = c.offset
        else:                                              # 18 Gated DeltaNet layers
            f = ArraysCache(size=len(c.cache))
            f.cache = [None if x is None else mx.broadcast_to(x, (n,) + x.shape[1:])
                       for x in c.cache]
        forked.append(f)
    return forked + [None] * (len(cache) - n_layers)

I drive the decoder layers directly instead of calling mlx-vlm's forward. Its forward has special paths for left padding and per-row splitting that I do not need. Owning the loop also gives per-layer hidden states and early exit (n_layers=k) for free. The prefix/suffix split comes from rendering the real chat template around a placeholder query. That picked up something easy to miss by hand: Qwen3.5's generation prompt ends in an empty <think>\n\n</think>\n\n block, so the decision token sits after it.

A control that barely moved

The forked batch is bit-identical to full-sequence runs in both fp32 and bf16, at every layer and at early exit. But the wrong-position control only moved the output to cosine 0.9987, compared with 0.89 for Qwen3-VL. Only 6 of the 24 layers use rotary positions at all. A tolerance of 0.999, which looked strict, would barely have caught a position bug. Since the correct path is exact, I tightened the tolerance to 0.99999 in both precisions.

Latency

MLX, bf16, p50 of 15 runs. The vision encoder alone takes 29.6 ms at 224 and 70.1 ms at 448. Six questions as independent full sequences take 479 ms at 224 and 977 ms at 448, so sharing is 3.9× and 4.9× faster.

At 224 the vision encoder alone uses 30 of a 33 ms budget, so a 2B VLM cannot reach 30 Hz on this machine at any resolution. The tiers exist because of that floor. On accuracy, Qwen3.5 tied Qwen3-VL (spatial +0.014, GQA −0.007), with a smaller language-model state and an earlier plateau.

06From frames to video

Qwen3.5 reads video natively. Frames go in pairs (temporal patch size 2), and each pair is rendered as a timestamp marker followed by its vision block:

<|im_start|>user\n
<0.1 seconds><|vision_start|><|video_pad|> × 48 <|vision_end|>
<0.6 seconds><|vision_start|><|video_pad|> × 48 <|vision_end|>
…
Please answer the following question based on the video:\n

The vision encoder's attention is block-diagonal by group: cu_seqlens repeats h × w once per temporal group. A group's vision features therefore do not depend on its neighbours. Each 2-frame group can be encoded once, cached, and reused by every window that contains it.

video_engine.py builds a window from cached groups:

• render the window's token sequence with mlx-vlm's own _timestamped_video_placeholder;
• take positions from get_rope_index;
• scatter the cached features with merge_input_ids_with_image_features;
• run the language model.

Reusing the library's helpers means the token layout cannot drift from the reference path.

07Does the model use time?

Every video eval runs three conditions per clip, each a separate window prefill with the clip's questions in one batch:

• full: frames in order;
• single: the middle frame alone;
• shuffled: the same groups in a random order.

Temporal gain is full minus single.

MVBench

I downloaded the MVBench subsets whose videos ship publicly (STAR, CLEVRER, SSv2, Moments in Time, VLN-QA): 11 tasks with about 200 questions each. I sampled 16 frames uniformly and set timestamps to match real time, then read the answer as a softmax over option-letter logits.

Qwen3.5-2B, MLX bf16, 224 budget, 16 frames. Chance is the mean of 1 / options.

Synthetic motion with known answers

MVBench says little about the questions a robot asks, such as "is that person approaching?", so I generated 600 labelled clips (4 s at 8 fps, 3,000 questions) from COCO:

• Camera motion over a still photo: zoom in/out (approach/recede), pan left/right, and static.
• Object motion: a real COCO person, cut out with its segmentation polygon and pasted onto a person-free background. It grows (approaching), shrinks, moves sideways, or stays still.

Every moving clip has an exact reversal, frames[::-1], whose answers must flip. A model that answers from appearance cannot pass that test.

Shuffled groups scored 0.623 on average. Window length barely mattered (1 s: 0.647, 2 s: 0.665, 4 s: 0.673), nor did sampling at 4 instead of 8 fps (0.659). Reversal consistency is the share of forward/reverse pairs whose P(yes) moves in the correct direction; 0.5 is chance.

08Motion lives in the features

A wrong zero-shot answer does not mean the information is missing. I cached the decision-token hidden state at every layer for every synthetic clip, trained probes on sources 0–39, and tested on sources 40–59: unseen photos and unseen people. A forward clip and its reversal always stay on the same side of the split.

The vision-only baseline uses no language model at all. It is a logistic probe on [pooled(last group) − pooled(first group), mean(pooled)], a 4,096-dimensional vector the runtime already has for free.

Test AUROC on held-out sources 40–59. The language-model probes read the decision token of the full 2 s window; layer 0 of that token is uninformative by construction (0.47–0.52).

The runtime ships these as motion heads (heads/motion_*.npz). Each head is trained on sources 0–29, with a temperature fitted on sources 30–39 and a test on 40–59. The approach head reaches 0.945 AUROC and 88% accuracy at a 0.5 cutoff. It runs as a Tier 1 question, answered about 43 ms after the newest frame, with no language model.

09The runtime

python -m runtime.run --source webcam|clip.mp4 turns all of the above into a live loop. It runs as one MLX process, with no torch at runtime.

%%{init: {'theme':'neutral'}}%%
flowchart LR
  SRC["Frame source
webcam or file"] --> CAP["Capture thread
sample 8 fps into 2-frame groups"]
  CAP --> T0["Tier 0 · camera health
brightness, blur, frozen, occlusion
every frame, CPU"]
  CAP --> Q[("Group queue
bounded, drop-oldest")]
  Q --> VIS["Vision encoder
once per group, cached"]
  VIS --> T1["Tier 1 · linear heads
presence: newest group
motion: window feature change"]
  VIS --> ST["Running stream
append group once
rebuild every 2 windows"]
  ST --> T2["Tier 2 · VLM
fork + all due questions
in one batch"]
  T0 --> BUS["Decision bus
EMA + hysteresis"]
  T1 --> BUS
  T2 --> BUS
  BUS --> OUT["Overlay viewer · JSONL log"]

The decision contract

@dataclass
class Decision:
    question_id: str
    answer: str        # "yes" | "no" | "uncertain" | an option
    p: float           # smoothed probability
    tier: int          # 0 camera health, 1 linear heads, 2 VLM
    t_start: float     # oldest frame the answer is based on
    t_end: float       # newest frame the answer is based on
    t_decided: float
    p_raw: float       # unsmoothed model output
    depth: int         # LM layers used
    # age_ms = t_decided - t_end: consumers can reject stale answers

A VLM answer describes frames that are already a couple of hundred milliseconds old. The contract carries that age explicitly, so a consumer never mistakes it for the present.

Scheduling rules

• Latency stays bounded. In realtime mode, when the worker falls behind, the oldest queued groups are dropped and the window restarts. A window must be contiguous or its timestamps would be false. Tier 2 is skipped while a backlog exists.
• Tier 2 fires on every new group, up to each question's rate. Scene changes (a jump in pooled-feature cosine distance) ask every question immediately.
• Tier 1 publishes before any language-model work, so presence and motion answers never wait behind a prefill.

Asking your own questions

Questions are data, not code. Each lives in a YAML file with a short id for the overlay panel; --questions my.yaml loads a file and --ask "…" adds one-off questions on the command line. Yes/no and multiple-choice questions share one batch per tick:

- id: coffee
  text: Is the person drinking from a cup or mug?
  enter: 0.55     # switch on fast: a sip lasts a second or two
  ema: 0.7
- id: activity
  kind: choice
  text: What is the person mainly doing?
  options: [typing, writing, drinking, on phone, reading, idle, away]
  rate_hz: 1      # slow-changing: once a second is enough

A desk-work set of 18 questions (pen, coffee, phone, glasses, headphones, looking away, a second person, and an activity choice) batches into 136 ms per tick, 220 ms for the whole tick, just inside the 250 ms needed for 4 Hz. Phrasing matters in ways specific to the camera: from a laptop's own webcam the laptop is not in view, so "Is the person sitting in front of a laptop?" scored 0.23 on a frame of someone doing exactly that, while "Is the person looking toward the camera?" scored 0.98.

Heads must be trained on the path they run on

The first Tier 1 heads were trained on the image-path features from the ladder study. The runtime computes features on the video path (encode_group with the frame duplicated), and the two processors resize differently. The features agreed on average (cosine 0.97) but not everywhere (minimum 0.84). Ranking quality survived, but individual probabilities moved by up to 0.66, enough to break any calibrated threshold. I retrained on video-path features: AUROC 0.979 for human, 0.966 for vehicle and 0.964 for animal, with a 7% miss rate for humans at 0.5. That was still not enough, as the first real webcam session showed (section 11).

Bugs the benchmark found

I built a replay benchmark from held-out synthetic clips: 128 s, alternating motion and static segments, with ground truth per segment. I played it through the runtime in real time. It found three scheduling bugs that unit tests would not have:

1. Rate limit on the wrong clock. A question's "last asked" time was the decision time, which lands about 230 ms after the frames. The next group, 250 ms later, looked too soon, so every other tick was skipped: 2.0 Hz. The rate is now measured on video time.
2. Tolerance smaller than jitter. Realtime file replay stamps frames after a sleep, so group gaps ran from 242 to 258 ms around a nominal 250. A 1 ms tolerance rejected 38% of groups. It is now 50 ms, still at most one answer per group.
3. Tier 1 behind the language model. Adding streaming (next section) put the append before the heads, and Tier 1 age rose from 28 ms to 132 ms.

10Streaming prefill

With a sliding window, every tick re-prefills 2 s of video, about 450 tokens. That took 133–149 ms, 65% of the tick. The model is causal, so appending group k to a state that already holds groups 0…k−1 gives the same result as encoding all k+1 groups at once. The one obstacle is the template: the instruction text comes after the video. A stream therefore holds only the header and the groups, and the instruction moves to the start of every question's suffix. That is still exact, and the test suite confirms stream = window = native path.

Where an append spends its time

A small append costs about 1.6 ms per layer. That is fixed per-call overhead rather than per-token compute: 8× fewer tokens were only 3.6× cheaper. This decided the design:

• Two staggered streams were worse than one. Keeping two streams, so one always has enough history, meant two appends per group and paid the fixed cost twice. The tick dropped only from 221 to 196 ms.
• One stream plus a rebuild. When the stream reaches 2 windows' worth of groups, it is rebuilt from the last window's cached groups in a single forward, about 130 ms once every 2 s. Questions always see 2–4 s of history, and the eval showed window length barely affects accuracy.
• Positions are precomputed. They depend only on the token layout, never on pixel content, so they are computed once per frame size and sliced. That removed 6 ms per append.

Result

Questions cost a little more in stream mode because they read up to 4 s of history and each suffix carries the instruction. The net result is a 33% shorter tick, answers 76 ms fresher, and a third of the GPU free for more questions or a higher resolution. Stream mode is the runtime's default.

11Field test: the webcam said I wasn't there

The first real webcam session showed T1 human_present: no (0.20), steadily, with a person sitting in the middle of the frame. Benchmarks had put this head at 0.98 AUROC.

Ruling things out

video_eval/webcam_probe.py grabs frames through exactly the runtime's capture and vision path and reports what the head receives. Nothing about the frames was wrong:

Isolating it on the real frame

The model saw the person; the cheap head did not. The first hypothesis was dilution: the head reads the mean of all visual tokens, and a strongly patterned curtain filled much of the frame. Adding the max over tokens to the mean made it worse (0.10–0.12). The real cause was simpler: the head was a logistic regression over 2,048 dimensions trained on 307 images. That is enough to score well on more COCO photos and not enough to leave their distribution.

The fix: data first, then pooling

I added 3,000 COCO train2017 images (half with people, many indoor), which never overlap the val-based test split, and chose mean or mean+max pooling per head on that test split.

• Webcam frames: 0.16–0.22 → 0.56–0.69, above the 0.4 "yes" threshold. Pure background stays near 0.01.
• Why mean + max for humans: better calibrated (error 0.018 vs 0.030), and a crop of a shirt hanging on a door scored 0.15 instead of 0.43.
• A cross-check: the default questions now include a Tier 2 "Is there a person in the video?". On the webcam clip, Tier 1 says yes (0.61) and Tier 2 says yes (1.00), so a safety-relevant answer never rests on the cheap head alone.

12What I learned

• Every exactness claim needs a control that must fail. The wrong-position control worked well on Qwen2-VL (cosine 0.89) and barely at all on Qwen3.5 (0.9987). Without it, a loose tolerance would have blessed a position bug.
• Tokenization and positions caused more of the early bugs than attention did. A trailing space and an index convention broke equivalence long before the attention logic did.
• A VLM is worth its cost for relational decisions, not detection. Presence is solved by the vision encoder alone or an 8 ms classifier. Spatial and compositional questions go from chance to 0.85+.
• Zero-shot answers rank better than they decide. Uncalibrated probabilities, and for motion direction systematically wrong ones, sit on top of internal states that a linear probe reads well.
• On Apple Silicon at 2B scale, fixed per-layer overhead dominates small calls. This made 4-bit weights slower than bf16, made two streams slower than one, and set the vision encoder as the floor on frame rate.
• Train heads on the exact runtime path, with enough variety. Two processors that resize differently moved calibrated probabilities by up to 0.66, and a head trained on 307 images scored 0.98 on COCO and 0.2 on a webcam.
• Test on the device, early. A six-frame webcam probe found a failure that 1,100 benchmark images, a 10k-question ladder and a replay benchmark had not.
• End-to-end replay finds what unit tests cannot. Clock domains, jitter and ordering mistakes only show up with real timing.

Open problems

• Real footage. Validate the motion heads on recorded clips of people approaching, receding and passing, and evaluate the presence heads across rooms, lighting and people.
• Per-question thresholds. The default hysteresis thresholds (0.65 / 0.35) can erase a correct but uncalibrated zero-shot signal. Each question needs thresholds fitted to its own calibration.
• Accuracy of the live replay. The replay has only two segments per motion kind, which is too few to rank methods. The offline held-out evaluation remains the accuracy measure.
• Reasoning accuracy. 77–80% on spatial and compositional questions is the weakest number here. The next test is Qwen3.5-4B on the reasoning tier.

13Reproducing it

Two environments: venv (PyTorch, transformers 5.17, sklearn) for training probes and heads, and venv-mlx (MLX 0.32, mlx-vlm 0.7.2) for everything that runs.

# exactness
venv-mlx/bin/python mlx_check_equivalence.py --fp32            # image engine, Qwen3.5
venv-mlx/bin/python mlx_check_equivalence.py --fp32 --wrong-pos  # control: must fail
venv-mlx/bin/python -m pytest tests/test_video_equivalence.py -q # video + streaming, 6 tests

# data
venv/bin/python data/build_eval_set.py                          # COCO presence + synthetic faults
venv/bin/python data/build_ladder.py                            # presence, VSR, GQA
venv-mlx/bin/python video_eval/build_synthetic_motion.py        # 600 motion clips

# studies
venv/bin/python eval.py --report                                # section 3
venv-mlx/bin/python mlx_ladder_extract.py --budget 448 && venv/bin/python ladder.py report
venv-mlx/bin/python video_eval/eval_video.py mvbench            # section 7
venv-mlx/bin/python video_eval/motion_probes.py extract && venv/bin/python video_eval/motion_probes.py fit

# heads
venv-mlx/bin/python data/build_head_trainset.py                 # 3,000 extra COCO train2017 images
venv-mlx/bin/python video_eval/validate_heads.py --dump --dump-train   # runtime-path features
venv/bin/python export_heads.py                                 # presence heads, pooling per head
venv/bin/python video_eval/motion_probes.py export              # motion heads

# runtime
venv-mlx/bin/python -m runtime.run --source webcam
venv-mlx/bin/python -m runtime.run --source webcam --questions my.yaml --ask "Is someone waving?"
venv-mlx/bin/python video_eval/webcam_probe.py                  # what the heads see on your camera
venv-mlx/bin/python -m runtime.run --source data/video_eval/replay.mp4 \
    --questions video_eval/replay_questions.yaml --no-view --log results/replay_run_stream.jsonl
venv-mlx/bin/python video_eval/runtime_bench.py analyze results/replay_run_stream.jsonl \
    --segments data/video_eval/replay.json