A 16-frame grid costs 2.1% of 1 fps video on GPT-4o-mini, and the least accurate random forest reached the most correct conclusions

What Zhu and Gligorić posted on 30 September 2026

Zhixi Zhu and Kristina Gligorić, both at Johns Hopkins University, posted Paying for Too Many Tokens? Valid and Cost-Efficient Multimodal LLM Annotation with Simple Heuristics as arXiv:2610.00809v1 on 30 September 2026. The repository README lists it as an AACL-IJCNLP 2026 paper. It is a short benchmark paper with a 15-page PDF including appendices.

The setup has three knobs. Frame selection: uniform (Uni) or shot-transition (ShotT), where ShotT picks local maxima of a grayscale-difference plus histogram chi-square score. Grid: 9 frames (3x3) or 16 frames (4x4 for MELD, 2x8 for TikTok) packed into one image. Modality: frames only, transcript only, or both. Each configuration is scored three ways: accuracy and Cohen's kappa, downstream inferential validity, and per-video token cost. The main model is GPT-4o-mini at temperature 0.3; Claude Sonnet 4.6, GPT-5.1 and Grok 4.3 are run on 100-item subsets (Tables 6 and 7).

Two tasks. MELD: 900 clips of 1 to 6 seconds from Friends, 3-way sentiment, and a downstream logistic regression of each sentiment on main-cast versus supporting-cast. TikTok: 900 US videos from October 2025 to January 2026 under 18 trending hashtags, median about 46 seconds, mean 62.3 seconds, maximum 701 seconds, with the creator's hashtag as the reference label and per-hashtag frequency as the downstream estimand. Bias correction uses prediction-powered inference (PPI) with 100 human-labelled items.

The headline claims: accuracy and validity diverge; text alone wins on MELD while grids win on TikTok; and a single 2x8 grid comes within kappa .05 of full-video understanding at about 15% of the token cost.

Does the cost table rebuild from first principles, and which ratio is the 14.8%?

Every token count in Table 2 is an integer number of OpenAI image tiles. OpenAI's vision guide prices a high-detail gpt-4o-mini image at 2,833 base tokens plus 5,667 per 512px tile, after scaling the image to fit 2048 by 2048 and its short side to 768. A vertical 9:16 frame becomes 768 by 1365, which is 2 by 3 tiles, or 36,835 tokens. Divide the paper's 1 fps column by that and you get exactly 3.0 frames at the minimum, 701.0 at the maximum and 61.8 on average, matching the stated 3 second to 701 second range and 62.3 second mean.

Prices read from developers.openai.com (gpt-4o-mini $0.15 input, $0.60 output; gpt-4o $2.50 input) and ai.google.dev (Gemini 2.5 Flash $0.30 input for text, image and video) on 4 October 2026. The repository's Gemini script calls google/gemini-2.5-flash through OpenRouter; the paper does not name the Gemini version.

The rebuild says something the paper does not: these are image-token estimates only. The 2x8 maximum is exactly 48,169, the ceiling for one image under OpenAI's rule (2048 by 768 is 4 by 2 tiles), so no prompt or transcript tokens can be inside it. The paper's footnote that the MELD 4x4 and TikTok 2x8 grids "incur the same per-image token cost" also fails the formula if MELD frames are 16:9: a 4x4 grid of 16:9 frames keeps a 16:9 shape and lands on 6 tiles (36,835 tokens), against 8 tiles (48,169) for the 2x8 of vertical frames. And a 3x3 grid of 9:16 frames has the same shape as one frame, so it is billed as one 1 fps frame.

The abstract's "kappa within .05 at about 15% of the token cost" joins two comparisons. The 14.8% is the Gemini grid against Gemini full video, where Table 4 shows no gap at all (.860 and .860). The kappa gap belongs to GPT-4o-mini's grid against Gemini video, and it is .054 (.860 minus .806), just outside .05. Read on its own numbers, the Gemini result is stronger than the abstract says.

Priced at list rates, the model choice outweighs the frame heuristic. Gemini 2.5 Flash watching the whole video costs about $0.00473 per video at kappa .860; GPT-4o-mini reading one 2x8 grid costs $0.00715 at kappa .806 on the same 100 videos. The reason is the gpt-4o-mini tile price: 5,667 tokens per tile against 170 for gpt-4o, 33.3 times more tokens at 16.7 times lower price per token, so the same 8-tile grid costs $0.00723 of image tokens on gpt-4o-mini and $0.00361 on gpt-4o. This is our arithmetic on published price sheets; the paper does not price Gemini and does not run GPT-4o.

Why every GPT-4o-mini count is 2,833 plus a multiple of 5,667

The cost decoupling the paper reports is a property of the provider's image tokenizer, which the authors acknowledge in their limitations ("contingent on ... API image-token policies"). The diagram traces both paths through OpenAI's resize and tile rule.

Two consequences follow. First, grid cost scales with the grid's aspect ratio, not its frame count: 16 frames in a wide 2x8 strip cost 31% more than 9 frames in a 3x3 square, because the strip covers 8 tiles and the square 6. Second, the grids pay for resolution they cannot use. A 1728 by 768 grid of 16 frames gives each frame about 216 by 384 pixels, the detail a single low-resolution tile could almost carry, at 8 tiles of price. Whether a smaller tile budget would hold accuracy is untested in the paper; we flag it as an open lever, not a finding.

Which MELD setup reaches the right conclusion? The one ranked third on accuracy

The paper's own Table 1a supports a stronger version of its "accuracy and validity diverge" claim than the text makes. Ground truth on MELD has one null and two effects: Positive (beta .13, p=.096), Neutral (beta -.31, p below .001) and Negative (beta .27, p below .001). We scored each configuration on whether a researcher would draw the same conclusion, meaning the same sign and the same side of p=.05, and on mean absolute coefficient error.

Spearman's rho between accuracy rank and correct-conclusion rank is -0.65 over these four points. With n=4 that is illustrative, not a test. The random forest on video plus transcript embeddings, ranked third of four on accuracy, is the only raw source that detects both true effects in the right direction; it overstates the Negative coefficient four-fold (1.08 against .27), which the paper reports as distortion. Text-only, the accuracy leader, gets the Negative effect significant in the wrong direction (beta -.43, p=.038). ShotT 4x4 points the wrong way on both real effects and calls neither significant.

PPI behaves as designed and still leaves a researcher with nothing. Its three 95% intervals, [-1.0, 1.9], [-1.6, .7] and [-1.0, 1.5], contain the ground truth and also zero, so with 100 labels it reports no effect for all three sentiments, two of which are significant at p below .001 in the full data. The intervals are 2.3 to 2.9 units wide around effects of .13 to .31. The paper's Appendix Table 8 points the same way: at 200 human labels the across-draw spread of the estimates is still .36 to .51.

TikTok repeats the pattern on frequencies. From Table 1b, mean absolute frequency error across the 18 hashtags is 1.06 points for the raw ShotT 2x8 VLM (accuracy .819), 0.88 for the video random forest (.717) and 0.83 for the video plus transcript forest (.718). Accuracy rank and error rank are reversed again (rho -0.5 on three points). PPI's point estimates average 0.99 points of error and are closer to ground truth than the raw VLM on 9 of 18 hashtags, further on 8, tied on 1. All 18 intervals cover ground truth, as the paper states; their mean width is 5.53 points, against a mean hashtag frequency of 5.56%.

Is the code released where the paper says it is?

No. The paper points to github.com/JHU-CSS/multimodal-annotation, which returned HTTP 404 on 4 October 2026, both the web page and the GitHub API. The JHU-CSS organisation has three public repositories; the matching one is JHU-CSS/multimodal-llm-annotation, created 26 September 2026, last pushed 27 September 2026, 2 commits, no licence file, 26 files. It holds frame sampling with the shot-transition detector, prompts, annotation runners for OpenAI and OpenRouter, a Gemini runner, evaluation and a PPI script.

What the repository does not hold matters for anyone trying to reproduce Tables 1 and 2. The TikTok list has 2,157 video IDs (120 per hashtag, 117 for dance), not the 900 annotated or the 1,000 in the PPI script, and no split file says which. There are no model annotations, no human labels, no MELD clip list, and no token or cost script. Three settings differ from the paper: --temperature defaults to 0.0 (paper: 0.3); the Gemini runner fixes temperature 0; and ppi/ppi_correct.py builds normal-approximation intervals from one draw, while Section 3.1 describes bootstrap intervals averaged over 100 trials. The paper's sensitivity tables sample from "a pool of 1,000 videos" while the main text uses 900, and Table 9's ground-truth frequencies (funnyvideos 5.11%, travel 6.01%, anime 4.60%) differ from Table 1b's (5.0%, 6.0%, 4.4%), consistent with a different base set.

Smaller numerical slips: the grid-versus-video crossover is 2,332 / 281 = 8.3 seconds, which the paper rounds to "roughly 9 seconds", and the saving at the 701-second maximum is 98.8% on those rates, where the text says about 97%. Neither changes a conclusion.

What this means if you annotate video with a VLM

Price the provider's image tokenizer before you choose a heuristic. On GPT-4o-mini the grid saves 97.9% to 98.4% against 1 fps, but the tile rule makes that model's images expensive: the same grid is half the price on GPT-4o, and Gemini 2.5 Flash on the raw video was cheaper than either GPT-4o-mini grid in our pricing. Compute tokens per image from the published rule for your frame size and grid shape; a 3x3 of vertical frames is billed as one frame, a 2x8 as the image ceiling.

Once the grid is in place, gold labels are the bill. The full 3x3 pass over 900 TikTok videos costs $4.94 in input tokens. The paper recommends 100 to 200 human labels for PPI (about 200 for regression coefficients). Any gold label priced above 2.5 cents makes 200 of them cost more than the whole VLM pass; the saving against 1 fps, $302.24, pays for 200 labels at $1.51 each. Spend the effort on label quality and adjudication, which is where the variance now sits.

Select configurations on downstream agreement, not accuracy. On MELD the accuracy leader got a significant effect backwards. If the pilot set is labelled anyway, score each candidate on the estimand you will report: run the regression or frequency estimate on the pilot with VLM labels and with human labels and compare. In this paper that procedure would have picked a random forest over both VLM setups for MELD.

Read PPI width as the result, not a footnote. PPI keeps coverage, but at 100 labels it erased two real MELD effects and its TikTok intervals are as wide as the frequencies they estimate. Our judgement: report the corrected interval and the uncorrected point estimate side by side, and size the gold set from a pilot power calculation on the estimand rather than a fixed 100.

Treat the reference label as part of the design. The TikTok target is the creator's hashtag, which the authors note captures intent and reach-seeking. A hashtag classifier at kappa .816 agrees with creators, not with a content codebook. For content categories, a content-grounded human gold set is the reference that makes the validity claim mean what a reader assumes. Related reading: the cost and accuracy tradeoff for text annotation models and reproducibility and construct validity of LLM labels.

Check it yourself

Everything above uses the PDF tables, two provider price pages and the repository. No videos or API calls are needed. The tile rule, Table 2 rebuild and MELD recount:

python3 - <<'PY'
import math
def tiles(w,h):                      # OpenAI high-detail rule
    s=min(1,2048/max(w,h)); w,h=w*s,h*s
    if min(w,h)>768: s=768/min(w,h); w,h=w*s,h*s
    return math.ceil(w/512)*math.ceil(h/512)
mini=lambda t:2833+5667*t; g4o=lambda t:85+170*t
for n,(w,h) in {'frame 9:16':(1080,1920),'3x3':(3240,5760),'2x8':(8640,3840)}.items():
    print(n,tiles(w,h),mini(tiles(w,h)),g4o(tiles(w,h)))
print(2275411/36835,110505/36835,25821335/36835)   # 61.77 3.0 701.0
for t in (36596,47665): print(t/2275411, 2275411/t) # 1.61%, 2.09%
print(25821335/48169, 2332/15774, 2332/281)        # 536x, 14.8%, 8.3 s
GT={'P':(.13,.096),'U':(-.31,.0009),'N':(.27,.0009)}
C={'ShotT':{'P':(.05,.8),'U':(.25,.205),'N':(-.39,.07)},
   'Text':{'P':(.29,.186),'U':(.12,.542),'N':(-.43,.038)},
   'RF':{'P':(-.61,.039),'U':(-.82,.0009),'N':(2.07,.0009)},
   'RFVT':{'P':(-.04,.879),'U':(-.58,.015),'N':(1.08,.003)},
   'PPI':{'P':(.48,.528),'U':(-.48,.411),'N':(.19,.47)}}
f=lambda b,p:0 if p>=.05 else (1 if b>0 else -1)
for k,v in C.items():
    print(k,sum(f(*v[s])==f(*GT[s]) for s in GT),
          round(sum(abs(v[s][0]-GT[s][0]) for s in GT)/3,3))
PY

# release check (4 October 2026)
curl -s -o /dev/null -w "%{http_code}\n" https://api.github.com/repos/JHU-CSS/multimodal-annotation      # 404
curl -s https://api.github.com/orgs/JHU-CSS/repos | grep '"full_name"'
git clone -q https://github.com/JHU-CSS/multimodal-llm-annotation r && cd r
git log --format='%ad %s' | wc -l                     # 2
tail -n +2 dataset/tiktok_data.csv | wc -l           # 2157
grep -n 'temperature", type=float, default' annotation/annotate.py   # default=0.0
grep -n 'norm.ppf\|"detail"' ppi/ppi_correct.py annotation/annotate.py
grep -n 'MODEL =' annotation/annotate_gemini.py     # google/gemini-2.5-flash

Paper fields used: Table 1a (beta and p per sentiment per configuration), Table 1b (per-hashtag frequency, delta, 95% CI), Table 2 (avg, min, max tokens and cost), Table 3 (overall accuracy and kappa), Table 4 (Gemini subset), Sec. 4.3 (2,332 and 15,774 tokens, 281 tokens per second). Prices: OpenAI vision guide (2,833 base and 5,667 per tile for gpt-4o-mini; 85 and 170 for gpt-4o), gpt-4o-mini model page, Gemini API pricing. The 4x4 MELD tile count assumes 16:9 source frames; MELD frame size is not stated in the paper.

What would prove this wrong

Our reading is that Table 2 counts image tokens only, computed from OpenAI's tile rule rather than measured from API usage, and that the paper's accuracy ordering on MELD runs opposite to its conclusion ordering. Both are checkable.

First, if the authors release per-call usage.prompt_tokens logs for the TikTok 2x8 runs, our reading predicts the average will exceed 47,665 by the prompt and transcript tokens and that some videos will exceed 48,169. If the logged average equals 47,665 and nothing exceeds 48,169, then the table did include text and our tile reconstruction is a coincidence of rounding, which we would correct.

Second, a dated prediction: by 31 March 2027, any revised version or released annotation files that rerun MELD with 200 or more PPI labels will still show the text-only GPT-4o-mini Negative coefficient significant in the wrong direction before correction, and the RF video plus transcript source will still match ground truth on sign and significance for Neutral and Negative. If a rerun flips either, the ranking reversal we report was sampling noise in one draw.

Sources

1. Zhu, Z., Gligorić, K. Paying for Too Many Tokens? Valid and Cost-Efficient Multimodal LLM Annotation with Simple Heuristics. arXiv:2610.00809v1, 30 September 2026. Johns Hopkins University. Sections 3 to 5, Tables 1 to 9, Appendix A.
2. JHU-CSS. multimodal-llm-annotation, created 26 September 2026, 2 commits, last push 27 September 2026, no licence. Accessed 4 October 2026. The URL printed in the paper, github.com/JHU-CSS/multimodal-annotation, returned 404 the same day.
3. OpenAI. Images and vision guide (tile token rule) and gpt-4o-mini and gpt-4o model pages. Accessed 4 October 2026.
4. Google. Gemini Developer API pricing, Gemini 2.5 Flash. Accessed 4 October 2026.
5. Angelopoulos, A. N., Bates, S., Fannjiang, C., Jordan, M. I., Zrnic, T. Prediction-powered inference. Science 382(6671), 2023.
6. Poria, S., et al. MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversations. ACL 2019.
7. BLOMEGA. Decision models for text annotation: cost and accuracy. 2026.
8. BLOMEGA. LLM annotation reproducibility and construct validity. 2026.

FAQ

How much cheaper is an image grid than 1 fps video for VLM annotation?

On GPT-4o-mini, arXiv:2610.00809 reports 47,665 tokens per TikTok video for one 2x8 grid against 2,275,411 for 1 fps frames, 2.09% of the tokens (47.7 times fewer), and 36,596 for a 3x3 grid, 1.61% (62.2 times fewer). BLOMEGA rebuilt every count in that table from OpenAI's published 512px tile rule: each 1 fps frame is 6 tiles or 36,835 tokens, and a single image can never exceed 8 tiles or 48,169 tokens. The 14.8% figure in the abstract is a different ratio: a Gemini grid against Gemini full-video input.

Does the most accurate VLM annotation configuration give the right downstream conclusions?

Not in this paper. On MELD, text-only GPT-4o-mini has the highest accuracy (.610) and reaches the ground-truth conclusion (sign and significance at p below .05) on 1 of 3 sentiment coefficients; the ShotT 4x4 grid (.589) also reaches 1 of 3. A random forest on video and transcript embeddings, at .560 accuracy, reaches 3 of 3. BLOMEGA counted this from Table 1a. Spearman correlation between accuracy rank and correct-conclusion rank is -0.65 over the four raw configurations.

Are the code and data for arXiv:2610.00809 released?

Partly. The URL printed in the paper, github.com/JHU-CSS/multimodal-annotation, returned HTTP 404 on 4 October 2026. The code is at github.com/JHU-CSS/multimodal-llm-annotation, created 26 September 2026, 2 commits, no licence file. It holds 2,157 TikTok video IDs but not which 900 or 1,000 were used, no model annotations, no MELD clip list and no cost script. Its default temperature is 0.0 against 0.3 in the paper, and its PPI script uses normal-approximation intervals from a single draw rather than the bootstrap over 100 trials the paper describes.