@ttywisp, under the heading “AI-written report,” benchmarks two zero-output-token System-1 decision APIs, TypeSafe's jev-1.14.0 and Google's gemini-decision-1 via OpenRouter, on 21 Zelda games: 840 pairwise choices and three passes of a 126-question, 5-axis rubric. All 1,218 Gemini evaluations cost $0.026. On exact repeats, Gemini recorded 0.000 MAE and 0/126 flips; Jev recorded 0.034 MAE and 7/126 flips, and quantized probabilities to 0.01.
- 168 views
- 1 likes
- 0 bookmarks
- 0 reposts
About this case
Benchmark. ttywisp (@ttywisp) writes that both APIs showed positional bias: Jev favored the second choice by 4.37%, and Gemini favored opt_1 by 4.63%. Rearranging JSON key order left Gemini unchanged. Jev placed Tears of the Kingdom first in its pairwise ranking. Mean inter-axis correlation of the five rubric axes was 0.725 for Jev and 0.400 for Gemini. Gemini separated criteria more: GameCube Wind Waker ranked 3rd in atmosphere and 19th in pacing, and A Link to the Past, Majora's Mask, and Ocarina of Time were its overall top three. The post says Gemini's rubric reproduced most of its pairwise ranking at linear cost, with finer probabilities, while Jev's 0.01 quantization discarded probability resolution and its rubric axes stayed more correlated. Exact 0.00 values from Jev made unadjusted log-ratio calculations undefined. Each 126-question rubric pass on Gemini cost $0.004, and one pass recovered more than 92% of pairwise tournament rank variance. The dashboard is https://zelda-evaluation-dashboard.mail-22c.workers.dev/ . Text only.
Original posthttps://x.com/ttywisp/status/2121022508581150911