Case wall
Jev AI Benchmarks cases, page 2
63 public posts filed under Benchmarks. English summaries on this page, with a link to each case.
Last updated
These are the public posts filed under the Benchmarks category.
No posts match these filters.
@brunocabado built Decision Lab and ran JEV against LLMs on support tickets he labeled himself. He writes that decision models answer classification questions with a probability per option instead of generated text. The linked page is titled "Decision models vs LLMs: JEV, measured on tickets".
Decision models like JEV answer classification questions with a probability per option instead of generated text. I wanted my own numbers before using them, so…
@sardine says Jev caught 27 of 29 fraudulent transactions in a September fraud ring, the most of any approach they tested, and that Jev is now a node in Sardine Flow.
We ran Jev, TypeSafe AI's new System One model, on a fraud ring that hit one of our customers in September. Jev caught 27 of the ring's 29 fraudulent transactio…
@noahsark769 had @typesafeai Jev look at and categorize all ads for the top 680 brands on @replohq's industry landscape list. About 67% of these brands have some ads with customer quotes, and of those brands about 4–10% of the ads they're running have customer quotes.
dove deep into this to figure out how many of the top brands are doing this type of ad. I had @typesafeai Jev look at and categorize all ads for the top 680 bra…
@Centific_CAIR reports Jev matched Claude Sonnet 4 spotting harmful AI replies (F1 0.758 vs 0.717) at 176× lower cost on 498 samples; newer Claude models often refused to judge.
Switching the AI judge cut evaluation costs far more than trimming prompts. We tested @typesafeai's Jev, which gives verdicts without writing out its reasoning,…
@Mav1972 checked 54 storefront screenshots with short yes/no questions. Jev answered in 0.4 s a call, caught 19 of 22 planted defects, and raised 0 of 24 false alarms on harmless changes.
I checked 54 storefront screenshots with short yes/no questions about what each screen must show. Jev answered them in 0.4 s a call, caught 19 of 22 planted def…

Paper: Jev matches GPT-6 Astra on RAG claim checks within 0.6 pts at ~187× lower cost, but lets more unsupported claims through.
This paper finds that a cheap verifier can match a frontier LLM's accuracy at checking RAG claims, while letting more unsupported claims slip through. RAG pipel…
One Jev read of a trace ranks agent failures, including quiet ones where many runs agree, and guides hill-climbing. No answer key.
Can you catch an AI agent's wrong answer from one run, with no answer key, and use it to generate tasks that expose new failures? One JEV read of the trace can…

@yonyoniz tests Jev on Hebrew tweets: same stance as English on 8 of 9; language isn't the bottleneck.
Does Jev read Hebrew? I tested it on my own tweets https://x.com/i/article/2113860659519598592 Liran Tal asked a fair question: does TypeSafe's small decision m…

@0xasten classified 64,946 GitHub repos with TypeSafe Jev for under $2, ~90% accurate on 170 hand labels.
I classified 64,946 GitHub repos using @typesafeai's Jev — for under $2. Here is what I found: 🤖 Agents are dominating the AI space Agents alone (3,029) outnumb…

@theio666 runs a prompt-injection bench: Jev vs Cloudflare Clef vs four trained classifiers on a wambosec subset.
Finally got my hands on doing Jev vs classifier for prompt injection detection benchmark. Subset of dataset by wambosec, with categories I found interesting, an…

@honnibal ran Jev with Ellf on data from Opus few-shot classifiers; Jev agreed with Opus on 95% of the inputs.
Very impressed with @typesafeai Jev. We've been working on a user project that was built with a lot of few-shot classifiers powered by Opus. I used Ellf to run…

@ikkistan benchmarks guest-photo moderation for Candids on UnsafeBench: Cloudflare clef-flash, TypeSafe Jev, Claude Haiku 5.5, and Amazon Rekognition.
I’m building Candids, an app for wedding and party guests add photos to one shared album from their phones. Guest uploads need moderation, so I benchmarked four…
@ikkistan: Jev, Haiku, and clef-flash reach AUROC 0.86 to 0.87 on unsafe images, Rekognition 0.70, at 4 to 18 cents per 1,000 images versus $1.
Jev, Haiku and clef-flash rank unsafe images about equally well (AUROC 0.86 to 0.87, Rekognition 0.70) at 4 to 18 cents per 1,000 images vs $1. Code, per-image…

@instantricecook runs 66 Where's Waldo scenes across decision models; Jev 1.14 is quickest, with a public Waldo bench site.
I wanted to see how well decision models are at... Finding Waldo. After 66 scenes, here are the results: - @liquidai d1-ultra found the most Waldos - @typesafea…
@gcai: GC AI Engineering built 38 projects in 12 hours with TypeSafe Jev while @ChrisDzoba tried to break it.
GC AI Engineering built 38 projects in 12 hours with TypeSafe's Jev, a classification model. Director of Engineering @ChrisDzoba spent the day trying to break i…

@SaadMRP: Jev diagnoses SREGym-Lite faults; LLM review lifts pass rate 78.1%→89.5% across 21 faults, ~38s avg.
1/8 In our latest experiment, we let Jev, a decision model, investigate system failures and asked an LLM to check its diagnosis. Across 21 SREGym-Lite faults, r…

@gbolagadehq compares Jev 1.14, OpenAI Decisions API, and Opus 5.5 on five public tests including 1,999 product photos.
https://x.com/i/article/2112290824528576512

@a_latner benchmarks three-day-old Jev image classification against Haiku 5.5 on real production data and reports matching quality.
@typesafeai Jev image classification is 3 days old. I benchmarked it against Haiku 5.5 on real production data. We know it's faster, but are the results as good…
@huuw_khoa benchmarks Jev against Gemma 4 12B and gpt-oss-20b on 60 knowledge-graph questions in open-source hopwise.
Can a model built for typed decisions walk a knowledge graph better than a general LLM? I tried Jev, Gemma 4 12B and gpt-oss-20b on 60 questions. Feedback and i…
@Adityakuma8784 built jev-harness to test coding-agent guardrails; a typical hook let 331 of 503 disguised dangerous commands through.
Your AI coding agent has guardrails: hooks, permission rules, CLAUDE.md, AI reviewers. Almost nobody tests them. I built jev-harness to test them. Against a typ…

@_vMyth’s jevals runs agent evals and guardrails as one Jev decision request per trace, cheap enough to score every run.
Runs agent evals and guardrails as 1 request to a Jev decision model, not an LLM judge. Cheap enough to score every trace. Change 2 defaults: the quick-start se…
@OpenRouter posted Decision Model Rankings for spend share and token share by model and task type, and says @typesafeai is leading all categories today.
New: Decision Model Rankings! See the spend share and token share of different decision models and task types, including Relevance, Correctness, Instruction Fol…
@h_nilforoshan benchmarked the OpenAI Decisions API against Jev for HiringCafe.com: relevance of a query or resume to a job, scored 1-10. The post says OpenAI costs 2x more and is 5-10% worse.
I benchmarked "Jev-killer" OpenAI Decisions API against Jev for https://HiringCafe.com, serving 2.5 million users. The task is to predict how relevant a user qu…
Quoted @OpenAIDevs

@ValsAI says they independently tested TypeSafe's Jev against 11 other models. On 400 claim-verification questions it matched GPT-6 Astra's 97.5% accuracy at about 1/500th the cost.
Can a model that doesn’t generate text compete with frontier LLMs? We independently tested TypeSafe’s Jev against 11 other models. On 400 claim-verification que…