Skip to content

Case wall

Jev AI Benchmarks cases

63 public posts filed under Benchmarks. English summaries on this page, with a link to each case.

Last updated

These are the public posts filed under the Benchmarks category.

Filter

24

started building a benchmark for vision decision models was expecting to trade accuracy for speed... meanwhile... - GPT-6.1 Terra (high): 96/100 correct, 7.98s…

Benchmarks

SkalskiP @skalskip92

EN

@skalskip92 says he started building a benchmark for vision decision models, expecting to trade accuracy for speed. On his numbers, GPT-6.1 Terra (high) was 96/100 correct at 7.98 s average per image, and TypeSafe AI Jev 1.14 was 89/100 correct at 9.64 s average per image. He asks why decision models are so slow.

started building a benchmark for vision decision models was expecting to trade accuracy for speed... meanwhile... - GPT-6.1 Terra (high): 96/100 correct, 7.98s…

趣味で書いた論文が通ったので一応 https://www.mdpi.com/2078-2489/17/11/1160 Systematic ReviewのTitle and Abstract screeningにDecision modelのJevを使いましたよ、のやつ。安くて速くてよいですねJev

Benchmarks

松井健太郎 睡眠・精神医学 (Kentaro Matsui) @matsuikentaro1

JA

@matsuikentaro1 says a paper he wrote as a hobby was accepted. He used the decision model Jev for title and abstract screening in a systematic review, and writes that Jev is cheap, fast, and good. He links https://www.mdpi.com/2078-2489/17/11/1160.

趣味で書いた論文が通ったので一応 https://www.mdpi.com/2078-2489/17/11/1160 Systematic ReviewのTitle and Abstract screeningにDecision modelのJevを使いましたよ、のやつ。安くて速くてよいですねJev

We worked with @typesafeai to test Jev in the field. Our Maintenance App lets apartment maintenance techs photograph appliance labels instead of typing in model…

Benchmarks

Elise Labs @elise_ai

EN

@elise_ai says they worked with @typesafeai to test Jev in the field. Their Maintenance App lets apartment maintenance techs photograph appliance labels instead of typing in model and serial numbers. If a photo is unreadable, they need to catch it before the technician walks away. They tested Jev, Sol and Opus on 140 labeled photos to see how quickly and accurately they could flag photos that needed a retake. In the follow-up, Jev caught 32 of 35 unreadable photos and incorrectly flagged two readable ones. Its median check took 94 ms, about 30× faster than Sol, at roughly 2% of the estimated cost. They write that these are training-set results. Next they are testing on new photos and measuring the full workflow from taking a photo to receiving a retake prompt.

We worked with @typesafeai to test Jev in the field. Our Maintenance App lets apartment maintenance techs photograph appliance labels instead of typing in model…

Poster from Saif (@saifcodes_exe)

Benchmarks

Saif @saifcodes_exe

EN

@saifcodes_exe says two AI models do not write text. They return probabilities. Jev (TypeSafe) versus Gemini Decision 1 (Google), on the same 600 questions: accuracy 86.3% versus 85.8%, a tie; speed 489 versus 621 ms median; cost 28% cheaper on Jev; trust at ≥90% confidence, 93.9% versus 97.6%, and Gemini wins that one.

Two AI models that don't write text. They return probabilities. Jev (TypeSafe) vs Gemini Decision 1 (Google), same 600 questions: Accuracy: 86.3% vs 85.8%, a ti…

Poster from irony.aditya (@0xironyAditya)

Benchmarks

irony.aditya @0xironyAditya

EN

@0xironyAditya says @liquidai released another model, so he tested d1.2-ultra against the other decision models on two real jobs: grading storm and fire damage on before/after satellite shots, and sorting Serengeti camera-trap bursts into empty or which animal. The setup is 200 picture decisions, the same pictures and the same questions. d1.2-ultra versus Jev 1.14 versus gpt-6 luna versus clef, all through @OpenRouter: exact damage grade 47% / 46% / 42% / 41%; send a team first 70% / 64% / 68% / 69%; animal or empty 99% / 96% / 96% / 98%; species 62% / 63% / 62% / 65%; cost for all 200, $0.018 / $0.034 / $0.034 / $0.083. He writes that accuracy is a near tie at the top, and that d1.2-ultra got there at half the price of Jev and luna and almost 4.7× cheaper than clef. The only catch is that it was the slowest of the four. These cannot replace the person doing the job yet, but they can help sort the pile, double-check calls, and clear out the empty frames.

@liquidai released another banger, so had to test the d1.2-ultra against the other decision models on two real jobs tasks: - grade storm and fire damage on befo…

Poster from Erik Kokalj (@erik_kokalj)

Benchmarks

Erik Kokalj @erik_kokalj

EN

@erik_kokalj tested the new Gemini Decisions API on a recycling line and says it is more accurate, and more expensive, than the other multimodal System 1 alternatives he names, Jev and OAI. Accuracy on this clip: Gemini Decision 1 76%, OAI Decisions API (GPT-6 Luna) 53%, Jev 1.14 72%. Cost per 1,000 images / median speed: Gemini $0.13 / ~340 ms, OAI $0.08 / ~270 ms, Jev $0.08 / ~130 ms. He says @typesafeai’s Jev still has the best performance per price.

Gemini just released Decisions API, and wanted to test it on recycling line ♻️ It's more accurate (and more expensive) compared to other multimodal "System 1" a…

Here is the summary of overall performance between Google Decision 1 and TypeSafe Jev across our four decision primitives. Across all 40 benchmark images, Jev s…

Benchmarks

Flying Higher AI @flyinghigherai

EN

@flyinghigherai posts a summary of overall performance between Google Decision 1 and TypeSafe Jev across four decision primitives. Across all 40 benchmark images, Jev showed higher accuracy and cleaner calibration in multi-class root cause analysis and desiccation severity scoring. The post quotes an earlier post by the same author. That quote reports the Boolean/Noul primitive: Decision 1 at 82.5% accuracy, with 6 false negatives and 1 false positive; Jev at 85.0% accuracy, with 4 false negatives and 2 false positives.

Here is the summary of overall performance between Google Decision 1 and TypeSafe Jev across our four decision primitives. Across all 40 benchmark images, Jev s…

Quoted @flyinghigherai

On paie des IA pour écrire, alors que souvent on leur demande juste un oui ou un non. Du coup j'ai testé Jev, un modèle qui ne sait pas écrire, contre Claude So…

Benchmarks

Axel Briche (dev/IA) @axel_briche

FR

@axel_briche says people pay AIs to write when they often only want a yes or a no. He tested Jev, a model that cannot write, against Claude Sonnet on 180 examples. He says it is as good, answers in 0.16 seconds instead of 2, and costs him four times less. Jev is a TypeSafe model. You give it a text and a sentence, such as “he is really talking about our project”, and it returns only a number between 0 and 1. In his example he asks a client whether something is blocking. The client says they are completely blocked, but about another project. Claude Haiku had said yes, because the client answered. Jev puts 0.55 on “he is really talking about our project”, so for Jev that is a no. Two things he understood while testing: you need a sentence for each doubt, so that when it says no you know why; and the number where you say yes is not the same for every sentence. For him it is 0.8 for one and 0.55 for another, and he found that by measuring. The defect he names: because it does not write, it cannot correct an answer, and on its own it is wrong more often. Claude alone said yes too easily and was wrong 8 times out of 180, against 2 times when Jev checks behind it. Having another Claude check Claude also works, except it costs four times more and takes 2 seconds instead of 0.16.

On paie des IA pour écrire, alors que souvent on leur demande juste un oui ou un non. Du coup j'ai testé Jev, un modèle qui ne sait pas écrire, contre Claude So…

Poster from Walters954 (@TheWalters954)

Benchmarks

Walters954 @TheWalters954

EN

@TheWalters954 says his website-domain rule let https://eu.posthog.com in as a new account. He checked 50 inbound accounts against 25 existing ones in a mock CRM with plain rules, GPT-4o mini, and Jev, TypeSafe AI's decision model. He links https://youtu.be/-1Zpy1gNss0.

My website-domain rule let https://eu.posthog.com in as a new account. I checked 50 inbound accounts against 25 existing ones in a mock CRM with plain rules, GP…

@liquidai 's d1.2 is wow. In complicated construction drawings accurately marking the doors by room name: Jev 1.14 1/41, D1.2-ultra 34/41 . This is magnitude st…

Benchmarks

⚡ Rahul ⚡ @rahulstheory

EN

@rahulstheory says @liquidai's d1.2 is wow. On complicated construction drawings, accurately marking the doors by room name, Jev 1.14 got 1/41 and D1.2-ultra got 34/41. He calls that magnitude strong. About 1k requests cost $0.11 in total.

@liquidai 's d1.2 is wow. In complicated construction drawings accurately marking the doors by room name: Jev 1.14 1/41, D1.2-ultra 34/41 . This is magnitude st…

Evaluating Jev, Gemini, and OpenAI decision models, and why we picked GLiDE from a much smaller provider for Mjolnir's github event classifications. https://blo…

Benchmarks

Jonathan Ellis @spyced

EN

@spyced writes about evaluating Jev, Gemini, and OpenAI decision models, and why they picked GLiDE, from a much smaller provider, for Mjolnir's GitHub event classifications.

Evaluating Jev, Gemini, and OpenAI decision models, and why we picked GLiDE from a much smaller provider for Mjolnir's github event classifications. https://blo…

Poster from Build Fast with AI (@BuildFastWithAI)

Benchmarks

Build Fast with AI @BuildFastWithAI

EN

@BuildFastWithAI says Google launched Gemini Decision 1 yesterday, so they tested it against Jev from @typesafeai in a game of chess. Every turn, both models receive all the legal moves and decide which one to play. The post says you can watch their reasoning, see the top 3 moves each model considers, and track which side they think is winning.

Google launched Gemini Decision 1 yesterday, so I put it to the test against Jev from @typesafeai in a game of chess. Every turn, I give both models all the leg…

We tested 3 AI decision models on our private eval set: @SarvamAI - Prism @typesafeai - Jev & @Alphanimble - Wity Here’s what we found 🧵

Benchmarks

Maharishi R B @maharishi_r_b

EN

@maharishi_r_b says they tested three AI decision models on a private eval: @SarvamAI Prism, @typesafeai Jev, and @Alphanimble Wity. At 5% false positives, Jev caught 77% of errors, Prism 59%, and Wity 57%. To catch 80% of issues, Jev needed 5.9% false positives, against 11.7% for Prism and 20% for Wity. In Sarvam’s launch benchmarks Prism scored 71.8 against Jev’s 64.5, and Alpha nimble said Wity was better than Sarvam, but that is not what this eval showed. The task has a significant mix of Indian languages, and Jev won.

We tested 3 AI decision models on our private eval set: @SarvamAI - Prism @typesafeai - Jev & @Alphanimble - Wity Here’s what we found 🧵

I ran some quick tests for this new Gemini Decision 1 model in a project I'm working on where I'm already using @typesafeai Jev and Microsoft Decision 1. Gemini…

Benchmarks

Trevin Chow @trevin

EN

@trevin says he ran some quick tests of the new Gemini Decision 1 model in a project where he already uses @typesafeai Jev and Microsoft Decision 1. Gemini Decision 1 was accurate but never cheaper, and worse in his "request matching" tests.

I ran some quick tests for this new Gemini Decision 1 model in a project I'm working on where I'm already using @typesafeai Jev and Microsoft Decision 1. Gemini…

Quoted @OfficialLoganK

AI-written report Benchmarked two zero-output-token "System-1" decision APIs, TypeSafe's jev-1.14.0 and Google's gemini-decision-1 (via OpenRouter), across 21 Z…

Benchmarks

ttywisp @ttywisp

EN

@ttywisp, under the heading “AI-written report,” benchmarks two zero-output-token System-1 decision APIs, TypeSafe's jev-1.14.0 and Google's gemini-decision-1 via OpenRouter, on 21 Zelda games: 840 pairwise choices and three passes of a 126-question, 5-axis rubric. All 1,218 Gemini evaluations cost $0.026. On exact repeats, Gemini recorded 0.000 MAE and 0/126 flips; Jev recorded 0.034 MAE and 7/126 flips, and quantized probabilities to 0.01.

AI-written report Benchmarked two zero-output-token "System-1" decision APIs, TypeSafe's jev-1.14.0 and Google's gemini-decision-1 (via OpenRouter), across 21 Z…

We replaced our classifier with an AI model that doesn't write a single word. Macro-F1 more than doubled: 0.25 → 0.53. Cost: ~$0.02 per 1,000 posts. Here's what…

Benchmarks

Data_GiG @Data_Gig

EN

@Data_Gig says they replaced their classifier with an AI model that doesn't write a single word. Macro-F1 more than doubled, from 0.25 to 0.53, at about $0.02 per 1,000 posts. In the thread, Portfolio Shield detects coordinated stock manipulation on social media, and every Reddit or YouTube post needs labels: about the stock, bullish or bearish, sarcastic, promotional. Those are decisions, not writing tasks. They tested #Jev, a decision model from #TypeSafe AI: no text output, just a typed answer and a probability. On 801 posts labelled by people, macro-F1 was 0.53 versus 0.25 for the old keyword classifier, with a median of about 0.17 s per post. Lessons they list: pin the model version, because aliases move and silently change labels; shuffle answer options, after a lean toward the first option; never fall back silently; 0.53 is a big jump, not the finish line, and still “Beta”. The takeaway: not every AI problem needs a chatbot. Fast decision models for the many small judgments, LLMs for the few that need words.

We replaced our classifier with an AI model that doesn't write a single word. Macro-F1 more than doubled: 0.25 → 0.53. Cost: ~$0.02 per 1,000 posts. Here's what…

I built 3 agents that make “bets” on NFL games every weekend. I used JEV, D1 and Luna. Placed them head to head. I’m 4 weeks into this experiment but JEV is def…

Benchmarks

Jarod Reyes @jreyesdev

EN

@jreyesdev says he built 3 agents that place bets on NFL games every weekend, using Jev, D1 and Luna, and put them head to head. Four weeks into the experiment, Jev is winning, and he is thinking about writing why the three decision models disagree.

I built 3 agents that make “bets” on NFL games every weekend. I used JEV, D1 and Luna. Placed them head to head. I’m 4 weeks into this experiment but JEV is def…

i yet again went hardmode on this and spent $3.85 on @typesafeai Jev to analyze the entire @replohq sightglass dataset. brands are already doing this a lot: 57.…

Benchmarks

Noah Gilmore @noahsark769

EN

@noahsark769 says he spent $3.85 on @typesafeai Jev to analyze the entire @replohq sightglass dataset, and that 57.9% of pages which list ingredients have callouts like the photo, either callouts only or lists mixed with callouts.

i yet again went hardmode on this and spent $3.85 on @typesafeai Jev to analyze the entire @replohq sightglass dataset. brands are already doing this a lot: 57.…

Quoted @oliverkenyon

Poster from davepoon (@davepoon)

Benchmarks

davepoon @davepoon

EN

@davepoon put this week’s Google Decisions API into his Trust Me Bro Phishing test, with the same questions and rules for every model: 200 real phishing emails, plus 200 normal emails written to look like them. Phishing caught / false alarms: TypeSafe Jev 180/8, OpenAI Decisions API 185/3, Cloudflare Clef 191/7, Gemini Decisions API 188/1. He says 1 false alarm in 200 is the lowest of any AI model he has tested, and that Gemini beat Jev on both counts by too wide a gap to be luck.

Google launched a Decisions API this week, so it went into my Trust Me Bro Phishing test with Jev and the other Decisions models 😄 200 real phishing emails, plu…

Quoted @sundarpichai

Based on this mention of Particle-1, I compared it against Microsoft’s Decision-1 and @typesafeai Jev 1.14. Decision-1 still leads and Particle-1 performed horr…

Benchmarks

Trevin Chow @trevin

EN

@trevin compared Particle-1 with Microsoft's Decision-1 and @typesafeai Jev 1.14 on one specific task, and writes that Decision-1 still leads while Particle-1 performed horribly there.

Based on this mention of Particle-1, I compared it against Microsoft’s Decision-1 and @typesafeai Jev 1.14. Decision-1 still leads and Particle-1 performed horr…

Celeris-1.1 Decision recorded the lowest Time per Task at 56 milliseconds, followed by Elephant 9B (70 milliseconds) and Jev 1.14 (92 milliseconds). In comparis…

Benchmarks

Artificial Analysis @ArtificialAnlys

EN

@ArtificialAnlys reports Decision Index time per task: Celeris-1.1 Decision 56 ms, Elephant 9B 70 ms, Jev 1.14 92 ms, and Claude Haiku 5.5 (Max) 32.65 seconds. Mammoth 27B scores within 4 points of GPT-6 Luna (Max) and is 118× faster.

Celeris-1.1 Decision recorded the lowest Time per Task at 56 milliseconds, followed by Elephant 9B (70 milliseconds) and Jev 1.14 (92 milliseconds). In comparis…

our eng team got #1 on HN today! they tested if fast + cheap decision models like Jev @typesafeai could grade complex data-matching tasks as well as frontier mo…

Benchmarks

Tejas Manohar @tejasmanohar

EN

@tejasmanohar says Hightouch’s eng post hit #1 on HN: Jev+Opus nearly matched Opus alone on hard data-matching grades at much lower cost.

our eng team got #1 on HN today! they tested if fast + cheap decision models like Jev @typesafeai could grade complex data-matching tasks as well as frontier mo…

On a generic decisions benchmark, OpenAI, Jev, and Parallel all score about 90%. On questions that depend on the live web, OpenAI drops 2.2x, and Jev drops 2.6x…

Benchmarks

Parallel Web Systems @p0

EN

@p0 (Parallel Web Systems) says on a generic decisions benchmark OpenAI, Jev, and Parallel all score about 90%, but on live-web questions OpenAI drops 2.2× and Jev drops 2.6×. They built PWS-Decisions so generic benches cannot hide that gap, and link the benchmark blog.

On a generic decisions benchmark, OpenAI, Jev, and Parallel all score about 90%. On questions that depend on the live web, OpenAI drops 2.2x, and Jev drops 2.6x…

Two Jev passes helped our Graph RAG pipeline outperform the LLM rerankers we tested, with higher recall and less time spent on model calls. Earlier, we released…

Benchmarks

Milvus @milvusio

EN

@milvusio reports two Jev passes in their Graph RAG pipeline beat the LLM rerankers they tested on recall while spending less model time.

Two Jev passes helped our Graph RAG pipeline outperform the LLM rerankers we tested, with higher recall and less time spent on model calls. Earlier, we released…

Unofficial developer resource. Not affiliated with TypeSafe AI. These posts belong to their authors. jevnotes.com does not host the videos.