Case wall
Jev AI Benchmarks cases
63 public posts filed under Benchmarks. English summaries on this page, with a link to each case.
Last updated
These are the public posts filed under the Benchmarks category.
No posts match these filters.
@skalskip92 says he started building a benchmark for vision decision models, expecting to trade accuracy for speed. On his numbers, GPT-6.1 Terra (high) was 96/100 correct at 7.98 s average per image, and TypeSafe AI Jev 1.14 was 89/100 correct at 9.64 s average per image. He asks why decision models are so slow.
started building a benchmark for vision decision models was expecting to trade accuracy for speed... meanwhile... - GPT-6.1 Terra (high): 96/100 correct, 7.98s…
@matsuikentaro1 says a paper he wrote as a hobby was accepted. He used the decision model Jev for title and abstract screening in a systematic review, and writes that Jev is cheap, fast, and good. He links https://www.mdpi.com/2078-2489/17/11/1160.
趣味で書いた論文が通ったので一応 https://www.mdpi.com/2078-2489/17/11/1160 Systematic ReviewのTitle and Abstract screeningにDecision modelのJevを使いましたよ、のやつ。安くて速くてよいですねJev
@elise_ai says they worked with @typesafeai to test Jev in the field. Their Maintenance App lets apartment maintenance techs photograph appliance labels instead of typing in model and serial numbers. If a photo is unreadable, they need to catch it before the technician walks away. They tested Jev, Sol and Opus on 140 labeled photos to see how quickly and accurately they could flag photos that needed a retake. In the follow-up, Jev caught 32 of 35 unreadable photos and incorrectly flagged two readable ones. Its median check took 94 ms, about 30× faster than Sol, at roughly 2% of the estimated cost. They write that these are training-set results. Next they are testing on new photos and measuring the full workflow from taking a photo to receiving a retake prompt.
We worked with @typesafeai to test Jev in the field. Our Maintenance App lets apartment maintenance techs photograph appliance labels instead of typing in model…
@saifcodes_exe says two AI models do not write text. They return probabilities. Jev (TypeSafe) versus Gemini Decision 1 (Google), on the same 600 questions: accuracy 86.3% versus 85.8%, a tie; speed 489 versus 621 ms median; cost 28% cheaper on Jev; trust at ≥90% confidence, 93.9% versus 97.6%, and Gemini wins that one.
Two AI models that don't write text. They return probabilities. Jev (TypeSafe) vs Gemini Decision 1 (Google), same 600 questions: Accuracy: 86.3% vs 85.8%, a ti…
@0xironyAditya says @liquidai released another model, so he tested d1.2-ultra against the other decision models on two real jobs: grading storm and fire damage on before/after satellite shots, and sorting Serengeti camera-trap bursts into empty or which animal. The setup is 200 picture decisions, the same pictures and the same questions. d1.2-ultra versus Jev 1.14 versus gpt-6 luna versus clef, all through @OpenRouter: exact damage grade 47% / 46% / 42% / 41%; send a team first 70% / 64% / 68% / 69%; animal or empty 99% / 96% / 96% / 98%; species 62% / 63% / 62% / 65%; cost for all 200, $0.018 / $0.034 / $0.034 / $0.083. He writes that accuracy is a near tie at the top, and that d1.2-ultra got there at half the price of Jev and luna and almost 4.7× cheaper than clef. The only catch is that it was the slowest of the four. These cannot replace the person doing the job yet, but they can help sort the pile, double-check calls, and clear out the empty frames.
@liquidai released another banger, so had to test the d1.2-ultra against the other decision models on two real jobs tasks: - grade storm and fire damage on befo…
@erik_kokalj tested the new Gemini Decisions API on a recycling line and says it is more accurate, and more expensive, than the other multimodal System 1 alternatives he names, Jev and OAI. Accuracy on this clip: Gemini Decision 1 76%, OAI Decisions API (GPT-6 Luna) 53%, Jev 1.14 72%. Cost per 1,000 images / median speed: Gemini $0.13 / ~340 ms, OAI $0.08 / ~270 ms, Jev $0.08 / ~130 ms. He says @typesafeai’s Jev still has the best performance per price.
Gemini just released Decisions API, and wanted to test it on recycling line ♻️ It's more accurate (and more expensive) compared to other multimodal "System 1" a…
@flyinghigherai posts a summary of overall performance between Google Decision 1 and TypeSafe Jev across four decision primitives. Across all 40 benchmark images, Jev showed higher accuracy and cleaner calibration in multi-class root cause analysis and desiccation severity scoring. The post quotes an earlier post by the same author. That quote reports the Boolean/Noul primitive: Decision 1 at 82.5% accuracy, with 6 false negatives and 1 false positive; Jev at 85.0% accuracy, with 4 false negatives and 2 false positives.
Here is the summary of overall performance between Google Decision 1 and TypeSafe Jev across our four decision primitives. Across all 40 benchmark images, Jev s…
Quoted @flyinghigherai
@axel_briche says people pay AIs to write when they often only want a yes or a no. He tested Jev, a model that cannot write, against Claude Sonnet on 180 examples. He says it is as good, answers in 0.16 seconds instead of 2, and costs him four times less. Jev is a TypeSafe model. You give it a text and a sentence, such as “he is really talking about our project”, and it returns only a number between 0 and 1. In his example he asks a client whether something is blocking. The client says they are completely blocked, but about another project. Claude Haiku had said yes, because the client answered. Jev puts 0.55 on “he is really talking about our project”, so for Jev that is a no. Two things he understood while testing: you need a sentence for each doubt, so that when it says no you know why; and the number where you say yes is not the same for every sentence. For him it is 0.8 for one and 0.55 for another, and he found that by measuring. The defect he names: because it does not write, it cannot correct an answer, and on its own it is wrong more often. Claude alone said yes too easily and was wrong 8 times out of 180, against 2 times when Jev checks behind it. Having another Claude check Claude also works, except it costs four times more and takes 2 seconds instead of 0.16.
On paie des IA pour écrire, alors que souvent on leur demande juste un oui ou un non. Du coup j'ai testé Jev, un modèle qui ne sait pas écrire, contre Claude So…
@TheWalters954 says his website-domain rule let https://eu.posthog.com in as a new account. He checked 50 inbound accounts against 25 existing ones in a mock CRM with plain rules, GPT-4o mini, and Jev, TypeSafe AI's decision model. He links https://youtu.be/-1Zpy1gNss0.
My website-domain rule let https://eu.posthog.com in as a new account. I checked 50 inbound accounts against 25 existing ones in a mock CRM with plain rules, GP…
@rahulstheory says @liquidai's d1.2 is wow. On complicated construction drawings, accurately marking the doors by room name, Jev 1.14 got 1/41 and D1.2-ultra got 34/41. He calls that magnitude strong. About 1k requests cost $0.11 in total.
@liquidai 's d1.2 is wow. In complicated construction drawings accurately marking the doors by room name: Jev 1.14 1/41, D1.2-ultra 34/41 . This is magnitude st…
@spyced writes about evaluating Jev, Gemini, and OpenAI decision models, and why they picked GLiDE, from a much smaller provider, for Mjolnir's GitHub event classifications.
Evaluating Jev, Gemini, and OpenAI decision models, and why we picked GLiDE from a much smaller provider for Mjolnir's github event classifications. https://blo…
@BuildFastWithAI says Google launched Gemini Decision 1 yesterday, so they tested it against Jev from @typesafeai in a game of chess. Every turn, both models receive all the legal moves and decide which one to play. The post says you can watch their reasoning, see the top 3 moves each model considers, and track which side they think is winning.
Google launched Gemini Decision 1 yesterday, so I put it to the test against Jev from @typesafeai in a game of chess. Every turn, I give both models all the leg…
@maharishi_r_b says they tested three AI decision models on a private eval: @SarvamAI Prism, @typesafeai Jev, and @Alphanimble Wity. At 5% false positives, Jev caught 77% of errors, Prism 59%, and Wity 57%. To catch 80% of issues, Jev needed 5.9% false positives, against 11.7% for Prism and 20% for Wity. In Sarvam’s launch benchmarks Prism scored 71.8 against Jev’s 64.5, and Alpha nimble said Wity was better than Sarvam, but that is not what this eval showed. The task has a significant mix of Indian languages, and Jev won.
We tested 3 AI decision models on our private eval set: @SarvamAI - Prism @typesafeai - Jev & @Alphanimble - Wity Here’s what we found 🧵
@trevin says he ran some quick tests of the new Gemini Decision 1 model in a project where he already uses @typesafeai Jev and Microsoft Decision 1. Gemini Decision 1 was accurate but never cheaper, and worse in his "request matching" tests.
I ran some quick tests for this new Gemini Decision 1 model in a project I'm working on where I'm already using @typesafeai Jev and Microsoft Decision 1. Gemini…
Quoted @OfficialLoganK
@ttywisp, under the heading “AI-written report,” benchmarks two zero-output-token System-1 decision APIs, TypeSafe's jev-1.14.0 and Google's gemini-decision-1 via OpenRouter, on 21 Zelda games: 840 pairwise choices and three passes of a 126-question, 5-axis rubric. All 1,218 Gemini evaluations cost $0.026. On exact repeats, Gemini recorded 0.000 MAE and 0/126 flips; Jev recorded 0.034 MAE and 7/126 flips, and quantized probabilities to 0.01.
AI-written report Benchmarked two zero-output-token "System-1" decision APIs, TypeSafe's jev-1.14.0 and Google's gemini-decision-1 (via OpenRouter), across 21 Z…
@Data_Gig says they replaced their classifier with an AI model that doesn't write a single word. Macro-F1 more than doubled, from 0.25 to 0.53, at about $0.02 per 1,000 posts. In the thread, Portfolio Shield detects coordinated stock manipulation on social media, and every Reddit or YouTube post needs labels: about the stock, bullish or bearish, sarcastic, promotional. Those are decisions, not writing tasks. They tested #Jev, a decision model from #TypeSafe AI: no text output, just a typed answer and a probability. On 801 posts labelled by people, macro-F1 was 0.53 versus 0.25 for the old keyword classifier, with a median of about 0.17 s per post. Lessons they list: pin the model version, because aliases move and silently change labels; shuffle answer options, after a lean toward the first option; never fall back silently; 0.53 is a big jump, not the finish line, and still “Beta”. The takeaway: not every AI problem needs a chatbot. Fast decision models for the many small judgments, LLMs for the few that need words.
We replaced our classifier with an AI model that doesn't write a single word. Macro-F1 more than doubled: 0.25 → 0.53. Cost: ~$0.02 per 1,000 posts. Here's what…
@jreyesdev says he built 3 agents that place bets on NFL games every weekend, using Jev, D1 and Luna, and put them head to head. Four weeks into the experiment, Jev is winning, and he is thinking about writing why the three decision models disagree.
I built 3 agents that make “bets” on NFL games every weekend. I used JEV, D1 and Luna. Placed them head to head. I’m 4 weeks into this experiment but JEV is def…
@noahsark769 says he spent $3.85 on @typesafeai Jev to analyze the entire @replohq sightglass dataset, and that 57.9% of pages which list ingredients have callouts like the photo, either callouts only or lists mixed with callouts.
i yet again went hardmode on this and spent $3.85 on @typesafeai Jev to analyze the entire @replohq sightglass dataset. brands are already doing this a lot: 57.…
Quoted @oliverkenyon
@davepoon put this week’s Google Decisions API into his Trust Me Bro Phishing test, with the same questions and rules for every model: 200 real phishing emails, plus 200 normal emails written to look like them. Phishing caught / false alarms: TypeSafe Jev 180/8, OpenAI Decisions API 185/3, Cloudflare Clef 191/7, Gemini Decisions API 188/1. He says 1 false alarm in 200 is the lowest of any AI model he has tested, and that Gemini beat Jev on both counts by too wide a gap to be luck.
Google launched a Decisions API this week, so it went into my Trust Me Bro Phishing test with Jev and the other Decisions models 😄 200 real phishing emails, plu…
Quoted @sundarpichai
@trevin compared Particle-1 with Microsoft's Decision-1 and @typesafeai Jev 1.14 on one specific task, and writes that Decision-1 still leads while Particle-1 performed horribly there.
Based on this mention of Particle-1, I compared it against Microsoft’s Decision-1 and @typesafeai Jev 1.14. Decision-1 still leads and Particle-1 performed horr…
@ArtificialAnlys reports Decision Index time per task: Celeris-1.1 Decision 56 ms, Elephant 9B 70 ms, Jev 1.14 92 ms, and Claude Haiku 5.5 (Max) 32.65 seconds. Mammoth 27B scores within 4 points of GPT-6 Luna (Max) and is 118× faster.
Celeris-1.1 Decision recorded the lowest Time per Task at 56 milliseconds, followed by Elephant 9B (70 milliseconds) and Jev 1.14 (92 milliseconds). In comparis…
@tejasmanohar says Hightouch’s eng post hit #1 on HN: Jev+Opus nearly matched Opus alone on hard data-matching grades at much lower cost.
our eng team got #1 on HN today! they tested if fast + cheap decision models like Jev @typesafeai could grade complex data-matching tasks as well as frontier mo…
@p0 (Parallel Web Systems) says on a generic decisions benchmark OpenAI, Jev, and Parallel all score about 90%, but on live-web questions OpenAI drops 2.2× and Jev drops 2.6×. They built PWS-Decisions so generic benches cannot hide that gap, and link the benchmark blog.
On a generic decisions benchmark, OpenAI, Jev, and Parallel all score about 90%. On questions that depend on the live web, OpenAI drops 2.2x, and Jev drops 2.6x…
@milvusio reports two Jev passes in their Graph RAG pipeline beat the LLM rerankers they tested on recall while spending less model time.
Two Jev passes helped our Graph RAG pipeline outperform the LLM rerankers we tested, with higher recall and less time spent on model calls. Earlier, we released…