Skip to content

@axel_briche says people pay AIs to write when they often only want a yes or a no. He tested Jev, a model that cannot write, against Claude Sonnet on 180 examples. He says it is as good, answers in 0.16 seconds instead of 2, and costs him four times less. Jev is a TypeSafe model. You give it a text and a sentence, such as “he is really talking about our project”, and it returns only a number between 0 and 1. In his example he asks a client whether something is blocking. The client says they are completely blocked, but about another project. Claude Haiku had said yes, because the client answered. Jev puts 0.55 on “he is really talking about our project”, so for Jev that is a no. Two things he understood while testing: you need a sentence for each doubt, so that when it says no you know why; and the number where you say yes is not the same for every sentence. For him it is 0.8 for one and 0.55 for another, and he found that by measuring. The defect he names: because it does not write, it cannot correct an answer, and on its own it is wrong more often. Claude alone said yes too easily and was wrong 8 times out of 180, against 2 times when Jev checks behind it. Having another Claude check Claude also works, except it costs four times more and takes 2 seconds instead of 0.16.

Benchmarks

Axel Briche (dev/IA) @axel_briche

FR

  • 43 views
  • 1 likes
  • 0 bookmarks
  • 0 reposts
On paie des IA pour écrire, alors que souvent on leur demande juste un oui ou un non. Du coup j'ai testé Jev, un modèle qui ne sait pas écrire, contre Claude So…

About this case

Benchmark. Axel Briche (@axel_briche) posts in French. Text only. No link.

Original posthttps://x.com/axel_briche/status/2122587038105166132

Related cases