Skip to content

@davepoon put this week’s Google Decisions API into his Trust Me Bro Phishing test, with the same questions and rules for every model: 200 real phishing emails, plus 200 normal emails written to look like them. Phishing caught / false alarms: TypeSafe Jev 180/8, OpenAI Decisions API 185/3, Cloudflare Clef 191/7, Gemini Decisions API 188/1. He says 1 false alarm in 200 is the lowest of any AI model he has tested, and that Gemini beat Jev on both counts by too wide a gap to be luck.

Benchmarks

davepoon @davepoon

EN

  • 16 views
  • 0 likes
  • 0 bookmarks
  • 0 reposts
Poster from davepoon (@davepoon)

About this case

Benchmark. davepoon (@davepoon) quotes Sundar Pichai (@sundarpichai) announcing the Gemini Decisions API. On his own numbers, Clef still catches a few more phishing emails, but he says that gap is too small to call. Downsides he names for Gemini: about 2x Jev’s price, and 709 ms per email from Sydney, of which around 350 ms is the trip to Google’s servers. He attaches his own video. The quoted launch video is the quoted post’s media.

Original posthttps://x.com/davepoon/status/2121390376447774858

Related cases