@Data_Gig says they replaced their classifier with an AI model that doesn't write a single word. Macro-F1 more than doubled, from 0.25 to 0.53, at about $0.02 per 1,000 posts. In the thread, Portfolio Shield detects coordinated stock manipulation on social media, and every Reddit or YouTube post needs labels: about the stock, bullish or bearish, sarcastic, promotional. Those are decisions, not writing tasks. They tested #Jev, a decision model from #TypeSafe AI: no text output, just a typed answer and a probability. On 801 posts labelled by people, macro-F1 was 0.53 versus 0.25 for the old keyword classifier, with a median of about 0.17 s per post. Lessons they list: pin the model version, because aliases move and silently change labels; shuffle answer options, after a lean toward the first option; never fall back silently; 0.53 is a big jump, not the finish line, and still “Beta”. The takeaway: not every AI problem needs a chatbot. Fast decision models for the many small judgments, LLMs for the few that need words.
- 3 views
- 0 likes
- 0 bookmarks
- 0 reposts
We replaced our classifier with an AI model that doesn't write a single word. Macro-F1 more than doubled: 0.25 → 0.53. Cost: ~$0.02 per 1,000 posts. Here's what…
About this case
Benchmark. Data_GiG (@Data_Gig) attaches one photo and ends the root post with “Here's what we learned 🧵”. The root text has no URL. A later post in the author's thread links the application, https://portfolio-shield.app/.
Original posthttps://x.com/Data_Gig/status/2122857186883686807