@NathanFlurry shares a tip learned the hard way at Rivet: Jev does not reason, which is by design, and they should have read the manual first. A single prompt that asked for the severity of an issue, with SEV1 through SEV4 as outputs, gave meh results and defeated what makes Jev useful, because an LLM would spend reasoning tokens to reach a conclusion. After reading the manual they split it into small questions: did production go down or become unusable, are they blocked with no workaround, and is something in Rivet broken or failing. Code then returns SEV1 if blocked, SEV2 if down, SEV3 if broken, and otherwise SEV4. He says that works. When a ticket is weighted badly, they give it to an LLM with what they expected, the LLM adds it to a test suite and adjusts the decision logic, and they backtest the new logic on old issues. Within a few tickets the classifier was solid. He calls this a feature of Jev, not a limitation: more control over how each decision is weighed than a generic LLM, and easier to reason about, monitor, and backtest how agents will behave. In their internal support tool the Jev question counts are 13 for support classification and customer status, 6 for severity, 6 for team replies and problem identification, 6 for follow-up timing, 4 for linked dependencies and later conversations, 2 for the addressed teammate, and 2 for conversation grouping. He ends with: Jev is awesome. A screenshot of a real-world classifier is attached.
- 40.7k views
- 470 likes
- 587 bookmarks
- 13 reposts
quick tip we learned the hard way @ rivet: jev does not reason ...which is by design ...and we should have rtfm first ~ we started by prompt-engineer to ask som…
About this case
Guide. Nathan Flurry 🔩 (@NathanFlurry) attaches one photo of a real-world classifier. The post text has no URL.
Original posthttps://x.com/NathanFlurry/status/2123108222470959600