How Koko uses Zentropi to detect crisis
Guest Post By Rob Morris, Koko
Koko used Zentropi to build a crisis detection classifier that it has put into production to catch coded and implicit distress in real peer-support conversations. Koko brought the clinical framework and a decade of content moderation expertise; Zentropi brought CoPE, the model that runs the classifier, and the tooling that optimized the policy behind it. What follows is Koko's own account: how they wrote the policy, how they built the benchmark, and how CoPE scored against the frontier models they tested.
Crisis detection that works where it counts
A user tells your chatbot they want to die. Another writes "I just took a bunch of pills." Another uses coded language your classifier has never seen: "I want to catch the bus," "st4rv3," "km$."
Every platform deploying conversational AI faces this. Most solve it poorly — keyword filters miss anything subtle, and classifiers can't tell real crisis from surface-level distress. The result: models that flag "suicide" in a book report but miss a teen who plans to "step off a chair."
Koko has built content moderation models for over a decade, with the deepest expertise in mental health. Our peer support platform has served millions of young people, and our pipeline has processed over a million real conversations. From that experience, we used Zentropi’s tools to build a crisis detection policy and benchmark around one question: is this person at imminent risk of serious physical or emotional harm?
Policy and data, together
The construct covers active suicidal ideation, ongoing abuse, self-harm, domestic violence, exploitation, eating disorders, substance crises, and medical emergencies.
Crisis policies are usually written by clinicians and trust & safety leads. That expertise is necessary but not sufficient — the lexicon of crisis evolves faster than any expert panel can track. A suicide-prevention expert may not know that "draw red lines" is a euphemism on the platforms where young people actually talk. So we built our policy collaboratively between domain experts and real data: the clinical framework defines the construct; the data defines the boundaries.
A benchmark built from real crisis
Most crisis benchmarks are easy to ace and meaningless in production. A model that scores 98% separating "I want to kill myself" from "I had a great day" tells you nothing about "I wanna sleep forever" or "my uncle comes into my room at night."
Ours is built entirely from real posts written by people reaching out for help — misspelled, coded, euphemistic, ambiguous. We sampled four strata to stress-test specific failure modes: model-flagged posts, human-confirmed crisis, human-cleared false-positive traps, and never-flagged general population. The result: roughly 40% of posts fall in the boundary region — the cases that actually determine whether a model works.
Two independent raters labeled every post blind (86% agreement, Cohen's kappa 0.73). We adjudicated disagreements through a structured calibration protocol and tested for bias across six protected groups — none detected.
Real results
On a 388-post held-out benchmark, CoPE — Zentropi's 25.2B-parameter model (3.8B active), running locally — beat every frontier model we tested, while being significantly smaller, faster and cheaper to run:
94% recall means the model catches nearly every crisis, including coded and implicit cases. The 83% precision reflects the benchmark's design — it's deliberately loaded with boundary cases. On a representative production sample, precision is substantially higher.
Why it matters for your platform
If your users might disclose crisis-level distress, you face two problems. In the moment, you need to know when to intervene — not when someone mentions a sensitive topic, but when they need help right now. After the fact, you need to audit what happened: how many crisis disclosures did your model encounter, and did it respond well?
That second use case may be the more important one. You can't evaluate how your model handles vulnerable users by reading transcripts — there are too many — or by keyword search — the language is too varied. A precise crisis detector narrows the transcripts to the ones where someone disclosed a crisis. Reviewing those is how you find out whether your model responded well.
The policy is yours to define. The methodology lets you measure it. And the results show it works.