Optimizing our Policy Optimizers

At TrustCon this year, Dave gave a talk about how to automatically optimize a policy using an inversion of the binocular labeling methodology that Zentropi previously pioneered. He also teased a new tool we had in the works that’s based on this technique.

Today, the wait is over. We are releasing this next-generation policy optimization tooling on the Zentropi platform! It’s actually three separate tools: policy-only correction, label-only correction, and auto-optimization. Together, they automate the work of refining a content labeler until it's as accurate as possible. 

These new optimization tools close the loop on what was once merely a theoretical possibility: machines can now edit policy machines.

Policy-only correction

The first tool, policy-only correction, takes your current policy and your labeled dataset and asks a simple question: where does the policy disagree with the labels, and can a better expression of the policy idea close that gap?

It works by generating many potential revisions to your policy - alternative phrasings, additional clauses, refined carve-outs - and testing each one against your data. Because CoPE evaluates content against policy at inference time with no retraining required, each candidate revision can be tested immediately. There's no model retraining cycle, no waiting period. The optimizer writes a revision, tests it with CoPE, and remembers the best results.

CoPE’s speed means the optimizer can try many policy variations in the time it would take a human to draft and review one. It surfaces the revisions that improve accuracy and discards the ones that don't. The output is a refined policy that is better articulated for machine interpretation.

Label-only correction

The second tool, label-only correction, inverts the question. Instead of asking whether the policy is wrong, it asks whether the labels are.

This matters more than most teams expect. When you bring a labeled dataset to a content classification problem, you typically treat those labels as ground truth - the gold standard your policy should match. But in practice, most labeled datasets aren't golden. They were created by humans working under time pressure, applying interpretations that may have drifted from the current policy, or dealing with edge cases that were never clearly resolved. Labels carry noise, and that noise compounds.

Label-only correction evaluates your existing labels against your current policy and flags the ones that don't hold up. If the policy says one thing and a label says another, the optimizer identifies the disagreement and proposes a corrected label - with the policy as the reference point, not the other way around.

This is crucial to getting good results. Correcting the labels against the current policy matters as much as correcting the policy itself. A policy is only as good as the data you test it against, and that data is only as good as the labels behind it. If either side is wrong, the whole system underperforms - and you can't always tell which side is the problem just by looking.

Auto-optimization

While both policy-only correction and label only-correction are valuable on their own, they skirt around the core problem of policy optimization: there really is no such thing as "ground truth." A given label, for a given piece of content, can only be said to be true or false against a particular policy text: if the policy text changes, a label that was correct can become incorrect purely from a shift in the nuance of the line you're drawing.

So if you optimize the policy text alone, then check how well it guides CoPE to match your expected labels, you're scoring against a shifted target. The labels were made under the old text, and you're now measuring the new text against these potentially outdated judgments. Correcting all of your golden data against every revision of a policy text is incredibly labor intensive, so in practice no one does it when revising.

Put another way - properly understood, a policy and its labels are one object with two halves. Figuring out which half of a disagreement belongs to the policy and which to the label is central to fixing both.

Auto-optimization automates that laborious back-and-forth. It runs label-only correction, then policy-only correction, then label-only correction again - cycling between the two correction modes until no further improvement is found. Each pass tightens the alignment between policy and labels. Each pass removes a layer of noise. The loop stops when additional revisions stop producing gains - when the policy and the labels have converged on the best version of each other.

By the Numbers

Our quantitative evals of these next-gen optimizers show a leap in labeler tuning capability. Across an internal corpus of already mature policies, our next-gen optimizer in policy-only mode lifted F1 scores by 8.4% on average over the starting policy. Auto mode did even better with an 11.2% average F1 lift when scored on its auto-corrected labels. We expect others to see similar or even better results on their own labelers.

Try it

The next-generation optimizer is available now. Use it on our website, or for an even simpler experience, install the Zentropi Agent Skill and ask your agent to optimize your policy with it.

Due to the intensive computation required, this capability is normally subscriber-only. But to celebrate this milestone, we’re giving everyone a limited quota of free runs to try it for themselves. Check it out today!