The Consensus Loop: AI Translation You Can Trust, With the Numbers to Prove It
A single language model will translate anything you give it, and it will always sound sure of itself, and that confidence is precisely the problem, because the output reads well even when it is wrong. If you don't compare it with the source text, it seems fine. But a legal term can be translated into something that it does not mean without you noticing it, because there is nothing in the sentence to warn you, so unless a human reads that line, and reads the source text against it, the error goes out to the client unedited.
When I started rebuilding our translation engine, it took me seconds to get a translation back from any model, but knowing which of those translations I could trust, and which were correct, without any human having to check them, took me months.
I run a translation company. Two years ago, generative AI started eating into our revenue, and I made a decision that most LSP owners are still considering: instead of buying someone else's black box, I decided to rebuild our production engine myself, with large language models at the core. This article is the full picture of its centrepiece, the consensus loop.
The first half is written for people who run language businesses, and the second half is the engine room, with the design decisions and the failures, for whoever wants to build one of these themselves.
Machine translation got good enough, which means that its raw output is no longer what a client pays for: they can get it for free.
Anyone can get a fluent draft in seconds, free or nearly so, and clients know it. They do it every day!
What they are still willing to pay for is what the draft does not give them, which is the assurance that the translation is right, which can only come from human review, which is human time, the most expensive resource in the cost structure of an LSP.
And that is exactly where the economics of translation got complicated. If you cannot tell which segments to trust, then you have to pay a human to read every single line, which means your costs stay at human levels while the market is repricing the work at machine levels. That is, clients see translation as almost free, need it to be checked, but are not willing to pay the prices of yesteryear.
And the reality is that the review effort is spread evenly across work that mostly does not need it, and the segments that genuinely need attention end up getting the same minutes as the ones that were already right. The economics of an LSP in the LLM era live and die on one question: which segments can skip the human?
The business half: which segments can skip the human?
Answering that question badly is worse than not answering it at all. A language model hands you fluent, confident text even when it is wrong, and the errors that hurt are exactly the ones that are invisible at a glance.
Worse than that, the standard quality scores can be fooled. I found segments where the "translation" was the source text copied over, completely untranslated, and the automated quality score rated them as near-perfect. If the score lets an untranslated copy through, what else is it letting through? In our case, about 3,000 segments had been wrongly marked as trusted before my safety net caught the problem. If your pipeline cannot catch that class of error, the quality score is not protecting you.
So the system I built does not just translate. What it produces, for every single segment, is a defensible answer to the question "does a human need to see this?". Several models translate, critique and rewrite each segment until they agree with each other, and deterministic checks catch the numbers, the tags, the units and the negations. A quality model scores every segment on my own GPU, at a fixed cost. In the end the score becomes a grade, from "locked, ships untouched" all the way down to "failed, human required".
The design is nothing exotic, and that is the point. It is the workflow that every serious translation company already runs, but instead of humans, rebuilt with models. No LSP ships a single translator's first draft: there is a translator, then a reviser who challenges the work, then a quality check before anything goes out the door. Each of those steps exists because one pass by one person is not enough, and we have known that for decades. If we would never trust one human's first attempt, why would we trust one model's?
| In a translation team | In the loop |
|---|---|
| The translator does a first draft | A model proposes a translation, with glossary and TM as context |
| The reviser challenges it | A second model flags problems and the segment is rewritten, re-looping until they agree |
| QA reads the doubtful ones back | On the lowest-scoring segments, a third model back-translates to check the meaning held |
| QA gives it a verdict | A reference-free quality score sets the grade that decides what a human reviews |
The benchmark: what humans actually deliver
Before any number from my pipeline can mean anything, one thing has to be settled first: compared with what? The instinctive answer is "human quality", imagined as zero errors. The problem is that this standard has never existed. The industry's own acceptance thresholds say so: the QA frameworks LSPs have used for decades, the LISA QA Model and SAE J2450, score translations in penalty points per thousand words, and they pass any work that stays under a threshold. A pass mark above zero would not need to exist if human output were expected to be error-free.
And the research agrees. In the largest MQM study of translation quality (Freitag et al., 2021, run with professional annotators), professional human translations were clearly preferred over machine output, and even so they still carried a measurable error load of their own.
And the step that everyone trusts to catch what remains, the revision, is leakier than the standard assumes. Revision research has repeatedly found revisers catching roughly half to two thirds of the errors in front of them, while introducing some new ones of their own along the way (Künzli 2007; Robert 2013; Brunette et al. 2005).
I see the same thing in my own data, and not at the number I expected, to be honest. Of the segments that reach a reviser as 100% exact matches from our own human translation memory, built and maintained by professional translators and revisers over 20 years, the reviser still changes roughly 12% across all language pairs, and about 10% in our core English–Portuguese memory. That counts any edit at all, cosmetic and stylistic changes included, so it is a ceiling on disagreement, not an error rate.
But the direction is the point here: the asset that humans built and polished for the longest time does not pass a second human untouched, one time in ten. If that is what trusted human work looks like under review, then "untouched by a reviser" was never the bar.
"Human quality" has always been an error rate, not an absence of errors. The honest question for any pipeline, human or machine, is where its error rate actually sits, and whether you can prove it. Nobody measures a traditional TEP pipeline, it is simply trusted. Mine is measured, and the numbers below are that measurement.
The numbers, honestly stated
A design only matters if it survives contact with the reviewers, so here are the numbers, with their caveats attached to each one, because a number without its caveat is just marketing. But before the numbers themselves, you need to know where they come from, because that is what makes them worth anything.
First, the grades. Every segment that comes out of the loop gets a grade, and the grade is a summary of everything the system knows about that segment: the quality score, how many rounds it took, whether it had to be re-looped, whether any risk was flagged, and whether it passed the safety net.
At the top sits the locked grade, which is a trust label: top score, first round, no re-loops, no flags, safety net clear. Below it, A is top quality without the locking commitment, B passed, C and D only passed after re-loops, and E failed. Everything below the locked grade still goes to human revision, but with one difference: E arrives flagged as failed, so the reviser knows to expect trouble, while A, B, C and D arrive showing no status to the reviser at all. The full mechanics are in the engine room below; for the business half, what matters is that a grade is a claim about risk.
Second, the measurement. For months, every segment went to human revision, whatever its grade said, and we recorded every word that the reviewers changed. Each change gets classified: cosmetic (punctuation, spacing, capitalisation), preference (a synonym, a style choice), or a real error (wrong terminology, changed meaning, a critical factual slip). Strip out the cosmetic and the preference changes, and what remains is a real-error rate per grade, measured in words.
Third, the lock came last, and it came slowly. The top grade spent its first few months as a label only: segments that met all of its conditions were marked as such, but they still went to human revision, unlocked, like everything else, while the edit data kept accumulating. And fourth, only when the measured rate held up did we make the lock effective in the delivered files, and that was recent.
So when I say that today about one in five segments ships locked, with no human reading it, that number stands on that history. For the work that still gets reviewed, the recording never stopped, and this is what it shows, per grade, in words, across every language pair we run:
| Grade | Real-error rate (in words) |
|---|---|
| Locked (ships without review) | 0.9% |
| A | 1.6% |
| B | 2.6% |
| C | 3.8% |
| D | 2.0% (a thin slice, too small to read much into) |
| E (failed, human required) | 4.9% |
Across everything graded, the rate is 3.1%.
That table reads three ways, and all three are honest. The ladder predicts risk before a human ever sees the segment: the error rate climbs as the grade drops, from under 1% at the top to 4.9% at the bottom, and that gradient is the whole point of having grades. And remember that the revisers who produced those numbers could not see the grades, so the ladder was predicting the behaviour of people who did not know it existed.
The second reading is that most of what reviewers touch is not error. Of all the words the reviewers edited, around three quarters are preference or cosmetic, a different word order, a synonym, an accent on a heading. The real-error rate is what survives after stripping all of those out.
And the third reading: 0.9% is not zero, and I will not pretend that it is. Locking is a bet that the residual error rate is low enough to be worth the review capacity it frees. The data says it is, but it does not say the tier is perfect.
And the measuring did not stop when the lock went live. Some segments of the locked tier still meet a reviser, simply because a project manager sometimes unlocks them manually, when they want that content revised anyway. On top of that, the system audits itself: per language pair, once a running word counter crosses a threshold, the next whole job is delivered for full review with nothing locked. The audit jobs carry no visible flag, so the reviser does not know that the job is an audit, and the measurement stays blind.
Locked never means hidden either: even in normal jobs, the segment arrives in the CAT tool confirmed and locked, and a reviser can always open it. The 0.9% is not a launch-day number that froze. It keeps having to earn the lock.
That is the business half. The rest is the engine room: the loop, the exit conditions, the checks, the safety net, and the prompts. What follows are the design decisions, the rules, and the failures, with just enough of the production system showing to prove that it is real. And if you run an LSP and want the outcome without building any of this, you now know what to ask of whoever builds it for you.
The shape of the loop
Each segment goes through a team of models with fixed roles. One model translates, with the glossary and the translation memory as context. A second model, always from a different provider, reads the result and flags what looks wrong, and it never proposes text. The first model then rewrites the flagged parts. They go back and forth until they agree, and the deterministic checks for numbers, tags, units and negation run inside the loop, so the obvious things never slip through.
The rounds have a small, fixed ceiling and an early exit. My first version ran every round every single time, whether the segment needed them or not. The spec said six. Today the ceiling is four rounds, or three on heavy jobs, and the loop exits the moment there is agreement, which means the average sits far below the ceiling.
Two rules are what make this a consensus rather than an echo chamber. The verifier is never the provider that translated the batch. And the critic never proposes text, it only flags problems, so the translator cannot blindly copy a suggestion that might itself be wrong. Two independent voices, by construction. Of course, independence has its limits, because different providers still share training data and blind spots, and that is exactly why the loop is never the only signal: the deterministic checks and the quality score do not depend on what the models agreed on.
What "agreement" means
Agreement means zero blocking issues, zero regressions, and stability. And stability here is stricter than just "the text stopped changing": it means fewer than 3% of the segments changed AND no critical token (a number, a negation, a unit, a modal, a comparator) changed.
When the loop does not reach agreement, it never silently ships the result. Every single non-agreement exit marks the segment for human review:
| Exit | When |
|---|---|
max_rounds | rounds exhausted with blocking issues remaining |
convergence_breaker | the blocking count stops improving for 3 rounds |
false_positive | 2 or fewer blockers, stable and repeated between rounds (the critic is probably wrong) |
parse_failure | 2 consecutive unreadable critic responses |
timeout | the batch exceeded its time limit |
The convergence breaker is what stops the loop from burning tokens forever: when the blocking count stops improving for three rounds, the loop accepts the current state, records the open issues as comments for the reviewer, and exits.
There is one guard on this exit that matters more than its size suggests. The false_positive exit exists for critics that keep flagging the same one or two issues that the other models keep refusing to change: after enough repetition, the loop concludes that the flags are noise, and it moves on.
But there is one flag that is never treated as noise. Before accepting the exit, the loop checks whether the segment in question is source equal to target in a recognisable source language, in other words, an untranslated copy. If it is, then the critic was right all along: the exit is blocked, and the segment keeps its path to a human. This is the same class of error that fooled the quality score back in the business half, and it gets no benefit of the doubt anywhere in this pipeline.
The deterministic layer: predictable, auditable checks
Before and around the critic, purely algorithmic checks run on every segment. They are cheap and reproducible, and when one of them flags something, you can trace exactly why. They still produce false positives and false negatives of their own, of course, but they fail in ways that you can read and fix. They are the first line of defence, and their flags map to the blocking issue types that the critic uses.
The number check holds a lesson about translation specifically. "1,234.5" in English and "1.234,5" in Portuguese are the same value written in two different ways, and a naive string comparison would flag every properly localised number as an error. So the check normalises the locale first, and it only fires when the value changes, not when the notation does.
The negation check gets domain-aware strictness. In legal and medical content, any asymmetry between the source and the target negation is a risk. Everywhere else, only a lost negation fires.
And before that runs, abbreviations are stripped, so that "No. 12345" is never confused with the negation "no". Tags and placeholders are extracted with their IDs and counted per type, with missing and extra computed on both sides. Units demand exact equality in the technical and medical domains.
The safety net: when the "translation" is the original, untranslated
Here is the failure I learned the most from, and the reason why no single signal in this system gets the final say.
The pipeline sometimes copied the source straight into the target, leaving it untranslated. And the quality score came back near-perfect. A reference-free quality model is blind to this case, because an identical source and target look like a flawless match to it, so it scores a copy at 0.95 or higher. A whole project once came back this way, hundreds of fresh segments marked as top quality and locked against editing, until an internal reviewer caught it.
My first fix over-corrected. I loosened the filters, and it started demoting legitimate names and numbers, so I reverted it within a day. The fix that held is a conservative heuristic, and its two hard-won details are worth spelling out.
Detail one: the comparison is done on plain text, not on the raw markup. The bug that made this necessary was a subtle one. A cache served the target already stripped of tags while the source still had them, so a byte-for-byte if (src !== tgt) return false exited early and let the copy through. Stripping the markup on both sides before comparing is what closed it. About 3,000 segments had been wrongly trusted before that fix landed.
Detail two: the anti-false-positive filters exist because source==target is very often legitimate. Emails, URLs, phone prefixes, variables, all-numeric strings, proper names, all-caps headings. The philosophy is written in the code as a comment: in this net, a false positive costs more than a false negative. That reads backwards until you see what each one actually does here. A false positive demotes a legitimate segment, and when I briefly loosened these filters, that happened at scale, to names and numbers, across live jobs. A false negative lets one copy slip past this net, but not past the system, because the consensus loop runs its own source-equals-target guard, and the worst-scoring segments get back-translated. The net is one signal of several, so it can afford to be conservative; an aggressive classifier here would do more damage than the errors it is meant to catch.
From score to decision: how a grade is born
After the loop, every segment is scored by xCOMET, a reference-free quality estimation model that judges from the source and the translation alone, with no human reference needed. The score is a number between 0 and 1, and the thresholds on it are operating points rather than probabilities: 0.95 does not mean 95% certain, it means a line that earns its place against what the reviewers actually change. But a number is not a decision, so one canonical function turns every score into a grade, the same way on every path.
The ladder reads like this. A_locked is a trust label: score at 0.95 or above, first round, no re-loops, risk not high, and it passed the safety net. It means "good enough to lock against further editing", and no human reads it. A is top quality without the locking commitment. B passed QE, or was demoted there by the safety net. C and D only passed after one or two re-loops. E failed. On the worst segments, a third model back-translates the result into the source language, and a divergent meaning flags the segment for a human, but that layer deserves an article of its own.
One structural rule cost me a bug to learn: this function is canonical on purpose. There used to be two places in the pipeline deciding grades, and they diverged. A safety fix got applied to one of them and not to the other. Now A_locked means exactly the same thing on every path, and if structural inputs are missing, the grade is capped at A, never at A_locked: no trust label gets issued on incomplete data.
The prompts: three roles, one rule each
The prompts are where the independence gets enforced in language. There are three roles, and the rule given to the critic is the one that makes the whole loop work.
The translator gets hard constraints, numbered, with no exceptions: segment completeness with exact IDs, output format, placeholder preservation, glossary enforcement as an obligation rather than a suggestion, no assuming "already translated" in close languages, target naturalness over mirroring the structure of the source, and a self-check that it must run but not print.
The critic gets the rule that matters most:
You are a professional translation quality reviewer ...
YOUR ROLE IS PURELY CRITICAL. You MUST:
- ONLY identify what is wrong or can be improved
- NEVER suggest alternative translations
- NEVER propose corrected text
- If you find no issues, respond with isValid: true and an empty issues array
It returns structured JSON with a closed list of issue types, a severity level, an isBlocking flag, evidence, and a quote of the problematic part of the target. Scope rules keep the noise down: a placeholder mismatch can only be reported if the source actually has markers, and brand or product names never count as placeholders.
The corrector closes the triangle: rewrite only the flagged segments, preserve every tag and placeholder, apply the glossary, and never invent tags that are not in the source. For regression detection, each round hands the critic the issues from the previous round, and the critic marks anything that got worse.
Why build instead of buy
I built this myself, with Claude Code on Linux, instead of buying an out-of-the-box tool. The truth is that off-the-shelf tools almost never fit. They either do far more than you need, or they stop just short of your actual case, and closing that last gap means changing how your team works to suit the tool. And the ones that do fit tend to sit behind a monthly licence steep enough that paying for it feels like adding another salary to the payroll. So I built our own, shaped to how we already work.
This is one piece of a larger rebuild. The next piece goes deep on that scorer: how a reference-free quality model judges a translation with no human reference, what it is blind to, and how its score becomes a decision that you can defend.