The contract came with a number in it, and the number was the problem. A mid-size language-service provider had won a framework agreement to post-edit a German consumer-electronics account: roughly forty thousand words a week of manuals, support articles, and UI strings, machine-translated first and post-edited by a rotating pool of nine linguists. Buried in clause 7.3 was a single sentence the sales team had skimmed past: "Deliverables shall be evaluated to a per-thousand-word error score not exceeding 15, with zero Critical errors, scored to ISO 5060." For the first three weeks nobody scored anything. Files shipped, the client said nothing, and everyone assumed the clause was boilerplate. In week four the client sent back a sample they had scored themselves: a quality score of 41 against a ceiling of 15, two Criticals, and a one-line email asking the provider to "share your evaluation records for the same file." The provider had no records, because the provider had no workflow. They had nine people post-editing well and nobody scoring repeatably, which meant that when the client produced a number, the provider could not produce a number back. This lesson builds the thing they did not have: a severity-scored evaluation workflow that is a real, repeatable step inside the pipeline, the same way for every evaluator, on every file, producing a number you can stand behind when a client produces a different one. Not the theory of severity, which you already have, but the operational machine that turns marked errors into a defensible score, file after file, evaluator after evaluator.

From Scoring an Error to Scoring a File

You already know how to assign a severity to a single error. That is a necessary skill and it is not this lesson. The gap between "I can decide this error is a Major" and "our pipeline produces a defensible quality score on every file" is the gap between a judgment and a workflow, and the provider in the opening story fell straight into it. They had nine linguists who could each look at an error and call it correctly. What they lacked was the connective tissue that turns nine private judgments into one repeatable number: a defined unit to score, a sample to score it on, a typology to mark against, a weight per severity, a normalization that makes a long file and a short file comparable, and an output format a client and an auditor can read. Each of those is a step. Skip any one and the score stops being reproducible, which means it stops being defensible.

The vocabulary first, used precisely throughout, because this lesson assembles several pieces of machinery and the names have to be exact. Machine translation (MT) is any system that renders text from a source language into a target language with no human writing the words. A large language model (LLM) is a general-purpose text predictor that translates as a side effect of its broad training and is fluent even when it is wrong. Machine-translation post-editing (MTPE), often shortened to PE, is the workflow where a human edits machine output rather than translating from scratch. A segment is the unit a translation tool works in, usually a sentence, the row a linguist sees in a CAT tool (a computer-assisted translation tool, the editing environment the linguist works in). A termbase is the controlled glossary of a client's approved terms. MQM is Multidimensional Quality Metrics, the analytic error-typology framework that classifies translation errors by dimension (the kind of error) and severity (how much it matters). ISO 5060:2024 is the international standard that formalizes an MQM-aligned model for the human analytic evaluation of translation output, including the Critical, Major, and Minor severity bands and the principle of computing a normalized error score. Quality estimation (QE) is an automatic, model-produced confidence signal about a translation, distinct from the human evaluation this lesson builds. A quality score is the single normalized number the workflow outputs. Define each acronym once, then trust it.

The workflow this lesson builds has six steps in a fixed order: sample the content, mark errors by dimension, assign a severity to each, weight the severities into penalty points, normalize the penalty against the size of what you scored, and output a score with the records behind it. Around those six steps wraps a seventh concern that is not a step but a property: repeatability, the discipline that makes two evaluators scoring the same file land on the same number. We build the six steps in order, then we make them repeatable, then we run the whole thing on a real file end to end.

A single error severity is a judgment. A file score is a workflow: sample, mark, assign, weight, normalize, output, and make it repeatable. Skip a step and the number stops being defensible.

Step One: Sampling, or What You Actually Score

The first decision is the one most teams never make consciously, and it is the one that quietly breaks everything downstream: what, exactly, are you scoring? You almost never score every segment of a large delivery. Forty thousand words a week scored in full would cost more than the post-editing itself and defeat the economics that put MT in the pipeline in the first place. So you score a sample, a defined subset of the content, and the score on the sample stands in for the quality of the whole. That substitution is only valid if the sample is chosen by a rule, not by convenience, because a sample chosen by convenience can be gamed, cannot be reproduced, and tells a client nothing trustworthy about the file they are paying for.

A defensible sampling rule fixes three things in advance and writes them down. First, the size: how many words or segments you score, stated as an absolute number or a percentage with a floor. A common shape is "the greater of 1,000 words or 10% of the delivery," so a small file is scored heavily and a large file is scored proportionally without exploding the cost. Second, the selection method: how the specific segments are chosen. Random selection across the whole file is the baseline because it cannot be steered toward the cleanest passages. Stratified selection is stronger still: you deliberately sample across content types, across the different linguists who worked the file, and across the risk tiers present, so that no slice of the delivery escapes evaluation by luck of the draw. Third, the coverage rule for high-risk content: any segment in a high-liability tier (a dosage, a contraindication, a legal obligation, a financial figure) is scored at 100%, never sampled, because the whole point of sampling is to save effort on low-consequence content, and there is no acceptable amount of effort to save on a drug label.

Why the Sample Rule Must Precede the File

The sampling rule has to exist before you open the file, and this is not a bureaucratic nicety, it is what makes the number mean something. If you choose which segments to score after you have seen the file, you will, consciously or not, choose in a way that flatters or condemns the result, and a client who knows you chose after the fact has every reason to distrust the score. A rule fixed in advance, "random stratified sample, 10% with a 1,000-word floor, 100% coverage of the safety tier," is reproducible: a second evaluator handed the same rule and the same file scores the same segments. Reproducibility is the property that lets your score and the client's score be compared at all. The provider in the opening story could not even have that conversation, because they had no rule, so the client's sample and their non-existent sample had nothing in common to argue about.

One more discipline lives in the sample step: record the scored word count precisely, because it is the denominator of the entire calculation that follows. Normalization divides penalty points by the size of what you scored, so if you score 1,240 words, that exact figure, not the file's total word count and not a round-number guess, is what the score is computed against. Sloppiness here silently corrupts every number downstream.

Step Two: Marking Errors by Dimension

With the sample fixed, the evaluator reads the scored segments against the source and marks every error. Marking has two parts that must stay separate: identifying that something is wrong, and classifying what kind of wrong it is. The classification is the dimension, and the workflow uses the MQM-aligned dimensions that ISO 5060 organizes its model around. Four carry the overwhelming majority of localization errors, and naming them is what makes a marked error a structured data point rather than a margin note.

• Accuracy: the target does not say what the source says. Mistranslations, dropped or added content, dropped negations, flipped instructions, invented numbers. This is the dimension where the silent critical error lives, because an accuracy error can read perfectly and mean the opposite.
• Terminology: the target uses a term that violates the client's approved termbase or domain convention, even if the substitute is a real and fluent word. The engine rendering the approved "concentrator" as "the unit" is a terminology error.
• Locale conventions: the target gets a locale-specific convention wrong: date format, number and decimal separators, units, currency, address format, formality register. The convention is wrong for the target market even when the words are right.
• Fluency: the target reads badly in its own right: grammar, spelling, punctuation, register, awkward phrasing. Fluency errors are about the target language as language, independent of the source, and they are the dimension most likely to be Minor.

Marking by dimension does real work beyond tidiness. It forces the evaluator to name what is wrong before deciding how much it matters, which prevents the gut-feel collapse where "this segment feels bad" becomes a severity with no stated cause. It produces a typed error record that aggregates: at the end you can tell the client "your file failed on terminology, not accuracy," which is actionable in a way that "the file scored 41" is not. And it lets you spot patterns across files: an engine that produces terminology errors at ten times the rate of accuracy errors needs a glossary intervention, not a post-editing intervention, and only dimension-tagged data reveals that.

The Span and the Note Make the Error Auditable

Each marked error needs more than a dimension to be defensible later. The workflow records, for every error, the segment identifier (which row), the error span (the exact text that is wrong, highlighted in the target), the source span it should have rendered, the dimension, and a one-line note stating the divergence: "source says 'or', target says 'and'." This is the atomic unit of the quality record. An error marked this completely can be reconstructed by anyone, argued by the client, and confirmed or overturned by an auditor without re-reading the whole file. An error marked as a bare "Major, accuracy, segment 14" with no span and no note is an assertion, not evidence, and it collapses the moment someone challenges it. The discipline of capturing the span and the note at marking time, not reconstructing it later from memory, is what separates a quality record from a tally.

Mark the dimension, the segment, the exact span in source and target, and a one-line note on the divergence. An error you can reconstruct is evidence. An error you merely asserted is a tally, and a tally loses every dispute.

Step Three: Assigning Severity to Each Marked Error

Now the severity decision you already know how to make gets applied inside the workflow, once per marked error. For each error the evaluator assigns one of three bands. A Minor error does not meaningfully affect meaning or usability: it annoys but does not mislead. A Major error meaningfully impairs comprehension or usability: it misleads the reader to a wrong understanding but acting on it does not cause real-world harm. A Critical error renders the content dangerous, unusable, or legally exposed: it misleads on a point where acting on it causes safety, legal, or financial harm. The three-question test, does it mislead, does it harm, or does it merely annoy, sets the floor: merely annoys is Minor, misleads without harm is Major, misleads and harms is Critical.

Two things matter about doing this inside a workflow rather than in the abstract. First, severity is assigned independently of dimension. Every dimension spans the full severity range: a terminology error is Minor for a generic UI label and Critical for a wrong drug name; an accuracy error is Minor for an imprecise adjective and Critical for a flipped negation. The workflow keeps the two fields separate precisely because conflating them, "terminology errors are always Major," is a common and expensive shortcut that produces wrong scores. Second, severity is assigned against the content's risk tier, which the sampling step already established. The same date-format error is Minor on a coupon and Critical on a medication expiry, and the evaluator must know which tier the segment sits in before answering the harm question. This is why the workflow carries the risk tier alongside the content into the scoring step rather than asking the evaluator to guess it.

The Client Scoring Profile Overrides Judgment Where It Speaks

A mature workflow does not leave every severity to the evaluator's judgment, because judgment is where evaluators diverge. Many clients pre-declare a scoring profile: a set of rules that fix severity for whole categories of error and remove the judgment entirely. "Any termbase violation is scored Major minimum." "Any accuracy error in safety content is Critical." "Locale errors on dates are Major minimum." Where the profile speaks, it overrides the evaluator's instinct, and following it is itself the defensible move because the rule, not the evaluator, made the call. The workflow loads the client profile before scoring begins, applies its declared rules automatically where they match, and falls back to the three-question test only where the profile is silent. This single mechanism removes a large fraction of inter-evaluator disagreement, because it converts the most-disputed categories from judgment calls into lookups.

Step Four: Weighting Severities into Penalty Points

So far the workflow has produced a list of marked errors, each tagged with a dimension and a severity. To turn that list into a number you need a weight per severity band: a penalty value that says how many points each Minor, Major, and Critical costs. Weighting is what makes the severities commensurable, so that "three Minors and one Major" becomes an arithmetic quantity instead of a description.

The classical MQM weighting, the one most ISO 5060-aligned profiles start from, is 1 point per Minor, 5 points per Major, and a large penalty (commonly 25, sometimes higher) per Critical. The ratios encode the discipline's core belief: a Major matters roughly five times as much as a Minor, and a Critical is in a different league entirely. The exact Critical weight matters less than you might think, for a reason we come to in the next step: in most workflows a single Critical does not merely add 25 points, it fails the file outright through a separate rule, so the 25 is a placeholder that almost never decides the outcome by arithmetic. The Minor and Major weights, by contrast, decide real outcomes on Critical-free files all the time, which is why the 5-to-1 Major-to-Minor ratio is the number that actually moves scores.

Why the Weights Are a Policy, Not a Fact

The weights are not laws of nature; they are a policy the workflow declares and the client agrees to. A client running life-safety content might widen the gap, weighting Majors at 10 and Criticals at 100 to make the score react violently to serious errors. A client running high-volume marketing content might narrow it. What matters for repeatability is not which weights you pick but that the weights are fixed and written into the profile before scoring, so that every evaluator on every file applies the same multipliers. A workflow where the weights drift between evaluators produces scores that cannot be compared even within the same provider, which is the disease the opening-story provider had in its most basic form: not only no rule, but no shared arithmetic. Declare the weights once, version them with the profile, and never let an evaluator improvise them on a file.

The weighting step itself is mechanical once the weights are fixed: multiply the count of each band by its weight and sum. Three Minors, two Majors, zero Criticals, under classical weights, is (3 × 1) + (2 × 5) + 0 = 13 penalty points. That 13 is the raw penalty, and it is meaningless until the next step makes it comparable across files of different sizes.

Weights are a declared policy, not a discovery. Fix them in the client profile, version them, and apply them identically on every file. The 5-to-1 Major-to-Minor ratio is the number that decides Critical-free files; the Critical weight rarely decides anything because the Critical rule fires first.

Step Five: Normalization, or Making Files Comparable

Thirteen penalty points means nothing on its own. Thirteen points on a 200-word sample is a disaster; thirteen points on a 13,000-word sample is nearly spotless. The raw penalty has to be divided by the size of what you scored to become a rate, and that division is normalization. It is the step that lets you set one quality threshold that applies to a tweet and to a technical manual, because it converts an absolute count of pain into a density of pain per unit of content.

The standard normalization in ISO 5060-aligned scoring is the error score per thousand words: take the total penalty points, divide by the scored word count, and multiply by one thousand. The thirteen-point sample, if those thirteen points came from a 1,000-word scored sample, is a score of 13.0 per thousand words. If the same thirteen points came from a 2,600-word sample, the score is (13 ÷ 2,600) × 1,000 = 5.0 per thousand words, a very different file even though the raw penalty is identical. The formula is small and it is the hinge of the whole workflow:

• Penalty points = (Minor count × Minor weight) + (Major count × Major weight) + (Critical count × Critical weight).
• Normalized score = (Penalty points ÷ scored word count) × 1,000.
• The result is an error score per thousand words, lower is better, compared against the client's declared passing threshold.

The passing threshold is itself part of the profile: the opening-story contract set it at "not exceeding 15 per thousand words." A normalized score of 13.0 passes that threshold; a 16.0 fails it. Note what normalization does and does not do: it makes files of different lengths comparable on the weighted Major-and-Minor axis, but it sits entirely underneath the absolute Critical rule. A file can have a beautiful normalized score of 2.0 and still fail, because the Critical rule overrides the arithmetic. Normalization decides Critical-free files against the threshold; it never rescues a file that contains a Critical.

The Denominator Discipline

The single most common arithmetic error in this workflow is using the wrong denominator. The normalized score must divide by the scored word count, the size of the sample, not the total word count of the delivery. If you scored a 1,000-word sample of a 40,000-word file and accidentally normalize against 40,000, your score collapses to a fortieth of its true value and a failing file sails through. The sample step recorded the scored word count precisely for exactly this reason. Tie the denominator to the recorded sample size, automate the division so a human never types the wrong number, and the most embarrassing failure mode in evaluation, a passing score on a failing file because of a divide-by-the-wrong-thing, disappears.

Step Six: The Score Output and What It Has to Contain

The final step produces the artifact the workflow exists to produce: not just a number, but a score output that a client and an auditor can read and trust. A bare "41" is what the opening-story client sent the provider, and it started a fight precisely because it was bare, the provider could not see what was in it or argue with it. A defensible score output makes the number transparent and reconstructible. It contains, at minimum, the following.

• The headline result: the normalized error score per thousand words, the pass/fail verdict against the threshold, and the Critical count called out separately because one Critical is its own verdict.
• The sample definition: the sampling rule used, the scored word count, and which segments were in scope, so the reader knows what the score is a score of.
• The error list: every marked error with its segment, source span, target span, dimension, severity, and the one-line note, the atomic records from the marking step.
• The arithmetic: the error counts per band, the weights applied, the penalty total, and the normalization, shown so the number can be recomputed by hand.
• The profile reference: which client scoring profile and version governed the scoring, including the weights, the threshold, and any declared severity rules that overrode evaluator judgment.

The reason every one of these belongs in the output is the Friday-afternoon escalation. When a client disputes a score, they do not dispute the headline number, they dispute a specific call: "why is segment 14 a Critical?" An output that carries the error list with spans and notes answers that in seconds. An output that carries only the headline forces you to re-open the file, re-find the error, and re-argue it from memory, three weeks later, with the relationship already strained. The output is not paperwork after the real work; it is the evidence the real work produces, and capturing it at scoring time costs almost nothing while reconstructing it later costs the account.

The output is the product. A number nobody can reconstruct starts fights; a number backed by the sample, the error list with spans, the arithmetic, and the profile version ends them. Capture the evidence while you score, not when you are challenged.

Making It Repeatable Across Evaluators

Six steps, run by one person on one file, produce a score. The same six steps run by nine people on nine files produce nine scores that may not be comparable, and non-comparable scores are worse than no scores, because they carry the false authority of a number while encoding the private biases of whoever produced them. The opening-story provider's deeper problem was never that they lacked a formula; it was that nine linguists scoring independently would have produced nine different numbers on the same file. Repeatability, the property that two evaluators scoring the same content land on the same score, is what turns the workflow from a personal habit into a pipeline capability, and it is built deliberately through three mechanisms.

Guidelines: The Written Rule Set

The first mechanism is a written scoring guideline, the document that fixes every decision the workflow does not leave to per-error judgment. It states the sampling rule, the dimension definitions with examples, the severity definitions with the three-question test, the weights, the normalization formula, the threshold, and the client profile's overriding rules. It is the single source of truth that every evaluator reads before scoring and refers back to during scoring. Without it, each evaluator carries a slightly different version of the rules in their head, and the scores diverge in ways no one can trace. The guideline is mundane and it is load-bearing: it is the artifact that makes "we all score the same way" a checkable claim rather than a hope. ISO 5060 expects this kind of documented, repeatable procedure precisely because a standard that produced different numbers in different hands would certify nothing.

Calibration: Converging on the Ambiguous Middle

The second mechanism is calibration, the periodic exercise where the evaluators score the same sample independently, then compare and reconcile. Calibration is not a test that someone passes or fails; it is the process that surfaces and resolves the disagreements the guideline did not anticipate. The clean cases never need it: every evaluator scores the dropped negation in the safety warning as Critical and the missing comma as Minor. The disagreement lives entirely in the ambiguous middle, the term drift that might be Minor or Major, the date error whose harm depends on what the date governs, and calibration is where the team argues those cases to a shared rule and writes the resolution back into the guideline. A team that calibrates monthly converges; a team that never calibrates drifts apart silently, each evaluator's instinct hardening in a slightly different direction until two of them score the same file ten points apart and neither can say who is right.

Calibration also produces a measurable artifact: inter-evaluator agreement, the degree to which independent scorers match. A workflow that tracks agreement can prove its own repeatability to a client, "our evaluators agree within two points on calibration samples," which is exactly the kind of evidence an ISO 5060 conformance audit looks for. Disagreement on the clean cases signals that someone misunderstands the bands and needs retraining; disagreement on the ambiguous cases is normal and is resolved by argument and a guideline update, not by deferring to the most senior person in the room.

The Profile and the Tooling Remove Drift

The third mechanism is structural: load as much as possible into the profile and the tooling so the evaluator has fewer chances to drift. Every severity rule the client pre-declares is one fewer judgment call that can vary. Every weight and threshold fixed in the profile is arithmetic no one can improvise. Automating the weighting and normalization, so the evaluator marks errors and the system computes the score, removes the denominator mistake and the multiplication slip entirely. The general principle is that repeatability is highest when the workflow asks the evaluator for the smallest possible number of irreducible judgments, the genuine "does this mislead, does this harm" calls, and handles everything else, sampling, weighting, normalizing, profile rules, by mechanism. Human judgment is the scarce, valuable, and variable input; spend it only where it is irreplaceable, and let the workflow be rigid everywhere else.

A Worked Build of the Evaluation Step on a Real File

Abstract steps become a workflow only when you run them once, end to end, on something concrete. Take a slice of the opening-story account: a 1,000-word scored sample drawn from a delivered batch of German consumer-electronics support content, machine-translated and post-edited, scored under a client profile with classical weights (Minor 1, Major 5, Critical 25), a passing threshold of 15 per thousand words, a declared rule that "termbase violations are Major minimum," and the absolute "zero Criticals" rule from the contract. We walk all six steps and produce the output.

Sample. The sampling rule is "random stratified, the greater of 1,000 words or 10%, with 100% coverage of any safety-tier segment." The batch is 9,400 words; 10% is 940, so the 1,000-word floor governs. The tooling draws a stratified random sample across the three linguists who worked the batch and the two content types present (UI strings and support prose), and pulls in the four safety-tier segments in full. The scored word count is recorded exactly: 1,000 words. That figure is the denominator for everything that follows.

Mark and assign. The evaluator reads the sample against the source and marks each error with its dimension, span, note, and severity. The sample yields the following.

• Segment 12, terminology, Major. Source "battery pack"; the termbase mandates "Akkupack"; the target reads "Batterie". A real word, fluent, but a termbase violation, and the profile's declared rule makes it Major minimum. Note: "approved term Akkupack rendered as Batterie."
• Segment 31, fluency, Minor. An awkward but fully understandable clause ordering in a support sentence. It annoys, it does not mislead. Note: "stiff phrasing, meaning intact."
• Segment 33, locale, Minor. A US-style thousands separator on a non-critical spec figure in body prose. Misreads slightly but the value is unmistakable in context. Note: "comma separator where period expected, low-stakes figure."
• Segment 47, accuracy, Major. The source says the device "can be" wall-mounted; the target says it "must be" wall-mounted. Misleads the reader about an option versus a requirement, but mounting it on the wall when it did not have to be causes no harm. Note: "modal can be rendered must be."
• Segment 58, accuracy, Critical. A safety-tier segment. Source: "Do not charge the battery near an open flame." Target: the negation is dropped, reading "Charge the battery near an open flame." Misleads on a safety instruction and acting on it risks physical harm. Note: "dropped negation in safety warning, instruction inverted."

Weight. The counts are: 2 Minor, 2 Major, 1 Critical. Apply the classical weights. Penalty points = (2 × 1) + (2 × 5) + (1 × 25) = 2 + 10 + 25 = 37 penalty points.

Normalize. Divide by the scored word count and scale to a thousand. Normalized score = (37 ÷ 1,000) × 1,000 = 37.0 per thousand words, against a threshold of 15. On the weighted arithmetic alone the file is already failing, more than double the ceiling. But the arithmetic is not the verdict.

The verdict and the output. The Critical rule fires first and overrides everything: the sample contains one Critical (segment 58), so the file fails, full stop, regardless of the normalized score, and regardless of how clean the other 995 words read. The output records the headline (normalized score 37.0, threshold 15, Critical count 1, verdict FAIL), the sample definition (1,000-word stratified sample, 100% safety coverage, scored word count 1,000), the five error records with their spans and notes, the arithmetic (2 Minor, 2 Major, 1 Critical; weights 1/5/25; 37 points; normalized 37.0), and the profile reference (classical weights, threshold 15, termbase-violation-Major rule, zero-Criticals rule, profile version). That output is the document the opening-story provider could not produce in week four. With it, the conversation with the client stops being "trust us" and becomes "here is the file, here is the sample, here is the dropped negation in segment 58, here is the arithmetic, and here is why it fails, scored exactly the way your profile told us to score it." The number now defends itself.

The worked file fails not because 37 exceeds 15, but because one dropped negation in a safety segment is a Critical and the Critical rule overrides the arithmetic. The normalized score describes the file; the Critical rule decides it.

What the Workflow Produces Beyond the Pass/Fail

The pass/fail verdict is the headline, but a repeatable scoring step produces three more things the provider needs as much as the verdict. It produces a diagnosis: the dimension breakdown shows whether the engine and the post-editing are failing on accuracy, terminology, locale, or fluency, which routes the fix to the right place, a glossary update for terminology, a prompt or engine change for accuracy, a post-editor retraining for fluency. It produces a trend: scored the same way on every batch, the per-thousand-word score becomes a time series that shows whether quality is improving, holding, or drifting, which is the difference between managing quality and hoping for it. And it produces a defensible record: the accumulated outputs are the evidence of conformance the revised standards expect and a client audit demands, the proof that the human accountability for quality was exercised systematically rather than asserted. A workflow that produces only a verdict is leaving most of its value on the table; the records are where the score becomes management information and audit evidence, not just a gate.

Key Takeaways

• A file score is a six-step workflow, not a single judgment: sample what you score, mark errors by dimension, assign a severity to each, weight the severities into penalty points, normalize against the scored word count, and output a transparent score with the records behind it. Skip any step and the number stops being reproducible, which means it stops being defensible.
• The sampling rule must be fixed before you open the file: size (an absolute number or a percentage with a floor), selection method (random or stratified, never convenience), and 100% coverage of high-liability content. Record the scored word count exactly, because it is the denominator of the entire calculation, and a rule chosen after seeing the file is one a client has every reason to distrust.
• Mark each error with its dimension (accuracy, terminology, locale, fluency), its segment, the exact source and target spans, and a one-line note on the divergence. That complete atomic record is evidence that survives a dispute; a bare "Major, segment 14" with no span is a tally that loses one.
• Severity is assigned independently of dimension and against the content's risk tier: every dimension spans the full Minor-to-Critical range, and the same error is Minor on a coupon and Critical on a drug label. Where a client scoring profile pre-declares severity rules, those rules override evaluator judgment and following them is the defensible move.
• Weights are a declared policy, not a fact: the classical MQM ratio is 1 per Minor, 5 per Major, and a large penalty (commonly 25) per Critical, fixed in the profile and applied identically by every evaluator. The 5-to-1 Major-to-Minor ratio decides Critical-free files; the Critical weight rarely decides anything by arithmetic because the absolute Critical rule fails the file first.
• Normalization converts raw penalty points into an error score per thousand words: (penalty points ÷ scored word count) × 1,000, compared against the client's declared threshold. It makes files of different lengths comparable, but it sits entirely beneath the absolute Critical rule and never rescues a file that contains a Critical. The most common arithmetic failure is dividing by the total file size instead of the scored sample size.
• Repeatability across evaluators is built with three mechanisms: a written scoring guideline that fixes every non-judgment decision, periodic calibration that converges the team on the ambiguous middle and produces measurable inter-evaluator agreement, and loading as much as possible into the profile and tooling so judgment is spent only on the irreducible mislead-and-harm calls.
• The score output is the product, not the paperwork: it carries the headline verdict and Critical count, the sample definition, the full error list with spans and notes, the arithmetic, and the profile version, so any call can be reconstructed and defended. Beyond the pass/fail it yields a diagnosis (which dimension is failing), a trend (the per-thousand-word score over time), and the conformance record the revised standards expect.