Reward model QA is the missing layer that turns step-level preference data into trustable training signal. When sparse outcome signals fail at the reasoning-trace layer, as the recent LongTraceRL work has made unavoidable, step-level human evaluation is what fills the gap, and step-level human evaluation needs longitudinal evaluator consistency tracking to be defended as signal rather than noise.

The shape of the reward-modelling problem changed quietly over the last year, and the recent LongTraceRL work is the moment it became visible to anyone paying attention to the training-recipe debates. Sparse outcome signals (one preference judgement per completed trace) are sufficient at the chat-assistant scale and clearly insufficient at the reasoning-model scale. The traces are longer, the failures are more subtle, and a single end-of-trace thumbs-up does not propagate enough signal back through the steps for a reward model to learn what good reasoning looks like.

The downstream consequence almost nobody is naming yet. If sparse outcome signals fail, the field has to evaluate intermediate reasoning steps, not just final outputs. Step-level evaluation is a substantially different operation than outcome evaluation. The evaluator has to follow the trace, understand the local move at each step, and judge whether the step was good given what the model had to work with at that point. The judgement is finer-grained, the cognitive load on the evaluator is higher, and the noise floor on any individual rating is correspondingly worse.

Which means the reward model trained on step-level data is more sensitive to evaluator quality than the reward model trained on outcome-level data ever was. Sloppy step-level judgement does not just add noise; it actively miscalibrates the reward model along dimensions the team training the model may not even be measuring. The reward model becomes a faithful encoding of whatever the step-level evaluator cohort actually did, and the team has no way to tell the difference between strong signal and confidently-wrong signal.

This is the layer that does not currently exist in most pipelines. Reward-model QA. The question is not whether a reward model passes some held-out benchmark; it is whether the human judgement that trained it was reliable enough to defend.

What reward-model QA actually has to mean

Reward-model QA is the property that every step-level preference judgement in a reward-model training set traces back to a stable evaluator identity, a signed and timestamped contribution, a versioned methodology attestation, and a status trail that surfaces revocations or methodology supersessions. The standards stack is the same one Issue 02 Wednesday set out for longitudinal evaluation: W3C Decentralized Identifiers anchor the evaluator; W3C Verifiable Credentials carry the signed step-level judgements and the rubric attestation; W3C Bitstring Status Lists handle revocation when an evaluator credential or methodology version is superseded.

With that stack in place, the reward-model team can answer three questions any third party will eventually want answered. First, who actually made each step-level judgement, identified by a stable DID that survives a labelling-vendor switch. Second, what step-level methodology was attested to at the time the judgement was made, with the rubric version signed into each credential. Third, what fraction of the step-level training data came from evaluators whose calibration history the team has tracked over multiple batches, versus first-time entrants whose calibration is asserted rather than measured.

Without those three properties, a step-level preference dataset is a flat list of per-step ratings whose generative process is opaque, and the reward model trained on it is also opaque. The team can describe the methodology in detail; what the team cannot do is hand a downstream auditor a chain that lets the auditor independently verify what actually happened step by step.

Why step-level evaluation makes the cost calculation different

Outcome-level preference data has an implicit cost insurance: a single bad judgement is one rating amongst many, the noise is roughly i.i.d. across the dataset, and the reward model averages it out with enough volume. Step-level preference data does not have that insurance. A miscalibrated evaluator can lock in a systematic bias on a specific class of reasoning step (early-trace exploration moves, intermediate verification steps, late-trace decisive moves) that the volume does not wash out, because the bias is structured rather than random.

The reward model learns the structured bias. Distillation propagates it (the argument Issue 02 Tuesday made at the outcome-data layer applies here at the step-data layer, with more force). The downstream model behaves the way the contaminated upstream said it should. The team measuring final-output benchmarks may not even see the bias unless the benchmark is itself step-aware, which most current benchmarks are not.

Reward-model QA is the layer that closes this exposure. It is not a quality-control afterthought; it is the part of the pipeline that makes the rest of the pipeline defensible.

What audit-ready step-level evaluation looks like in practice

An audit-ready step-level preference dataset has four properties any third party can verify.

First, each step-level judgement carries the signed credential of the evaluator who made it. The credential resolves to a DID the evaluator controls, not a platform-internal account ID. When the evaluator works across multiple reasoning-trace projects or labelling vendors, the DID is the durable handle that lets a third party assemble a complete picture of what that evaluator has been involved in across the model’s training history.

Second, each judgement is bound to the step-level rubric version that applied at the time. Rubric updates are tracked as versioned issuer attestations, not silent edits to a methods doc. A downstream auditor can reconstruct the dataset against any historical rubric version and compare the resulting reward signals.

Third, evaluator-specific calibration history is observable. Per-evaluator inter-rater agreement on step-level hold-out items becomes a primary statistic, signed into the evaluator credential as a versioned attestation. The reward-model team can decompose the version delta between two reward models into model change, evaluator-cohort change, and per-evaluator calibration change. None of these are answerable today in the median pipeline.

Fourth, revocation is first-class. When a step-level rubric version is superseded or an evaluator credential is rescinded for cause, the W3C Bitstring Status List surfaces the change immediately to any downstream verifier. The team training the reward model can decide whether to retrain on the affected batches, reweight them, or accept them as legacy. The decision becomes informed; today, it is not even possible.

Where Ontology fits

Ontology has been deploying decentralised identity standards for years. ONT ID issues credentials that hold across systems and across the multi-batch, multi-vendor lifetime of a reward-model training programme. ONTO Wallet gives the evaluator direct custody of their step-level contribution record. The identity substrate that makes preference data, longitudinal evaluation, and sybil resistance auditable (the through-line of Issue 02) makes step-level reward-model QA auditable. Same primitives, different consumer.

Reward-model QA is not a product Ontology ships. It is a property that any reward-model training pipeline can achieve once the team decides to build on top of an identity substrate rather than a labelling-vendor account database. The standards work has been done. The deployment cost is engineering effort, not research. The question is which teams build the QA layer before the next reasoning-model release that the field cannot reproduce reveals what was missing from the training pipeline.

Continue reading this week

Tomorrow: Your benchmark is only as good as your evaluators, on why MLE-Bench skepticism and benchmark gaming are the same structural problem at the user-facing capability claim. Background reading: Issue 02 Tuesday on preference data integrity and Wednesday on longitudinal evaluation are the load-bearing prior pieces.