Evaluator uniqueness, the property that one person can prove they are one unique evaluator without disclosing identity, is the W3C primitive that closes the chronic sybil-contamination problem in preference data and opens the next round of agent decision evaluation. The standards work has been done. The closing argument Issue 02 has been circling all week comes down to this: human judgement with verifiable uniqueness is the substrate everything else compounds against.<\/p>\n\n\n\n
Issue 02 opened with the METR teardown, used it to argue that benchmark publishers without traceable evaluator provenance are one critique away from a credibility event, and then traced the same structural problem through preference data integrity and longitudinal evaluation. All three pieces ended at the same place. The standards stack for stable, signed, portable evaluator identity is mature. The question is whether teams running the next round of model training, evaluation, and deployment build the human-side infrastructure with the same rigour as the model side.<\/p>\n\n\n\n
This closing piece folds together the two threads still open. The first is the chronic problem the field has known about and quietly absorbed for years: sybil contamination in preference data, where one person controls multiple evaluator accounts and biases the upstream toward whatever the contaminating actor wanted. The second is the emerging problem that does not yet have a defined name in the literature: agent decision evaluation, where the agent architectures shipping into production today have well-understood execution metrics and no standard framework for measuring whether the decisions those executions came from were any good. The two problems look unrelated. They are not. They are both solved by the same primitive.<\/p>\n\n\n\n
Sybil contamination is what happens when one person controls multiple evaluator accounts on a labelling pipeline, or when a coordinated group games a preference-data marketplace. It is the structural counterpart to ad fraud in display advertising: a chronic background cost most teams know exists, most teams quietly absorb, and most teams do not measure rigorously enough to defend a number on. Anecdotally the fraction of a typical paid-per-judgement preference dataset that is sybil-contaminated is non-trivial. Rigorous published baselines are scarce because the labelling pipelines that would publish them are also the ones whose business model depends on not having the answer.<\/p>\n\n\n\n
At the reward-model training layer the cost is structural rather than statistical. A sybil cluster does not just add noise. It biases preference data systematically toward whatever the contaminating actor wanted, which is sometimes payment optimisation, sometimes ideological, sometimes adversarial. The reward model treats the sybil-weighted preferences as signal. Tuesday’s piece on preference data integrity made the point that distillation propagates the resulting defects faithfully at lower latency. The piece left the actual fix to today.<\/p>\n\n\n\n
The fix is not better fraud detection at the application layer. Application-layer fraud detection is an arms race against actors who already have economic incentive to win it. The fix is a primitive: evaluator uniqueness as a verifiable, privacy-preserving property of any preference judgement, irrespective of which labelling vendor or eval platform mediated it.<\/p>\n\n\n\n
Selective disclosure, formalised in the W3C Verifiable Credentials Data Model 2.0<\/a> and implemented in IETF RFC 9901 (SD-JWT)<\/a>, lets a credentialed issuer attest that an evaluator is one unique person, certified by a trust framework whose criteria are publicly documented, without disclosing the evaluator’s name, location, demographic envelope, or any other attribute the methodology does not require. The reward-model team gets the uniqueness guarantee. The evaluator gets the privacy guarantee. Neither has to compromise.<\/p>\n\n\n\n This argument was made in detail in Issue 01’s Thursday piece on\u00a0selective disclosure<\/a>, which traced the standards work, the deployment ergonomics, and why selective disclosure is the privacy primitive AI evaluation has been quietly waiting for. The argument extends naturally to sybil resistance because the same mechanic that proves rubric eligibility without revealing identity proves uniqueness without revealing identity. The credential issuer is a trust framework rather than a labelling vendor. The verifier is the reward-model team rather than the platform. The holder is the evaluator. The selective-disclosure proof is the same shape.<\/p>\n\n\n\n With that primitive in place, the chronic sybil problem becomes structurally tractable. Every preference judgement carries a signed credential anchored to a W3C Decentralized Identifier<\/a> the evaluator controls. The reward-model team verifies the credential, confirms the uniqueness attestation, and does not have to know who the evaluator is. The trust framework attests, the holder holds, the verifier verifies. The labelling vendor is the issuer of the rubric attestation, not the owner of the evaluator identity. The economics of running sybil clusters collapse because the cluster operator can no longer mint new evaluator accounts faster than the trust framework can revoke them through a W3C Bitstring Status List<\/a>.<\/p>\n\n\n\n Agent architectures have matured fast over the last twelve months. The execution side is competitive: tool use, planning loops, multi-step reasoning, and the scaffolding around them are now a crowded technical landscape with well-understood benchmarks. The decision side, by which we mean the part where the agent decides what to attempt, in what order, and whether to stop and ask, is a comparatively quiet vacuum. There is no standard framework for evaluating whether an agent is making good decisions, as distinct from competently executing whatever decision came out of the planner.<\/p>\n\n\n\n This is going to crystallise into a defined problem this year. The market pressure is obvious: anyone deploying agents at scale is currently buying execution competence against an undefined decision-quality ceiling, and the credibility events that will follow look structurally identical to the METR teardown the week opened with. The next dispute will be a high-profile agent failure attributed to a planning decision the eval framework had no way to measure, and the field will discover, in the same uncomfortable way it discovered the benchmark-provenance problem, that the standard tooling does not surface the inputs.<\/p>\n\n\n\n When the agent decision evaluation problem does crystallise, the ground-truth layer it lands on is human judgement with verifiable uniqueness. Each decision-quality rating attaches to a uniquely identified evaluator. The eval cohort composition becomes auditable along the same dimensions Wednesday’s piece on\u00a0longitudinal evaluation<\/a>\u00a0set out. The publishing organisation can answer who rated each decision, under what version of the rubric, and what fraction of the cohort returned across batches. The teams that build agent decision benchmarks on top of audit-ready evaluator infrastructure first will be the ones whose benchmarks survive the inevitable credibility event. The teams that build them on opaque cohorts will inherit the credibility risk the METR situation distributed across the benchmark publishers who got there first.<\/p>\n\n\n\n Issue 01 closed with a piece on DIDs for agents that argued identity is actor-agnostic and that the W3C primitives extend naturally to AI agents themselves. That piece never published; the cadence slipped at the end of Issue 01. The argument it set up is still load-bearing and still correct. Agent identity is one half of the agent evaluation answer; evaluator uniqueness is the other. The same standards stack is the substrate for both.<\/p>\n\n\n\n Issue 02’s four shipped pieces all end at the same point. The standards stack for stable, signed, portable evaluator identity is mature. The standards stack for selective-disclosure uniqueness proofs is mature. The standards stack for revocation when a methodology, evaluator, or rubric version is superseded is mature. The teams whose evaluation pipelines were built before any of that was deployable now have a choice about whether to retrofit, replace, or absorb the credibility risk that comes with not doing either.<\/p>\n\n\n\n Issue 02 did not argue for a single technology or a single product. It argued that the next twelve months of AI evaluation will be a sustained credibility event for any publisher whose evaluator chain is opaque. METR was the warning shot. Preference data integrity is the chronic version inside training pipelines. Longitudinal evaluation is the version that surfaces over time as model cadences accelerate. Sybil resistance is the version that has been sitting unaddressed for years and is the easiest to fix the day teams decide to start. Agent decision evaluation is the version everyone will be talking about by the end of the year. The primitive that closes all five is human judgement with verifiable uniqueness.<\/p>\n\n\n\n Ontology has been deploying decentralised identity standards for years.\u00a0ONT ID<\/a>\u00a0issues credentials that hold across systems, including the selective-disclosure credentials that prove uniqueness without revealing identity.\u00a0ONTO Wallet<\/a>\u00a0gives the evaluator direct custody, including across the platform switches and labelling-vendor changes that snapshot eval pools struggle with. The Decentralized Identity Foundation<\/a>\u00a0has been stewarding the broader standards work for almost a decade. The stack is deployable today. Issue 02’s argument was not that Ontology ships a turnkey evaluator-platform product. It was that the identity substrate any evaluator platform, preference-data marketplace, eval research group, or agent-decision benchmark publisher needs already exists, and the question is which teams adopt it before the credibility events keep finding them.<\/p>\n\n\n\n Issue 02 ran Monday to Friday, four pieces shipped, and folded the original five-piece arc into four by combining the sybil and agent-decision threads into this closer. Issue 03 will open against whatever the week’s editorial brief surfaces. The candidate threads are already visible: agent decision evaluation will likely have crystallised into a named problem; preference-data marketplaces will likely face their own credibility event; the next round of frontier-model benchmarks will arrive with whatever provenance infrastructure their publishers chose to invest in. The standards stack does not change. The audience expanding awareness of it is the variable to watch.<\/p>\n","protected":false},"excerpt":{"rendered":" Evaluator uniqueness, the property that one person can prove they are one unique evaluator without disclosing identity, is the W3C primitive that closes the chronic sybil-contamination problem in preference data and opens the next round of agent decision evaluation. The standards work has been done. The closing argument Issue 02 has been circling all week<\/p>\n","protected":false},"author":5,"featured_media":913,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[170,113],"tags":[188,189,117,133,187],"class_list":["post-912","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-ai","category-data","tag-evaluator-uniqueness","tag-agent-evaluation","tag-decentralised-identity","tag-selective-disclosure","tag-sybil-resistance"],"_links":{"self":[{"href":"https:\/\/ont.io\/news\/wp-json\/wp\/v2\/posts\/912","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/ont.io\/news\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/ont.io\/news\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/ont.io\/news\/wp-json\/wp\/v2\/users\/5"}],"replies":[{"embeddable":true,"href":"https:\/\/ont.io\/news\/wp-json\/wp\/v2\/comments?post=912"}],"version-history":[{"count":1,"href":"https:\/\/ont.io\/news\/wp-json\/wp\/v2\/posts\/912\/revisions"}],"predecessor-version":[{"id":914,"href":"https:\/\/ont.io\/news\/wp-json\/wp\/v2\/posts\/912\/revisions\/914"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/ont.io\/news\/wp-json\/wp\/v2\/media\/913"}],"wp:attachment":[{"href":"https:\/\/ont.io\/news\/wp-json\/wp\/v2\/media?parent=912"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/ont.io\/news\/wp-json\/wp\/v2\/categories?post=912"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/ont.io\/news\/wp-json\/wp\/v2\/tags?post=912"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}The agent decision vacuum is where the same primitive lands next<\/h2>\n\n\n\n
Where the week’s pieces meet<\/h2>\n\n\n\n
Where Ontology fits<\/h2>\n\n\n\n
Where Issue 03 begins<\/h2>\n\n\n\n