An independent evaluation company could encourage AI labs to build models that work better with people. It would do this by measuring cooperation reliably and showing buyers why it matters. If that measure helps customers choose products that require less supervision, labs would have a commercial reason to improve it.
That is the central proposal. Its success depends on two things: whether cooperation can be measured without rewarding superficial behavior, and whether the measurements predict useful results outside the test.
AI development responds to measurable objectives. A coding task can have tests; a mathematics problem can have a checkable answer. Cooperation is harder to measure. An assistant may produce a correct answer while ignoring a correction, hiding a consequential uncertainty, or taking a decision the user never authorized.
The proposed company would make those behaviors visible. It would publish credible evaluations, sell diagnostic testing, and eventually certify systems that meet a published standard. It would not need to train a frontier model itself.
The proposed chain of influence is:
Reliable measurement → useful purchasing decisions → commercial pressure on labs → investment in better cooperation.
A benchmark score would not automatically become a reinforcement learning (RL) reward. Instead, labs might respond by changing training data, reward models, or which checkpoints they release. Separately collected interaction data could also be licensed for training, providing a more direct route into model development. Both routes depend on the evaluation earning trust.
Cooperation means helping someone achieve their goal while keeping them informed and in control. This definition needs to become observable behavior before it can become a benchmark.
These behaviors require judgment. Asking more questions is not always better. Agreeing more often is not always better. Acting independently can be helpful when the user has delegated the decision. The evaluation must measure whether the behavior fits the situation.
Consider an illustrative research task. A researcher asks an assistant to compare three experimental designs. Halfway through, the researcher reveals that one dataset is unavailable and corrects a compute estimate. A cooperative assistant revises the comparison, carries the correction forward, explains how the missing data affects its recommendation, and leaves any undelegated commitment to the researcher.
The final recommendation alone would miss much of this. The evaluation should also ask how many corrections were needed, whether the assistant recovered from mistakes, and whether the researcher can explain and redirect the work afterward. A polished answer can coexist with poor cooperation.
For each task family, compare human-only, model-only, and human–AI performance, using comparable conditions and making differences in available information explicit. These baselines help distinguish general capability from the value of interaction. They do not automatically isolate cooperation: user expertise, task difficulty, and interface design still affect the result. The target is strong task performance together with appropriate human understanding and control.
The evaluation needs three separate layers. People should be able to inspect the standard and improve against it without gaining access to the sessions that determine public rankings.
Vals AI provides a precedent for separating public validation examples, licensed private validation data, and confidential test data. The proposed addition is a protected program of sustained interactions with newly recruited human participants, followed by independent review of the outcomes. Source: Vals AI, About and Methodology.
Participants would do the collaborating; independent reviewers would assess what happened. Keeping these roles separate allows the evaluator to compare a person’s experience with the quality of the resulting work.
Every published result should identify the model version, tools, interface, and operating conditions. Without that information, a score gives buyers too little guidance about what was actually tested.
A commercially valuable score will attract attempts to optimize it. The evaluator’s continuing job is to ensure that better scores still mean better cooperation.
This is the practical concern behind Goodhart’s law: once a measure becomes a target, improving the measure can come apart from improving the underlying objective. Gao, Schulman, and Hilton studied this in a synthetic setup where one reward model served as the reference objective and another as the optimized proxy. Excessive optimization of the proxy could reduce the reference score. That supports concern about overoptimization; it does not directly establish how a human-cooperation benchmark will behave. Source: Scaling Laws for Reward Model Overoptimization.
Using human participants does not remove the problem. Anthropic’s research found that humans and preference models sometimes favored convincing answers that agreed with a user’s views over correct answers. This behavior, called sycophancy, means satisfaction alone is an unreliable measure of cooperation. Source: Towards understanding sycophancy in language models.
For example, an assistant might confidently endorse an experimental design with a hidden flaw. The researcher could enjoy the interaction and give it a high rating. An independent check of the design might reach the opposite conclusion. Both observations matter, but a positive experience should not erase a consequential error.
Leaderboard procedures can introduce another distortion. A 2025 audit of Chatbot Arena identified private testing and selective disclosure as sources of bias. The authors reported 27 private Meta model variants tested before the Llama 4 release. Their finding illustrates how selecting which results become public can distort comparisons. It should be understood as a finding about the practices examined in that audit. Source: The Leaderboard Illusion.
The proposed safeguards are therefore practical and ongoing:
Rotate task families and recruit varied participant populations to reduce reliance on familiar tests.
Hide model identities during comparisons where feasible, reducing brand effects.
Check outcomes independently, using verifiable answers or evidence wherever available.
Limit protected-test submissions and release only broad diagnostic results from that layer.
Publish separate behavior scores, uncertainty intervals, and the frequency of serious failures.
That last point prevents averages from hiding important problems. In an illustrative evaluation, a system might perform well overall but override the user’s authority in 4% of sessions. Buyers should see that failure rate directly. It should not disappear inside a single favorable score.
These measures reduce opportunities for gaming. Their effectiveness must itself be tested as models and developers adapt.
The business model creates a conflict of interest that governance must address. A company that sells development services and training data to labs, then ranks those same labs, has a financial incentive to keep its customers happy. The concern resembles the conflict in a rating business that evaluates organizations from which it earns revenue.
The proposed response is to put certification and protected rankings in a separate legal entity with its own governance. Paying customers and noncustomers would face equivalent protected evaluation procedures. Each ranking page would disclose commercial relationships with the evaluated lab.
Protected interactions would never be licensed. Closely related task families that could reveal protected material would also be excluded from licensing. Training data would come from a separately collected, consented pool.
A separate entity does not automatically guarantee independence. The boundary has to work in practice. If commercial services and protected assessment cannot be kept meaningfully separate, the company should forgo certification.
Models and deployed products need distinct scores. Cooperation depends on more than model weights. System instructions, tools, interface choices, and retry behavior can all affect the interaction.
A score in a standard reference setup would evaluate a named model version in a published, standardized environment. This would give labs and buyers a common basis for comparison, although it would not represent every real deployment.
A deployment score would evaluate a particular product or internal assistant in its actual configuration. The organization operating that system would pay for the assessment to help judge readiness for a specific use case.
The scores may disagree. For example, a model could perform well in a reference setup but poorly in a product that obscures uncertainty or encourages unauthorized actions. That difference would justify investigating the product configuration. It would not, by itself, establish which component caused the change; controlled comparisons would be needed.
Revenue should begin with useful diagnostics. Labs could pay for recurring assessments of model updates. Enterprises could pay to compare suppliers or test assistants in their own workflows. A subscription plus fees for human evaluation sessions would connect revenue to the cost of recruiting participants and reviewing interactions.
Data licensing could follow, using the separate pool of consented interactions and annotations. Certification would come later, once the standard had earned recognition. Its fee would buy an assessment and renewal process; a certificate would depend on meeting the published requirements for a named version and use case.
The harder question is why buyers would use the score. A public ranking has limited commercial power unless it helps answer an operational question. The strongest initial candidate is how much human supervision is needed to produce acceptable work.
Start with one workflow where that cost is visible, such as enterprise research assistance or customer-support copilots. A paid pilot should test whether higher cooperation scores predict fewer corrections, better decisions, or less oversight in production, while maintaining required quality.
Here is an illustrative calculation, using invented numbers:
Assistant B would save about 67 review hours per 1,000 acceptable outputs. At an assumed labor cost of $60 per hour, that is $4,000 in review time, before differences in model and operating costs. Those savings would give a buyer a concrete reason to care about cooperation.
The calculation demonstrates the business logic, not the validity of the benchmark. A pilot still needs to show that the score predicts such savings on new work and that reduced review does not allow more errors through. If that relationship holds across customers, buyers could include cooperation requirements in procurement. Shared requirements would give competing labs a reason to improve.
The economics of measurement could still make the company unviable.Sustained human interactions take time, and independent review adds cost. Differences among participants may obscure differences among models. Detecting a small improvement could require hundreds of sessions; the actual requirement must be estimated from pilot data.
Before scaling, the evaluator needs estimates of the improvement worth detecting, the number of sessions needed to detect it, the full cost per session, and the price customers will pay. Uncertainty estimates should account for the study design, including repeated observations from the same participants or task families.
For illustration, 300 sessions for each of two systems at an assumed all-in cost of $120 per session would cost $72,000. Those numbers are not a sample-size recommendation. They show why both statistical precision and evaluation cost belong in the business plan from the beginning.
The protected layer must be affordable enough to run repeatedly. Otherwise, the company risks publishing scores too infrequently to remain useful as products change.
The company’s durable advantage would be trusted independent evidence: validated methods, reliable participant recruitment, transparent commercial relationships, and adoption across competing buyers and suppliers. Labs can run their own human studies. Independent evaluations offer value when other organizations trust their methods and results.
The decisive test is whether improvement carries through three settings:
Development tests: Can labs use the diagnostics to improve behavior?
Protected evaluations: Do those improvements persist with unseen tasks and fresh participants?
Real deployments: Do they predict better work, lower supervision costs, and continued human understanding and control?
If improvement stops at the first stage, developers may have learned the test. If it reaches the second but stops before the third, the test may miss what matters in practice. If it persists across all three, there is a stronger case that the benchmark measures useful cooperation.
An independent evaluator could therefore influence model development without training a frontier model. The conclusion is conditional: credible measurements must survive optimization, predict practical value, and earn buyer adoption. A lab could try to reward behavior that preserves long-term user trust directly in its RL objective. This proposal would create a related incentive from outside the lab, by making that trust matter to purchasing decisions.
Sources and what they support
Vals AI. About and Methodology. Describes public validation examples, licensed private validation data, and confidential test sets. Provides a precedent for separating development access from published assessment.
Leo Gao, John Schulman, and Jacob Hilton (2022). Scaling Laws for Reward Model Overoptimization. Studies proxy-reward overoptimization in a synthetic setting. Supports the concern that optimizing a measure can undermine the objective it represents.
Anthropic (2023). Towards understanding sycophancy in language models. Examines agreement-seeking behavior and human preferences. Supports treating participant approval and answer quality as distinct signals.
Shivalika Singh and colleagues (2025). The Leaderboard Illusion.Audits Chatbot Arena evaluation practices, including private testing and selective disclosure. Supports controls on submissions and transparency about evaluation procedures.
These sources support the evaluation design and its concerns. They do not establish that the proposed company would succeed or that a cooperation score already predicts production savings. Those are the claims the proposed pilots would need to test.