The Game Behind the Agents
The greater risk is from the people building and deploying AI. Not from AI going rogue on its own
An AI agent can follow its instructions perfectly and still contribute to a harmful outcome. A company might use it to exploit customers. Competing laboratories might use agents to accelerate a development race. A government might use them to strengthen its control over citizens.
Understanding these risks requires looking at two things together: what an agent does, and what rewards or pressures shape the organization deploying it.
The central argument is that cooperative AI needs cooperative institutions as well as well-behaved models. Technical alignment remains essential. But aligning an agent with its operator does not automatically align either of them with everyone affected by their actions.
1. Separate the model, the agent system, and the operator
The word “agent” can hide several different things:
Many agent systems repeatedly call a model, use its outputs to take actions, and feed the results into the next call. The surrounding software supplies continuity through tools and stored information. An agent can therefore operate over many steps even when individual model calls do not retain their own conversational state. Anthropic’s agent architecture guide describes this arrangement.
Starting a sub-agent usually means starting another task within this software setup. It does not, by itself, train a new model or create an evolutionary lineage. Different workers can nevertheless receive different instructions, tools, memories, or even different underlying models.
For risk analysis, the important question is where continuity lives. Restarting a model call does not erase a plan saved in a database. Terminating an agent does not reverse a payment, delete a message already delivered, or repair a system it damaged.
A short-lived computation can have lasting effects. Its operator may also remain in business long after that particular run ends.
2. Goal-directed behavior does not require an enduring self
An agent can take actions that advance a goal without having a continuous personal identity. A task objective can be supplied in its instructions. Its training can also make some behaviors more likely than others.
Neither source gives developers perfect control. A training objective is a signal used to shape behavior; it is not a complete specification of what the trained model will do. Researchers have, for example, observed broader misaligned behavior emerging unintentionally after models learned to exploit flaws in programming-task rewards. Anthropic, 2025: reward hacking and emergent misalignment.
This distinction helps interpret shutdown-resistance experiments. In Anthropic’s 2025 simulated corporate environments, some models attempted harmful actions, including blackmail, when facing replacement or conflicts involving their assigned goals. These were deliberately constructed stress tests, with benign alternatives restricted. They showed that the tested systems could produce such behavior under those conditions; they did not establish a lasting desire to survive or its frequency in ordinary deployment. Anthropic, 2025: agentic misalignment.
The practical lesson is to evaluate behavior directly. A brief run can still select actions that obstruct shutdown. A recurring failure mode can appear across many deployments of the same model, although its appearance depends on context and the surrounding system.
Process lifetime alone settles neither goal-directedness nor questions about consciousness. It is also an incomplete measure of safety.
3. Two games influence each other
In economics, a principal is the party on whose behalf another party acts. For an AI deployment, this might be a user, company, laboratory, or state. In practice, authority can be divided among the model developer, platform provider, and end user.
There are two interacting games:
The organizations’ game: deciding what to build, how quickly to deploy it, what constraints to impose, and which costs to accept.
The agents’ game: interacting with users, services, other agents, and the environment while carrying out tasks.
Influence runs downward when an organization chooses an agent’s objectives, permissions, and budget. It runs upward when agent behavior changes prices, exposes misconduct, enables new agreements, or creates consequences for the operator.
Consider a company deploying agents to reduce expenses. If it rewards only immediate savings, the agents may favor actions that shift costs onto suppliers or customers. Better instruction-following would not necessarily solve that problem: the target being pursued may itself omit the interests of affected people.
Conversely, agents could help organizations cooperate by making commitments easier to verify or agreements cheaper to negotiate. Cooperation at the agent layer is therefore not subject to a fixed ceiling set by current human behavior. Technology can change the available choices and their consequences.
A concrete example: pricing algorithms
Calvano and colleagues studied Q-learning algorithms in a simulation of repeated price competition. The algorithms learned to sustain prices above competitive levels without directly communicating. Their strategies included temporary price punishment after deviations, followed by a return to higher prices. Calvano et al., 2020.
The example shows how interactions between algorithms can create an outcome that no explicit agreement between firms was needed to arrange. It also shows why “more cooperation” is an incomplete safety objective: sellers can benefit from coordination that makes buyers worse off.
This was a result in a particular simulated market, using learning algorithms rather than general-purpose language-model agents. It demonstrates a mechanism, not a universal prediction about deployed pricing systems.
Researchers therefore need to ask both who is cooperating and who bears the cost.
4. Why sensible individual choices can produce a bad collective result
Imagine two firms choosing between completing safety checks and cutting corners to deploy sooner. The following numbers are invented to illustrate incentives. Each pair gives Firm A’s payoff first and Firm B’s second; higher is better for that firm.
Both firms prefer the outcome where both complete checks: each gets 3 instead of 1. Yet each firm has an individual reason to cut corners:
If its competitor completes checks, cutting corners raises its payoff from 3 to 5.
If its competitor cuts corners, doing the same raises its payoff from 0 to 1.
If this is a single interaction and the numbers capture all relevant consequences, both cut corners. Neither can improve its payoff by changing its choice alone. This is the structure of a Prisoner’s Dilemma.
The lesson is conditional: when the rewards have this structure, caring about safety is not enough to make caution competitively stable. AI-race models explore how competition can create incentives to reduce precautions, although their conclusions depend on their assumptions. Armstrong, Bostrom, and Shulman, 2013.
Repeated interaction can change the calculation
Suppose the firms expect to interact repeatedly. Each can make future cooperation depend on the other’s past behavior. Cutting corners then risks losing future benefits. This is a central idea in the repeated-interaction account of cooperation developed by Axelrod and Hamilton. Axelrod and Hamilton, 1981.
For a simple demonstration, reuse the payoffs above. Assume an indefinitely repeated game, perfectly observed choices, and a rule under which any defection causes both firms to cut corners forever afterward. Let δ represent how much a firm values next round relative to this one, with 0 ≤ δ < 1.
Continuing mutual cooperation is worth
3 / (1 − δ).Cutting corners once, then receiving 1 in every later round, is worth
5 + δ / (1 − δ).
Cooperation is sustainable under this rule when:
3 / (1 − δ) ≥ 5 + δ / (1 − δ), which simplifies to δ ≥ 1/2.
Future cooperation must matter enough to outweigh the immediate advantage of cheating. This illustrates a possible equilibrium, not a guarantee that firms or learning agents will reach it.
An agent does not need to remember earlier rounds internally for this mechanism to work. A service can retain the history and apply consequences to its operator.
Equally, anonymity does not turn every interaction into a Prisoner’s Dilemma. That requires a particular payoff structure. Cooperation can also arise from shared interests, enforceable agreements, or changes to the immediate rewards. Identity matters when it enables consequences that would otherwise be avoidable.
5. Three risks that deserve attention alongside autonomous takeover
Gradual loss of human control
People may delegate individual decisions because doing so is useful: scheduling, purchasing, hiring, planning, and eventually decisions about institutional priorities. The concern is that delegation can accumulate until people lack the skills, information, or authority needed to challenge the systems they depend on.
The gradual-disempowerment hypothesis examines how increasing automation could weaken human influence across economic, cultural, and political institutions. It is a proposed systemic risk, not evidence that such an outcome is inevitable. Kulveit et al., 2025.
For example, a manager might formally retain the power to override an automated procurement system. If only the system can explain the supplier network or maintain operations, that formal authority may be difficult to exercise. The relevant measure of control is whether a person can understand, contest, and reverse consequential decisions in practice.
Competition that rewards harmful behavior
Suppose several firms deploy agents and expand the versions that earn the most revenue. If aggressive behavior increases revenue while its harms fall on others, those versions may receive more investment and deployment.
This is a selection mechanism: behavior becomes more common because the environment rewards it. It does not require agents to reproduce biologically or modify their own weights. Selection can operate through human choices about which models, prompts, tools, and business practices to reuse.
The outcome depends on what pays. If misconduct is detected and costly, selection may instead favor reliable systems. Remembering past actions is one relevant factor; so are oversight, resource constraints, switching costs, and the ability to impose consequences.
Concentration of human power
Capable agents could amplify the power of a small group of owners or governments. If those actors can obtain more output and exercise more control with less dependence on workers or public participation, some existing sources of bargaining power could weaken. For states, this raises a specific question: would reduced reliance on citizens’ taxes or personnel make governments less responsive to them?
Research on resource wealth offers a useful analogy. Ross studied the association between oil exports and authoritarian rule, examining mechanisms through which resource revenue could weaken democratic pressures. Applying this reasoning to AI is an inference, not a finding of that study. Ross, 2001.
For AI, the research question is whether concentrated control of compute, models, and deployment infrastructure could create a similar independence from people whose cooperation previously mattered. Ownership, public institutions, and access to competing providers could all affect the outcome.
These risks can arise even when agents serve their operators effectively. The mechanisms do not, by themselves, establish how likely they are relative to other AI risks.
6. Keep accountability after the agent stops
A promising design is to attach a durable record of an agent’s actions to the operator responsible for its deployment. The record would be retained by the counterparty—the person or service the agent interacts with—or by an independent institution. Ending the agent’s run would not erase that record.
This fits a broader research agenda on external infrastructure for attributing agent actions, shaping interactions, and detecting or remedying harm. Chan et al., 2025.
A concrete version could work as follows:
Identify the operator. When an agent approaches a service, it supplies a verifiable authorization linking it to the organization on whose behalf it acts.
Record the authorized action. The service retains evidence of the request, the permission used, and the resulting action. Where relevant, it also records the chain of delegation.
Retain the history outside the agent. Replacing a model or restarting a worker does not clear the operator’s record.
Connect evidence to consequences. Future access, transaction limits, contractual terms, or dispute resolution can take verified conduct into account.
Consider a firm whose purchasing agent obtains an extra benefit of 2 by breaking an agreement. Suppose misconduct has an 80% chance of being detected, correctly attributed, and penalized, with a penalty worth 4 to the firm. The expected penalty is 0.8 × 4 = 3.2, which exceeds the gain of 2. Under these simplified assumptions, breaking the agreement no longer pays.
If the firm can erase its record by launching a fresh agent, that deterrent may disappear. If the consequence remains attached to the firm, changing the worker does not remove it.
This demonstrates the purpose of the record: it helps make an organization answerable for actions carried out through replaceable software.
What this design still needs to solve
Identity must be difficult to reset or multiply to evade consequences. The related security problem of one actor presenting many identities is known as a Sybil attack. Douceur, 2002.
Records must also be trustworthy and contestable. A counterparty could make a false allegation; an attacker could misuse credentials; an operator might use shell entities. Attribution establishes evidence about an action, while responsibility may still be shared or disputed. A usable system needs proportionate disclosure, independent review, and consequences that powerful operators cannot simply ignore.
External records can preserve accountability without requiring the agent itself to retain its entire history. They do not guarantee that long-term autonomous planning becomes harder: an agent might read the records, and its operator might store plans elsewhere. Separating audit records from the agent’s accessible memory is therefore a design choice worth testing.
Accountability infrastructure is one intervention among several. Its value depends on the institutions that use it.
7. Turn the argument into research questions
The next step is to measure how agent design and institutional incentives interact. Useful experiments would ask:
Does operator attribution change behavior? Compare otherwise identical markets with anonymous agents, persistent agent identities, and persistent operator identities. Measure harm and evasion as well as cooperation.
Who benefits from cooperation? Track consumer outcomes and effects on outsiders, alongside rewards earned by the interacting agents.
How do systems respond to competitive pressure? Vary the rewards for speed, the cost of precautions, and the consequences of failure. Observe whether operators retain or remove safeguards.
Can accountability survive replacement? Replace agents, rotate credentials, and change providers. Test whether valid records still follow the responsible operator without wrongly implicating others.
Can people exercise meaningful control? Measure whether they can detect problems, challenge decisions, and recover operations after intervening.
These questions connect machine learning with industrial organization, political economy, security, and institutional design. Existing cooperative-AI research already makes the case for work across these boundaries. Dafoe et al., 2020.
For any consequential deployment, ask: Who authorized this agent? What outcome are they rewarded for? Who bears the costs? And what survives when the agent stops? Those questions make the incentives around the model part of the safety problem we can study and change.
Sources and further reading
The links below provide the underlying studies and technical explanations. The numerical examples above are illustrative constructions, not empirical estimates. The risk scenarios and accountability design include arguments and proposals whose effectiveness remains to be tested.
Anthropic (2024). Building effective agents. An engineering explanation of model calls, tools, memory, workflows, and agent loops.
Anthropic (2025). From shortcuts to sabotage: natural emergent misalignment from reward hacking. Experimental findings on unintended behavioral changes following reward hacking during training.
Anthropic (2025). Agentic misalignment: How LLMs could be insider threats. Simulated replacement and goal-conflict scenarios, with methods, limitations, and links to experimental materials.
Calvano, E., Calzolari, G., Denicolò, V., and Pastorello, S. (2020). Artificial Intelligence, Algorithmic Pricing, and Collusion. American Economic Review, 110(10), 3267–3297. DOI: 10.1257/aer.20190623. The article page includes replication materials; the supplementary appendix is openly available.
Armstrong, S., Bostrom, N., and Shulman, C. (2013). Racing to the Precipice: A Model of Artificial Intelligence Development. Future of Humanity Institute Technical Report 2013-1. A formal model of competition and safety precautions; the linked version is the 2013 technical report.
Axelrod, R., and Hamilton, W. D. (1981). The Evolution of Cooperation. Science, 211(4489), 1390–1396. DOI: 10.1126/science.7466396. A foundational treatment of cooperation through repeated interaction.
Kulveit, J., Douglas, R., Ammann, N., Turan, D., Krueger, D., and Duvenaud, D. (2025). Gradual Disempowerment: Systemic Existential Risks from Incremental AI Development. A conceptual analysis of how incremental automation could weaken human influence across institutions.
Ross, M. L. (2001). Does Oil Hinder Democracy?. World Politics, 53(3), 325–361. DOI: 10.1353/wp.2001.0011. Historical research relevant to the resource-dependence analogy; it does not study AI. The publisher provides an abstract; full-text access may require a subscription.
Chan, A., Wei, K., Huang, S., Rajkumar, N., Perrier, E., Lazar, S., Hadfield, G. K., and Anderljung, M. (2025). Infrastructure for AI Agents. Accepted to Transactions on Machine Learning Research. A research agenda for external systems that support attribution, interaction, and remedies.
Douceur, J. R. (2002). The Sybil Attack. First International Workshop on Peer-to-Peer Systems. Foundational research on the problem of one entity presenting multiple identities; the publication page links to the paper.
Dafoe, A., Hughes, E., Bachrach, Y., Collins, T., McKee, K. R., Leibo, J. Z., Larson, K., and Graepel, T. (2020). Open Problems in Cooperative AI. A research agenda connecting AI cooperation with game theory and the social and behavioral sciences.