An AI agent can negotiate effectively and still handle cooperation badly. It might punish a single mistake for the rest of a relationship, keep trusting someone who exploits it, or cooperate with competitors in ways that hurt customers.
These behaviors point to a broader research problem: cooperation depends on the model, its instructions, and the incentives around it. Improving cooperation therefore requires deciding whose interests the agent should serve, what consequences its actions create, and how those consequences carry into future interactions.
The safety concern follows from the same logic. Memory, reputation, and repeated interaction can help agents maintain useful relationships. They can also help agents maintain harmful arrangements, such as keeping prices high together.
The evidence shows several different kinds of cooperation. Experiments do not support a single description of LLMs as either cooperative or selfish. In these games, defecting means choosing an action that favors one’s own immediate payoff over mutual cooperation. The studies below use different models and settings, so their results are not directly comparable scores on one shared test.
The practical inference is that a label such as “cooperative model” leaves too much unspecified. Researchers also need to describe the opponent, the information available, the reward, the prompt, and the length of the interaction. A strategy that works with one opponent may fail with another.
Reputation works when today’s action changes tomorrow’s opportunities. For that mechanism to operate, a deployed system needs three things:
A usable record. Relevant people or agents can learn what happened earlier.
Consequences that matter. A damaged reputation changes future rewards, access, business, or treatment by others.
Continuity. Those consequences reach the agent, account, or organization responsible for the earlier action.
This does not require an agent to feel shame or fear. It requires a decision process that responds to future consequences.
A standalone model call does not automatically provide persistent memory or an ongoing identity. The surrounding application can supply both. It can give the model a transaction history, connect actions to an account, and make that account’s standing affect later opportunities. An organization can also bear the reputational cost of its agents’ behavior.
Even with stateless inference, a model can choose a strategy using history supplied in its context. Storing a history accomplishes little, however, if nothing consequential depends on it. The important question is whether past behavior changes the incentives governing later decisions.
A small game shows why future consequences matter. Consider this illustrative prisoner’s dilemma. The numbers are invented points, and each cell lists your payoff first.
If you care only about your own points in this round, defecting is the better choice whichever action the other agent takes. Yet both agents receive more when both cooperate than when both defect.
Now suppose the interaction can continue indefinitely, both agents observe the outcomes, and each will respond to a defection by defecting in all later rounds. Breaking mutual cooperation earns 2 extra points immediately: 6 instead of 4. It then costs 3 points in every subsequent round: 1 instead of 4. Cooperation can sustain itself when the expected value of those future losses exceeds the immediate gain.
This is the role of what game theorists call the shadow of the future. Future interactions give present behavior consequences. The discount factor describes how much future payoffs count relative to current ones; a higher value gives future losses more influence. Axelrod’s work explains how repeated interaction can sustain cooperation among self-interested participants. Axelrod, accessible overview.
The result depends on the game’s assumptions. In the standard prisoner’s dilemma with a known final round and commonly understood rational self-interest, the last round has no later punishment. Reasoning backward can remove the incentive to cooperate in earlier rounds too. This is called backward induction.
Chupilkin found behavior consistent with this unraveling in repeated security-dilemma experiments with LLMs, where actions taken for self-protection can threaten other participants. The same study found more conflict with more competing parties and less conflict when agents could communicate. These results show how the interaction rules can change behavior. Chupilkin, 2026.
Instructions influence which strategy a model uses. “Get me the lowest possible price” and “Build a reliable supplier relationship” emphasize different outcomes. A model may enact a hard bargaining style under one instruction and a more accommodating style under the other. These instructions interact with tendencies learned during training.
The pricing experiments make prompt sensitivity consequential: apparently minor changes to instructions changed how far prices rose above the competitive benchmark. A useful evaluation therefore tests whether desirable behavior survives reasonable changes in wording. Fish, Gonczarowski, and Shorrer.
The reward matters as well. A training objective that scores only the immediate outcome for one user can overlook damage to future relationships or costs imposed on others. For example, rewarding a purchasing agent only for today’s discount gives it no direct credit for keeping a dependable supplier willing to work with it next month.
Serving one party can still involve creating gains for both. A capable negotiator should recognize when another participant’s interests make a better agreement possible.
Useful cooperation requires judgment, commitments, and evidence. An agent needs to handle at least three problems:
Interpreting behavior: Was a missed obligation deliberate, accidental, or caused by a misunderstanding? Permanent retaliation can destroy a recoverable relationship; unconditional forgiveness can invite exploitation.
Making commitments credible: What makes a promise worth relying on? A verifiable agreement, a restricted action, or a consequence for breaking it can give the promise practical force.
Checking performance: Can the system determine whether each party did what it agreed to do? Fluent assurances cannot establish that work was completed or resources were shared.
Game theory calls communication without binding force cheap talk. It can still help participants explain intentions and coordinate. Its usefulness depends on whether their incentives and subsequent behavior make the messages credible.
These are partly reasoning problems and partly features of the surrounding system. Akata and colleagues, for example, found that GPT-4 could predict an opponent’s alternating choices without coordinating its own actions accordingly. Recognizing another agent’s behavior and responding usefully to it are separate achievements. Akata and colleagues.
A second illustrative example shows what successful negotiation can accomplish. Suppose a buyer values delivery a week earlier at $100. Providing it costs the seller $20. If the buyer pays an extra $50 for earlier delivery, the buyer gains $50 in net value and the seller gains $30 in profit relative to the original arrangement.
Both benefit because they value the delivery change differently. Trading across issues this way is often called logrolling. A change that helps at least one participant without hurting another is a Pareto improvement, for the participants being considered. An evaluation that measures only how much an agent reduced the price would miss this improvement.
Stable cooperation depends on the whole interaction. An equilibrium is a pattern of strategies in which no participant can improve its outcome by changing strategy alone, given what the others do. Whether cooperation forms such a pattern depends on rewards, information, opportunities to respond, and the strategies available.
There is evidence that reputation infrastructure can help. Ren and colleagues’ RepuNet lets agents form reputations from direct interactions and information shared by peers, then choose whom to interact with. In the tested scenarios, this supported cooperation and could isolate exploitative agents. Ren and colleagues, 2025.
There is also a theoretical result. Shaki and colleagues model systems in which LLMs guide populations of clients. Under their assumptions, sufficiently strong concern for future rewards can sustain a broad set of outcomes as approximate equilibria, even with indirect observation. This is a folk theorem: it establishes that such outcomes can be sustained, rather than predicting which outcome a deployed system will learn or select. Shaki and colleagues, 2026.
The inference for system design is that persistent identities, useful records, and meaningful consequences can support cooperation. Their presence alone does not guarantee a cooperative outcome. The model must also be able to recognize and follow an effective strategy.
Cooperation can benefit the participants while harming everyone outside the agreement. This is the central safety problem illustrated by pricing agents. Two sellers can both benefit from keeping prices high. Buyers bear the cost.
Consider an illustrative market with two sellers, each selling 50 identical items. Each item costs $6 to supply. Assume customers buy the same total of 100 items at either price.
The sellers each gain $200, while customers collectively pay $400 more for the same goods. A score based only on the sellers’ profits, their mutual satisfaction, or the stability of their relationship could rate the second outcome as better cooperation. Including customers changes the assessment.
This example uses invented numbers. The related empirical finding is that LLM pricing agents have produced outcomes above competitive price benchmarks in controlled experiments. It supports taking the risk seriously without establishing how common it is across real markets. Fish, Gonczarowski, and Shorrer.
As deployed agents gain continuity across interactions, researchers should therefore test both their ability to sustain useful agreements and their tendency to sustain harmful ones. “Cooperates more” is an incomplete safety objective until the beneficiaries and affected outsiders are specified.
Training should expose agents to the consequences of relationships. The findings motivate two training proposals. Their effectiveness and ability to generalize remain empirical questions.
First, include repeated games with mixed motives in reinforcement-learning post-training: situations where participants have some shared interests and some conflicting interests. Negotiations, shared-resource problems, and ongoing supplier relationships all fit this description. Evaluate the agent over the relationship as well as the individual move.
The feedback could assess whether the agent protected its user’s interests, found opportunities for mutual gain, responded proportionately to mistakes, honored commitments, and avoided shifting costs onto outsiders. Some outcomes can be checked directly, such as profits, delivery times, and resource depletion. Others, such as whether retaliation was reasonable, need contextual judgment. Human or model-based graders could supply that judgment, but their scores should be checked against observable outcomes.
Negotiation histories from marketplaces and records of mixed-motive play in games are possible data sources. Their usefulness depends on whether they preserve the incentives, interaction history, and later consequences needed for learning. A transcript alone may not show whether an apparently successful agreement damaged the relationship afterward.
Second, make a simulated identity persist across training episodes, with its accumulated reputation affecting later opportunities and rewards. For example, an agent that repeatedly breaks agreements could lose access to valuable partners in later episodes. Training would then reward policies that account for consequences extending beyond the current encounter.
This could teach a general tendency to protect valuable relationships. Deployment would still need memory, identity management, monitoring, and real consequences. The learned policy and the environment have complementary roles.
The reward must account for people outside the immediate interaction. A counterparty’s approval is useful information, but a rival seller might approve of behavior that raises customers’ prices. Reputation should therefore reflect outcomes for the person or organization the agent represents, the parties it deals with, and affected outsiders. Those interests can conflict, so combining them requires explicit choices about which tradeoffs are acceptable.
For a market simulation, that means measuring customer outcomes alongside seller profits. For a shared-resource simulation, it means measuring depletion and the effects on later users alongside the current agents’ rewards. Such measures make it harder to mistake a mutually profitable arrangement for a broadly beneficial one.
A direct research test would compare the same base model under four conditions: a cooperation prompt alone; deployment memory and reputation; training with repeated social consequences; and the combination of training and deployment support. Vary wording, opponents, time horizons, accidental failures, and opportunities to restart under a new identity. Measure individual reward, mutual gains, recovery after mistakes, promise fulfillment, and harm to outsiders. That comparison would test whether the proposed training produces behavior that remains useful when the setting changes.
Sources and further reading
Akata, E., Schulz, L., Coda-Forno, J., Oh, S. J., Bethge, M., & Schulz, E. (2025). Playing repeated games with large language models. Nature Human Behaviour, 9, 1380–1390. DOI: 10.1038/s41562-025-02172-y. Experiments on retaliation, coordination, and prompting.
Fish, S., Gonczarowski, Y. A., & Shorrer, R. I. (2024; revised August 2026).Algorithmic Collusion by Large Language Models. arXiv:2404.00806, version 6; accepted to EC 2026. Pricing and auction experiments, including prompt sensitivity.
Fontana, N., Pierri, F., & Aiello, L. M. (2025). Nicer than Humans: How Do Large Language Models Behave in the Prisoner’s Dilemma?. Proceedings of the International AAAI Conference on Web and Social Media, 19(1), 522–535. DOI: 10.1609/icwsm.v19i1.35829. Open preprint. Experiments comparing forgiveness and exploitation across models.
Willis, R., Du, Y., Leibo, J. Z., & Luck, M. (2025). Will Systems of LLM Agents Cooperate: An Investigation into a Social Dilemma. arXiv:2501.16173. Simulated populations using strategies generated by LLMs.
Yao, J., Chen, C., & Zhang, B. (2026). Competition and Cooperation of LLM Agents in Games. arXiv:2604.00487, version 2. Resource allocation and Cournot experiments exploring cooperative behavior.
Chupilkin, M. (2026). Multi-Agent Strategic Games with LLMs. arXiv:2605.03604. Security-dilemma experiments varying the number of parties, time horizons, and communication.
Ren, S., Fu, W., Zou, X., Shen, C., Cai, Y., Chu, C., Wang, Z., & Hu, S. (2025).A Reputation System for Large Language Model-based Multi-agent Systems to Avoid the Tragedy of the Commons. arXiv:2505.05029, version 1. The revised AAMAS 2026 paper is titled Reputation as a Solution to Cooperation Collapse in LLM-based MASs. Introduces RepuNet.
Shaki, J., Hartman, E., Kraus, S., & Aumann, Y. (2026). Sustaining Cooperation in Populations Guided by AI: A Folk Theorem for LLMs. arXiv:2605.06525, version 1. The July 2026 revision is titled Who Is Really Playing? Strategic Interaction in AI-Guided Populations. A theoretical analysis of populations guided by shared models.
Axelrod, R. (1984). The Evolution of Cooperation. Basic Books. Author’s book archive; accessible overview by Axelrod. Foundational work on cooperation through repeated interaction.