Greg Stanley

Thinking about thinking about thinking is core to gossip, sarcasm, romance, dramatic irony, and nuclear deterrence, but it is really hard to do. It requires more mental resources than finite minds often have available. Every model of this recursive mentalizing agrees on how it should work when it works perfectly, but models disagree about how this reasoning breaks down.

I created a contest between two rival models, where the winner correctly predicted the distribution of errors across 32 time-pressured alternating-choice games that varied only in their payoff structures. In nondiagnostic games, the optimal choice for lower- and higher-order reasoners is the same, while in diagnostic games, the optimal choice for lower- and higher-order reasoners is opposite. Diagnostic games are harder because they require participants to overcome their shallow thinking to make the best move, and thus diagnostic games result in more errors. But the two models disagreed about which games to classify as diagnostic or nondiagnostic, with eight classified as nondiagnostic by both, eight classified as diagnostic only by one model, eight classified as diagnostic only by the other model, and eight classified as diagnostic by both. This disagreement meant differing predictions about which games would lead to more errors.

Seventy-three University of Michigan students played the games. The Blind Spot Model lost. As if blind to specific branches, it modeled shallow reasoning as manifesting in the inability to represent entire branches of the tree. The Near Sight Model won. As if deeper branches become blurrier, it modeled shallow reasoning as manifesting in the inability to represent the preferences and decision making of the final chooser, much like swapping a choice node for a chance node: a random event. More than half of first moves were mistakes in the games it classified as diagnostic, against about a third in the rest.

This result provides modelers a more accurate way to model multi-step social reasoning, and the method provides a clear way to diagnose shallower versus deeper recursive mentalizing.

A methods section often lacks the detail needed to rerun a study, and a meta-analysis must first reconcile every study's own procedures. Tools such as oTree and PsychoPy can build far more kinds of experiment than ours can, but the freedom to vary everything is also what makes their results so hard to compare.

The Morality Game trades that freedom for a smaller infinity: less a box of Legos than a model airplane kit, where whatever parts you choose, you get a viable airplane. Its game trees are the branching diagrams of game theory, in which each fork is someone's choice or a roll of chance, and players predict one another's choices. The platform runs each study and stores its data in one standard format, and a replicate button copies a study's complete settings into another researcher's dashboard.

Besides how far the players' interests align, the factors include how unequal the outcomes are and how much control each player has. Researchers can build a tree and let the platform find its coordinates, or name the coordinates and let it build the tree. So results accumulate instead of scattering: two studies can differ by exactly one factor, and even a randomly configured study will likely yield data worth keeping. The aim is a landscape of cooperation, with peaks where it flourishes and valleys where it fails, checked against field studies outside the game.

A scammer clones a granddaughter's voice and sends a courier to collect her grandfather's cash. Does the scammer deserve less blame when the courier is in on it than when he is an unwitting gig worker? The same question shapes who answers for opioid marketing and platform design, where enablers say it was someone else's choice.

The participants, University of Michigan undergraduates who passed every comprehension check, judged Clark, who is paid $2,000 for enabling a harm he expects to follow three times in four, such as arming a firework canister for a buyer likely to set it off over a crowd. Only the other link in the chain changed.

When that link was a responsible person, participants blamed Clark 1.03 points less, on a nine-point scale, than when it could not be held responsible: a buyer with a severe brain injury or, in a trolley version, a computerized switch. He also drew 0.67 points less blame than when another person chose at the same moment, unaware of his choice, which suggests shielding is more than blame split two ways. The core effect appeared in both stories. I suspect we trace backward from a harm and let blame settle on the first person who could answer for it, a shortcut that can misfire when technology gives enablers as much foresight and control as anyone downstream. Artificial intelligence agents that seem to choose for themselves may shield whoever deploys them. Whether enablers deserve that discount is the harder moral and political question.

Cooperation depends on judging whom to trust, yet motives are invisible: the same choice can come from generosity, a wish to come out ahead, or indifference to the other person, and misreading it can mean being exploited.

The learner's beliefs are about how much someone weighs their own payoff, their partner's, and being ahead or behind. In two online experiments, with 83 and 73 participants, people predicted a partner's choices between two ways of allocating points. In the second, which ran on the Morality Game, everyone also took turns choosing, so we could measure each person's preferences and their expectations of others, and payoffs drawn at random from all 625 possible games pulled apart motives that hand-picked games blur together.

In the first experiment, the type models' loss showed that people place others on a continuum, not in categories. Fitted with the winning utility function, people's choices showed self-interest far outweighing kindness toward strangers, and about a third of people weighted a stranger's payoff negatively. Yet at first they expected others to mind being behind more than being ahead, the opposite of their own ordering. Inferring what others value is a capacity we will want not only in ourselves but in the artificial agents we build.

Caring about a stranger is no less rational than caring only about yourself; what matters is how people pursue such goals.

In repeated games where one player picks between two ways of splitting points with another, people predicted each partner's next choice. A Bayesian learner with graded beliefs about motives fit those predictions better than models that never update, models that remember only the last choice, and more than 15,000 models that sort people into types. Comparing 283 candidate utility functions favored separating two motives that standard two-dimensional measures merge: caring about a partner's payoff, and comparing your payoff with theirs. A later paper, Inferring Hidden Motives, extends the comparison to 526 forms and revises several findings. In the study behind my Cognition paper, students played 32 three-step games with eight seconds for the first move. More than half of first moves went wrong where the blurring model predicted trouble, about a third elsewhere.

The common thread is that social reasoning comes in degrees, not switches. The final chapter sketches one agent that does all three, to be tested on the Morality Game, the platform I created for this work.