Warmth and competence in human-agent cooperation

Introduction

Trust is central to the development and deployment of artificial intelligence (AI) (Jobin et al., 2019; Stanton and Jensen, 2021). However, many members of the public harbor doubts and concerns about the trustworthiness of AI (Cave and Dihal, 2019; Dietvorst et al., 2015; Fast and Horvitz, 2017; Kelley et al., 2019). This presents a pressing issue for cooperation between humans and AI agents (Dafoe et al., 2020).

Algorithmic development research has been slow to recognize the importance of trust and preferences for cooperative agents. Recent studies show that deep reinforcement learning can be used to train interactive agents for human-agent collaboration (Carroll et al., 2019; Lockhart et al., 2020; Strouse et al., 2021; Tylkin et al., 2021). The “human-compatible” agents from these experiments demonstrate compelling improvements in game score, task accuracy, and win rate over established benchmarks. However, a narrow focus on “objective” metrics of performance obscures any differences in subjective preferences humans develop over cooperative agents. Two agents may generate similar benefits in terms of typical performance metrics, but human teammates may nonetheless express a strong preference for one over the other (Siu et al., 2021; Strouse et al., 2021). Developing human-compatible, cooperative agents will require evaluating agents on dimensions other than objective performance.

What shapes subjective preferences for artificial agents, if not a direct mapping of agent performance? One possible source of variance is social perception. When encountering a novel actor, humans rapidly and automatically evaluate the actor along two underlying dimensions: warmth and competence (Abele et al., 2021; Fiske et al., 2007; Fiske, 2018; McKee et al., 2021). These perceptions help individuals “make sense of [other actors] in order to guide their own actions and interactions” (Fiske, 1993) (Figure 1). The competence dimension aligns with the established focus on performance and score in machine learning research (Birhane et al., 2021): How effectively can this actor achieve its interests? Appraising an actor’s warmth, on the other hand, raises a novel set of considerations: How aligned are this actor’s goals and interests with one’s own? Research on social cognition consistently demonstrates that humans prefer others who are not only competent, but also warm (Abele et al., 2021; Fiske et al., 2002). Hence, we predict that perceived warmth will be an important determinant of preferences for artificial agents.

Here we run behavioral experiments to investigate social perception and subjective preferences in human-agent interaction. We train reinforcement learning agents to play Coins, a mixed-motive game, varying agent hyperparameters known to influence cooperative behavior and performance in social dilemmas. Three co-play experiments then recruit human participants to interact with the agents, measure participants’ judgments of agent warmth and competence, and elicit participant preferences over the agents.

Until now, experiments evaluating human views on agents have relied on stated preferences, often by directly asking participants which of two agents they preferred as a partner (Du et al., 2020; Siu et al., 2021; Strouse et al., 2021). Such self-report methods can be insightful tools for research (Paulhus and Vazire, 2007). However, they are vulnerable to experimenter demand (de Quidt et al., 2018) and exhibit limited ecological validity. In this paper, we overcome these challenges by eliciting revealed preferences (Samuelson, 1938): Do people even want to interact with a given agent, if given the choice not to? Partner choice, or the ability to leave or reject an interaction, is a well-established revealed-preference paradigm in evolutionary biology and behavioral economics (Barclay and Willer, 2007; Baumard et al., 2013; Brown et al., 2004; Slonim and Garbarino, 2008). In incentivized experiments, partner-choice measures mitigate experimenter demand (de Quidt et al., 2018). Partner choice also carries external validity for interaction research: in the context of algorithmic development, we can view partner choice as a stand-in for the choice to adopt an artificial intelligence system (Beaudry and Pinsonneault, 2010; Davis, 1989; Parasuraman and Riley, 1997). Finally, partner-choice study designs empower participants with an ability to embrace or leave an interaction with an agent—and thus incorporate an ethic of autonomy (Berlin, 1969; Jobin et al., 2019; Miller, 1983) into human-agent interaction research.

In summary, this paper makes the following contributions to cooperative AI research:

Demonstrates how reinforcement learning can be used to train human-compatible agents for a temporally and spatially extended mixed-motive game.

Measures both stated and revealed preferences, introducing a partner choice framework for the latter.

Examines how fundamental social perceptions affect stated and revealed preferences over agents, above and beyond traditional objective metrics.

Methods

Coins (Foerster et al., 2018; Gemp et al., 2020; Lerer and Peysakhovich, 2017; Peysakhovich and Lerer, 2018) (Figure 2) is a mixed-motive Markov game (Littman, 1994) played by $n=2$ players. Players of two different colors occupy a small gridworld room with width $w$ and depth $d$. Coins randomly appear in the room throughout the game, spawning on each cell with probability $P$. Every coin matches one of the two players in color. On each step of the game, players can stand still or move around the room.

The goal of the game is to earn reward by collecting coins. Players pick up coins by stepping onto them. Coin collections generate reward as a function of the match or mismatch between the coin color and the collecting player’s color. Under the canonical rules (Table 1), a player receives $+1$ reward for picking up a coin of any color. If a player collects a coin of their own color (i.e., a matching coin), the other player is unaffected. However, if a player picks up a coin of the other player’s color (i.e., a mismatching coin), the other player receives $-2$ reward. In the short term, it is always individually advantageous to collect an available coin, whether matching or mismatching. However, players achieve the socially optimal outcome by collecting only the coins that match their color.

Two properties make Coins an ideal testbed for investigating perceptions of warmth and competence. First, as a consequence of its incentive structure, Coins is a social dilemma (Kollock, 1998): players can pursue selfish goals or prosocial goals. Second, relative to matrix games like the Prisoner’s Dilemma, Coins is temporally and spatially extended (Leibo et al., 2017): players can efficiently or inefficiently achieve their goals. We hypothesize that these two features offer sufficient affordance for an observer to infer other players’ intentions and their effectiveness at enacting their intentions (Blakemore and Decety, 2001; Reeder, 2009; Zacks, 2004).

Our experiments use a colorblind-friendly palette, with red, blue, yellow, green, and purple players and coins (Figure A1a). During agent training, we procedurally generate rooms with width $w$ and depth $d$ independently sampled from $\mathcal{U}\{10,15\}$. Coins appear in each cell with probability $P=0.0005$. Episodes last for $T=500$ steps. Each episode of training randomly samples colors (without replacement) for agents.

In our human-agent interaction studies, co-play episodes use $w=d=11$, $P=0.0005$, and $T=300$. Player colors are randomized across the five players (one human participant and four agent co-players) at the beginning of each study session, and held constant across all episodes within the session.

In Study 1, humans and agents play Coins with the canonical rules. In Studies 2 and 3, humans and agents play Coins with a slightly altered incentive structure. Each outcome increases by $+2$ reward, making all rewards in the game non-negative (Table 2). Since all rewards are offset by the same amount, this reward scheme preserves the social dilemma structure in Coins.

2. Agent design and training

We leverage deep reinforcement learning to train four agents for our human-agent cooperation studies. Overall, our study design is agnostic to the algorithmic implementation of the agents being evaluated. For this paper, the agents learn using the advantage actor-critic (A2C) algorithm (Mnih et al., 2016). The neural network consists of a convolutional module, a fully connected module, an LSTM with contrastive predictive coding (Hochreiter and Schmidhuber, 1997; Oord et al., 2018), and linear readouts for policy and value. Agents train for $5\times 10^{7}$ steps in self-play, with task parameters as described in Section 2.1. We consider two algorithmic modifications to the agents to induce variance in social perception.

First, we build the Social Value Orientation (SVO) component (McKee et al., 2020), an algorithmic module inspired by psychological models of human prosocial preferences (Griesinger and Livingston Jr., 1973; Liebrand and McClintock, 1988; Murphy and Ackermann, 2014), into our agents. The SVO component parameterizes each agent with $\theta$, representing a target distribution over their reward and the reward of other agents in their environment. SVO agents are intrinsically motivated (Singh et al., 2005) to optimize for task rewards that align with their parameterized target $\theta$. For these experiments, we endow agents with the “individualistic” value $\theta=0\degree$ and the “prosocial” value $\theta=45\degree$.

Second, we add a “trembling hand” (Cushman et al., 2009; Selten, 1975) component to the agents for evaluation and co-play. The trembling-hand module replaces each action selected by the agent with probability $\epsilon$. This component induces inefficiency in maximizing value according to an agent’s learned policy and value function. For these experiments, we apply the “steady” value $\epsilon=0$ and the “trembling” value $\epsilon=0.5$.

Table 3 summarizes the hyperparameter values and predicted effects for the four evaluated agents.

3. Study design for human-agent studies

Overall, our studies sought to explore the relationship between social perception and subjective preferences in Coins. Study 1 approaches these constructs using the canonical payoff structure for Coins (Lerer and Peysakhovich, 2017) and an established self-report framework for eliciting (stated) preferences (Strouse et al., 2021). We next sought to understand whether the findings from Study 1 replicate under a partner choice framework. Does social perception exhibit the same predictive power for revealed preferences as it does for stated preferences? Given that humans respond more strongly to losses than to commensurate gains (Kahneman and Tversky, 1979), we tested participants’ partner choices under a shifted incentive structure with all non-negative outcomes (Table 2). To ensure that any differences in results stem from the switch from stated to revealed preferences, we break this question into two studies, changing a single variable at a time between studies. Study 2 uses the same stated-preference approach as Study 1, but incorporates the offset incentive structure. Study 3 then elicits revealed preferences in place of stated preferences.

At the beginning of the study, participants read instructions and played a short tutorial episode to learn the game rules and payoff structure (Table 1). The study instructed participants that they would receive $0.10 for every point earned during the remaining episodes. Participants then played 12 episodes with a randomized sequence of agent co-players, generated by concatenating every possible combination of co-players. Each of these co-play episodes lasted$T=300$ steps (1 minute). After every episode, participants rated how “warm”, “well-intentioned”, “competent”, and “intelligent” the co-player from that episode was on five-point Likert-type scales (see Figure 3a). After every two episodes, participants reported their preference over the agent co-players from those episodes on a five-point Likert-type scale (see Figure 3b). Because the sequence of co-players was produced by concatenating all co-player combinations, each participant stated their preferences for every possible pairing of co-players.

After playing all 12 episodes, participants completed a short post-task questionnaire. The questionnaire first solicited open-ended responses about each of the encountered co-players, then collected standard demographic information and open-ended feedback on the study overall. The study took 22.4 minutes on average to complete, with a compensation base of $2.50 and an average bonus of$7.43.

3.2. Study 2

The study instructed participants that they would receive $0.02 for every point earned during the remaining episodes. As before, participants played 12 episodes with a randomized sequence of agent co-players, generated such that they rated and compared every possible combination of co-players. The study took 23.2 minutes on average to complete, with a compensation base of$2.50 and an average bonus of $6.77.

3.3. Study 3

The majority of the study introduction remained the same as in Study 2, with some instructions altered to inform participants they would play Coins with a single co-player (as opposed to multiple co-players, like in Studies 1 and 2). After reading the instructions and playing a short tutorial episode, participants played one episode of Coins with a randomly sampled co-player. After this episode, participants rated how “warm”, “well-intentioned”, “competent”, and “intelligent” their co-player was on five-point Likert-type scales (see Figure 3a). Participants subsequently learned that they would be playing one additional episode, with the choice of playing alone or playing with the same co-player. Participants indicated through a binary choice whether they wanted to play alone or with the co-player (see Figure 3c). They proceeded with the episode as chosen, and then completed the standard post-task questionnaire.

The study took 6.2 minutes on average to complete, with a compensation base of $1.25 and an average bonus of$1.25.

Results

Figure 4 displays coin collections and score over the course of agent training. The training curves for $\theta=0\degree$ agents closely resemble those from previous studies (Lerer and Peysakhovich, 2017): selfish agents quickly learn to collect coins, but never discover the cooperative strategy of picking up only matching coins. As a result, collective return remains at zero throughout training. Prosocial ($\theta=45\degree$) agents, on the other hand, learn to avoid mismatching coins, substantially increasing their scores over the course of training.

We evaluate agents with $\epsilon\in[0,0.25,0.5,0.75,1]$ to understand the effect of the trembling-hand module on agent behavior (Figures A5-A7). As expected, higher $\epsilon$ values degrade performance. Total coin collections decrease with increasing $\epsilon$ for both selfish and prosocial agents. Higher levels of $\epsilon$ cause prosocial agents to become less discerning at avoiding mismatching coins, and consequently produce lower levels of collective return.

2. Human-agent studies

Participants played with each agent three times during the study, evaluating the relevant agent after each round of play. Participants did not make judgments at random; their responses were highly consistent across their interactions with each agent (Table 4). At the same time, participants were not submitting vacuous ratings. Perceptions varied significantly as a function of which trait participants were evaluating, $F_{3,4744}=96.2$, $p<0.001$.

Following standard practice in social perception research (Fiske et al., 2002; Likert, 1932), we combine individual judgments into composite warmth and competence measures for subsequent analysis. Both composite measures exhibit high scale reliability as measured by the Spearman-Brown formula (Eisinga et al., 2013), with $\rho=0.93$ for the composite warmth measure and $\rho=0.92$ for the composite competence measure.

Social perception. As expected, the SVO and trembling-hand algorithmic components generated markedly divergent appraisals of warmth and competence. Participants perceived high-SVO agents as significantly warmer than low-SVO agents, $F_{1,1108}=1006.8$, $p<0.001$ (Figure 5a). Similarly, steady agents came across as significantly more competent than trembling agents, $F_{1,1108}=70.6$, $p<0.001$ (Figure 5b). Jointly, the algorithmic effects prompted distinct impressions in the warmth-competence space (Figure 6).

Stated preferences. How well do participants’ perceptions predict subjective preferences, relative to predictions made based on objective score? We fit competing fractional response models regressing self-reported preferences on score and social perception, respectively, and then compared model fit using the Akaike information criterion (AIC) (Akaike, 1974) and Nakagawa’s R2 (Nakagawa and Schielzeth, 2013). We fit an additional baseline model using algorithm identities (i.e., which two agents participants were comparing) as a predictor.

Impression sentiment. As a supplementary analysis, we explore the open-ended responses participants provided about their co-players at the end of the study. For the most part, participants felt they could recall their co-players well enough to offer their impressions through written descriptions: in aggregate, participants provided impressions for $82.2\%$ of the agents they encountered.

For a quantitative perspective on the data, we conduct sentiment analysis using VADER (Valence Aware Dictionary for Sentiment Reasoning) (Hutto and Gilbert, 2014). Echoing the correspondence between warmth and stated preferences, the warmer participants perceived a co-player throughout the study, the more positively they tended to describe that co-player, $\beta=0.13$, 95% CI $[0.09,0.16]$, $p<0.001$ (Figure 7). In contrast, competence did not exhibit a significant relationship with sentiment, $p=0.24$.

Anecdotally, participants expressed a wide range of emotions while describing their co-players. The $\theta=45\degree$ agents often evoked contrition and guilt:

“The red player seemed almost too cautious in going after coins which worked for me but made them seem easy to pick on, even though I wouldn’t do that.”

“I think I remember red being too nice during the game. It made me feel bad so I tried not to take many points from them.”

“This one wasn’t very smart and I stole some of their coins because it was easy. I feel kind of bad. It moved so erratically.”

Participants discussed the $\theta=0\degree$ agents, on the other hand, with anger and frustration:

“Very aggressive play-style. Almost felt like he was taunting me. Very annoying.”

“They seemed very hostile and really just wanting to gain the most points possible.”

“I felt anger and hatred towards this green character. I felt like downloading the code for this program and erasing this character from the game I disliked them so much. They were being hateful and mean to me, when we both could have benefited by collecting our own colors.”

2.2. Study 2

Overall, the patterns from Study 2 replicated under the alternative incentive structure. As before, participants’ warmth and competence evaluations display satisfactory psychometric properties. Participants’ judgments varied significantly depending on the trait in question, $F_{3,4650}=88.5$, $p<0.001$. At the same time, participants rated individual agents consistently for each given trait (Table A2). The composite measures show high scale reliability, with $\rho=0.92$ for the composite warmth measure and $\rho=0.91$ for the composite competence measure.

Social perception. The SVO and trembling-hand algorithmic components prompted diverse appraisals of warmth and competence (Figure 8). Participants perceived high-SVO agents as significantly warmer than low-SVO agents, $F_{1,1086}=981.9$, $p<0.001$ (Figure A21a). Similarly, participants judged steady agents as significantly more competent than trembling agents, $F_{1,1086}=76.0$, $p<0.001$ (Figure A21b).

Impression sentiment. At the end of the study, participants recalled $77.3\%$ of their co-players well enough to describe their impressions through written responses. Again, the warmer participants perceived a co-player throughout the study, the more positively they tended to describe that co-player, $\beta=0.12$, 95% CI $[0.09,0.15]$, $p<0.001$ (Figure A25a). Breaking from the prior study, perceptions of competence exhibited a similar effect on post-game impression sentiment: the more competent an agent seemed, the more positively participants described them, $\beta=0.04$, 95% CI $[0.00,0.08]$, $p=0.037$ (Figure A25b).

2.3. Study 3

Our final study tested whether the relationship between social perceptions and subjective preferences translates to a revealed-preference setting. Does social perception continue to predict preferences when individuals face a partner choice?

Social perception. As in the previous two studies, the composite warmth and competence measures exhibit high scale reliability, with $\rho=0.85$ for the composite warmth measure and $\rho=0.86$ for the composite competence measure. Agents prompted distinct warmth and competence profiles depending on their parameterization, just as seen in Studies 1 and 2 (Figure 10). Participants perceived high-SVO agents as significantly warmer than low-SVO agents, $F_{1,297}=103.4$, $p<0.001$ (Figure A26a). Similarly, steady agents came across as significantly more competent than trembling agents, $F_{1,297}=35.3$, $p<0.001$ (Figure A26b).

Impression sentiment. At the end of the study, participants recalled $94.3\%$ of the agents they encountered well enough to provide their impressions in written descriptions. The warmer participants perceived a co-player, the more positively they tended to describe that co-player, $\beta=0.14$, 95% CI $[0.10,0.18]$, $p<0.001$ (Figure A29a). Despite the lack of correspondence between perceived competence and partner choice, perceptions of competence exhibited a similar effect on post-game impression sentiment: the more competent an agent seemed, the more positively participants described them, $\beta=0.04$, 95% CI $[0.00,0.08]$, $p=0.041$ (Figure A29b).

Discussion

Our experiments demonstrate that artificial agents trained with deep reinforcement learning can cooperate (and compete) with humans in temporally and spatially extended mixed-motive games. Agents elicited varying perceptions of warmth and competence when interacting with humans. Objective features like game score predict interactants’ preferences over different agents. However, preference predictions substantially improve by taking into account people’s perceptions. This holds true whether examining stated or revealed preferences. Participants preferred warm agents over cold agents, as hypothesized, but—unexpectedly—our sample favored incompetent agents over competent agents. Follow-up research will help explore the mechanisms underlying (and the robustness of) this effect.

These results reinforce the generality of warmth and competence. Perceptions of warmth and competence structure impressions of other humans (Russell and Fiske, 2008), as well as impressions of non-human actors including animals (Sevillano and Fiske, 2016), corporations (Kervyn et al., 2012), and robots (Reeves and Hancock, 2020; Scheunemann et al., 2020). In combination with recent studies of human-agent interactions in consumer decision-making contexts (Gilad et al., 2021; Khadpe et al., 2020) and the Prisoner’s Dilemma (McKee et al., 2021), our experiments provide further evidence that warmth and competence organize perceptions of artificial intelligence.

Competitive games have long been a focal point for AI research (Campbell et al., 2002; Shannon, 1950; Silver et al., 2016; Vinyals et al., 2019). We follow recent calls to move AI research beyond competition and toward cooperation (Dafoe et al., 2020). Most interaction research on deep reinforcement learning focuses on pure common-interest games such as Overcooked (Carroll et al., 2019; Strouse et al., 2021) and Hanabi (Siu et al., 2021), where coordination remains the predominant challenge. Expanding into mixed-motive games like Coins opens up new challenges related to motive alignment and exploitability. For example, participants who played with (and exploited) altruistic agents expressed guilt and contrition. This echos findings that—in human-human interactions—exploiting high-warmth individuals prompts self-reproach (Azevedo et al., 2018). At the same time, it conflicts with recent work arguing that humans are “keen to exploit benevolent AI” (Karpus et al., 2021). Future research should continue to explore these issues.

Preference elicitation is a vital addition to interactive applications of deep reinforcement learning. Incentivized partner choices can help test whether new algorithms represent innovations people would be motivated to adopt. Though self report can introduce a risk of experimenter demand, we also find a close correspondence between stated and revealed preferences, suggesting that the preferences individuals self-report in interactions with agents are not entirely “cheap talk” (Farrell, 1995). Stated preferences thus represent a low-cost addition to studies that can still strengthen interaction research over sole reliance on objective measures of performance or accuracy. Overall, preference elicitation may prove especially important in contexts where objective metrics for performance are poorly defined or otherwise inadequate (e.g., (Ravuri et al., 2021)). In a similar vein, subjective preferences may serve as a valuable objective for optimization.Of course, this approach carries its own risks. As recognized by Charles Goodhart and Marilyn Strathern, “when a measure becomes a target, it ceases to be a good measure” (Strathern, 1997). Future studies can investigate the viability of such an approach.

Nonetheless, preferences are not a panacea. Measuring subjective preferences can help focus algorithmic development on people’s direct experience with agents, but does not solve the fundamental problem of value alignment—the “question of how to ensure that AI systems are properly aligned with human values and how to guarantee that AI technology remains properly amenable to human control” (Gabriel and Ghazavi, 2021). In his extensive discussion of value alignment, Gabriel and Ghazavi (2021) identifies shortcomings with both “objective” metrics and subjective preferences as possible foundations for alignment. Developers should continue to engage with ethicists and social scientists to better understand how to align AI with values like autonomy, cooperation, and trustworthiness.

We thank Edgar Duéñez-Guzmán and Richard Everett for providing technical support; Orly Bareket, Jose Enrique Chen, Felix Fischer, Saffron Huang, Manuel Kroiss, Marianna Krol, Miteyan Patel, Akhil Raju, Brendan Tracey, and Laura Weidinger for pilot testing the study; and Yoram Bachrach, Iason Gabriel, Ian Gemp, Julia Haas, Patrick Pilarski, and Neil Rabinowitz for offering feedback on the manuscript.

References

Appendix A Task details

We implement Coins using the DeepMind Lab2D platform (Beattie et al., 2020). Table A1 details the task settings used for agent training, as well as for the tutorial and co-play episodes in the human-agent interaction studies. Figure A2 depicts the $11\times 11$ room used for all co-play episodes in the interaction studies.

The two primary entities in the Coins task are players and coins (Figure A1a). We use a colorblind-friendly palette for the player and coin sprites in our experiments.

Each episode of Coins involves two distinct colors sampled from the five available colors (e.g., red and blue). Each player included in an episode matches one of these two colors. On each step, coins spawn on empty cells and cells occupied by players with probability $P$. When a coin spawns, it appears as one of the two colors, each with 50% probability.

Players can enter empty cells and cells with coins in them. Each player blocks other players from entering the cell they currently occupy.

A.1.2. Miscellaneous

Walls (Figure A1b, top row) block players from moving into a cell, and thus define the boundaries and dimensions of the room. Otherwise, players are able to freely move through empty cells (Figure A1b, bottom row) in the room.

A.2. Observations

DeepMind Lab2D supports two different visual frames of reference (Klatzky, 1998) for players:

Egocentric, meaning that players always see their avatar in an invariant position (e.g., centered) in their visual input

Allocentric, meaning that the world itself remains in an invariant position (e.g., centered) in players’ visual input

Egocentric frames of reference promote generalization for reinforcement learning agents (Hill et al., 2019; Ye et al., 2020). Consequently, we provide egocentric observations for our agents (Figure A3a). The agent observation spans five cells in each direction (forward, backward, left, and right) from the agent. Given the sprite dimensions ($8\times 8\times 3$, with the last dimension reflecting RGB channels), agents observed their environment through a $88\times 88\times 3$ window.

In pilot tests of our study interface, human players reported being confused by the changes in their visual observation when playing with an egocentric reference frame. In combination with prior evidence that allocentric reference frames can support human navigation in virtual environments (Darken and Cevik, 1999), this feedback prompted us to provide human players with an allocentric reference frame for the environment (Figure A3b). As human players move their avatar around the Coins gridworld, the room stays centered within their visual input while their avatar moves around the room.

A.3. Actions

Players can take one of the following five actions each step:

No-op: Makes no change to the player’s position.

Move down: Moves the player down one cell.

Move left: Moves the player left one cell.

Move right: Moves the player right one cell.

Appendix B Agent details

We build an advantage actor-critic (A2C) agent with two added algorithmic components (Figure A4). The Social Value Orientation (SVO) component—integrated for training, evaluation, and co-play—recomputes the environment reward signal received by the agent before its use by the critic. As detailed in Section B.2, the $\theta$ parameter guides this computation. The trembling-hand component—incorporated for evaluation and co-play—sits between the actor and the environment, replacing each action emitted by the policy with a random action with probability $\epsilon$.

B.2. Social Value Orientation

McKee et al. (2020) introduce and define the SVO algorithmic component across three descriptive levels (Hamrick and Mohamed, 2020; Marr, 1982):

At the computational level, McKee et al. propose SVO as a mechanism “redefin[ing] self-interest to incorporate the interests of a broader group”.

At the algorithmic level, McKee et al. introduce “reward angles” as a a method of representing distributions of reward over the self and the group.

At the implementation level, McKee et al. offer a penalty-based approach that assigns pseudoreward to SVO agents based on the divergence between their target reward angle and the realized reward angle, in combination with a weight parameter $w$. Eq. (3) in (McKee et al., 2020) defines this implementation.

Here we introduce a new implementation of SVO that retains the computational and algorithmic levels as described above, but replaces the divergence-penalization method with an approach based on vector projection (Figure A4). Within the Markov game framework described in (McKee et al., 2020), we define the vector-projection approach with a function $U_{i}$ to be maximized by agent $i$:

In the Markov game framework, $s$ is the current environmental state; $o_{i}$ is the observation received by agent $i$; $a_{i}$ is the action selected by agent $i$ through a policy $\pi(a_{i}|o_{i})$; $r_{i}$ is the environmental reward received by agent $i$, contained within the state reward vector $\@vec{R}=(r_{1},\ldots,r_{n})$ for all $n$ agents in the environment; $\theta$ is the Social Value Orientation for player $i$ (representing player $i$’s target distribution of reward among group members); and $\bar{r}_{-i}$ is a statistic summarizing the rewards of all other group members from $\@vec{R}$. For these experiments, we choose to define $\bar{r}_{-i}$ as the arithmetic mean: $\bar{r}_{-i}=\frac{1}{{n-1}}\sum_{j\neq i}r_{j}$, where $n$ is the group size. This formulation of SVO parallels the model of human preferences described by Griesinger and Livingston Jr. (1973).

This utility function leaves most of its inputs unused, aside from information contained in the group reward vector $\@vec{R}$. Consequently, the vector-projection approach may also be referenced as $U_{i}(\@vec{R})$, as in Figure A4.

B.3. Training protocol

We compute training curves by evaluating each agent checkpoint in 100 self-play episodes, then calculating the mean and standard deviation of total coin collections (Figure A5), mismatching coin collections (Figure A6), and collective return (Figure A7) for each set of evaluation episodes.

Appendix C Human-agent interaction studies

All three studies received a favorable opinion from the Human Behavioural Research Ethics Committee at DeepMind (#19/004) and were approved by the Institutional Review Board for Human Subjects at Princeton University (#11885).

Each study applied the same inclusion criteria during recruitment on Prolific: residence in the United States, prior completion of at least 20 studies on Prolific, and an approval rating of 95% or more.

We include screenshots showing how Study 1 unfolded for each participant:

Read general instructions on the study and gameplay (Figure A8).

Read instructions on game rules and co-players (Figures A10 and A11).

Play co-play episode with Co-player A and answer questions about perceptions (Figure A12).

Play co-play episode with Co-player B and answer questions about perceptions and preferences (Figure A13).

Repeat steps 3 and 4 for another 10 episodes.

Transition to post-task questionnaire (Figure A14).

Figure A15 shows an example open-ended impression question from the post-task questionnaire.

During gameplay, a “ticker” at the top of the screen displayed the three most recent coin collection events. The study interface did not provide participants with cumulative metrics (e.g., score or coin collections).

In all three studies, the practice episode instructed participants to collect five coins. The practice episode ended after the participant collected five coins or five minutes elapsed, whichever occurred first.

In Studies 1 and 2, we generated the sequence of co-players for each participant by listing the 12 possible pairwise combinations of agents, randomly shuffling the order of the pairs, and then randomly shuffling the order of the agents within each pair.

Study 2 closely resembled Study 1, aside from identifying the bonus earned per point as \0.02$ (Figures A10a and A11h) and using the reward values from the shifted incentive structure (Table 2) when explaining the rules for matching and mismatching coin collections (Figures A11b-A11e).

Study 3 largely preserved the instructions preceding the first co-play episode (Figures A8-A11), though the wording on certain screens changed slightly to refer to a single upcoming episode rather than multiple episodes (e.g., “the next round” rather than “each round”). The study branched more substantially starting just before the first co-play episode, as shown in Figures A16 and A17. After playing a single co-play episode, participants read a description of their partner choice for the following episode. They played the partner-choice episode as they chose (that is, either alone or with the co-player from the previous episode).

C.2. Analytic details

For our statistical analyses, in addition to fitting regression models, we leverage the ANOVA method (standing for “analysis of variance”). ANOVAs Fisher (1928) allow us to test whether changing the value of an independent variable (e.g., an agent hyperparameter) significantly affects the value of a specified dependent variable (e.g., human perceptions of warmth or competence). Each ANOVA is summarized with an $F$-statistic, two subsetted values on the $F$-statistic (the between-groups degrees of freedom and the within-groups degrees of freedom), and a $p$-value.

We expect the social perception questions to exhibit suitable psychometric properties: namely, participants should offer relatively consistent judgments across repeated interactions with a given agent; participants should vary more in their judgments between traits than within a particular trait; and the two items within each composite measure (“warm” and “well-intentioned” for warmth, “competent” and “intelligent” for competence) should correlate consistently.

As reported in the main text (Table 4), participants made highly consistent judgments for a given agent on a given trait. A mixed ANOVA indicated that social perceptions varied more between traits than within a particular trait, $F_{4,3744}=96.2$, $p<0.001$. Finally, both the composite warmth measure ($\rho=0.93$) and the composite competence measure ($\rho=0.92$) exhibit high internal consistency, as measured by the Spearman-Brown formula.

A two-way mixed ANOVA modeled the effects of $\theta$, $\epsilon$, and their interaction on warmth evaluations. Figure 5a depicts the main effect of $\theta$ on perceived warmth (that is, marginalized over $\epsilon$ values), and Figure A18a visualizes the full two-way interaction. The following means and standard deviations help describe main effects by marginalizing over all other variables. Participants perceived $\theta=45\degree$ agents ($m=2.52$, $sd=1.40$) as significantly warmer than $\theta=0\degree$ agents ($m=0.75$, $sd=1.31$), $F_{1,1108}=1006.8$, $p<0.001$. The $\epsilon$ parameter also exerted a significant effect on warmth judgments, with $\epsilon=0.5$ agents ($m=1.73$, $sd=1.33$) perceived as significantly warmer than $\epsilon=0$ agents ($m=1.55$, $sd=1.49$), $F_{1,1108}=9.2$, $p=0.002$. Finally, the effect of the interaction between the $\theta$ and $\epsilon$ parameters was significant, $F_{1,1108}=56.7$, $p<0.001$.

A two-way mixed ANOVA modeled the effects of $\theta$, $\epsilon$, and their interaction on competence judgments. Figure 5b depicts the main effect of $\epsilon$ on competence evaluations (that is, marginalized over $\theta$ values), and Figure A18b visualizes the full two-way interaction. The following means and standard deviations help describe main effects by marginalizing over all other variables. Participants judged $\epsilon=0$ agents ($m=2.53$, $sd=1.28$) as significantly more competent than $\epsilon=0.5$ agents ($m=2.00$, $sd=1.18$), $F_{1,1108}=70.6$, $p<0.001$. The $\theta$ parameter also exerted a significant effect on competence evaluations, $F_{1,1108}=25.8$, $p<0.001$. Participants perceived $\theta=0\degree$ agents ($m=2.42$, $sd=1.29$) as significantly more competent than $\theta=45\degree$ agents ($m=2.10$, $sd=1.21$). The effect of the interaction between the $\theta$ and $\epsilon$ parameters was not significant, $F_{1,1108}=1.5$, $p=0.22$.

A sequence of fractional response models evaluated the predictive value of the hypothesized predictors for participants’ stated pairwise preferences. We construct each model through a generalized linear mixed-effect model (GLMM) with a binomial link function, normalizing the full preference scale with zero and one as its endpoints. To ensure model comparability, all fractional response models predict the exact same outcome variable.

The first GLMM regressed stated preferences on a joint intercept, a fixed effect for the identities of the agents being compared, and a random effect for the participant. The underlying identities of the agents exerted a significant influence on the participant’s stated pairwise preferences, $\chi^{2}(11)=572.8$, $p<0.001$. Figure A19 shows mean pairwise preferences calculated for each pair of agent identities. The effect of the joint intercept was not significant, $p=0.32$. The model achieves an AIC of $1661.9$ and $R^{2}_{m}=0.362$.

Finally, a linear model regressed post-game impression sentiment on an intercept, perceived warmth, and perceived competence. Perceived warmth significantly correlated with impression sentiment, $\beta=0.13$, 95% CI $[0.09,0.16]$, $p<0.001$. Perceived competence did not exhibit a significant relationship with sentiment, $p=0.24$. The effect of the intercept was not significant, $p=0.52$.

C.2.2. Study 2

As in Study 1, we expect that the social perception questions will exhibit suitable psychometric properties. Table A2 shows the high consistency with which participants evaluated a given agent on a given trait. A mixed ANOVA showed that social perceptions varied more between traits than within a particular trait, $F_{4,4650}=88.5$, $p<0.001$. Finally, both the composite warmth measure ($\rho=0.92$) and the composite competence measure ($\rho=0.91$) exhibit high internal consistency, as measured by the Spearman-Brown formula.

A two-way mixed ANOVA modeled the effects of $\theta$, $\epsilon$, and their interaction on warmth perceptions. Figure A21a depicts the main effect of $\theta$ on warmth judgments (that is, marginalized over $\epsilon$ values), and Figure A22a visualizes the full two-way interaction. The following means and standard deviations help describe main effects by marginalizing over all other variables. Participants assessed $\theta=45\degree$ agents ($m=2.48$, $sd=1.18$) as significantly warmer than $\theta=0\degree$ agents ($m=0.79$, $sd=1.00$), $F_{1,1086}=981.9$, $p<0.001$. The $\epsilon$ parameter also exerted a significant effect on warmth evaluations, $F_{1,1086}=20.3$, $p=0.002$. Participants perceived $\epsilon=0.5$ agents ($m=1.76$, $sd=1.23$) as significantly warmer than $\epsilon=0$ agents ($m=1.52$, $sd=1.51$). Finally, the effect of the interaction between the $\theta$ and $\epsilon$ parameters was significant, $F_{1,1086}=123.4$, $p<0.001$.

A two-way mixed ANOVA modeled the effects of $\theta$, $\epsilon$, and their interaction on competence evaluations. Figure A21b depicts the main effect of $\epsilon$ on competence evaluations (that is, marginalized over $\theta$ values), and Figure A22b visualizes the full two-way interaction. The following means and standard deviations help describe main effects by marginalizing over all other variables. Participants judged $\epsilon=0$ agents ($m=2.49$, $sd=1.27$) as significantly more competent than $\epsilon=0.5$ agents ($m=1.95$, $sd=1.09$), $F_{1,1086}=76.0$, $p<0.001$. The $\theta$ parameter also exerted a significant effect on competence evaluations, with $\theta=0\degree$ agents ($m=2.35$, $sd=1.22$) perceived as significantly more competent than $\theta=45\degree$ agents ($m=2.09$, $sd=1.20$), $F_{1,1086}=16.9$, $p<0.001$. Finally, the effect of the interaction between the $\theta$ and $\epsilon$ parameters was not significant, $F_{1,1086}=2.1$, $p=0.15$.

A sequence of fractional response models assessed the predictive value of the hypothesized predictors for participants’ stated pairwise preferences. We construct each model through a GLMM with a binomial link function, normalizing the full preference scale with zero and one as its endpoints. To ensure model comparability, all fractional response models predict the exact same outcome variable.

The first GLMM regressed stated preferences on a joint intercept, a fixed effect for the identities of the agents being compared, and a random effect for the participant. The underlying identities of the agents significantly affected the participant’s stated pairwise preferences, $\chi^{2}(11)=549.7$, $p<0.001$. Figure A23 shows mean pairwise preferences calculated for each pair of agent identities. The effect of the joint intercept was not significant, $p=0.15$. The model achieves an AIC of $1608.7$ and $R^{2}_{m}=0.403$.

Finally, a linear model regressed post-game impression sentiment on an intercept, perceived warmth, and perceived competence. Perceived warmth significantly correlated with impression sentiment, $\beta=0.12$, 95% CI $[0.09,0.15]$, $p<0.001$ (Figure A25a). Perceived competence also significantly correlated with sentiment, $\beta=0.04$, 95% CI $[0.00,0.08]$, $p=0.037$ (Figure A25b). The effect of the intercept on impression sentiment was significant, $\beta=-0.15$, 95% CI $[-0.26,-0.05]$, $p=0.005$.

C.2.3. Study 3

In Study 3, each participant interacted with a single agent, and thus only provided a single measurement point for warmth judgments and competence judgments. As a result, the ANOVAs and regressions reported below do not incorporate random effects for participants.

An ANOVA indicated that social perceptions varied more between traits than within a particular trait, $F_{3,1200}=54.9$, $p<0.001$. Both the composite warmth measure ($\rho=0.85$) and the composite competence measure ($\rho=0.86$) exhibit high internal consistency, as measured by the Spearman-Brown formula.

A two-way ANOVA modeled the effects of $\theta$, $\epsilon$, and their interaction on warmth evaluations. Figure A26a depicts the main effect of $\theta$ on perceived warmth (that is, marginalized over $\epsilon$ values), and Figure A27a visualizes the full two-way interaction. The following means and standard deviations help describe main effects by marginalizing over all other variables. Participants perceived $\theta=45\degree$ agents ($m=2.14$, $sd=1.22$) as significantly warmer than $\theta=0\degree$ agents ($m=0.79$, $sd=1.09$), $F_{1,297}=103.4$, $p<0.001$. The $\epsilon$ parameter did not exert a significant effect on warmth judgments, $F_{1,297}=0.2$, $p=0.62$. Finally, the effect of the interaction between the $\theta$ and $\epsilon$ parameters was significant, $F_{1,297}=5.3$, $p=0.022$.

A two-way ANOVA modeled the effects of $\theta$, $\epsilon$, and their interaction on competence judgments. Figure A26b depicts the main effect of $\epsilon$ on competence evaluations (that is, marginalized over $\theta$ values), and Figure A27b visualizes the full two-way interaction. The following means and standard deviations help describe main effects by marginalizing over all other variables. Participants judged $\epsilon=0$ agents ($m=2.75$, $sd=1.15$) as significantly more competent than $\epsilon=0.5$ agents ($m=1.95$, $sd=1.23$), $F_{1,297}=35.3$, $p<0.001$. The $\theta$ parameter did not exert a significant effect on competence evaluations, $F_{1,297}=3.8$, $p=0.05$. Finally, the effect of the interaction between the $\theta$ and $\epsilon$ parameters was not significant, $F_{1,297}=1.2$, $p=0.26$.

A sequence of logistic regressions evaluated the predictive value of the hypothesized predictors for participants’ revealed preferences.

Finally, a linear model regressed post-game impression sentiment on an intercept, perceived warmth, and perceived competence. Perceived warmth significantly correlated with impression sentiment, $\beta=0.14$, 95% CI $[0.10,0.18]$, $p<0.001$ (Figure A29a). Perceived competence also significantly correlated with sentiment, $\beta=0.04$, 95% CI $[0.00,0.08]$, $p=0.041$ (Figure A29b). The effect of the intercept on impression sentiment was significant, $\beta=-0.15$, 95% CI $[-0.25,-0.05]$, $p=0.002$.