This study was approved by the Ethics Committee of the Max Planck Institute for Human Development (protocol A 2022-24). All participants provided informed consent prior to participation.
Participant Recruitment, Exclusion and Compensation
Participants (n = 1155) were recruited through the online platform Prolific55, while screening for individuals residing in the United States and the United Kingdom, being proficient in English, and maintaining an approval rating of at least 95%. Upon successful completion of the study, participants received a fixed payment of €2.25, complemented by performance-dependent bonus compensation payments averaging €0.70. The median time for completing the study was 13:45 minutes. The average hourly compensation rate was €9.82.
Participants ranged in age from 18 to 92 years (M = 39.67, SD = 12.28). Of those with available demographic data (n = 1272), 56.4% identified as male, 43.2% as female, and 0.3% preferred not to say. The majority resided in the United Kingdom (72.9%), versus the United States (27.1%). Sex and gender were not analysed as variables in this study as they were not relevant to the research question.
As preregistered, participants were excluded based on two criteria: (1) Participants who took more than 25 minutes, deemed unreasonably long; (2) Participants with missing moves in any of their demonstration trials, as subsequent generations could not replicate incomplete solutions. These criteria also implicitly served as an attention check. Of the 1303 participants who completed the experiment, 148 were excluded under these conditions. Their spots were then filled by newly recruited participants. No other exclusions were made. The investigators were not blinded to allocation during outcome assessment, as the analysis followed a preregistered plan.
Participants who passed the exclusion criteria were eligible for bonus compensation. Participants’ compensation depended in part on their own performance in the Demonstration Phase, and in part on the performance of people that learned from them. To ensure a fair chance of gaining a bonus payment independently of the condition a participant was sampled in, we ranked these scores separately per generation and condition. The final bonus was then computed as the percentile score rank multiplied by €1.40.
Preregistration
The study was preregistered on asPredicted.org on 15 February 2024 (https://aspredicted.org/d765-77fm.pdf), including a detailed description of predictions, measures, conditions, analyses, exclusion criteria and sample size. No deviations from the preregistered analysis plan were made. In particular, we preregistered the following predictions:
1.
Main predictions
a.
General benefit of AI. Overall, participants from generation 1 onwards in AI trees will outperform participants in human trees.
b.
Long-lasting benefit of AI. The subset of participants in the final generation of AI trees will outperform the final generation of the human trees.
2.
Secondary predictions
a.
Transmission from AI to humans. The subset of participants in generation 1 of the AI trees will outperform generation 1 in the human trees.
b.
Explicit recognition of counterintuitive strategy. Overall, participants from generation 1 onwards in the AI trees will describe the counterintuitive strategy more frequently in their written strategies than those in the human trees.
Network task
We adapted the Reward Network task35,36 consisting of a network g with 12 nodes and 30 edges, where each edge is associated with a reward r ∈ [ − 50, 0, 100, 200, 400] (see Fig. 2 panel c). Hidden to participants, each node was associated with a level l ∈ [0, 1, 2, 3]. Constraints in the network generation ensured that any path connecting a node of level 0 to a node of level 3 included at least three edges with negative rewards. The largest reward of 400 points was accessible only from level 3. These two constraints required participants to accept at least three losses to access the highest rewards. Transitions between nodes of level 0 contained the second-highest reward of 200 points, making myopic behaviour a reasonable but generally suboptimal strategy.
Participants began the game at a predetermined starting node (e.g., node B in figure 2 panel c). Initially, only the outgoing edges from this node were revealed. As participants traversed the network, they sequentially uncovered it, and the outgoing edges of each visited node were displayed. Each participant would be playing on, in total, 10 different networks drawn randomly from the pool of experimental networks. The assignment of networks to individual participant trials ensured that no participant experienced the same network twice.
We generated three sets of networks (training, validation, and experiment) with 1000 networks each. The first two sets were used only during the training of the neural policy, while the last set was used exclusively during the experiment. After generation, we computed scores for a myopic rule-based strategy and a loss-seeking rule-based strategy. The myopic strategy always selects the move with the immediate highest reward, while the loss-seeking strategy prioritises edges with a − 50 reward over all others. We excluded networks from all three sets where the myopic strategy outperformed the loss-seeking strategy, which accounted for approximately 9% of all generated networks.
Populations and Social Learning Opportunities
The experiment consisted of 30 populations, each with 5 generations and 8 players per generation (see Fig. 2a). Half of the populations were assigned to the “Human-Machine” condition, and the other half to the “Human-Only” condition. In the “Human-Machine” condition, machines took three player positions in the first generation. Starting from generation 1, each participant could choose one of five possible demonstrators drawn from the eight positions in the previous generation. Participants were dynamically assigned to available positions in the populations, prioritising the completion of existing populations before adding participants to new ones. Positions for later generations became available only after all potential demonstrators had completed the experiment.
Experimental flow
Panel B in Figure 2 depicts the main phases of the experiment. A static version mimicking the experience for participants in later generations is available here: https://center-for-humans-and-machines.github.io/reward-network-iii.
Below, we summarise the main phases of our experiment:
1.
Introduction Phase: Participants learned about the study’s purpose, procedures, data handling, estimated time, and financial compensation. They then completed a tutorial guiding them through the task and interface.
2.
Individual Learning Phase: Participants solved the task individually. Generation 0 worked through 6 networks, while later generations worked through only 2 networks (given they encountered four more in the Social Learning Phase, see below) to familiarise themselves with the task. See also Fig. S3.
3.
Social Learning Phase (only from generation 1: Participants selected a demonstrator (see Supplementary Fig. S4). They could base their decision on demonstrators’ average scores from the Demonstration Phase (they were also informed about their own average score as a reference point). They then went through four networks by observing a replay of the demonstrator’s solution (see Supplementary Fig. S5), repeating that solution (see Supplementary Fig. S6), and trying to solve the same network on their own. Replay delays between moves were equalised to eliminate timing differences between human and machine-generated demonstrations. Last, they received feedback comparing their solution with the demonstration.
4.
Demonstration Phase: Participants independently solved four networks. The solutions entered were used for the Social Learning Phase of the next generation for those selected as a demonstrator.
Participants also documented their personal strategy in written form, before and after the Social Learning Phase. These strategies were collected solely for later analysis and were not presented to subsequent generations.
Neural policy and its training
We used a deep Q-learning-based reinforcement learning approach to train a neural policy for navigating and solving reward network tasks, as outlined by ref. 56. The neural policy featured a gated recurrent unit (GRU)57, flanked by two linear layers with ReLU activation, each containing 15 hidden units. The input Q(ot) consisted of a one-hot-encoded reward of each target node in a vector of shape Nodes × Rewards, while the output represented the Q-value of each target node, with unreachable nodes set to negative infinity. Thus, similar to the initial experience of human participants, the algorithm could only process the immediate outgoing edges from the current node without engaging in any explicit planning.
The neural network approximated Q-values Q(o, a; θ) where θ represented the network parameters. These parameters were optimised by minimising the loss function:
$$L(\theta )={\mathbb{E}}\left[{\left({y}_{t}-Q(o,a;\theta )\right)}^{2}\right]\,{\mbox{,}}\,$$
(1)
with the target value yt defined by the Bellman equation:
$${y}_{t}=r+\gamma \,{{\max }_{a^{\prime} }}\,Q(o^{\prime},a^{\prime} ;{\theta }^{-})\,{\mbox{,}}\,$$
(2)
and γ = 0.99 as the discount rate. The target policy weights θ− were only updated every 200 steps to stabilise training.
An epsilon-greedy strategy guided training with ϵ starting at 1 and reducing according to ϵe = 0.99⌊e/1, 000⌋, reaching a minimum of 0.01. Trajectories were stored in a replay buffer of 500 episodes, and a batch of 16 episodes was sampled for updating the policy. The policy was optimised using the Adam optimiser with an initial learning rate of 1 × 10−3, adjusted downwards by a factor of 0.8 every 2000 episodes. The agent underwent 20,000 episodes of training, with evaluations every 100th episode using 1000 test networks from a hold-out set. Training took approximately 5 minutes on an RTX 5000 GPU. The neural policy was trained using Python 3.10 and PyTorch 2.1.1. Complete package versions are listed in the code repository. We depict the average performance on a hold-out test set in Supplementary Fig. S7, illustrating that the policies initially discovered the myopic strategy before eventually settling on the optimal strategy.
MeasuresTask performance
We measured task performance by calculating the total score for each trial, which involved summing the rewards from the 10 moves. The maximum achievable reward was 2650, which corresponded to taking three losses of 50 points each and then gaining 400 points seven times. In contrast, the maximum score attainable with a myopic strategy was 2000, yielded by ten gains of 200 points each. The exact scores achievable with these two strategies varied between networks.
Behavioural congruency
We assessed human–machine behavioural congruency by comparing the actions taken each human l, denoted as \({a}_{l,t^{\prime} }^{H}\), with those predicted by an ensemble of three machines. For each machine k ∈ {1, 2, 3}, the predicted action was defined as
$${a}_{l,t^{\prime} }^{{M}_{k}}=\arg {\max }_{a}\,{Q}_{k}({a}_{l,0}^{H},\ldots,{a}_{l,t^{\prime} -1}^{H})\,{\mbox{,}}\,$$
(3)
based on the trajectory of moves \({a}_{lt}^{H}\) selected by the human participant. The ensemble prediction was then determined by majority vote,
$${a}_{l,t^{\prime} }^{M}={{{\rm{mode}}}}\left({a}_{l,t^{\prime} }^{{M}_{1}},{a}_{l,t^{\prime} }^{{M}_{2}},{a}_{l,t^{\prime} }^{{M}_{3}}\right).$$
(4)
A match \({a}_{l,t^{\prime} }^{H}={a}_{l,t^{\prime} }^{M}\), where the ensemble’s action corresponded with the human’s action, was encoded as ‘1’; a mismatch as ‘0’. This method required processing the entire trajectory to ensure that the recurrent neural units of the neural policies could utilise accumulated historical information to influence their decisions.
Written strategies
To measure the explicit recognition of the loss strategy, we manually coded all 2310 strategy descriptions (covering both time steps for every participant) with the aid of three trained human annotators. The annotators familiarised themselves with the task and independently coded the written strategies using a binary variable to indicate the presence of the loss strategy. Before coding the full experimental data, the annotators calibrated their ratings by coding a small independent dataset (n = 76) from two previous pilot studies. After individual coding, interrater agreement was high (Light’s κ = 0.83). They then discussed any discrepancies before proceeding to code the full dataset. Annotators were blind to participant conditions during both calibration and full data coding, and data was randomly shuffled prior to coding to prevent any bias linked to participant order. For the full dataset, interrater agreement was excellent (Light’s κ = 0.92).
Statistical procedure
All analyses were performed using the lme4 package (version 1.1-35.1;58) in R (version 4.3.2; R Core Team), and the structure and procedure of these analyses were part of our preregistration. We first subset the data to only include demonstration trials of participants. Next, we aggregated the data to the appropriate level for each measure of interest: trial-level for reward, move-level for behavioural congruency, and participant-level for the written strategies.
In all models, the predictor of interest was a fixed effect of condition, assessed by computing the bootstrapped confidence intervals around the estimate at 95% confidence level. All confidence intervals are two-sided at the 95% level. For the models predicting the measure across generations 1-4, we included random intercepts for participant and population, along with a random slope for generation within population. For models predicting measures within individual generations, we incorporated the random intercepts for participant and population. Reward outcomes were modelled using a linear link function, whereas behavioural congruency and strategies were modelled using a logistic link function. As preregistered, we centred and scaled the variable generation to enhance model convergence and to make the main effect of condition more interpretable. Residual diagnostics for the linear mixed-effects models confirmed approximate normality and homoscedasticity of residuals.
Power analysis
We determined the goal of 15 trees per condition by simulating data via an agent-based model resembling a variant of the model described in the next section. In this model, rates of learning success and individual discovery rates were calibrated based on pilot data. We created synthetic data for 100 independent runs of the full experimental design in this way. In a power analysis on this simulated data, we resampled runs with 10 or 15 populations, each 1000 times. We then checked the distribution of parameters for the main preregistered models 1a and 1b, and based our final decision on 1b (since it has the smaller subsample of the two main predictions). We found our predicted effect, as assessed by a t-value greater than 2 for “condition”, in 957 (96%) of the samples involving 15 populations, suggesting sufficient power to assess all predicted effects.
Agent-based simulation
We developed a parsimonious agent-based model to capture key mechanisms of multi-generational cultural transmission. Each generation consists of sequential individual learning, reward-based teacher selection, social transmission, strategy integration, demonstration, and reward updating.
Simulations comprised M = 50 replications, G = 20 generations, and Ngen = 20 agents per generation. Depending on the condition, Nm = 5 agents were machines with systematically manipulated exploration capacities Km and optimality biases qm.
During individual learning, human agents performed kh trials, allocating exploration between optimal and myopic strategies according to an exploration bias qh. Optimal and myopic strategies were discovered with cumulative probabilities determined by per-trial rates dopt (manipulated) and dmyo = 0.5, reflecting the greater accessibility of the myopic solution. Agents adopted an optimal, myopic, or null strategy, with priority given to the optimal strategy when both were discovered. Successful agents produced Kdemo = 10 demonstrations of the adopted strategy. Rewards were computed as
$$R={n}_{{{{\rm{opt}}}}}{R}_{{{{\rm{opt}}}}}+{n}_{{{{\rm{myo}}}}}{R}_{{{{\rm{myo}}}}}+{\epsilon }_{{{{\rm{eff}}}}},\quad {\epsilon }_{{{{\rm{eff}}}}} \sim {{{\mathcal{N}}}}\,\left(0,{\epsilon }^{2}{K}_{{{{\rm{demo}}}}}\right),$$
(5)
with Ropt = 1, Rmyo = 0.5, and ϵ = 0.1.
The initial generation relied exclusively on individual learning. For generations g > 0, learners sampled candidate teachers from the previous generation and selected among them via softmax weighting over rewards with temperature τ. Social transmission occurred with probability λ. Individual and social learning outcomes were integrated such that optimal strategies dominated whenever acquired. Provided that evolutionary selection operates through teacher choice, we adopt this simple prioritisation of demonstrated behaviour, assuming that agents aim to maximise individual performance during demonstration.
We examined homogeneous and mixed populations across ranges of exploration bias, discoverability (\(-{\log }_{10}{d}_{{{{\rm{opt}}}}}\)), and learnability (1 − λ), and systematically varied machine capacity and bias. Additional analyses varied selection strength τ, reward noise ϵ, machine placement, population size, and machine prevalence to assess robustness. Full implementation details and extended analyses are provided in the Supplementary Information.
Reporting summary
Further information on research design is available in the Nature Portfolio Reporting Summary linked to this article.