Study 1: humans versus AI
This page sets the two kinds of recommender side by side. Humans are the doctor, nurse, remote doctor, therapist, personal trainer and layperson (six). AI is AI in general, an AI diagnostic tool, an AI chatbot and an AI fitness coach (four). A doctor using AI is a person plus a tool, so it is shown on its own. Each person’s score for a group is the average of their ratings of its recommenders. The doctor-versus-AI comparison in the dissertation (H6, H7, E) is one pair; this is the exploratory version across all of them. It was not preregistered.
The gap
People rated humans higher than AI on all three things. On average, trust was 1.6 points higher for humans (d = 0.70), intuition 1.7 points higher (d = 0.67), and perceived quantitative reasoning 0.8 points higher (d = 0.35), the smallest of the three.
What to look for. The difference and d columns. Humans were rated higher than AI on trust, intuition and quantitative reasoning.
| Rating | Humans, mean (SD) | AI, mean (SD) | Difference [95% CI] | t (df) | p | d [95% CI] |
|---|---|---|---|---|---|---|
| Trust | 6.02 (2.01) | 4.41 (2.79) | 1.60 [1.27, 1.94] | 9.43 (180) | < .001 | 0.70 [0.54, 0.86] |
| Quantitative reasoning | 5.82 (2.05) | 5.03 (2.79) | 0.80 [0.47, 1.12] | 4.79 (186) | < .001 | 0.35 [0.20, 0.50] |
| Intuition | 5.98 (2.22) | 4.23 (2.76) | 1.74 [1.37, 2.12] | 9.15 (185) | < .001 | 0.67 [0.51, 0.83] |
Paired t tests on each person’s average rating. Only people who rated every recommender in both groups are included (181 to 187 of 188). d is the mean difference over the SD of the differences.
Figure 9. Every person, humans against AI
Each outline is how people’s averages are spread, each dot is one person, and the diamond is the group average with its bootstrapped 95% interval. The doctor using AI sits between the two groups on all three.
Trust
Outline: how answers are spread (smoothed). Dots: one person each. Diamond and bar: the average and its bootstrapped 95% interval.
View as a table
| Measure | n | Mean | 95% interval |
|---|---|---|---|
| Human recommenders (average of 6) | 185 | 6.0 | 5.7 to 6.3 |
| Doctor using AI | 187 | 5.7 | 5.2 to 6.1 |
| AI recommenders (average of 4) | 183 | 4.4 | 4.0 to 4.8 |
Quantitative reasoning
Outline: how answers are spread (smoothed). Dots: one person each. Diamond and bar: the average and its bootstrapped 95% interval.
View as a table
| Measure | n | Mean | 95% interval |
|---|---|---|---|
| Human recommenders (average of 6) | 188 | 5.8 | 5.5 to 6.1 |
| Doctor using AI | 188 | 5.8 | 5.4 to 6.2 |
| AI recommenders (average of 4) | 187 | 5.0 | 4.6 to 5.4 |
Intuition
Outline: how answers are spread (smoothed). Dots: one person each. Diamond and bar: the average and its bootstrapped 95% interval.
View as a table
| Measure | n | Mean | 95% interval |
|---|---|---|---|
| Human recommenders (average of 6) | 187 | 6.0 | 5.7 to 6.3 |
| Doctor using AI | 188 | 5.2 | 4.7 to 5.6 |
| AI recommenders (average of 4) | 187 | 4.3 | 3.9 to 4.6 |
Figure 10. How many people show the gap?
The averages hide the people who go the other way. Counting each person’s own difference: 126 of 181 rated humans higher on trust, 42 rated AI higher, and 13 rated them equal.
- First item rated higher
- Rated the same
- Second item rated higher
- Doctor is blue, AI is coral, wherever they appear
If a doctor using AI is counted differently
What to look for. Humans stay above AI however the doctor using AI is counted. Humans against a doctor using AI differ little on trust and not at all on quantitative reasoning.
| Comparison | Means (SD) | Difference [95% CI] | t (df) | p | d [95% CI] |
|---|---|---|---|---|---|
| Trust: humans versus a doctor using AI | 6.01 (1.99), 5.69 (3.06) | 0.33 [-0.05, 0.70] | 1.71 (184) | .089 | 0.13 [-0.02, 0.27] |
| Trust: a doctor using AI versus the four AI tools | 5.69 (3.06), 4.42 (2.77) | 1.27 [0.92, 1.62] | 7.17 (182) | < .001 | 0.53 [0.37, 0.68] |
| Trust: humans versus AI, counting a doctor using AI as AI (the notebook’s rule) | 6.02 (2.01), 4.68 (2.68) | 1.34 [1.03, 1.66] | 8.38 (180) | < .001 | 0.62 [0.46, 0.78] |
| Quantitative reasoning: humans versus a doctor using AI | 5.82 (2.04), 5.78 (2.96) | 0.04 [-0.31, 0.39] | 0.22 (187) | .826 | 0.02 [-0.13, 0.16] |
| Quantitative reasoning: a doctor using AI versus the four AI tools | 5.76 (2.95), 5.03 (2.79) | 0.73 [0.41, 1.05] | 4.48 (186) | < .001 | 0.33 [0.18, 0.47] |
| Quantitative reasoning: humans versus AI, counting a doctor using AI as AI (the notebook’s rule) | 5.82 (2.05), 5.17 (2.68) | 0.65 [0.34, 0.96] | 4.20 (186) | < .001 | 0.31 [0.16, 0.45] |
| Intuition: humans versus a doctor using AI | 5.98 (2.21), 5.14 (3.15) | 0.83 [0.46, 1.20] | 4.40 (186) | < .001 | 0.32 [0.17, 0.47] |
| Intuition: a doctor using AI versus the four AI tools | 5.14 (3.15), 4.25 (2.76) | 0.90 [0.50, 1.30] | 4.40 (186) | < .001 | 0.32 [0.17, 0.47] |
| Intuition: humans versus AI, counting a doctor using AI as AI (the notebook’s rule) | 5.98 (2.22), 4.41 (2.61) | 1.56 [1.23, 1.90] | 9.11 (185) | < .001 | 0.67 [0.51, 0.83] |
The same paired tests with the doctor using AI shown on its own, compared with each group, or counted as AI as the notebook does. The conclusion does not change.
Recommender by recommender
The groups are not uniform. The doctor is trusted most and the layperson about as little as the AI tools. The gap is widest against the doctor and the nurse, and nearly closes against the layperson. The four AI tools are close to one another.
Figure 11. Each recommender on all three ratings
Each row is a recommender, with one panel each for trust, intuition and quantitative reasoning. The dot is the average rating, with a bar for the 95% interval.
View as a table
| Recommender | Trust: average [95% interval] | Quantitative reasoning: average [95% interval] | Intuition: average [95% interval] |
|---|---|---|---|
| Doctor | 7.5 [7.1, 7.8] | 7.2 [6.8, 7.6] | 6.7 [6.3, 7.2] |
| Nurse | 6.8 [6.5, 7.2] | 6.4 [6.0, 6.8] | 6.5 [6.2, 6.9] |
| Doctor, remote visit | 6.1 [5.7, 6.5] | 6.0 [5.5, 6.4] | 6.0 [5.6, 6.5] |
| Therapist | 6.0 [5.6, 6.4] | 5.9 [5.5, 6.3] | 6.2 [5.8, 6.6] |
| Personal trainer | 5.3 [4.9, 5.7] | 5.0 [4.7, 5.4] | 5.4 [5.0, 5.8] |
| Layperson | 4.3 [3.9, 4.8] | 4.4 [3.9, 4.8] | 4.9 [4.5, 5.4] |
| Doctor using AI | 5.7 [5.2, 6.1] | 5.8 [5.4, 6.2] | 5.2 [4.7, 5.6] |
| AI, general | 4.7 [4.2, 5.1] | 5.2 [4.7, 5.7] | 4.5 [4.1, 4.9] |
| AI diagnostic tool | 4.4 [4.0, 4.9] | 5.2 [4.7, 5.6] | 4.3 [3.9, 4.8] |
| AI fitness coach | 4.3 [3.9, 4.8] | 4.8 [4.4, 5.3] | 4.0 [3.6, 4.5] |
| AI chatbot | 4.3 [3.8, 4.7] | 4.8 [4.4, 5.3] | 4.2 [3.7, 4.6] |
Dots are averages and bars are 95% confidence intervals. Blue is human, coral is AI, grey is a doctor using AI.
Do people who trust humans also trust AI?
Yes, moderately: the more someone trusts human recommenders, the more they tend to trust AI (r = .586, n = 181). Most dots fall under the equal-trust line, which is the same gap seen person by person.
Figure 12. Trust in humans and trust in AI, one dot per person
Each dot is one person. Left to right is how much they trusted human recommenders; bottom to top is how much they trusted AI. A line rising to the right means people who trust humans also tend to trust AI.
One dot per person. The solid line is the best-fit line; the dashed line is where trust would be equal. Dots below it belong to people who trust AI less than humans.
What to look for. The correlation. A positive value means people who trust humans also tend to trust AI.
| Rating | n | r | 95% CI | p |
|---|---|---|---|---|
| Trust | 181 | .586 | .481 to .674 | < .001 |
| Quantitative reasoning | 187 | .597 | .496 to .682 | < .001 |
| Intuition | 186 | .472 | .352 to .577 | < .001 |
What to look for. Alpha shows how consistently each person rated the recommenders inside a group. Higher means the group average is a fair summary.
| Rating and group | Alpha | Average r |
|---|---|---|
| Trust: Human (6) | .81 | .43 |
| Trust: AI (4) | .92 | .75 |
| Quantitative reasoning: Human (6) | .81 | .43 |
| Quantitative reasoning: AI (4) | .92 | .74 |
| Intuition: Human (6) | .85 | .50 |
| Intuition: AI (4) | .93 | .77 |
Alpha rises with the number of items, so compare the average correlations instead: about .75 between two AI tools and about .43 between two human recommenders. People rate the AI tools much more alike than they rate the humans.
Who shows the gap?
The gap points the same way in every group of people shown for trust and intuition, but its size varies. In this sample it is largest among people aged 60 and over, among White participants compared with Black participants, and among people with little experience of AI. It does not differ much by education or political views.
Figure 13. The gap, group by group
Each row is a group of people. The dot is the average human-minus-AI gap in that group, with a bar for the 95% interval; the right panel gives the gap in standard deviations (Cohen's d), which can be read against the Funder and Ozer small (0.20), medium (0.41) and large (0.63).
View as a table
| Group | Trust: average gap [95% interval] | Quantitative reasoning: average gap [95% interval] | Intuition: average gap [95% interval] |
|---|---|---|---|
| 18-29 (n = 55) | 1.3 [0.7, 1.9] | 0.5 [−0.1, 1.0] | 1.1 [0.5, 1.7] |
| 30-44 (n = 51) | 1.1 [0.4, 1.8] | 0.7 [−0.1, 1.5] | 1.9 [1.1, 2.7] |
| 45-59 (n = 42) | 1.6 [0.9, 2.4] | 0.6 [0.0, 1.1] | 1.6 [0.8, 2.4] |
| 60+ (n = 39) | 2.7 [2.1, 3.3] | 1.7 [1.0, 2.4] | 2.6 [1.8, 3.5] |
| Woman (n = 113) | 1.9 [1.4, 2.3] | 0.9 [0.5, 1.3] | 2.0 [1.5, 2.5] |
| Man (n = 74) | 1.2 [0.7, 1.7] | 0.6 [0.1, 1.1] | 1.4 [0.8, 1.9] |
| White (n = 122) | 1.9 [1.5, 2.4] | 0.9 [0.5, 1.4] | 2.1 [1.6, 2.6] |
| Black (n = 44) | 0.6 [0.1, 1.1] | 0.2 [−0.4, 0.7] | 0.8 [0.2, 1.4] |
| No college degree (n = 116) | 1.6 [1.2, 2.1] | 1.0 [0.5, 1.4] | 1.8 [1.3, 2.3] |
| College degree or more (n = 72) | 1.5 [1.0, 2.0] | 0.5 [0.0, 1.0] | 1.6 [1.0, 2.2] |
| Liberal (n = 58) | 1.6 [1.0, 2.1] | 0.8 [0.2, 1.4] | 1.9 [1.1, 2.7] |
| Moderate (n = 74) | 1.4 [0.9, 1.9] | 0.9 [0.3, 1.4] | 1.6 [1.0, 2.1] |
| Conservative (n = 56) | 1.9 [1.2, 2.7] | 0.7 [0.1, 1.3] | 1.8 [1.1, 2.5] |
| None or a little (n = 119) | 2.1 [1.6, 2.5] | 1.1 [0.7, 1.6] | 2.2 [1.7, 2.7] |
| Moderate or more (n = 69) | 0.8 [0.3, 1.2] | 0.2 [−0.2, 0.6] | 0.9 [0.5, 1.4] |
Each dot is the average gap (human minus AI) for the people in that group, with its 95% confidence interval. Groups under 10 people are not shown, and race is shown only for groups of 30 or more. The tests ask whether the gap differs between the groups shown; they are exploratory and not corrected for the 18 comparisons.
What to look for. For trust, age, race and experience with AI show p values below .05; gender, education and political views do not. Treat single small p values as leads, since 18 tests were run.
| Compared by | Groups | Trust | Quantitative reasoning | Intuition |
|---|---|---|---|---|
| Age | 18-29, 30-44, 45-59, 60+ | F = 4.18, p .007 | F = 2.51, p .061 | F = 2.87, p .038 |
| Gender | Woman, Man | t = −1.96, p .051 | t = −0.89, p .373 | t = −1.75, p .082 |
| Race | White, Black | t = 3.39, p < .001 | t = 1.97, p .051 | t = 2.72, p .007 |
| Education | No college degree, College degree or more | t = −0.30, p .763 | t = −1.38, p .169 | t = −0.59, p .555 |
| Political views | Liberal, Moderate, Conservative | F = 0.89, p .412 | F = 0.09, p .913 | F = 0.32, p .729 |
| Experience with AI | None or a little, Moderate or more | t = 3.90, p < .001 | t = 2.59, p .010 | t = 3.26, p .001 |
t tests for two groups and one-way F tests for more. Exploratory, and not adjusted for running 18 tests, so treat a single small p as a lead, not a finding. The groups overlap (for example, older people and people with less AI experience), so this does not say which characteristic matters.
What goes with trust, for humans and for AI
For humans, perceived quantitative reasoning is the stronger partner of trust. For AI, it is perceived intuition. Letting the two slopes differ by recommender type improves the model (Table 13 on the full analysis page: χ² = 20.21, p < .001).
What to look for. The coefficients for humans and for AI. Compare which rating carries more weight in each group.
| Predictor | Humans, b [95% CI] | AI, b [95% CI] |
|---|---|---|
| Quantitative reasoning | 0.45 [0.40, 0.49] | 0.27 [0.21, 0.33] |
| Intuition | 0.35 [0.30, 0.41] | 0.43 [0.37, 0.49] |
Mixed models with a random intercept for each person, fitted separately to the human and AI recommenders (the doctor using AI counts as AI, as in the notebook). b is the change in trust for one more point on the predictor, holding the other predictor fixed.
Figure 14. Predicted trust, humans and AI
Each line shows predicted trust as a rating rises, for human recommenders and for AI. Steeper means that rating goes with trust more strongly.
- Quantitative reasoning
- Intuition
- The other rating is held at its average. Bands are 95% confidence intervals.
Predicted trust from the AI-only and human-only models, with the other rating held at its average. Bands are 95% confidence intervals for the fixed effects.