Study 3: dig deeper
Dig deeper
Two more things for readers who want them: where each figure and table of the dissertation sits on these pages, and a Bayesian check on every comparison, computed from the raw ratings.
Where each dissertation figure and table lives
| In the dissertation | What it shows | On this site | Notes |
|---|---|---|---|
| Figure 13 | The vignette images for the three sources | In the dissertation and on OSF | Materials, not data; not redrawn here |
| Figure 14 | Conceptual model of the study | In the dissertation | Not data; not redrawn here |
| Figure 15 | Mean trust by source, with 95% intervals | Figures 2 and 6 | |
| Figure 16 | Mean belief by source, with 95% intervals | Figures 3 and 6 | |
| Planned and follow-up t tests in the text | Doctor against the AIs, and humanlike against standard AI | Table 5 | Cohen's d here uses the pooled standard deviation; the text uses 2t over the root of the degrees of freedom |
| Regression tables for Study 3 | Trust and belief predicted by source, then by gender and age, then by warmth and competence | Tables 7 and 8 | Age is in four bands in the public file, so the age coefficients differ in scale |
When the test found no difference
A p value can only say whether a difference was detected. When it was not, it cannot tell you whether the difference is truly absent or whether the study was too small to see it. A Bayes factor can. It asks how many times more likely the data are under one story than under the other: there is no difference, or there is a difference of a plausible size. If the data are 10 times more likely when there is no difference, that is a Bayes factor of 10 for no difference.
The second story needs a guess about "a plausible size," so I ran it three ways: a small, a medium and a large effect, using the benchmarks Funder and Ozer (2019) proposed for psychology. The guess is a Cauchy prior, a bell-shaped curve centered on zero with long tails, so smaller effects are most likely and larger ones are not ruled out. Each row of a chart shows all three. When the three markers sit close together, the conclusion does not depend on the guess. When they spread out, it does.
| Effect | Correlation (r) | Cohen's d | What it means in daily life |
|---|---|---|---|
| Very small | .05 | 0.10 | Hard to see in one person, but it can add up over many people or many years |
| Small | .10 | 0.20 | Noticeable only when you look across many cases |
| Medium | .20 | 0.41 | A typical finding in published psychology, and a real difference in everyday life |
| Large | .30 | 0.63 | Rare in psychology, and clearly visible |
| Very large | .40 or more | 0.87 or more | Uncommon, and often an overestimate |
These benchmarks are more realistic than the older 0.2, 0.5 and 0.8 rule of thumb, which describes effects most psychology studies never reach. This site labels every effect size with them. For ratings from the same person, d is the mean difference divided by the spread of the differences, and I convert it to r the way you would for two equal groups, so the labels are a guide.
| Times more likely | How to describe the evidence |
|---|---|
| 1 to 3 | Anecdotal: barely worth mentioning |
| 3 to 10 | Moderate |
| 10 to 30 | Strong |
| 30 to 100 | Very strong |
| Over 100 | Extreme |
The same words apply in both directions. Data that are 12 times more likely under no difference are strong evidence for no difference, and data 12 times more likely under a difference are strong evidence for a difference. Near 1, the data cannot tell the two stories apart.
- Small, medium and large effect (the labelled dot is medium)
- Data favor no difference
- Data favor a difference
View as a table
| Test | If the effect is small | If medium | If large |
|---|---|---|---|
| Trust: humanlike AI versus standard AI | 2.5 times for no difference | 4.5 times for no difference | 6.7 times for no difference |
| Trust: doctor versus standard AI | 4.9 times for a difference | 4.3 times for a difference | 3.4 times for a difference |
| Trust: humanlike AI versus doctor | 1.7 times for a difference | 1.3 times for a difference | about equally likely for no difference |
| Belief: humanlike AI versus standard AI | 3.1 times for no difference | 5.6 times for no difference | 8.5 times for no difference |
| Belief: doctor versus standard AI | 2.9 times for a difference | 2.4 times for a difference | 1.8 times for a difference |
| Belief: humanlike AI versus doctor | 1.9 times for a difference | 1.4 times for a difference | about equally likely for a difference |
Humanizing the AI did not change trust or belief. For trust, the data are 4.5 times more likely under no difference between the humanlike and standard AI than under a medium difference. For belief the figure is 5.7 times more likely. Both are moderate evidence for no effect, a firmer reading than the p values (p = .45 and p = .76) give on their own.
The doctor's edge over the AIs is modest evidence at best. Against the standard AI the data are 4.3 times more likely under a difference for trust, moderate evidence, and 2.4 times more likely for belief, which is anecdotal. Against the humanlike AI they are 1.3 times more likely for trust and 1.4 times more likely for belief, also anecdotal, even where p is below .05. Studies 4 and 5 give a sharper test.
Bayes factors in this section come from the raw ratings. Two-group tests use the Rouder default Bayes factor for the t test with a Cauchy prior on d, with widths set to a small (0.20), medium (0.41) and large (0.63) effect on the Funder and Ozer scale.