Study 4: dig deeper
Dig deeper
Two more things for readers who want them: where each figure and table of the dissertation sits on these pages, and a Bayesian check on every comparison, computed from the raw ratings.
Where each dissertation figure and table lives
| In the dissertation | What it shows | On this site | Notes |
|---|---|---|---|
| Figure 17 | The vignette images | In the dissertation and on OSF | Materials, not data; not redrawn here |
| Figures 18 and 19 | Trust and belief by first source, with 95% intervals | Figures 2, 3 and 6 | |
| Figures 20 and 21 | Trust and belief for the three recommenders, side by side | Figures 8, 9 and 12 | |
| The omnibus repeated-measures tests | Whether the three recommenders differ | Table 9 | The dissertation prints F(2, 1573) = 1.65 for trust; the data give F(2, 984) = 55.5 on the people who rated all three |
| Planned and follow-up t tests in the text | Humanlike against standard AI, and the doctor against each | Tables 6 and 10 | Part 1 d uses the pooled standard deviation; the text uses 2t over the root of the degrees of freedom |
| Regression and mixed-model tables for Study 4 | Trust and belief predicted by recommender, then by gender and age, then by warmth and competence | Tables 12 and 13 | Age is in four bands in the public file, so the age coefficients differ in scale |
What the Bayes factors say
A p value can only say whether a difference was detected. When it was not, it cannot tell you whether the difference is truly absent or whether the study was too small to see it. A Bayes factor can. It asks how many times more likely the data are under one story than under the other: there is no difference, or there is a difference of a plausible size. If the data are 10 times more likely when there is no difference, that is a Bayes factor of 10 for no difference.
The second story needs a guess about "a plausible size," so I ran it three ways: a small, a medium and a large effect, using the benchmarks Funder and Ozer (2019) proposed for psychology. The guess is a Cauchy prior, a bell-shaped curve centered on zero with long tails, so smaller effects are most likely and larger ones are not ruled out. Each row of a chart shows all three. When the three markers sit close together, the conclusion does not depend on the guess. When they spread out, it does.
| Effect | Correlation (r) | Cohen's d | What it means in daily life |
|---|---|---|---|
| Very small | .05 | 0.10 | Hard to see in one person, but it can add up over many people or many years |
| Small | .10 | 0.20 | Noticeable only when you look across many cases |
| Medium | .20 | 0.41 | A typical finding in published psychology, and a real difference in everyday life |
| Large | .30 | 0.63 | Rare in psychology, and clearly visible |
| Very large | .40 or more | 0.87 or more | Uncommon, and often an overestimate |
These benchmarks are more realistic than the older 0.2, 0.5 and 0.8 rule of thumb, which describes effects most psychology studies never reach. This site labels every effect size with them. For ratings from the same person, d is the mean difference divided by the spread of the differences, and I convert it to r the way you would for two equal groups, so the labels are a guide.
| Times more likely | How to describe the evidence |
|---|---|
| 1 to 3 | Anecdotal: barely worth mentioning |
| 3 to 10 | Moderate |
| 10 to 30 | Strong |
| 30 to 100 | Very strong |
| Over 100 | Extreme |
The same words apply in both directions. Data that are 12 times more likely under no difference are strong evidence for no difference, and data 12 times more likely under a difference are strong evidence for a difference. Near 1, the data cannot tell the two stories apart.
- Small, medium and large effect (the labelled dot is medium)
- Data favor no difference
- Data favor a difference
View as a table
| Test | If the effect is small | If medium | If large |
|---|---|---|---|
| Trust: humanlike AI versus standard AI | 1.6 times for no difference | 2.5 times for no difference | 3.6 times for no difference |
| Trust: doctor versus standard AI | 6.2 times for a difference | 5.7 times for a difference | 4.6 times for a difference |
| Trust: humanlike AI versus doctor | about equally likely for a difference | 1.4 times for no difference | 2 times for no difference |
| Belief: humanlike AI versus standard AI | about equally likely for no difference | 1.7 times for no difference | 2.5 times for no difference |
| Belief: doctor versus standard AI | 4.1 times for a difference | 3.6 times for a difference | 2.8 times for a difference |
| Belief: humanlike AI versus doctor | 1.7 times for no difference | 2.6 times for no difference | 3.8 times for no difference |
| Trust: humanlike AI versus standard AI | over a billion times for a difference | over a billion times for a difference | over a billion times for a difference |
| Trust: doctor versus standard AI | over a billion times for a difference | over a billion times for a difference | over a billion times for a difference |
| Trust: humanlike AI versus doctor | 1.3 times for a difference | 1.3 times for no difference | 2 times for no difference |
| Belief: humanlike AI versus standard AI | over a billion times for a difference | over a billion times for a difference | over a billion times for a difference |
| Belief: doctor versus standard AI | over a billion times for a difference | over a billion times for a difference | over a billion times for a difference |
| Belief: humanlike AI versus doctor | 2.1 times for no difference | 4 times for no difference | 6 times for no difference |
Seen alone, the evidence leans toward no effect of the framing, but weakly. The data are 2.5 times more likely under no difference for trust and 1.7 times more likely for belief, both anecdotal. With a voice added, Study 4 cannot rule out a small effect between people. The doctor's edge over the standard AI is moderate evidence: 5.7 times more likely under a difference for trust and 3.6 times more likely for belief.
Side by side, the framing effect is very clear. The humanlike AI against the standard AI, and the doctor against the standard AI, are both supported by extremely strong evidence, for trust and for belief.
The doctor and the humanlike AI are close, side by side. For trust the data are 1.3 times more likely under no difference, anecdotal. For belief they are 4 times more likely, moderate evidence for no difference. The p value for trust is .036, so I read this as a gap that is at most small.
Bayes factors in this section come from the raw ratings. Two-group tests use the Rouder default Bayes factor for the t test with a Cauchy prior on d, with widths set to a small (0.20), medium (0.41) and large (0.63) effect on the Funder and Ozer scale.