Study 5: dig deeper
Dig deeper
Two more things for readers who want them: where each figure and table of the dissertation sits on these pages, and a Bayesian check on every comparison, computed from the raw ratings.
Where each dissertation figure and table lives
| In the dissertation | What it shows | On this site | Notes |
|---|---|---|---|
| Figure 22 | A study material or conceptual model, not data | In the dissertation | Not redrawn here |
| Figures 23 and 24 | Trust and belief by recommender, with and without a voice | Figures 4 and 8 to 10 | |
| Figures 25 to 27 | Mediation through mind perception, for each pair of recommenders | Figures 15 to 18 | The dissertation prints the humanlike-versus-doctor effect with a reversed sign |
| Tables 20 to 22 | Mixed ANOVA and mixed models for Study 5 | Tables 5, 5b and 12 to 14 | Table 22, the model with mind perception, warmth, competence, age and gender, is Table 14 here. Age is in four bands, so its coefficient differs in scale. The dissertation's ANOVA degrees of freedom imply more people than the 727 who rated all three |
| Paired t tests in the text | Comparisons among the three recommenders | Tables 4 and 9 | dz is used for paired effect sizes; the text uses 2t over the root of the degrees of freedom |
| Mind-perception items | Seven items and their correlations | Tables 10 and 11 |
What the Bayes factors say
A p value can only say whether a difference was detected. When it was not, it cannot tell you whether the difference is truly absent or whether the study was too small to see it. A Bayes factor can. It asks how many times more likely the data are under one story than under the other: there is no difference, or there is a difference of a plausible size. If the data are 10 times more likely when there is no difference, that is a Bayes factor of 10 for no difference.
The second story needs a guess about "a plausible size," so I ran it three ways: a small, a medium and a large effect, using the benchmarks Funder and Ozer (2019) proposed for psychology. The guess is a Cauchy prior, a bell-shaped curve centered on zero with long tails, so smaller effects are most likely and larger ones are not ruled out. Each row of a chart shows all three. When the three markers sit close together, the conclusion does not depend on the guess. When they spread out, it does.
| Effect | Correlation (r) | Cohen's d | What it means in daily life |
|---|---|---|---|
| Very small | .05 | 0.10 | Hard to see in one person, but it can add up over many people or many years |
| Small | .10 | 0.20 | Noticeable only when you look across many cases |
| Medium | .20 | 0.41 | A typical finding in published psychology, and a real difference in everyday life |
| Large | .30 | 0.63 | Rare in psychology, and clearly visible |
| Very large | .40 or more | 0.87 or more | Uncommon, and often an overestimate |
These benchmarks are more realistic than the older 0.2, 0.5 and 0.8 rule of thumb, which describes effects most psychology studies never reach. This site labels every effect size with them. For ratings from the same person, d is the mean difference divided by the spread of the differences, and I convert it to r the way you would for two equal groups, so the labels are a guide.
| Times more likely | How to describe the evidence |
|---|---|
| 1 to 3 | Anecdotal: barely worth mentioning |
| 3 to 10 | Moderate |
| 10 to 30 | Strong |
| 30 to 100 | Very strong |
| Over 100 | Extreme |
The same words apply in both directions. Data that are 12 times more likely under no difference are strong evidence for no difference, and data 12 times more likely under a difference are strong evidence for a difference. Near 1, the data cannot tell the two stories apart.
- Small, medium and large effect (the labelled dot is medium)
- Data favor no difference
- Data favor a difference
View as a table
| Test | If the effect is small | If medium | If large |
|---|---|---|---|
| Doctor: voiced versus written | 2.4 times for no difference | 4.3 times for no difference | 6.5 times for no difference |
| Humanlike AI: voiced versus written | 2.9 times for a difference | 2 times for a difference | 1.4 times for a difference |
| Standard AI: voiced versus written | 1.7 times for a difference | about equally likely for a difference | 1.2 times for no difference |
| Humanlike AI versus standard AI | over a billion times for a difference | over a billion times for a difference | over a billion times for a difference |
| Doctor versus standard AI | over a billion times for a difference | over a billion times for a difference | over a billion times for a difference |
| Humanlike AI versus doctor | 118 times for a difference | 75 times for a difference | 52 times for a difference |
| Voice, across all recommenders | 2.6 times for no difference | 4.6 times for no difference | 6.8 times for no difference |
| Voice effect for the standard AI versus the humanlike AI | 2.6 times for no difference | 4.6 times for no difference | 6.9 times for no difference |
| Voice effect for the doctor versus the humanlike AI | 2.5 times for no difference | 4.4 times for no difference | 6.5 times for no difference |
| Doctor versus humanlike AI after adjusting for mind, warmth, competence | 2.2 times for no difference | 3.8 times for no difference | 5.7 times for no difference |
| Mind perception as a mediator: doctor versus standard AI | over a billion times for a difference | over a billion times for a difference | over a billion times for a difference |
| Mind perception as a mediator: humanlike AI versus standard AI | over a billion times for a difference | over a billion times for a difference | over a billion times for a difference |
| Mind perception as a mediator: humanlike AI versus doctor | 5.7 times for no difference | 12 times for no difference | 18 times for no difference |
Voice made no overall difference. The data are 4.6 times more likely under no voice effect, 4.6 times more likely under no voice-by-standard-AI interaction and 4.4 times more likely under no voice-by-doctor interaction. All are moderate evidence. A voice did not change the doctor's ratings either (4.4 times more likely under no change).
The two within-recommender voice effects are weak. For the humanlike AI the data are 2 times more likely under an effect, despite p = .017. For the standard AI they are about equally likely, about even. I would not lean on either.
The three recommenders are clearly ordered. The humanlike AI over the standard AI, and the doctor over the standard AI, are supported by extreme evidence. The doctor over the humanlike AI is supported too, 75 times more likely under a difference, very strong evidence even though the gap is small. Once warmth, competence, mind perception and the rest are in the model, the data move to 3.8 times more likely under no remaining doctor advantage, moderate evidence.
Mind perception explains the gaps against the standard AI, and none of the doctor's small edge over the humanlike AI. The indirect effects for the doctor and for the humanlike AI against the standard AI have extreme evidence behind them, while the indirect effect for humanlike versus doctor has strong evidence for none (12 times more likely under no effect). The indirect-effect factors use a bell-shaped approximation, so treat them as a guide.
Bayes factors in this section come from the raw ratings. Paired t tests use the Rouder default Bayes factor with a Cauchy prior on d, with widths set to a small (0.20), medium (0.41) and large (0.63) effect on the Funder and Ozer scale. Coefficients use a normal likelihood.