Study 2: dig deeper
Dig deeper
Two more things for readers who want them: where each figure and table of the dissertation sits on this page, and a Bayesian check on every comparison that found no difference.
Where each dissertation figure and table lives
| In the dissertation | What it shows | On this site | Notes |
|---|---|---|---|
| Table 4 | The twelve medical decisions and how they were grouped | Tables 2 and 3 | |
| Figure 9 | Quantitative reasoning needed, decision by decision | Figure 1 | |
| Figure 10 | Trust in the doctor and the AI, decision by decision | Figures 5 and 6 | Figure 4 shows each person |
| Table 5 and Hypothesis 1 | Paired t tests of trust, decision by decision | Table 5 | The dissertation text and its Table 5 differ slightly in a few means; this page reports the data |
| Tables 6 and 7 | The same test for the stage and domain composites | Table 6 | |
| Table 8 and Hypothesis 2 | Mixed model with the doctor-by-need interaction | Table 8, Figure 11 | Full model, with warmth and competence, is in the mixed models below |
| Tables 9 and 10 | Correlations among quantitative reasoning, intuition, warmth, competence, for each recommender | Table 9 | |
| Tables 11 and 12 | Within-person and between-person correlations | Table 10 | |
| Figure 11 | Mediation through warmth and competence | Figure 12 | |
| Figure 12 and Additional Mediation Models | Mediation through intuition and quantitative reasoning | Figure 13 and Table 15 |
When the test found no difference
A p value can only say whether a difference was detected. When it was not, it cannot tell you whether the difference is truly absent or whether the study was too small to see it. A Bayes factor can. It asks how many times more likely the data are under one story than under the other: there is no difference, or there is a difference of a plausible size. If the data are 10 times more likely when there is no difference, that is a Bayes factor of 10 for no difference.
The second story needs a guess about "a plausible size," so I ran it three ways: a small, a medium and a large effect, using the benchmarks Funder and Ozer (2019) proposed for psychology. The guess is a Cauchy prior, a bell-shaped curve centered on zero with long tails, so smaller effects are most likely and larger ones are not ruled out. Each row of a chart shows all three. When the three markers sit close together, the conclusion does not depend on the guess. When they spread out, it does.
| Effect | Correlation (r) | Cohen's d | What it means in daily life |
|---|---|---|---|
| Very small | .05 | 0.10 | Hard to see in one person, but it can add up over many people or many years |
| Small | .10 | 0.20 | Noticeable only when you look across many cases |
| Medium | .20 | 0.41 | A typical finding in published psychology, and a real difference in everyday life |
| Large | .30 | 0.63 | Rare in psychology, and clearly visible |
| Very large | .40 or more | 0.87 or more | Uncommon, and often an overestimate |
These benchmarks are more realistic than the older 0.2, 0.5 and 0.8 rule of thumb, which describes effects most psychology studies never reach. This site labels every effect size with them. For ratings from the same person, d is the mean difference divided by the spread of the differences, and I convert it to r the way you would for two equal groups, so the labels are a guide.
| Times more likely | How to describe the evidence |
|---|---|
| 1 to 3 | Anecdotal: barely worth mentioning |
| 3 to 10 | Moderate |
| 10 to 30 | Strong |
| 30 to 100 | Very strong |
| Over 100 | Extreme |
The same words apply in both directions. Data that are 12 times more likely under no difference are strong evidence for no difference, and data 12 times more likely under a difference are strong evidence for a difference. Near 1, the data cannot tell the two stories apart.
- Small, medium and large effect (the labelled dot is medium)
- Data favor no difference
- Data favor a difference
View as a table
| Test | If the effect is small | If medium | If large |
|---|---|---|---|
| Doctor-minus-AI gap: diagnostic versus prescriptive | 2.7 times for no difference | 4.7 times for no difference | 7 times for no difference |
| Doctor-minus-AI gap: diagnostic versus preventative | over 1 million times for a difference | over 1 million times for a difference | over 2 million times for a difference |
| Doctor-minus-AI gap: prescriptive versus preventative | over 59 million times for a difference | over 95 million times for a difference | over 107 million times for a difference |
| Doctor-minus-AI gap: physical health versus mental health | 2.7 times for no difference | 4.8 times for no difference | 7.1 times for no difference |
| Doctor-minus-AI gap: bodily injury versus acute sickness | 3 times for no difference | 5.5 times for no difference | 8.2 times for no difference |
| Doctor-minus-AI gap: bodily injury versus chronic sickness | 2.9 times for no difference | 5.2 times for no difference | 7.8 times for no difference |
| Doctor-minus-AI gap: acute sickness versus chronic sickness | 2.8 times for no difference | 5 times for no difference | 7.5 times for no difference |
| H2: Does the gap depend on the need for quantitative reasoning? | 4.5 times for no difference | 8.5 times for no difference | 13 times for no difference |
| Trust by how much quantitative reasoning the decision needs | 3.5 times for no difference | 6.5 times for no difference | 9.5 times for no difference |
| Trust by gender (man versus not) | 3.5 times for no difference | 6.5 times for no difference | 9.8 times for no difference |
| Trust by age group (four bands) | 2.3 times for no difference | 4 times for no difference | 5.7 times for no difference |
| Does the gap grow with age? (age band by doctor) | 2.9 times for a difference | 1.7 times for a difference | 1.2 times for a difference |
| H5: Does warmth explain the gap? (indirect effect) | 1.4 times for no difference | 2 times for no difference | 2.8 times for no difference |
The gap does not depend on how much quantitative reasoning a decision needs. The doctor-by-need interaction (hypothesis 2) was not significant. The data are 8.5 times more likely if the gap is the same whether a decision leans on numbers or on judgment than if it changes by a medium-sized amount. That is moderate evidence for no change, a firmer reading than the p value alone gives.
The gap is the same size across kinds of health problem. Physical against mental health, and bodily injury, acute sickness and chronic sickness against each other, each come out roughly 5 to 6 times more likely under no difference, moderate evidence. Stage of care is different: the gap for preventative decisions is clearly smaller than for diagnostic or prescriptive ones, with the data millions of times more likely under a difference. Diagnostic and prescriptive gaps match (4.7 times more likely under no difference).
Gender made no difference to trust; age is unclear. The data are 6.5 times more likely under no gender difference, moderate evidence. Age by itself is similar (4 times more likely under no difference), but whether the gap grows with age is inconclusive: the data are only 1.7 times more likely under a difference, despite p = .013, so I would not lean on that finding.
Warmth did not explain the gap, but the evidence is thin. The warmth pathway was not significant in the model that also included competence. The data are 2 times more likely under no indirect effect, which is anecdotal. This one assumes the indirect effect is roughly bell-shaped, which indirect effects often are not, so treat it as a guide.