Study 1: dig deeper
Dig deeper
Two more things for readers who want them: where each figure and table of the dissertation sits on this page, and a Bayesian check on every comparison, with extra attention to the ones that found no difference.
Where each dissertation figure and table lives
| In the dissertation | What it shows | On this site | Notes |
|---|---|---|---|
| Table 1 | The fifteen medical decisions and how they were grouped | Tables 2 and 3 | |
| Table 2 | The eleven medical recommenders | Table 8 | |
| Figure 3 | Quantitative reasoning needed, decision by decision | Figure 1 | Every person is plotted, not only the mean and interval |
| Figure 4 | The same, by stage of care | Figure 3 | |
| Figure 5 | The same, by kind of health problem | Figure 2 | |
| Figure 6 | Quantitative reasoning and intuition for each recommender | Figures 6 and 7 | Plus Figure 11 for the human-versus-AI split |
| Figure 7 | Trust in each recommender | Figure 6 | |
| Hypotheses 1 to 7, exploratory test | Paired t tests | Tables 6 and 7, Figure 5 | |
| Table 3 | Mixed model of trust, with within-person and between-person terms | Tables 9 to 14, Figure 8 | The released file groups age, so some terms differ in the third decimal |
When the test found no difference
A p value can only say whether a difference was detected. When it was not, it cannot tell you whether the difference is truly absent or whether the study was too small to see it. A Bayes factor can. It asks how many times more likely the data are under one story than under the other: there is no difference, or there is a difference of a plausible size. If the data are 10 times more likely when there is no difference, that is a Bayes factor of 10 for no difference.
The second story needs a guess about "a plausible size," so I ran it three ways: a small, a medium and a large effect, using the benchmarks Funder and Ozer (2019) proposed for psychology. The guess is a Cauchy prior, a bell-shaped curve centered on zero with long tails, so smaller effects are most likely and larger ones are not ruled out. Each row of a chart shows all three. When the three markers sit close together, the conclusion does not depend on the guess. When they spread out, it does.
| Effect | Correlation (r) | Cohen's d | What it means in daily life |
|---|---|---|---|
| Very small | .05 | 0.10 | Hard to see in one person, but it can add up over many people or many years |
| Small | .10 | 0.20 | Noticeable only when you look across many cases |
| Medium | .20 | 0.41 | A typical finding in published psychology, and a real difference in everyday life |
| Large | .30 | 0.63 | Rare in psychology, and clearly visible |
| Very large | .40 or more | 0.87 or more | Uncommon, and often an overestimate |
These benchmarks are more realistic than the older 0.2, 0.5 and 0.8 rule of thumb, which describes effects most psychology studies never reach. This site labels every effect size with them. For ratings from the same person, d is the mean difference divided by the spread of the differences, and I convert it to r the way you would for two equal groups, so the labels are a guide.
| Times more likely | How to describe the evidence |
|---|---|
| 1 to 3 | Anecdotal: barely worth mentioning |
| 3 to 10 | Moderate |
| 10 to 30 | Strong |
| 30 to 100 | Very strong |
| Over 100 | Extreme |
The same words apply in both directions. Data that are 12 times more likely under no difference are strong evidence for no difference, and data 12 times more likely under a difference are strong evidence for a difference. Near 1, the data cannot tell the two stories apart.
- Small, medium and large effect (the labelled dot is medium)
- Data favor no difference
- Data favor a difference
View as a table
| Test | If the effect is small | If medium | If large |
|---|---|---|---|
| H1: Diagnostic decisions need more quantitative reasoning than prescriptive ones | 3.5 times for no difference | 6.7 times for no difference | 10 times for no difference |
| H2: Diagnostic decisions need more quantitative reasoning than preventative ones | 114,000 times for a difference | 138,000 times for a difference | 125,000 times for a difference |
| H3: Prescriptive decisions need more quantitative reasoning than preventative ones | 165,000 times for a difference | 202,000 times for a difference | 183,000 times for a difference |
| H4: Physical health decisions need more quantitative reasoning than mental health ones | 4.4 times for a difference | 3.2 times for a difference | 2.3 times for a difference |
| H5: Bodily injury decisions need more quantitative reasoning than acute sickness ones | about equally likely for a difference | 1.3 times for no difference | 1.9 times for no difference |
| H6: A doctor is seen as more capable of intuition than an AI | over a billion times for a difference | over a billion times for a difference | over a billion times for a difference |
| H7: Doctors are judged more trustworthy than medical AI | over a billion times for a difference | over a billion times for a difference | over a billion times for a difference |
| Exploratory: A doctor is seen as more capable of quantitative reasoning than an AI (exploratory) | over a billion times for a difference | over a billion times for a difference | over a billion times for a difference |
| Trust: layperson versus AI | 1.5 times for no difference | 2.4 times for no difference | 3.6 times for no difference |
| Intuition: layperson versus AI | 1.3 times for a difference | 1.3 times for no difference | 1.8 times for no difference |
| AI: quantitative reasoning versus intuition | 13 times for a difference | 11 times for a difference | 7.8 times for a difference |
| Trust: doctor versus doctor using AI | over 371 million times for a difference | over 526 million times for a difference | over 527 million times for a difference |
| Trust: humans versus AI | over a billion times for a difference | over a billion times for a difference | over a billion times for a difference |
| Quantitative reasoning: humans versus AI | 4,100 times for a difference | 4,400 times for a difference | 3,700 times for a difference |
| Intuition: humans versus AI | over a billion times for a difference | over a billion times for a difference | over a billion times for a difference |
Diagnostic and prescriptive decisions are rated alike. Hypothesis 1 found no difference (p = .68). The data are 6.7 times more likely if the two stages need the same amount of quantitative reasoning than if they differ by a medium-sized amount. That is moderate evidence for no difference, a firmer conclusion than "not significant." It holds if you expect a small or a large difference instead.
Bodily injury versus acute sickness is unsettled. Hypothesis 5 missed the usual cutoff (p = .061). The data are only 1.3 times more likely under no difference than under a medium difference, which is the same as saying they lean neither way. A larger sample is the only way to settle it.
One result is weaker than its p value suggests. Physical health versus mental health reached p = .010, yet the data are only 3.2 times more likely under a medium difference than under none, which is anecdotal. I would call that difference suggestive. The comparisons behind hypotheses 2, 3, 6 and 7 are another matter: the data are many thousands to many billions of times more likely under a difference.
A layperson and an AI are trusted about equally. People trusted a layperson (M = 4.3) about as much as a medical AI (M = 4.7). The data are 2.5 times more likely under no difference, which is anecdotal, so I would not claim the two are the same, only that no gap showed up. The AI is trusted well below a doctor.