Klayman J, Soll JB, González-Vallejo C, & Barlas S (1999). Overconfidence: It depends on how, what, and whom you ask. Organizational Behavior and Human Decision Processes, 79, 216–247. doi: 10.1006/obhd.1999.2847 [ PubMed] [ CrossRef] [ Google Scholar] Mata A, & Almeida T (2014). Using metacognitive cues to infer others’ thinking. Judgment and Decision Making, 9, 349–359. [ Google Scholar] Thousands of students from MIT, Harvard, and Princeton had been put through the quiz, and you’d think that anyone in these prestigious universities would be able to solve this problem with an unerring ease. Not so fast. It turned out that more than 50% responded with the knee-jerk—incorrect—answer. The two systems that led to the choices. Lichtenstein S, & Fischhoff B (1977). Do those who know more also know more about how much they know? Organizational Behavior and Human Performance, 20, 159–183. doi: 10.1016/0030-5073(77)90001-0 [ CrossRef] [ Google Scholar]

The question really belongs more to the science of the mind than it does to mathematics and logic— it is about the assumptions we make, rather than whether or not we have the ability to solve the question.For the second and third samples, nine simple math problems were included between the experimental questions and the memory questions to serve as interference to limit recall and recognition based on working memory (see the Supplemental Materials). Neither math problems nor any other activity occurred between the De Neys et al. (2013) experimental and the memory questions for the first sample. Specifically, we started by testing 126 MTurk participants and then examined their data. We did not have a precise stopping rule for the sample size, but we decided from the outset to pause data collection after examining the data from an initial sample of MTurk participants. We noted poor memory performance by these initial participants. On the basis of our observations, we tested two additional samples of participants (one from MTurk and one from UCB), each approximately the same size as the initial sample (128 participants), and gave them the math problems to create interference. Finally, latency responses to both the standard and control versions were collected. Correlations between response time and response confidence were expected, in line with previous work (e.g., Johnson et al., 2016; Kelley & Lindsay, 1993; Thompson, Prowse Turner, & Pennycook, 2011; Thompson et al., 2013). Soll JB (1996). Determinants of overconfidence and miscalibration: The roles of random error and ecological structure. Organizational Behavior and Human Decision Processes, 65, 117–137. doi: 10.1006/obhd.1996.0011 [ CrossRef] [ Google Scholar] Lichtenstein S, Fischhoff B, & Phillips LD (1982). Calibration of probabilities: The state of the art to 1980. In Kahneman D, Slovic P, & Tversky A (Eds.), Judgment under uncertainty: Heuristics and biases (pp. 306–334). Cambridge, UK: Cambridge University Press. [ Google Scholar]

Baron J, Scott S, Fincher K, & Metz SE (2015). Why does the Cognitive Reflection Test (sometimes) predict utilitarian moral judgment (and other things)? Journal of Applied Research in Memory and Cognition, 4, 265–284. doi: 10.1016/j.jarmac.2014.09.003 [ CrossRef] [ Google Scholar] Kelley CM, & Lindsay DS (1993). Remembering mistaken for knowing: Ease of retrieval as a basis for confidence in answers to general knowledge questions. Journal of Memory and Language, 32, 1–24. doi: 10.1006/jmla.1993.1001 [ CrossRef] [ Google Scholar] Szollosi A, Bago B, Szaszi B, & Aczel B (2017). Exploring the determinants of confidence in the bat-and-ball problem. Acta Psychologica, 180, 1–7. doi: 10.1016/j.actpsy.2017.08.003 [ PubMed] [ CrossRef] [ Google Scholar]

Giving some thoughts to deliberation.

For the recall response, a mixed effects logistic regression (with subject as the random variable) was conducted due to the dichotomous dependent variable (with or without “more than”), considering only those participants who wrote down an answer that could be coded as with or without the “more than” phrase (e.g., “don’t know” responses were excluded). Incorrect reasoners usually recalled the standard problem, but not the control, as containing “more than” (see Table 1), with this effect of condition significant, b = 3.02, odds ratio ( OR) = 20.44, χ 2 = 21.69, p< .001, 95% confidence interval ( CI) [5.74, 72.75]. De Neys W, & Glumicic T (2008). Conflict monitoring in dual process theories of thinking. Cognition, 106, 1248–1299. doi: 10.1016/j.cognition.2007.06.002 [ PubMed] [ CrossRef] [ Google Scholar] Bago B, & De Neys W (2017). Fast logic? Examining the time course assumption of dual process theory. Cognition, 158, 90–109. doi: 10.1016/j.cognition.2016.10.014 [ PubMed] [ CrossRef] [ Google Scholar] De Neys W (2012). Bias and conflict: A case for logical intuitions. Perspectives on Psychological Science, 7, 28–38. doi: 10.1177/1745691611429354 [ PubMed] [ CrossRef] [ Google Scholar]

Keren G (1997). On the calibration of probability judgments: Some critical comments and alternative perspectives. Journal of Behavioral Decision Making, 10, 269–278. doi: 10.1002/(SICI)1099-0771(199709)10:3<269::AID-BDM281>3.0.CO;2-L [ CrossRef] [ Google Scholar] Fischhoff B, Slovic P, & Lichtenstein S (1977). Knowing with certainty: The appropriateness of extreme confidence. Journal of Experimental Psychology: Human Perception and Performance, 3, 552–564. doi: 10.1037/0096-1523.3.4.552 [ CrossRef] [ Google Scholar]De Neys W, & Bonnefon J-F (2013). The ‘whys’ and ‘whens’ of individual differences in thinking biases. Trends in Cognitive Sciences, 17, 172–178. doi: 10.1016/j.tics.2013.02.001 [ PubMed] [ CrossRef] [ Google Scholar] Mata A, & Ferreira MB (2018). Response: Commentary: Seeing the conflict: an attentional account of reasoning errors. Frontiers in Psychology, 9, 24. doi: 10.3389/fpsyg.2018.00024 [ PMC free article] [ PubMed] [ CrossRef] [ Google Scholar] Sinayev A, & Peters E (2015). Cognitive reflection vs. calculation in decision making. Frontiers in Psychology, 6, 532. doi: 10.3389/fpsyg.2015.00532 [ PMC free article] [ PubMed] [ CrossRef] [ Google Scholar]