Book: Rationality - From AI to Zombies
Author: Elezier Yudkowsky
Find Online:https://intelligence.org/rationality-ai-zombies/
Date Read: In progress
I heard about Yudkowsky through HPMOR, and after seeing a ton of online recommendations for this online, I figured it was the best way to get a well-rounded introduction to Yudkowsky's ideas.
If we want to be sure that learning more will help us, rather than making us worse off than we were before, we need to discover and correct for biases in our data.
Like statistical biases, cognitive biases can distort our view of reality, they can’t always be fixed by just gathering more data, and their effects can add up over time.
Psychologists who work on dual process theories distinguish the brain’s “System 1” processes (fast, implicit, associative, automatic cognition) from its “System 2” processes (slow, explicit, intellectual, controlled cognition)
Same as the two-system theory of consciousness Faris mentions in “Paid Attention”
The representativeness heuristic, for example, is our tendency to assess phenomena by how representative they seem of various categories
cognitive heuristics— rough shortcuts that get the right answer often, but not all the time
To solve problems, our brains have evolved to employ cognitive heuristics— rough shortcuts that get the right answer often, but not all the time. Cognitive biases arise when the corners cut by these heuristics result in a relatively consistent and discrete mistake
Studying biases can in fact make you more vulnerable to overconfidence and confirmation bias, as you come to see the influence of cognitive biases all around you—in everyone but yourself. And the bias blind spot, unlike many biases, is especially severe among people who are especially intelligent, thoughtful, and open-minded
People vary in how strongly they exhibit different biases, so there should be a host of yet-unknown ways to influence how biased we are
Influencing people’s bias sounds a lot like propaganda depending on your motivations.
The approach to debiasing in Rationality: From AI to Zombies is to communicate a systematic understanding of why good reasoning works, and of how the brain falls short of it
As a consequence, it might be necessary to distinguish between experience and expertise, with expertise meaning “the development of a schematic principle that involves conceptual understanding of the problem,” which in turn enables the decision maker to recognize particular biases. However, using expertise as countermeasure requires more than just being familiar with the situational content or being an expert in a particular domain. It requires that one fully understand the underlying rationale of the respective bias, is able to spot it in the particular setting, and also has the appropriate tools at hand to counteract the bias