Random Clusters

Random Clusters

The human mind is not good at randomness. The human mind is good at identifying and seeing patterns. The mind is so good at patter recognition and so bad at randomness that we will often perceive a pattern in a situation where no pattern exists. We have trouble accepting that statistics are messy and don’t always follow a set pattern that we can observe and understand.
 
 
Steven Pinker points this out in his book The Better Angels of Our Nature and I think it is an important point to keep in mind. He writes, “events that occur at random will seem to come in clusters, because it would take a nonrandom process to space them out.” This problem of our perception of randomness comes into play when our music streaming apps shuffle songs at random. If we have a large library of our favorite songs to chose from, some of those songs will be by the same artist. If we hear two or more songs from the artist back to back, we will assume there is some sort of problem with the random shuffling of the streaming service. We should expect to naturally get clusters of songs by the same artist or even off the same album, but it doesn’t feel random to us when it happens. To solve this problem, music streaming services deliberately add algorithms that stop songs from the same artist from appearing in clusters. This makes the shuffle less random overall, but makes the perception of the shuffle feel more random to us.
 
 
Pinker uses lightning to describe the process in more detail. “Lightning strikes are an example of what statisticians call a Poisson process,” he writes. “In a Poisson process, events occur continuously, randomly, and independently of one another. … in a Poisson process the intervals between events are distributed exponentially: there are lots of short intervals and fewer and fewer of them as they get longer and longer.”
 
 
To understand a Poisson process, we have to be able to understand having many independent events and we have to shift our perspective to look at the space between events as variables, not just look at the events themselves as variables. Both of these things are hard to do. It is hard to look at a basketball team and think that their next shot is independent of the previous shot (this is largely true). It is hard to look at customer complaints and see them as independent (also largely true), and it is hard to look at the history of human wars and think that events are also independent (Pinker shows this to be largely true as well). We tend to see events as connected even when they are not, a perspective error on our part. We also look just at the events, not at the time between the events. If we think that the time between the events will have a statistical dispersion that we can analyze, it shifts our focus away from the actual event itself. We can then think about what caused the pause and not what caused the even. This helps us see the independence between events and helps us see the statistics between both the event and the subsequent pause between the next event. Shifting our focus in this way can help us see Poisson distributions, random distributions with clusters, and patterns that we might miss or misinterpret. 
 
 
All of these factors are part of probability and statistics which our minds have trouble with. We like to see patterns and think causally. We don’t like to see larger complex perspective shifting statistics. We don’t like to think that there is a statistical probability without an easily distinguishable pattern that we can attribute to specific causal structures. However, as lightning and other Poisson processes show us, sometimes the statistical perspective is the better perspective to have, and sometimes our brains run amok with finding patterns that do not exist in random clusters.
The Location of Human Knowledge - Yuval Noah Harari Sapiens - Joe Abittan

The Location of Human Knowledge

“The average forager had wider, deeper, and more varied knowledge of her immediate surroundings than most of her modern descendants,” writes Yuval Noah Harari in Sapiens. Individual human hunter-gatherers had to know a lot about their environment, and they were not learning from text books and schools. They were learning by trial and error, by being shown what was edible and what was not edible older members of their tribe, and they had to develop a plethora of skills in order to do all the things necessary for survival.
Humans today are not very likely to be able to weave baskets from reeds (as much as we joke about basket weaving courses for college athletes). They also likely can’t sharpen a flint arrowhead, don’t know what animals are around their location and how to hunt them, and don’t know what wild plants are helpful or harmful. Individually, modern humans don’t seem to have the same regionalized knowledge as the ancient humans that came before them. But today we definitely know far more than the humans before us as a whole.
The difference, Harari explains, is that the location of human knowledge has changed. We no longer all hold the same helpful regional information in our heads. Instead, we have a collective knowledge spread across humanity. Each of us is an expert in our own little niche. For example, I studied political science and Spanish. I know a bit about theories of policy processes, such as the Multiple Streams Framework, and I know a bit about medieval literature from the Iberian Peninsula. On the other hand, I don’t really know much about how nuclear submarines work, how my city’s sewage system works, and I don’t really know what migratory birds can be found in the area during the spring versus the fall. Someone else in the collective of human knowledge is an expert in each of those things.
“The human collective,” writes Harari, “knows far more today than did the ancient bands. But at the individual level, ancient foragers were the most knowledgeable and skillful people in history.” This quote may be a broad overgeneralization, but it is nevertheless interesting and thought provoking. Some of us are incredibly skilled with a variety of things and have a great deal of knowledge about much of the world around us. Others don’t seem to have as much skill, and know more about celebrities than we do about what is happening in the world. Overall, the important thing to consider is that the modern world has seen a shift in the distribution of knowledge. We don’t all have to hold information about our immediate surroundings in our heads, and we don’t all have to be able to produce the things necessary for our survival. Only some humans need to know those things and have the skills to produce the basic necessities for life. The rest of us can then go off and explore different areas and learn different things, constantly increasing the collective human knowledge and skill base, even if we individually seem more narrowly skilled and less immediately knowledgeable compared with our ancient forager ancestors.