Tag: Causality

The Poisson Nature of War

The Poisson Nature of War

Posted on by jabittan
When we look back at history and explain why the world is the way it is, we rarely attribute specific causes and results to chance. We don’t say that a group of terrorists happened to choose to fly planes into the World Trade Center on 9/11. We don’t say that a new technology happened to come along to advance the economy. And we don’t say that a war between two countries happened to break out. But in some ways it would make more sense for us to look back at history and view events as chance contingencies. Steven Pinker argues that we should do this when we look back at history’s wars.
 
 
Specifically, when we take a statistical view of the history of war, we see that wars follow a Poisson distribution. When we record all the wars in human history we see lots of short intervals between wars and fewer long gaps between wars. When we look back at history and try to explain wars from a causal standpoint, we don’t look at the pauses and gaps between wars. We look instead at the triggering factors and buildup to war. But what the statistics argue is that we are often seeing causal patterns and narratives where none truly exist. Pinker writes, “the Poisson nature of war undermines historical narratives that see constellations in illusory clusters.”
 
 
We see one war as leading to another war. We see a large war as making people weary of fighting and death, ultimately leading to a large period of peace. We create narratives which explain the patterns we perceive, even if the patterns are not really there. Pinker continues,
 
 
“Statistical thinking, particularly an awareness of the cluster illusion, suggests that we are apt to exaggerate the narrative coherence of this history – to think that what did happen must have happened because of historical forces like cycles, crescendos, and collision courses.”
 
 
We don’t like to attribute history to chance events. We don’t like to attribute historical decisions to randomness. We like cohesive narratives that weave together multiple threads of history, even when examples of random individual choices or chance events shape the historical threads and narratives. Statistics shows us that the patterns we see are not always real, but that doesn’t stop us from trying to pull patterns out of the randomness or the Poisson distribution of history anyway.
Random Clusters

Random Clusters

Posted on by jabittan
The human mind is not good at randomness. The human mind is good at identifying and seeing patterns. The mind is so good at patter recognition and so bad at randomness that we will often perceive a pattern in a situation where no pattern exists. We have trouble accepting that statistics are messy and don’t always follow a set pattern that we can observe and understand.
 
 
Steven Pinker points this out in his book The Better Angels of Our Nature and I think it is an important point to keep in mind. He writes, “events that occur at random will seem to come in clusters, because it would take a nonrandom process to space them out.” This problem of our perception of randomness comes into play when our music streaming apps shuffle songs at random. If we have a large library of our favorite songs to chose from, some of those songs will be by the same artist. If we hear two or more songs from the artist back to back, we will assume there is some sort of problem with the random shuffling of the streaming service. We should expect to naturally get clusters of songs by the same artist or even off the same album, but it doesn’t feel random to us when it happens. To solve this problem, music streaming services deliberately add algorithms that stop songs from the same artist from appearing in clusters. This makes the shuffle less random overall, but makes the perception of the shuffle feel more random to us.
 
 
Pinker uses lightning to describe the process in more detail. “Lightning strikes are an example of what statisticians call a Poisson process,” he writes. “In a Poisson process, events occur continuously, randomly, and independently of one another. … in a Poisson process the intervals between events are distributed exponentially: there are lots of short intervals and fewer and fewer of them as they get longer and longer.”
 
 
To understand a Poisson process, we have to be able to understand having many independent events and we have to shift our perspective to look at the space between events as variables, not just look at the events themselves as variables. Both of these things are hard to do. It is hard to look at a basketball team and think that their next shot is independent of the previous shot (this is largely true). It is hard to look at customer complaints and see them as independent (also largely true), and it is hard to look at the history of human wars and think that events are also independent (Pinker shows this to be largely true as well). We tend to see events as connected even when they are not, a perspective error on our part. We also look just at the events, not at the time between the events. If we think that the time between the events will have a statistical dispersion that we can analyze, it shifts our focus away from the actual event itself. We can then think about what caused the pause and not what caused the even. This helps us see the independence between events and helps us see the statistics between both the event and the subsequent pause between the next event. Shifting our focus in this way can help us see Poisson distributions, random distributions with clusters, and patterns that we might miss or misinterpret. 
 
 
All of these factors are part of probability and statistics which our minds have trouble with. We like to see patterns and think causally. We don’t like to see larger complex perspective shifting statistics. We don’t like to think that there is a statistical probability without an easily distinguishable pattern that we can attribute to specific causal structures. However, as lightning and other Poisson processes show us, sometimes the statistical perspective is the better perspective to have, and sometimes our brains run amok with finding patterns that do not exist in random clusters.
The Complexities of Society

The Complexities of Society

Posted on by jabittan
I have a hard time debating and arguing with friends about how to think about society. A large reason why is because, at best, I often find myself making the argument of, “well, maybe?”  Politics is a never ending attempt to answer the question of who gets what and when. We have scarce resources like money, roads and infrastructure, and influence and fame. These things are distributed across individuals with deliberate decisions and sometimes seemingly by random chance. Occasionally we step in to try to change these allocations, to provide greater rewards and incentives for those who pursue certain resources and goals over others, and punish those who deviate from courses we find appropriate. But figuring out how people will react to any given decision and figuring out which levers will lead to which outcomes is nearly impossible. I almost always find myself unsure exactly that the changes people advocate for will really have the desired impact or that the problem they identify is really caused by the root cause they suggest. I often find myself saying, “well, maybe” but having a hard time convincing others that their thoughts should be less certain.
 
 
In his book The Better Angels of Our Nature, Steven Pinker discusses the complexities of society when writing about how hard it is to identify a single factor that has lead people to become less violent over time. Especially in WEIRD societies, there is a lot of evidence to demonstrate that people are less violent today than they used to be, but it is hard to point to a single (or even a few) key factor and explain how it (they) reduced human violence. As Pinker writes, “a society is an organic system that develops spontaneously, governed by myriad interactions and adjustments that no human mind can pretend to understand.”
 
 
The best social science experiments that we can develop and the best models from social science only manager to explain about 40% of the variance that we observe across societies. We cannot singularly point to racism, inequality, or the percent of high school graduates and understand a given social outcome. We can see correlations, but rarely do we see a correlation that explains anywhere close to 50% of the differences we observe between desired and undesired social outcomes. We are unable to point to a given factor (or even a handful of given factors) and confidently say that we have identified the most important or the clear driving factor(s) that determine(s) whether someone is a success or a failure, whether a society is peacefully democratic or violently autocratic, or whether a society’s economy will boom or bust.
 
 
This is why I am so frequently stuck with, “well, maybe,” as a response to so man of my friend’s arguments. When a friend or family member is convinced that people need to change one thing in order to make the world a better place I remember that the best social science models explain less than half the variance. So pointing to a single factor and claiming that the world would be dramatically better if we changed that factor doesn’t feel convincing to me. Maybe it would have an impact, but maybe it wouldn’t. The complexities of society prevent us from ever being certain that a single change or a single decision will ever have the intended outcome we expect or hope for.
Violence and Convenient Mysticism

Violence and Convenient Mysticism

Posted on by jabittan
Mysticism in the United States doesn’t really feel like it lends itself to violence. When we think of mystics, we probably think of someone close to a shaman, or maybe a modern mystic whose aesthetic is very homeopathic. Mystics don’t seem like they would be the most violent people today, but in the past, mysticism was a convenient motivating factor for violence.
 
 
In his book The Better Angels of Our Nature, Steven Pinker describes the way that mysticism lends itself to violence by writing, “the brain has evolved to ferret out hidden powers in nature, including those that no one can see. Once you start rummaging around in the realm of the unverifiable there is considerable room for creativity, and accusations of sorcery are often blended with self-serving motives.”
 
 
There are two important factors to recognize in this quote from Pinker, and both are often overlooked and misunderstood. First, our brains look for causal links between events. They are very good and very natural at thinking causally and pinpointing causation, however, as Daniel Kahneman wrote in Thinking Fast and Slow, the brain can often fall into cognitive fallacies and misattribute causation. Mystical thinking is a result of misplaced causal reasoning. It is important that we recognize that our brains can see causation that doesn’t truly exist and lead us to wrong conclusions.
 
 
The second important factor that we often manage to overlook is our own self-interest. As Kevin Simler and Robin Hanson explain in The Elephant in the Brain, our self-interest plays a much larger role in much of our decision-making and behavior than we like to admit. When combined with mysticism, self-interest can be dangerous.
 
 
If you have an enemy who boasts that they are special and offers mystical explanations for their special powers, then it suddenly becomes convenient to justify violence against your enemy. You don’t need actual proof of any wrong doing, you don’t need actual proof of their danger to society, you just need to convince others that their mystical powers could be dangerous, and you now have a convenient excuse for disposing of those who you dislike. You can promote your own self-interest without regard to reality if you can harness the power of mystical thinking.
 
 
Pinker explains that the world is becoming a more peaceful place in part because mystical thinking is moving to smaller and smaller corners of the world. Legal systems don’t recognize mystical explanations and justifications for behaviors and crimes. Empirical facts and verifiable evidence has superseded mysticism in our evaluations and judgments of crime and the use of violence. By moving beyond mysticism we have created systems, structures, and institutions that foster more peace and less violence among groups of people.
Level Two Chaos & History

Level Two Chaos and History

Posted on by jabittan
“It is an iron law of history that what looks inevitable in hindsight was far from obvious at the time,” writes Yuval Noah Harari in his book Sapiens. History seems pretty clear when we look backwards. It is certainly complex, but whether it is our own lives, a sports season, or the rise and fall of an empire, we generally do a pretty good job of creating a compelling narrative to explain how history unfolded and why certain events took place. But these narratives create what we call the hindsight bias, where past events (in this case the course of human history) appear nearly deterministic. The reality is that small changes could shape the course of history in dramatic ways, and that the future is never clear at any point – as our current uncertainty about social media, climate change, and political polarization demonstrate. Harari continues, “in a few decades, people will look back and think that the answers to all of these questions were obvious,” but for us, the right answers are certainly far from obvious.
 
 
History, for human beings, is shaped by the innumerable decisions that we make every day. The course of history is not deterministic, but is instead chaotic. Harari argues that history is a level two chaotic system and writes, “chaotic systems come in two shapes. Level one chaos is chaos that does not react to predictions about it … level two chaos is chaos that reacts to predictions about it, and therefore can never be predicted accurately.”
 
 
The weather is a level one chaotic system because it doesn’t respond (on a day to day basis) to our predictions. Whether our computer models suggest a 1%, 45%, or 88% chance of rain on a given day doesn’t change what is going to happen in the atmosphere and whether it will or will not rain. Despite our sometimes magical thinking, scheduling a wedding or stating our hopes for or against certain weather patterns does not influence the actual weather.
 
 
History, is not a level one chaotic system. History is shaped by elections, the general beliefs within the public, the actions that people take, and how key actors understand risk and probability. Our predictions can greatly influence all of these areas. Predicting a landslide victory in an election can demotivate the losing side of a political divide, possibly turning what could have been a marginal victory into the predicted landslide as a sort of self-fulfilling prophecy. Reporting that general beliefs are becoming more or less prevalent among a population can influence the rate and direction of changing beliefs as people hear the predictions about belief trends (this seems like it may have happened as marijuana and gay marriage gained acceptance across the US). Our reactions to predictions can influence the final outcomes, contributing more uncertainty and chaos to the system.
 
 
We cannot predict exactly how people will react to predictions about the systems they participate in. It makes the predictions and forecasts more challenging since they have to incorporate different levels of response to various inputs. History cannot be thought of deterministically because so many small factors could have changed the outcomes, and those small changes in turn would have influenced exactly what predictions were made, in turn influencing the reactions of the people involved in the various actions of history. Our confidence in understanding history and why history played out as it did is not warranted, and is simply a fallacy.
The Hindsight Fallacy &The Hindsight Fallacy & the How & Why of History - Jim Collins Good To Great Bias the How & Why of History

The Hindsight Fallacy & the How & Why of History

Posted on by jabittan
Looking forward and making predictions and judgments on what will happen in the future is incredibly difficult, and very few people can reliably make good predictions about what will happen. But when we look to the past, almost all of us can describe what did happen. It is easy to look back and see how a series of events unfolded, to make connections between certain conditions and eventual outcomes, and to be confident that we understood why things unfolded as they did. But this confidence is misleading and is something we can reliably expect from people.
 
 
The hindsight fallacy is the term which describes our overconfidence in describing what happened in the past and determining which causal factors influenced the outcomes we observed. When the college football playoff is over this year, sports commentators will have a compelling narrative as to why the winning team was able to pull through. When the stock market makes a jump or dip in the next year, analysts will be able to look backward to connect the dots that caused the rise or fall of the market. Their explanations will be confident and narratively coherent, making the analysts and commentators sound like well reasoned individuals.
 
 
However, “every point in history is a crossroads,” writes Yuval Noah Harari. Strange and unpredictable things could happen at any time in history, and the causal factors at work are hard to determine. It is worth remembering that the best social science studies return an R value of about .4 at most (the R value is a statistical value reflecting how well the model fits reality). This means that the best social science studies we can conduct barely reflect the reality of the world. It is unlikely that any commentator, even a seasoned football announcer or stock market analyst, really understands causality well enough to be confident in what caused what, even in hindsight. Major shifts could happen because someone was in a bad mood. Unexpected windfalls could create new and somewhat random outcomes. Humans can think causally, and this helps us better understand the world, but we can also be overconfident in our causal reasoning.
 
 
Harari continues, “the better you know a particular historical period, the harder it becomes to explain why things happened one way and not another. Those who have only a superficial knowledge of a certain period tend to focus only on the possibility that was eventually realized.” What Harari is saying in this quote is that we can get very good at describing how things happened in the past, but not exactly very good at describing why. We can look at each step and each development that unfolded ahead of a terrorist attack, a win by an army, or as Harari uses for demonstration in his book, the adoption of Christianity in the Roman Empire. But we can’t always explain the exact causal pathway of each step. If we could, then we could identify the specific historical crossroads where history took one path and not another and make reasonable predictions about how the world would have looked had the alternative option been the one that history followed. But we really can’t do this. We can look back and identify factors that seemed important in the historical development, but we can’t always explain exactly why those factors were important in one situation relative to another. There is too much randomness, too much chance, and too much complexity for us to be confident in the causal pathways we see. We won’t stop thinking in a causal way, of course, but we should at least be more open to a wild range of possibilities, and less confident in our assessments of history.
 
 
One of my favorite examples of the hindsight bias in action is in Good to Great by Jim Collins. In the book published in 2001, Collins identifies 11 companies that had jumped form being good companies to great companies. One of the companies identified, Circuit City, was out of business before Collins published his subsequent book. Another, Wells Fargo, is now one of the most hated companies in the United States.  A third, Fannie Mae, was at the center of the 2008 financial crisis, and a fourth, Gillette, was purchased by P&G and is no longer an independent entity. A quick search suggests that the companies in the Good to Great portfolio have underperformed the market since the books publication. It is likely that the success of the 11 companies included a substantial amount of randomness, which Collins and his team failed to incorporate in their analysis. Hindsight bias was at play in the selection of the 11 companies and the explanation for why they had such substantial growth in the period that Collins explored. 
The Chicken and Egg Problem of Mental Health Issues and Homelessness

The Chicken and Egg Problem of Mental Health Issues and Homelessness

Posted on by jabittan
I recently wrote about the challenges of mental health and homelessness, and how sometimes homelessness itself causes mental health disorders in individuals. In general, we assume that people become homeless because they have mental health disorders, not that homelessness causes people to have mental health disorders. Elliot Liebow looks at the issue with a much more careful eye in his book Tell Them Who I Am. Liebow writes,
“Mental health problems and homelessness stood in a chicken-and-egg relationship to one another. Homelessness was seen as a cause of mental health problems just as often as mental health problems were seen as a cause of homelessness. Indeed, it was not uncommon for the women to use their homelessness to explain their sometimes ungenerous behavior.”
Being homeless is stressful. Homeless individuals cannot maintain many basic possessions. They face uncertainty with meals, where they will sleep, how they will go to the bathroom, and whether they will be in danger from weather, animals, or other people. They don’t have a lot of people, besides other homeless individuals, to speak to and get support from. Liebow writes about the ways this stress can boil over for the homeless, and how sometimes the women he profiled for his book would lash out or act irrationally and blame it on the stress of homelessness. With no safe places, shelters that impose rules and ask prying questions, and with little to keep one’s mind engaged and hopeful for a better future, it is not hard to imagine how the stress of homelessness could become overwhelming and spark mental health problems.
At the end of the day, however, this chicken-and-egg relationship should be encouraging. Not all the people who end up homeless have mental health problems – at least not when they initially experience homelessness. This means that early interventions and support can help keep people from developing worse mental health problems that prevent them from rejoining society. It also means that providing stable housing and shelter can help reduce some of the mental health problems that the homeless face, easing their potential reintegration into society. We can also look at the relationship between mental health and homelessness to see that providing more mental healthcare to people currently working and maintaining jobs may support them and keep them from becoming homeless. Preventative mental health care can prevent stress and anxiety worsening to drive someone into homeless where their mental health could further deteriorate. The key idea is that we shouldn’t dismiss the homeless as helpless crazy people. We should see investments in mental healthcare at all levels of society as a beneficial preventative measure to reduce and address homelessness.
Data Mining is a First Step

Data Mining is a First Step

Posted on by jabittan
From big tech companies, sci-fi movies, and policy entrepreneurs data mining is presented as a solution to many of our problems. With traffic apps collecting mountains of movement data, governments collecting vast amounts of tax data, and heath-tech companies collecting data for every step we take, the promise of data mining is that our sci-fi fantasies will be realized here on earth in the coming years. However, data mining is only a first step on a long road to the development of real knowledge that will make our world a better place. The data alone is interesting and our computing power to work with big data is astounding, but data mining can’t give us answers, only interesting correlations and statistics.
In The Book of Why Judea Pearl writes:
“It’s easy to understand why some people would see data mining as the finish rather than the first step. It promises a solution using available technology. It saves us, as well as future machines, the work of having to consider and articulate substantive assumptions about how the world operates. In some fields our knowledge may be in such an embryonic state that we have no clue how to begin drawing a model of the world. But big data will not solve this problem. The most important part of the answer must come from such a model, whether sketched by us or hypothesized and fine-tuned by machines.”
Big data can give us insights and help us identify unexpected correlations and associations, but identifying unexpected correlations and associations doesn’t actually tell us what is causing the observations we make. The messaging of massive data mining is that we will suddenly understand the world and make it a better place. The reality is that we have to develop hypotheses about how the world works based on causal understandings of the interactions between various factors of reality. This is crucial or we won’t be able to take meaningful action based what comes from our data mining. Without developing causal hypotheses we cannot experiment with associations and continue to learn, we can only observe what correlations come from big data. Using the vast amounts of data we are collecting is important, but we have to have a goal to work toward and a causal hypothesis of how we can reach that goal in order for data mining to be meaningful.
Complex Causation Continued

Complex Causation Continued

Posted on by jabittan
Our brains are good at interpreting and detecting causal structures, but often, the real causal structures at play are more complicated than what we can easily see. A causal chain may include a mediator, such as citrus fruit providing vitamin C to prevent scurvy. A causal chain may have a complex mediator interaction, as in the example of my last post where a drug leads to the body creating an enzyme that then works with the drug to be effective. Additionally, causal chains can be long-term affairs.
In The Book of Why Judea Pearl discusses long-term causal chains writing, “how can you sort out the causal effect of treatment when it may occur in many stages and the intermediate variables (which you might want to use as controls) depend on earlier stages of treatment?”
This is an important question within medicine and occupational safety. Pearl writes about the fact that factory workers are often exposed to chemicals over a long period, not just in a single instance. If it was repeated exposure to chemicals that caused cancer or another disease, how do you pin that on the individual exposures themselves? Was the individual safe with 50 exposures but as soon as a 51st exposure occurred the individual developed a cancer? Long-term exposure to chemicals and an increased cancer risk seems pretty obvious to us, but the actual causal mechanism in this situation is a bit hazy.
The same can apply in the other direction within the field of medicine. Some cancer drugs or immune therapy treatments work for a long time, stop working, or require changes in combinations based on how disease has progressed or how other side effects have manifested. Additionally, as we have all learned over the past year with vaccines, some medical combinations work better with boosters or time delayed components. Thinking about causality in these kinds of situations is difficult because the differing time scopes and combinations make it hard to understand exactly what is affecting what and when. I don’t have any deep answers or insights into these questions, but simply highlight them to again demonstrate complex causation and how much work our minds must do to fully understand a causal chain.
Complex Causation

Complex Causation

Posted on by jabittan
In linear causal models the total effect of an action is equal to the direct effect of that action and its indirect effect. We can think of an oversimplified anti-tobacco public health campaign to conceptualize this equation. A campaign could be developed to use famous celebrities in advertisements against smoking. This approach may have a direct effect on teen smoking rates if teens see the advertisements and decide not to smoke as a result of the influential messaging from their favorite celebrity. This approach may also have indirect effects. Imagine a teen who didn’t see the advertising, but their best friend did see it. If their best friend was influenced, then they may adopt their friend’s anti-smoking stance. This would be an indirect effect of the advertising campaign in the positive direction. The total effect of the campaign would then be the kids who were directly deterred from smoking combined with those who didn’t smoke because their friends were deterred.
However, linear causal models don’t capture all of the complexity that can exist within causal models. As Judea Pearl explains in The book of Why, there can be complex causal models where the equation that I started this post with doesn’t hold. Pearl uses a drug used to treat a disease as an example of a situation where the direct effect and indirect effect of a drug don’t equal the total effect. He says that in situations where a drug causes the body to release an enzyme that then combines with the drug to treat a disease, we have to think beyond the equation above. In this case he writes, “the total effect is positive but the direct and indirect effects are zero.”
The drug itself doesn’t do anything to combat the disease. It stimulates the release of an enzyme and without that enzyme the drug is ineffective against the disease. The enzyme also doesn’t have a direct effect on the disease. The enzyme is only useful when combined with the drug, so there is no indirect effect that can be measured as a result of the original drug being introduced. The effect is mediated between the interaction of both the drug and enzyme together. In the model Pearl shows us, there is only the mediating effect, not a direct or indirect effect.
This model helps us see just how complicated ideas and conceptions of causation are. Most of the time we think about direct effects, and we don’t always get to thinking about indirect effects combined with direct effects. Good scientific studies are able to capture the direct and indirect effects, but to truly understand causation today, we have to be able to include mediating effects in complex causation models like the one Pearl describes.