Laboratory Proof

Laboratory Proof

“If the standard of laboratory proof had been applied to scurvy,” writes Judea Pearl in The Book of Why, “then sailors would have continued dying right up until the 1930’s, because until the discovery of vitamin C, there was no laboratory proof that citrus fruits prevented scurvy.” Pearl’s quote shows that high scientific standards for definitive and exact causality are not always for the greater good. Sometimes modern science will spurn clear statistical relationships and evidence because statistical relationships alone cannot be counted on as concrete causal evidence. A clear answer will not be given because some marginal unknowns may still exist, and this can have its own costs.
Sailors did not know why or how citrus fruits prevented scurvy, but observations demonstrated that citrus fruits managed to prevent scurvy. There was no clear understanding of what scurvy was or why citrus fruits were helpful, but it was commonly understood that a causal relationship existed. People acted on these observations and lives were saved.
On two episodes, the Don’t Panic Geocast has talked about journal articles in the British Medical Journal that make the same point as Pearl. As a critique of the need for randomized controlled trials, the two journal articles highlight the troubling reality that there have not been any randomized controlled trials on the effectiveness of parachute usage when jumping from airplanes. The articles are hilarious and clearly satirical, but ultimately come to the same point that Pearl does with the quote above – laboratory proof is not always necessary, practical, or reasonable when lives are on the line.
Pearl argues that we can rely on our abilities to identify causality even without laboratory proof when we have sufficient statistical analysis and understanding of relationships. Statisticians always tell us that correlation is not causation and that observational studies are not sufficient to determine causality, yet the citrus fruit and parachute examples highlight that this mindset is not always appropriate. Sometimes more realistic and common sense understanding of causation – even if supported with just correlational relationships and statistics – are more important than laboratory proof.
Nature Answers the Questions We Pose

Nature Answers the Questions We Pose

I have not read A Hitchhiker’s Guide to the Galaxy, but I know there is a point where a character asks what’s the meaning of life, the universe, and everything, and receives a response of 42. The answer was certainly not what anyone was expecting, but it was an answer. Much of science is like the answer 42. We ask grand questions of nature and receive answers we didn’t quite expect and can’t always make sense of.
In The Book of Why Judea Pearl writes, “Nature is like a genie that answers exactly the question we pose, not necessarily the one we intend to ask.” We learn by making observations about the world. We can make predictions about what we think will happen given certain conditions and we can develop and test hypotheses, but the answers we get may not be answers to the questions we intended to ask. I frequently listen to the Don’t Panic Geocast and the hosts often talk about scientific studies that go awry because of some unexpected interaction between lights, between an experimental set-up and the sun, or because an animal happened to have messed with equipment in the field. Real results are generated, but they don’t always mean what we think they do on first look. The hosts have a frequent line that, “any instrument can be a thermometer,” to note how subtle changes in temperature can cause misleading noise in the data.
Pearl’s quote is meant to demonstrate how challenging science can be and why so much of science has taken such a long time to develop. Humans have often thought they were receiving answers to the questions they were asking, only to find out that nature was answering a different question, not the one the scientists thought they had asked. Pearl states that randomness has been one of the ways that we have gotten past nature, but writes about how counter-intuitive randomized controlled trials were when first developed. No one realized that the right question had to be asked through experimental set-ups that involved randomness. On the benefits of randomness he writes, “first, it eliminates confounder bias (it asks Nature the right question). Second, it enables the researcher to quantify his uncertainty.”
In the book, Pearl takes observations and statistical methods combined with causal insights to a level that is honestly beyond my comprehension. What is important to note, however, is that nature is not obligated to answer the questions we intend to ask. It answers questions exactly as we pose them, influenced by seemingly irrelevant factors in our experimental design. The first answer we get may not be very reliable, but randomness and statistical methods, combined as Pearl would advocate, with a solid understanding of causality, can helps us better pose our questions to nature, to be more confident that the responses we get answer the questions we meant to ask.
The Cigarette Wars - Judea Pearl The Book of Why - Joe Abittan

The Cigarette Wars

Recently I have been writing about Judea Pearl’s The Book of Why, in which Pearl asks if our reliance on statistics and our adherence to the idea correlation is not causation has gone too far in science. For most people, especially students getting into science and those who have studied politics, reiterating the idea that correlation does not imply causation is important. There are plenty of ways to misinterpret data and there is no shortage of individuals and interest groups who would love to have an official scientist improperly assign causation to a correlation for their own political and economic gain. However, Pearl uses the cigarette wars to show us that failing to acknowledge that correlations can imply causation can also be dangerous.
“The cigarette wars were science’s first confrontation with organized denialism, and no one was prepared,” writes Pearl. For decades there was ample evidence from different fields and different approaches linking cigarette smoking to cancer. However, it isn’t the case that every single person who smokes a cigarette gets cancer. We all know people who smoked for 30 years, and seem to have great lungs. Sadly, we also all know people who developed lung cancer but never smoked. The causation between cigarettes is not a perfect 1:1 correlation with lung cancer, and tobacco companies jumped on this fact.
For years, it was abundantly clear that smoking greatly increased the risk of lung cancer, but no one was willing to say that smoking caused lung cancer, because powerful interest groups aligned against the idea and conditioned policy-makers and the public to believe that in the case of smoking and lung cancer, correlation was not causation. The evidence was obvious, but built on statistical information and the organized denial was stronger. Who was to say if people more susceptible to lung cancer were also more susceptible to start smoking in the first place? Arguments such as these hindered people’s willingness to adopt the clear causal picture that cigarettes caused cancer. People hid behind a possibility that the overwhelming evidence was wrong.
Today we are in a similar situation with climate change and other issues. It is clear that statistics cannot give us a 100% certain answer to a causal question, and it is true that correlation is not necessarily a sign of causation, but at a certain point we have to accept when the evidence is overwhelming. We have to accept when causal models that are not 100% proven have overwhelming support. We have to be able to make decisions without being derailed by organized denialism that seizes on the fact that correlation does not imply causation, just to create doubt and confusion. Pearl’s warning is that failing to be better with how we think about and understand causality can have real consequences (lung cancer in the cigarette wars, and devastating climate impacts today), and that we should take those consequences seriously when we look at the statistics and data that helps us understand our world.
The Fundamental Nature of Cause and Effect

The Fundamental Nature of Cause and Effect

In my undergraduate and graduate studies I had a few statistics classes and I remember the challenge of learning probability. Probability, odds, and statistics are not always easy to understand and interpret. There are some concepts that are pretty straightforward, and others that seem to contradict what we would expect if we had not gone through the math and if we had not studied the concepts in depth. To contrast the difficult and sometimes counter-intuitive nature of statistics, we can think about causality, which is a challenging concept, but unlike statistics, is something we are able to intuit from very young age.
In The Book of Why Judea Pearl writes, “In both a cognitive and a philosophical sense, the idea of cause and effect is much more fundamental than probability. We begin learning causes and effects before we understand language and before we understand mathematics.”
As Pearl explains, we see causality naturally and experience causality as we move through our lives. From a young child who learns that if they cry they receive attention to a nuclear physicist who learns what happens when two atoms collide at high energy levels, our minds are constantly looking at the world and looking for causes. It begins by making observations of phenomena around us and continues as we predict what outcomes would happen based on certain system inputs. Eventually, our minds reach a point where we can understand why our predictions are accurate or inaccurate, and we can imagine new ways to bring about certain outcomes. Even if we cannot explain all of this, we can still understand causation at a fundamental and intuitive level.
However, many of us deny that we can see and understand the world in a causal way. I am personally guilty of thinking in a purely statistical way and ignoring the causal. The classes I took in college helped me understand statistics and probability, but also told me not to trust my intuitive causal thinking. Books like Kahneman’s Thinking Fast and Slow cemented this mindset for me. Rationality, we believe, requires that we think statistically and discount our intuitions for fear of bias. Modern science says we can only trust evidence when it is backed by randomized controlled trials and directs us to think of the world through correlations and statistical relationships, not through a lens of causality.
Pearl pushes back against this notion. By arguing that causality is fundamental to the human mind, he implies that our causal reasoning can and should be trusted. Throughout the book he demonstrates that a purely statistical way of thinking leaves us falling short of the knowledge we really need to improve the world. He demonstrates that complex tactics to remove variables from equations in statistical methods are often unnecessary, and that we can accept the results of experiments and interventions even when they are not fully randomized controlled trials.  For much of human history our causal thinking nature has lead us astray, but I think that Pearl argues that we have overcorrected in modern statistics and science, and that we need to return to our causal roots to move forward and solve problems that statistics tells us are impossible to solve.
Predictions & Explanations

Predictions & Explanations

The human mind has incredible predictive abilities, but our explanatory abilities do not always turn out to be as equally incredible. Prediction is relatively easy when compared to explanation. Animals can predict where a food source will be without being able to explain how it got there. For most of human history our ancestors were able to predict that the sun would rise the next day without having any way of explaining why it would rise. Computer programs today can predict our next move in chess but few can explain their prediction or why we would make the choice that was predicted.
As Judea Pearl writes in The Book of Why, “Good predictions need not have good explanations. The owl can be a good hunter without understanding why the rat always goes from point A to point B.” Prediction is possible with statistics and good observations. With a large enough database, we can make a prediction about what percentage of individuals will have negative reactions to medications, we can predict when a traffic jam will occur, and we can predict how an animal will behave. What is harder, according to Pearl, is moving to the stage where we describe why we observe the relationships that statistics reveal.
Statistics alone cannot tell us why particular patterns emerge. Statistics cannot identify causal structures. As a result, we continually tell ourselves that correlation is not causation and that we can only determine what relationships are truly causal through randomized controlled trials. Pearl would argue that this is incorrect, and he would argue that this idea results from the fact that statistics is trying to answer a completely different question than causation. Approaching statistical questions from a causal lens may lead to inaccurate interpretations of data or “p-hacking” an academic term used to describe efforts to get the statistical results you wanted to see. The key is not hunting for causation within statistics, but understanding causation and supporting it through evidence uncovered via statistics.
Seeing the difference between causation and statistics is helpful when thinking about the world. Being stuck without a way to see and determine causation leads to situations like tobacco companies claiming that cigarettes don’t cause cancer or oil and gas companies claiming that humans don’t contribute to global warming. Causal thinking, however, utilizes our ability to develop explanations and applies those explanations to the world. Our ability to predict different outcomes based on different interventions helps us interpret and understand the data that the world produces. We may not see the exact picture in the data, but we can understand it and use it to help us make better decisions that will lead to more accurate causal understandings over time.
Correlation and Causation - Judea Pearl - The Book of Why - Joe Abittan

Correlation and Causation

I have an XKCD comic taped to the door of my office. The comic is about the mantra of statistics, that correlation is not causation. I taped the comic to my office door because I loved learning statistics in graduate school and thinking deeply about associations and how mere correlations cannot be used to demonstrate that one thing causes another. Two events can correlate, but have nothing to do with each other, and a third thing may influence both, causing them to correlate without any causal link between the two things.
But Judea Pearl thinks that science and researchers have fallen into a trap laid out by statisticians and the infinitely repeated correlation does not imply causation mantra. Regarding this perspective of statistics he writes, “it tells us that correlation is not causation, but it does not tell us what causation is.”
Pearl seems to suggest in The Book of Why that there was a time where there was too much data, too much humans didn’t know, and too many people ready to offer incomplete assessments based on anecdote and incomplete information. From this time sprouted the idea that correlation does not imply causation. We started to see that statistics could describe relationships and that statistics could be used to pull apart entangled causal webs, identifying each individual component and assessing its contribution to a given outcome. However, as his quote shows, this approach never actually answered what causation is. It never actually told us when we can know and ascertain that a causal structure and causal mechanism is in place.
“Over and over again,” writes Pearl, “in science and in business, we see situations where mere data aren’t enough.”
To demonstrate the shortcomings of our high regard for statistics and our mantra that correlation is not causation, Pearl walks us through the congressional testimonies and trials of big tobacco companies in the United States. The data told us there was a correlation between smoking and lung cancer. There was overwhelming statistical evidence that smoking was related or associated with lung cancer, but we couldn’t attain 100% certainty just through statistics that smoking caused lung cancer. The companies themselves muddied the water with misleading studies and cherry picked results. They hid behind a veil that said that correlation was not causation, and hid behind the confusion around causation that statistics could never fully clarify.
Failing to develop a real sense of causation, failing to move beyond big data, and failing to get beyond statistical correlations can have real harms. We need to be able to recognize causation, even without relying on randomized controlled trials, and we need to be able to make decisions to save lives. The lesson of the comic taped to my door is helpful when we are trying to be scientific and accurate in our thinking, but it can also lead us astray when we fail to trust a causal structure that we can see, but can’t definitively prove via statistics.
We Bet on Technology

We Bet On Technology

I am currently reading Steven Pinker’s book Enlightenment Now and he makes a good case for being optimistic about human progress. In an age when it is popular to write about human failures, whether it is wealthy but unhappy athletes wrecking their cars, the perilous state of democracy, or impending climate doom, the responsible message always see ms to be warning about how bad things are. But Pinker argues that things are not that bad and that they are getting better. Pinker’s writing directly contradicts some earlier reading that I have done, including the writing of Gerd Gigerenzer who argues that we unwisely bet on technology to save us when we should be focused on improving statistical thinking and living with risk rather than hoping for a savior technology.
In Risk Savvy, Gigerenzer writes about the importance of statistical thinking and how we need it in order to successfully navigate an increasingly complex world. He argues that betting on technology will in some ways be a waste of money, and while I think he is correct in many ways, I think that some parts of his message are wrong. He argues that instead of betting on technology, we need to develop improved statistical understandings of risk to help us better adapt to our world and make smarter decisions with how we use and prioritize resources and attention. He writes, “In the twenty-first century Western world, we can expect to live longer than ever, meaning that cancer will become more prevalent as well. We deal with cancer like we deal with other crises: We bet on technology. … As we have seen … early detection of cancer is also of very limited benefit: It saves none or few lives while harming many.”
Gigerenzer is correct to state that to this point broad cancer screening has been of questionable use. We identify a lot of cancers that people would likely live with and that are unlikely to cause serious metastatic or life threatening disease. Treating cancers that won’t become problematic during the natural course of an individual’s life causes a lot of pain and suffering for no discernable benefit, but does this mean we shouldn’t bet on technology? I would argue that it does not, and that we can see the current mistakes we make with cancer screening and early detection as lessons to help us get to a better technological cancer detection and treatment landscape. Much of our resources directed toward cancer may be misplaced right now, but wise people like Gigerenzer can help the technology be redirected to where it can be the most beneficial. We can learn from poor decisions around treatment and diagnosis, call out the actors who profit from misinformation, uncertainty, and fear, and build a new regime that harnesses technological progress in the most efficient and effective ways. As Pinker would argue, we bet on technology because it offers real promises of an improved world. It won’t be an immediate success, and it will have red herrings and loose ends, but incrementalism is a good way to move forward, even if it is slow and feels like it is inadequate to meet the challenges we really face.
Ultimately, we should bet on technology and pursue progress to eliminate more suffering, improve knowledge and understanding, and better diagnose, treat, and understand cancer. Arguing that we haven’t done a good job so far, and that current technology and uses of technology haven’t had the life saving impact we wish they had is not a reason to abandon the pursuit. Improving our statistical thinking is critical, but betting on technology and improving statistical thinking go hand in hand and need to be developed together without prioritizing one over the other.
Positive Test Strategies

Positive Test Strategies

A real danger for us, that I don’t know how to move beyond, is positive test strategy. It is the search for evidence that confirms what we want to believe or what we think is true. When we already have an intuition about something, we look for examples that support our intuition. Looking for examples that don’t support our thought, or situations where our idea seems to fall short, is uncomfortable, and not something we are very good at. Positive test strategies are a form of motivated rationality, where we find ways to justify what we want to believe, and find ways to align our beliefs with what happens to be best for us.

 

In Thinking Fast and Slow, Daniel Kahneman writes the following, “A deliberate search for confirming evidence, known as positive test strategy, is also how System 2 tests a hypothesis. Contrary  to the rules of philosophers of science, who advise testing hypothesis by trying to refute them, people (and scientists, quite often) seek data that are likely to be compatible with the beliefs they currently hold.” 

 

In science, the best way to conduct a study is to try to refute the null hypothesis, rather than to try to support the actual hypothesis. You take a condition about the world, try to make an informed guess about why you observe what you do, and then you formulate a null hypothesis before you begin any testing. Your null hypothesis says, actually nothing is happening here after all. So you might think that teenage drivers are more likely to get in car crashes at roundabouts than regular intersections, or that crickets are more likely to eat a certain type of grass. Your null hypothesis is that teenagers do not crash at roundabouts more than typical intersections and that crickets don’t display a preference for one type of grass over another.

 

In your experimental study, instead of seeking out confirmation to show that teenagers crash more at roundabouts or that crickets prefer a certain grass, you seek to prove that there is a difference in where teenagers crash and which grass crickets prefer. In other-words, you seek to disprove the null hypothesis (that there is no difference) rather than try to prove that something specific is happening. It is a subtle difference, but it is importance. Its also important to note that good science doesn’t seek to disprove the null hypothesis in a specific direction. Good science tries to avoid positive test strategies by showing that the nothing to see here hypothesis is wrong and that there is something to see, but it could be in any direction. If scientists do want to provide more evidence that it is in a given direction, they look for stronger evidence, and less chance of random sampling error.

 

In our minds however, we don’t often do this. We start to see a pattern of behavior or outcomes, and we start searching for explanations to what we see. We come up with a hypothesis, think of more things that would fit with our hypothesis, and we find ways to explain how things align with our hypothesis. In My Big Fat Greek Wedding, this is what the character Gus does when he tries to show that all words in the world are originally Greek.

 

Normally, we identify something that would be in our personal interest or would support our group identity in a way to help raise our social status. From there, we begin to adopt hypothesis about how the world should operate that support what is in our personal interest. We then look for ways to test our hypothesis that would support it, and we avoid situations where our hypothesis could be disproven. Finding things that support what we already want to believe is comforting and relatively easy compared to identifying a null hypothesis, testing it, and then examining the results without already having a pre-determined outcome that we want to see.
Seeing Causality

Seeing Causality

In Thinking Fast and Slow Daniel Kahneman describes how a Belgian psychologist changed the way that we understand our thinking in regard to causality. The traditional thinking held that we make observations about the world and come to understand causality through repeated exposure to phenomenological events. As Kahneman writes, “[Albert] Michotte [1945] had a different idea: he argued that we see causality just as directly as we see color.”

 

The argument from Michotte is that causality is an integral part of the human psyche. We think and understand the world through a causal lens. From the time we are infants, we interpret the world causally and we can see and understand causal links and connections in the things that happen around us. It is not through repeated experience and exposure that we learn to view an event as having a cause or as being the cause of another event. It is something we have within us from the beginning.

 

“We are evidently ready from birth to have impressions of causality, which do not depend on reasoning about patterns of causation.”

 

I try to remember this idea of our intuitive and automatic causal understanding of the world when I think about science and how I should relate to science. We go through a lot of effort to make sure that we are as clear as possible with our scientific thinking. We use randomized controlled trials (RCT) to test the accuracy of our hypothesis, but sometimes, an intensely rigorous scientific study isn’t necessary for us to make changes in our behavior based on simple scientific exploration via normal causal thinking. There are some times where we can trust our causal intuition, and without having to rely on an RCT for evidence. I don’t know where to draw the line between causal inferences that we can accept and those that need an RCT, but through honest self-awareness and reflection, we should be able to identify times when our causal interpretations demonstrate validity and are reasonably well insulated from our own self-interests.

 

The Don’t Panic Geocast has discussed two academic journal articles on the effectiveness of parachutes for preventing death when falling from an aircraft during the Fun Paper Friday segment of two episodes. The two papers, both published in the British Medical Journal, are satirical, but demonstrate an important point. We don’t need to conduct an RCT to determine whether using a parachute when jumping from a plane will be more effective at helping us survive the fall than not using a backpack. It is an extreme example, but it demonstrates that our minds can see and understand causality without always needing an experiment to confirm a causal link. In a more consequential example, we can trust our brains when they observe that smoking cigarettes has negative health consequences including increased likelihood of an individual developing lung cancer. An RCT to determine the exact nature and frequency of cancer development in smokers would certainly be helpful in building our scientific knowledge, but the scientific consensus around smoking and cancer should have been accepted much more readily than what it was. An RCT in this example would take years and would potentially be unethical or impossible. Tobacco companies obfuscated the science by taking advantage of the fact that an RCT in this case couldn’t be performed, and we failed to accept the causal link that our brains could see, but could not prove as definitively as we can prove something with an RCT. Nevertheless, we should have trusted our causal thinking brains, and accepted the intuitive answer.

 

We can’t always trust the causal conclusions that our mind reaches, but there are times where we should acknowledge that our brains think causally, and accept that the causal links that we intuit are accurate.
Familiarity vs Truth

Familiarity vs Truth

People who wish to spread disinformation don’t have to try very hard to get people to believe that what they are saying is true, or that their BS at least has some element of truth to it. All it takes, is frequent repetition. “A reliable way to make people believe in falsehoods,” writes Daniel Kahneman in his book Thinking Fast and Slow, “is frequent repetition, because familiarity is not easily distinguished from truth.”

 

Having accurate and correct representations of the world feels important to me. I really love science. I listen to lots of science based podcasts, love sciency discussions with family members and friends, and enjoy reading science books. By accurately understanding how the world operates, by seeking to better understand the truth of the universe, and by developing better models and systems to represent the way nature works, I believe we can find a better future. I try not to fall unthinkingly into techno-utopianism thinking, but I do think that having accurate beliefs and understandings are important for improving the lives of people across the planet.

 

Unfortunately, for many people, I don’t think that accurate and correct understandings of the worlds have such high priority in their lives. I fear that religion and science may be incompatible or at odds with each other, and there may be a willingness to accept inaccurate science or beliefs to support religious doctrine. I also fear that people in extractive industries may discount science, preferring to hold an inaccurate belief that supports their ability to profit through their extractive practices. Additionally, the findings, conclusions, and recommendations from science may just be scary for many ordinary people, and accepting what science says might be inconvenient or might require changes in lifestyles that people don’t want to make. When we are in this situations, it isn’t hard to imagine why we might turn away from scientific consensus in favor of something comfortable but wrong.

 

And this is where accurate representations of the universe face and uphill battle. Inaccuracies don’t need to be convincing, don’t really need to sound plausible, and don’t need to to come from credible authorities. They just need to be repeated on a regular basis. When we hear something over and over, we start to become familiar with the argument, and we start to have trouble telling the truth and falsehood apart. This happened in 2016 when the number one word associated with Hillary Clinton was Emails. It happened with global warming when enough people suggested that human related CO2 emissions were not related to the climate change we see. And it happens every day in trite sayings and ideas from trickle down economics to popping your knuckles causes arthritis.

 

I don’t think that disproving inaccuracies is the best route to solving the problem of familiarity vs truth. I think the only thing we can hope to do is amplify those ideas, conclusions, experiments, and findings which accurately reflect the true nature of reality. We have to focus on what is true, not on all the misleading nonsense that gets repeated. We must repeat accurate statements about the universe so that they are what become familiar, rather than the mistaken ideas that become hard to distinguish from the truth.