Positive Test Strategies

Positive Test Strategies

A real danger for us, that I don’t know how to move beyond, is positive test strategy. It is the search for evidence that confirms what we want to believe or what we think is true. When we already have an intuition about something, we look for examples that support our intuition. Looking for examples that don’t support our thought, or situations where our idea seems to fall short, is uncomfortable, and not something we are very good at. Positive test strategies are a form of motivated rationality, where we find ways to justify what we want to believe, and find ways to align our beliefs with what happens to be best for us.

 

In Thinking Fast and Slow, Daniel Kahneman writes the following, “A deliberate search for confirming evidence, known as positive test strategy, is also how System 2 tests a hypothesis. Contrary  to the rules of philosophers of science, who advise testing hypothesis by trying to refute them, people (and scientists, quite often) seek data that are likely to be compatible with the beliefs they currently hold.” 

 

In science, the best way to conduct a study is to try to refute the null hypothesis, rather than to try to support the actual hypothesis. You take a condition about the world, try to make an informed guess about why you observe what you do, and then you formulate a null hypothesis before you begin any testing. Your null hypothesis says, actually nothing is happening here after all. So you might think that teenage drivers are more likely to get in car crashes at roundabouts than regular intersections, or that crickets are more likely to eat a certain type of grass. Your null hypothesis is that teenagers do not crash at roundabouts more than typical intersections and that crickets don’t display a preference for one type of grass over another.

 

In your experimental study, instead of seeking out confirmation to show that teenagers crash more at roundabouts or that crickets prefer a certain grass, you seek to prove that there is a difference in where teenagers crash and which grass crickets prefer. In other-words, you seek to disprove the null hypothesis (that there is no difference) rather than try to prove that something specific is happening. It is a subtle difference, but it is importance. Its also important to note that good science doesn’t seek to disprove the null hypothesis in a specific direction. Good science tries to avoid positive test strategies by showing that the nothing to see here hypothesis is wrong and that there is something to see, but it could be in any direction. If scientists do want to provide more evidence that it is in a given direction, they look for stronger evidence, and less chance of random sampling error.

 

In our minds however, we don’t often do this. We start to see a pattern of behavior or outcomes, and we start searching for explanations to what we see. We come up with a hypothesis, think of more things that would fit with our hypothesis, and we find ways to explain how things align with our hypothesis. In My Big Fat Greek Wedding, this is what the character Gus does when he tries to show that all words in the world are originally Greek.

 

Normally, we identify something that would be in our personal interest or would support our group identity in a way to help raise our social status. From there, we begin to adopt hypothesis about how the world should operate that support what is in our personal interest. We then look for ways to test our hypothesis that would support it, and we avoid situations where our hypothesis could be disproven. Finding things that support what we already want to believe is comforting and relatively easy compared to identifying a null hypothesis, testing it, and then examining the results without already having a pre-determined outcome that we want to see.

Timing Impacts Test Scores and Life Outcomes

“Indeed, for every hour later in the day the test were administered, scores fell a little more. The effects of later-in-the-day testing were similar to having parents with slightly lower incomes or less education – or missing two weeks of school a year.”

 

The quote above is from Dan Pink’s book When in a section where he wrote about researchers who studied test scores for children in relation to the time of day that tests were administered. School children performed better on tests when they took place early in the day, and worse on tests when they came later. The difference was not enormous to the point where students were passing with flying colors in the morning and failing in the afternoon, but it was meaningful.

 

I find this incredibly interesting for a couple of reasons. The first is that I think 8 hour work-days are horrible, and the second is that I think we ascribe our success or failure in life to our own efforts far more than what we should.

 

Starting with the first point, if I were suddenly named king in the United States, my first act would be reduce the standard workday to 6 hours instead of 8. Most American’s no longer work at a factory where they need to be active on a line pushing a button or hammering a nail in order for widgets to be produced. When that was the case, 8 hour workdays may have been necessary, but in a knowledge economy where our main output comes from our mind and not our hands, 8 hour workdays likely harm our work more than they allow us to produce meaningful outputs. My big fear is that increased time outside of work will lead to more urban sprawl and longer commutes for people rather than to more valuable time spent in communities or with family.

 

Ultimately, however, I think giving us more time for sleep, for exercise, and for family will help us be better people. I believe that many people claim to be more busy than they actually are at work, and often spend a lot of time focusing on low value tasks that keep them busy but don’t provide much benefit for anyone. Shortening the work day would force people to be better schedulers and to use their time more wisely, and would hopefully create a happier workforce and nation.

 

Second, it is important to recognize that simply the time at which students took a test impacted the score they got. The test in some ways is measuring their knowledge, but it clearly is also measuring endogenous factors unrelated to their learning. We are grading and scoring our students in ways that can be influenced by meaningless factors beyond the students’ control.

 

I think this example is revealing of a basic fact of our lives. We feel like we are in control and as if the outcomes that people experience are directly related to their own effort and skill. However simply the time of day can have a big impact on our achievements, regardless of our skill or effort. Beyond test scores, perhaps you had a job interview in the afternoon, and both you and the interviewer were a little less sharp than you would have both been at 10 a.m. for an interview. You might not come across as the best candidate, and the time of day may be the biggest factor that prevented you from getting the job. In many ways small factors like timing can shape our lives in ways that we can’t even imagine. Sometimes luck is just as important as our skill and effort, and we should recognize that when we think about where we are and how we got to the point we are at.