Getting Policy Right Takes Time | Eastern North Carolina Now

    Publisher's note: This article appeared on John Hood's daily column in the Carolina Journal, which, because of Author / Publisher Hood, is linked to the John Locke Foundation.

    Well, never mind, then. According to the only large-scale, randomized controlled trial of a statewide preschool initiative - Tennessee's pre-K program -intervening early to boost long-term academic performance doesn't work.

    Thousands of low-income students in Tennessee were randomly assigned to the pre-K program or to a control group. While there were some academic gains for participants at the kindergarten stage, by the third grade the benefits disappeared. In fact, pre-K participants actually scored lower than their otherwise-comparable peers.

    States aren't just wasting taxpayer money, then. Preschool programs are actively harming their intended beneficiaries. If North Carolina leaders truly care about disadvantaged children, they should shut down our state's pre-K program immediately.

    Is that what you were expecting me to say? Do you think such a conclusion would be reasonable based on a single study, however well-constructed? If your answer to either or both of these questions is yes, I would respectfully urge you to consider carefully what it means to apply scientific concepts and findings to public policy.

    One of those concepts is that the scientific method isn't really about proving propositions true. As a great 20th-century philosopher of science, Karl Popper, argued persuasively, science is really about proving propositions false. Although a scientific theory may seem highly useful in explaining the past and predicting the future, there is always the possibility that some valid future experiment will disprove it.

    "No matter how many instances of white swans we may have observed," Popper wrote, "this does not justify the conclusion that all swans are white." But he didn't mean that science wasn't immensely useful. The point is that if a proposition can't be falsified by experiment, it isn't a scientific proposition. "Those among us who are unwilling to expose their ideas to the hazard of refutation do not take part in the scientific game," he said.

    But what does falsification look like in the real world? Another great 20th-century philosopher of science, Thomas Kuhn, argued persuasively that the model of a broadly accepted scientific theory being disproved by a single experiment is unrealistic. What really happens is that the defenders of the theory assume the experiment is flawed in some way, or only applies to some situations. In other words, they explain the inconvenient finding away.

    Moreover, Kuhn suggested, they ought to do that! Abandoning an established set of explanations, what he called a paradigm, based on a single contradictory experiment is contrary both to human nature and to the search for truth. "So long as the tools a paradigm supplies continue to prove capable of solving the problems it defines," he wrote, "science moves fastest and penetrates most deeply through confident employment of those tools."

    Instead, what ought to happen, and what does happen, is that scientists see a puzzle to be solved. They replicate the contradictory experiment. If the finding is the same, they seek to explain it without abandoning the established theory. Others do the same.

    If the contradictions persist and proliferate, a crisis comes. A good analogy would be flowing water and a dam. Sometimes the dam holds. But if the pressure grows strong enough, it bursts - and in the case of science, a new paradigm is formed.

    To return to the question of preschool intervention, the Tennessee study, an earlier and comprehensive evaluation of Head Start, and new statistical work by the Brookings Institution's Russ Whitehurst all suggest little to no benefit. But there are a number of other, smaller-scale studies pointing the other way.

    Similarly, while dozens of scholarly studies show that school choice and competition are associated with academic and other benefits, initial studies of statewide voucher programs showed scant or even negative effects, although subsequent findings were more promising.

    In response to such evidence, it would be reasonable to temper one's short-run expectations and recognize that scaling up policy ideas can be challenging. But it wouldn't be reasonable to abandon those ideas altogether. That's not how science works. And it's certainly not how public policy works.