the column of sieves

In the midst of controversial debates, it's too common for either side to demand a "smoking gun" proof from the other. They insist on an item of evidence that's easily understood, beyond all doubt, and strikingly dramatic. They need to be impressed before they'll budge an inch. (Of course, whether or not their demand is sincere is a separate question.)

The strategy works in the heat of a debate because grand indisputable tests are relatively rare. Tests that are actually workable tend to have built-in limitations. Ethical reports of such tests attach sober probabilities to the corresponding conclusions. Imperfection is normal. Tests have holes. The genuine reality could "slip through". A favorable result might still be wrong.

Nevertheless, the big picture is more hopeful. Uninformed people often fail to grasp the total value of an array of imperfect tests. Although one test has known weaknesses in isolation, combining it with other tests can make a vast difference. If one test's holes are like the holes in a sieve (...a sieve with fewer holes than a real sieve would have...), then an array of favorable tests is like a line of sieves arranged in a column. Through this arrangement, anything that slipped through one sieve's holes would probably not slip through another's as well.

The rules of probability correspond to this analogy. As long as the tests are statistically independent, i.e. running one test doesn't affect another, the likelihood is very low that all the favorable results of the array of tests are simultaneously false. That coincidence would be unlikely. One example is five successful tests, and each success had a one in six chance of being false. Having every success be false would be comparable to getting only sixes in five rolls of a die.

Unfortunately, the convincing persuasiveness of an array of tests is a much more difficult story to tell. The focus is abstract and awkward. In place of a lone heroic test to admire, there's a team of mediocrity. It's broad and messy, like reality. Sleek certainty is easier to find in fantasies.

And the situation is generally less clear than this simplified portrayal. The mathematics are more complex and the conclusions more arguable. Perhaps the test results aren't all successes. Or opponents may suggest that the probability that a test is wrong should be greater; they may assert that it's far more flawed than it was said to be. Or, if they're more subtle, they may try to claim that the tests in the array just have identical holes ("pervasive blind spots"), so multiple tests aren't an improvement over one.

Attacks on the details would be progress compared to the alternative, though. Better that than a childish sweeping rejection of tests that can't be faultless. Perhaps the usual tests are suspect, but the recommendation is to round up all these usual suspect tests anyway. Four honest approximations reached through unconcealed methods deserve more credit than an "ultimate answer" reached through methods which are unknown or secret or dictatorial or unrepeatable.