Cargo Cult Science by Richard Feynman : Feynman’s classic scathing critique of the pseudo-science typified by the “cargo cult” of South Sea islanders:
In the South Seas there is a cargo cult of people. During the war they saw airplanes with lots of good materials, and they want the same thing to happen now. So they’ve arranged to make things like runways, to put fires along the sides of the runways, to make a wooden hut for a man to sit in, with two wooden pieces on his head to headphones and bars of bamboo sticking out like antennas–he’s the controller–and they wait for the airplanes to land. They’re doing everything right. The form is perfect. It looks exactly the way it looked before. But it doesn’t work. No airplanes land. So I call these things cargo cult science, because they follow all the apparent precepts and forms of scientific investigation, but they’re missing something essential, because the planes don’t land.
You see this sort of thing often, usually done purposely in order to advance a certain agenda. As Feynman notes, one of the classic examples is advertising. “Wesson oil doesn’t soak through food” – well, that’s true. But what’s missing is that no oils soak through food (when operated at a certain temperature, which is an additional misleading implication). To do away with this, Feynman makes a few suggestions:
In summary, the idea is to give all of the information to help others to judge the value of your contribution; not just the information that leads to judgement in one particular direction or another.
…
If you’ve made up your mind to test a theory, or you want to explain some idea, you should always decide to publish it whichever way it comes out. If we only publish results of a certain kind, we can make the argument look good. We must publish BOTH kinds of results.
These practices are indeed very important, and are often glossed over in the name of brevity or to save money… don’t allow yourself to be fooled by silly correlations and inflated numbers. I’ve found that there are a lot of issues that are quite simply on the outside, but when you dig deep, you find lots of contradicting information, making the issue that much more complex… [link found via USS Clueless in the midst of a discussion of international law, though the entry about “benchmarks” of Macs also seems relevant]