Sometimes I find something in my data which agrees with my predictions. This makes me happy, and usually prompts me to casually mention this fact on twitter. And sometimes, as in this case, my followers give me good advice.
@TWeDK raises a very interesting point. It’s interesting, because the human brain is very good at recognising patterns and looking for things. It’s also very good at seeing things which aren’t necessarily there. It’s easy to “see” what you were expecting to see, but that doesn’t mean it’s real. Even so, the statistic that over 1 in 3 people will believe that scientists doctor their results this way is something of a smack in the face.
In 1974, Richard Feynman gave an address at Caltech on the topic of “cargo cult science”. This delightfully coined phrase describes a practice which superficially appears scientific, but is not. Actual scientific research requires integrity and earnestness. Any scientist may make hypotheses, have expectations, and know what they’re hoping to find. After all, they are human. But a good scientist will also rigorously check over their own results to ensure that what they’re looking at is real. We can’t afford to become too attached to our ideas while we’re constantly hitting them with hammers to see if they break.
A cargo cult scientist, on the other hand, will seem to be doing everything correctly, but they’ll actually be somehow missing the point. To quote from Feynman’s address:
“We’ve learned from experience that the truth will come out. Other experimenters will repeat your experiment and find out whether you were wrong or right. Nature’s phenomena will agree or they’ll disagree with your theory. And, although you may gain some temporary fame and excitement, you will not gain a good reputation as a scientist if you haven’t tried to be very careful in this kind of work. And it’s this type of integrity, this kind of care not to fool yourself, that is missing to a large extent in much of the research in cargo cult science.
The first principle is that you must not fool yourself–and you are the easiest person to fool. So you have to be very careful about that. After you’ve not fooled yourself, it’s easy not to fool other scientists. You just have to be honest in a conventional way after that.”
I like to believe that I’m a good scientist. Perhaps driven by an innate fear of being wrong and/or looking foolish, I always check things multiple times. While it is easy to fool yourself and get excited over a result, you need to make sure it is what you think it is. I’ll even confess that I have indeed fooled myself a couple of times. Then I checked over the data again. At that point, I usually discover that what I thought I was looking at before was nothing more than a quirk in the way I was analysing things.
But this does highlight a rather important fact – As Feynman was lamenting, these vitally important things which we all should do are not, in fact, taught to us at any point. As PhD students, we learn how to do science by doing science. While a lot of graduate schools may be awfully keen to get us to take classes in how to talk to the press and how to design a conference poster (both very useful courses to take, incidentally), there are usually no courses at all about how scientific method works. We have to learn these things for ourselves. Assuming we care enough to do so. It seems logical that a lot of people probably miss out on a lot of important facts. Facts which, let’s be honest, are well known and should be taught to PhD students more formally.
I feel this requires more thought and discussion than I can spare the concentration for right now. When my work schedule calms down a bit, I should probably write more on this…