What is science? There is actually an entire academic field, called the philosophy of science, devoted to answering this question. There are hundreds, if not thousands, of academics around the world who have spent their professional lives trying to answer this question.
Given that, we might expect that we have a good idea of what science is, and therefore what non-science or pseudoscience is. But we don’t. The conclusion from centuries of study into the philosophy of the science is this: we have no idea what the scientific method is; we have no coherent answer to why science is the most powerful method of knowledge acquisition we’ve ever known; and we have little clue as to what separates science from non-science.
This latter problem is known as the demarcation problem. Is physics a science? What about cognitive psychology? Economics? Education research? There is no reliable way of saying what is and isn’t a science.
This has led some philosophers of science to offer this incredibly rigorous definition: think of science as a bit like pornography; you know it when you see it. Others define science simply as that which scientists do, which is probably the least helpful definition, since it is tautological: how do we know who is a scientist if we don’t know what science is?
The situation is in such a sorry state that most scientists simply ignore or avoid the question altogether. You can try a little experiment. Ask any scientists you know “What is science?” You will get a thousand different answers. Scientists rarely concern themselves with the philosophical foundations of science: they just get on with being scientists. At no stage in my scientific career has anybody ever sat me down and said “This is what scientists do” or “This is what you need to do if you want to be a scientist.” You learn what being a scientist is about through diffusion, and each person has his or her own unique way of doing science.
At this point, you might concede that defining science so precisely and separating the boundary between science and non-science so rigidly is perhaps an impossible dream, but that there must be some common techniques or attributes that real science shares, and we can use those as a guide for separating science from pseudoscience? No, unfortunately not.
Here’s some common ideas put forward as common attributes that sciences share and that pseudosciences don’t contain:
Science involves experiments.
This seems to be self-evident, doesn’t it? All sciences involve experiments? This is part of the reason that social scientists revere the randomised control trial above all else. But no, experiments do not make a subject scientific. The entire field of astronomy is based on observations of light from the universe. You cannot go out into the universe and make two stars collide. You cannot destroy a galaxy and see what happens. You can only watch what the universe is doing. Controlled experiments are largely impossible, yet most people still consider astronomy to be a science.
Scientific claims are falsifiable.
This idea came from the great philosopher of science, Karl Popper. He argued that in order for a claim to be scientific, there must be a way of proving that claim wrong. It’s closely related to the problem of induction: will the Sun rise tomorrow? Any rational person would say yes, but only because it has risen every day they’ve lived. There is no logically sound reason to suppose the Sun will rise tomorrow. This is the so-called problem of induction.
Consider the claim that all swans are white. If you’ve only ever lived in the UK, you might think this is a strong hypothesis. You go out and search for swans, and it turns out every swan you see is white. After 1000 such swans you are very confident that all swans are white. And then you go to Australia, and you find that actually there are black swans too. Your entire hypothesis has been blown to pieces: it’s been falsified. This power of falsification for disproving claims and circumventing the problem of induction is what Popper argued is necessary for a claim to be scientific.
There’s only one issue with this: most modern philosophers of science disagree that Popper’s falsification hypothesis can demarcate science from pseudoscience.
Consider the question: does supersymmetry exist? Supersymmetry is a theory in physics that claims that every fundamental particle has an associated super-particle. The electron has a heavier supersymmetric particle called the selectron, for example.
But according to Popper, this claim is unfalsifiable, and therefore unscientific; no matter how hard you look for a supersymmetric particle, a believer can just say “You didn’t look hard enough.”
Yet this is the sort of question real scientists grapple with all the time. Physicists are currently looking for evidence of supersymmetric particles at CERN, and the longer they go without finding them, the more sceptical they are that they exist.
Scientists adjust their degrees of belief in a hypothesis based on evidence they do and do not receive. According to Popper, the whole process is pseudoscience.
Science doesn’t adjust its theories to fit the facts i.e. explain away the data.
This is the “no moving goal posts” argument. If a scientific theory makes a prediction, and the data do not agree with the theory, then the theory is thrown out, discarded, rejected: we don’t make up excuses and “explain away the data” so to put it. Popper came up with this one too.
But again, it’s wrong.
Newton’s laws predicted the motion of the planets in our solar system, but Uranus’s orbit consistently disagreed with the predictions. Nevertheless, scientists did not discard Newton’s theory. In 1846, it was suggested that the reason for the discrepancy was due to another planet, as yet undiscovered, and astronomers calculated the position and mass of this hypothetical planet. Shortly afterwards, Neptune was discovered.
In other words, scientists assumed the theory was correct, even after contradictory data, and instead tried to explain away the data with the theory. It turned out that the theory was correct, after all. There are numerous other examples like this in the history of science.
In fact, Thomas Kuhn’s seminal book “On the Structure of Scientific Revolutions” suggested that science progresses through two main stages. There are periods of “normal” or “paradigmatic” science, in which everybody assumes that the laws and theories so far established are correct, and they try to explain all their data using those laws.
It is only when the number of ad-hoc hypotheses – the explaining away of data – becomes too great to bear that the consensus shifts and scientists stop believing in the status quo, look for new theories, and a revolution occurs.
Here’s another example, which you can skip if you’re bored:
In 2011, scientists at the OPERA experiment in Italy claimed they’d observed faster-than-light neutrinos. But practically not a single physicist in the whole world believed these results. Jim Al-Khalili even said that if these findings turn out to be true he’d eat his shorts live on TV.
Special relativity is now such a good theory, verified so many times, that it can never be truly thrown away. Any new theory, borne from the falsification of special relativity, must incorporate special relativity in some way, in the same way that Einstein’s general theory of relativity boils down to Newton’s laws in specific circumstances.
The route to normal science: education research is currently pre-paradigmatic.
Many philosophers argue that the social sciences, which includes education research, haven’t even reached the stage of normal/paradigmatic science or research consensus yet.
Instead, education research is currently stuck in the rut of “Baconian” science, in which we’re just accumulating random facts and reporting observations.
I personally find it hard to argue with that conclusion. In education, we are deluged with study after study, report after report, and observation after observation, but there are few common themes behind the observations; there are few theoretical underpinnings to hold it all together.
As Henri Poincare famously said “Science is built up of facts, as a house is with stones. But a collection of facts is no more a science than a heap of stones is a house.”
And this is why I believe that cognitive psychology has been so important to education research. It explains the “why” and not just the “what”. Previous findings, such as the superiority of direct instruction compared with discovery learning now have a solid, consistent theoretical framework explaining these observations.
The confluence of education research and cognitive psychology offers a truly promising way forward. Eventually, we could reach a stage of fully understanding the observations we’ve already made, and predicting what will and will not work in the classroom before we’ve even done it.
Physics envy: making your research look more scientific.
Physics envy is a phrase used to describe the process of “mathematizing” a subject to make it look more scientific. Many researchers in social sciences are in awe of the predictive power of the physical sciences. They see that the physical sciences are heavily mathematical and conclude that using complicated mathematics will also transform their fields into something with the predictive power that the natural sciences have. Economics has a big problem with this; unable to predict even the most basic economic events, yet filled with maths so rigorous that it takes years of studying to understand it.
But it’s just a correlation. It’s perfectly possible to do some physics without formulae at all. It’s the ideas and concepts that are important. Mathematics is just an abstraction of those ideas and concepts, and mathematics is only used as a tool to enlighten, rather than obscure. Indeed, many physicists are deeply perplexed at the what is called “The Unreasonable Effectiveness of Mathematics in the Natural Sciences.”
Education research has some of the hallmarks of physics envy. It’s not uncommon to see pages of statistical formulae, giving the impression of scientific rigour. But it’s a charade: “We stuck a straight line through our data and measured how far each data point was from our line and averaged the results” doesn’t sound as cool as “linear regression” and certainly doesn’t look anywhere near as cool when you embellish your manuscript with the associated mathematical formulae.
Many of the statistical analyses and concepts used in education research are not much harder than linear regression, but the mathematics can be frightening for the uninitiated. Never be impressed by mathematics for mathematics’ sake. Always dig down to unearth the concepts. If the methodology of a paper isn’t clear, it’s the fault of the paper, and not you.
So how do we evaluate evidence in education research?
I was inspired to write this post after reading David Didau’s post “Is the growth mindset pseudoscience?”
David tries to evaluate the validity of the growth mindset based on the falsifiability of its predictions, and the moving-goal-posts argument. But as we’ve seen, trying to demarcate science from pseudoscience isn’t possible, and even if it were it’s not clear that it’s a good way to evaluate the validity of a claim. We must still decide what to do and what not to do with the evidence available.
So how do we actually make up our minds about what works and what doesn’t?
The answer is as simple as it is unsatisfying: we read it with our sceptical hats on and form a subjective opinion.
1. Be naturally sceptical.
If somebody makes a claim, the onus is on them to provide evidence that convinces you. As the famous saying goes: “Extraordinary claims require extraordinary evidence.”
2. First, do no harm.
Harm is the regression of student learning and the wasting of teachers’ time. Let’s not do anything unless the evidence first convinces us, and let’s not experiment unless we are confident it won’t harm. Be naturally resistant to any new claim or fad. Wait for the evidence to convince you.
3. Read the research.
There is no way of telling good research from bad research without reading it; we can’t rely on qualities for proxies such as the name of the author, the university they work at, or the prestige of the journal they publish in. Never be impressed by these things.
4. Form a subjective opinion.
After reading a piece of research, we should then come to a purely subjective opinion about the validity of the claims. Are the data convincing: do they show what the authors claim they show? Have the authors taken into account methodological downfalls? Have they even acknowledged them, or have they conveniently forgotten to mention them? Have they reached a fair conclusion based on the evidence presented, or have they sensationalised and over-reached? There are numerous other easy questions to ask, which I’ll write a list of one day, that can already take you 90% of the way to deciding whether a piece of research is useful.
The good news is that it doesn’t require too much expertise to evaluate papers critically. It mostly requires common sense, and classroom experience will give you a far better idea of the pitfalls of various studies than more statistical or mathematical knowledge; many education researchers will never have taught, and will therefore fall prey to methodological mistakes that stick out like a sore thumb to a teacher.
5. Weigh it up with other evidence.
Have the findings been successfully replicated? What might the reasons be for failed replications: do they sound plausible?
Are the findings consistent with other observations and/or principles of cognitive psychology?
6. Share your opinion with others; consensus and the wisdom of crowds
I have never read a single paper or report on climate science or climate change. Yet I believe in anthropogenic climate change. Why? Because I trust the consensus opinion of experts.
There have been numerous surveys of the opinions of climate change experts, and the consensus view is that climate change is man-made. Crucially, that doesn’t mean those scientists who disagree are wrong, or even crazy. We must always be open to the possibility that these mavericks could be right. Once upon a time, nobody believed in atoms, or that the Sun was the centre of our solar system, either. Ludwig Boltzmann even killed himself because nobody would believe his ideas about entropy. And yet all turned out to be true.
The wisdom of informed crowds will allow us to slowly piece together what is and isn’t good for the classroom. That’s why it’s so important to talk to people. If you think something is fishy, say it, and say why. You’ll soon discover you’re not the only one, and when discussing with others you’ll learn even more about issues that you didn’t consider. It forces the believers to justify their beliefs more strongly by coming up with ever more convincing evidence.
Given the current state of education research, that’s about all we can hope for right now.
Most importantly, be open-minded. Be prepared to be wrong and to change your mind. Education research is in such a fragile, pre-consensus state that it would be absurd to hold really strong beliefs about anything. As the great economist John Maynard Keynes said (or probably didn’t say):
“When the facts change, I change my mind. What do you do?”