I read The Tyranny of Metrics recently. It is as interesting as everyone says, but what I ended up thinking about wasn’t so much the problems and unintended consequences of the data schools and teachers are (or have been) asked to collect, although this is something Muller address well. What I found myself thinking about was the various ways students misunderstand and misuse data. They, like the rest of us, need to remember that ‘measurement may provide us with distorted knowledge – knowledge that seems solid but it is actually deceptive.’ Below are 5 key things that I would like students to remember about the data they receive, in the form of marks for tests and essays.
One test result does not necessarily encapsulate your ability.
If you did very well, ask yourself honestly why. Were you lucky with the topics or questions that came up? Would you have done equally well had the paper been differently focused? You must not lose focus just because you fulfilled or exceeded your expectations in one mock or practice paper. You still need to think carefully about what to work on. Even if you know the material well now, you need to keep revising it regularly to keep it fresh.
Equally, one below par performance should not be a reason for despair. Does it really mean that you ‘just can’t do history’ and that any further efforts on the subject will be wasted? It is more likely to mean that you didn’t revise effectively or that your question answering technique was flawed. If it has knocked your confidence, take the time to redo the questions with the benefit of the advice and feedback you have received, or do some better revision and sit another practice paper. Even in football it is not really true that you are ‘only as good as your last result’. Performance needs to be judged over time, and thought given to how to build your knowledge and refine your technique so that you maximise your potential.
Comparing marks between subjects is problematic.
At my school, all year groups do exams in every subject towards the end of the academic year. And, every year, several students will decide that they are ‘better’ at subject X than subject Y because they got a higher percentage in the former. The dangers of judging ability from one test do not occur. Nor does the idea that one test may have been easier than the other. Indeed, sometimes when a student finds one exam easier they will take this as further evidence that they are better at that subject. Students do accept that getting full marks is possible in some subjects but not in others but that seems to be as far as it goes. Students rarely consider the variations caused because one subject demanded the use of more detailed knowledge or more complex analysis and evaluation. That 70% in a challenging examination may represent a far greater achievement than 90% in a straightforward one is a truth students find it hard to absorb.
Standardising scores across subjects will solve some of these problems, making comparisons between subjects more meaningful. Quality control across subjects to ensure that students are being appropriately stretched in all aspects of the curriculum may also be necessary.
Similar looking questions can vary a lot in difficulty.
Consider the following two questions:
- How far do economic causes explain the outbreak of the French Revolution?
- How far do economic causes explain the outbreak of the American Revolution?
The uninitiated might think that these two are of similar difficulty, but, in fact, the economic problems that underlie the French Revolution are far more complex than those that helped to provoke the American colonies into rebellion. Indeed, overall, the question on the French Revolution represents a greater academic challenge than the question on the American Revolution because of the level of contextual understanding required. At least we would normally teach the American Revolution before embarking on the French but our need to teach chronologically in history means that we can’t always tackle the simpler stuff first and, even if we do, students might perceive their ability to have dipped because they did better on an earlier essay on a more straightforward topic than a later one addressing a trickier subject.
You will not acquire any useful information from a test or examination for which you are unprepared.
After mock examinations, there are always students who inform me that they ‘were seeing how well they could do without revision’, as if that is some useful guide to further action. As I said to my classes last year, well before their exams: ‘This isn’t complicated. If you don’t revise YOU. WON’T. DO. VERY. WELL. You don’t need a poor mark in a mock to tell you that you need to revise. Everybody needs to revise. Furthermore, if you haven’t done any revision, I won’t be able to give you any meaningful feedback because I won’t be able to gauge how well you understand the material. Marks are only one aspect of feedback, and generally not the most important.’
Many of the marks you receive will be influenced by subjective judgements.
Yes, exam board use ‘seed’ scripts (pre-marked scripts placed in examiners’ bundles) and an element of double marking in an attempt to maintain levels of reliability. Teachers cross mark with their colleagues to ensure consistency. With essays, in particular, though, some subjectivity will still creep in. On a different day, with a different examiner, your script might not have got exactly the same mark. The same examiner might be more generous with the first essays he marks than those towards the end of the batch.
The best possible remedy for this is the widespread use of comparative judgement, so that each script is judged against many others, many times, by a number of different markers. Anyone from the maintained sector wanting to know more about comparative judgement, is welcome to sign up to attend a free INSET with Daisy Christodoulou on 1st November, in which Daisy will both explain the fundamentals of CJ and give a lecture on the Seven Myths About Education.
The event URL is https://www.eventbrite.com/e/inset-at-south-hampstead-high-school-with-daisy-christodoulou-tickets-50682469631