, 6 minutes to read
“Adaptive” appears on the website of almost every online test. It has a precise meaning, and knowing it helps you to tell a test that adapts from one that only says so. No mathematics is needed to follow this guide.
The problem with a fixed test
Imagine a paper with fifty questions that has to place anyone from a beginner to a near-native speaker. To cover six or eight levels it can spend only six or eight questions on each. For any one student, then, about forty of the fifty are far too easy or far too hard, and tell you nothing you did not know. The decision rests on the handful of questions near the student’s level, and a handful is not many.
An adaptive test turns this round. It spends nearly all its questions near the student’s level, so twenty of them carry more information than the paper’s fifty.
How it works, in four moves
- Start with an open mind. Before the first answer, the student could be anywhere on the scale. The test begins in the middle.
- Ask where it is most useful. The most informative question is one the student has about an even chance of getting right. Too easy or too hard, and the answer is predictable.
- Update the estimate. A right answer moves the estimate up, a wrong one down. Early on the steps are large; as evidence accumulates they shrink.
- Stop when sure enough. When further questions would no longer change the level, the test ends. A student in the middle of a level finishes sooner than one who sits on a boundary.
The part that is easy to overlook: measured difficulty
Choosing “a question near the student’s level” only works if the test knows how hard each question really is. A label such as “B1” written by the author is a guess. Authors disagree, and when their labels are compared with how thousands of students actually answer, questions carrying the same label turn out to differ widely.
A properly built adaptive test therefore measures each question: it records who answered it correctly and who did not, and works out the difficulty from that. This is called calibration. It needs a great many answers, which is why a new test cannot simply declare itself adaptive, and why new questions have to be tried out before they count.
What about guessing?
On a question with four options, a student who knows nothing is right one time in four. A sound test allows for this: a correct answer to a multiple-choice question counts for a little less than a correct answer that had to be typed. Offering “I don’t know” helps too, because an honest “no idea” is more informative than a lucky guess.
What an adaptive test cannot do
- It cannot test what it does not ask. A test of grammar and vocabulary says nothing direct about speaking. Keep the short conversation.
- It cannot remove uncertainty. It can only tell you how large it is. Be suspicious of any test that reports a level with no margin.
- It is only as good as its questions. A faulty question, with a wrong key or a second correct answer, misleads an adaptive test just as it misleads a paper.
Seven questions to ask about any adaptive test
- Does every question have its own measured difficulty, or only a level label?
- How many real answers is that measurement based on?
- How does the test decide when to stop?
- Does the result show a margin, or only a level?
- How large is the question bank, and can two candidates get the same test?
- Is the scoring done on the server, or in the candidate’s browser where it can be inspected?
- What happens to a question that starts to behave oddly?
Our own answers to all seven are on the page How it works.