Two students both score 9 out of 10 on a quiz called “What is machine learning?” Then the teacher asks each one: “A model is very accurate on its training photos and fails on new ones. Why?” One answers straight away. The other stares at the page.
The quiz could not tell them apart. That is the core problem of AI literacy assessment. Recall is easy to test, and understanding is not.
What quiz scores can and cannot show
Source: A systematic review of AI literacy scales npj Science of Learning
The measurement problem is not unique to schools. Much of AI literacy research has also relied on what learners say they can do rather than evidence of what they can actually demonstrate.
Multiple choice checks vocabulary well. Understanding shows itself in other ways. Students explain a mechanism in their own words, predict what will happen, find out why a model failed, and defend a judgement. Each of those needs a different kind of task.
Six kinds of evidence, and what each one shows
| Evidence | What it tells you | Watch out for |
| Short recall quiz | Students know the key terms | Can be memorised |
| Worked calculation (for example, computing a perceptron’s output by hand) | Students follow the mechanism | Needs maths readiness |
| Error analysis (“why did the model fail?”) | Students reason about data and limits | Needs well-built prompts |
| Build-and-test artefact | Students can apply and iterate | Slow to mark |
| Defended position | Students can make and justify a judgement | Subjective without a rubric |
| Portfolio or capstone | Understanding joined up over time | Arrives late |
No single row is enough. A school that uses all six across a unit will see far more than one that relies on the quiz. For the quick, low-cost end of this list, see the small checks that reveal misconceptions. For the outcomes these tasks should be judged against, see what students should be able to do.
What the OECD and PISA 2029 are signalling
Two recent developments point the same way.
What the sources say. The OECD and the European Commission have published an AI literacy framework for primary and secondary education. A draft went public in June 2025 (Media & Learning), and the OECD lists the final version, Empowering Learners for the Age of AI, as published on 17 June 2026 (OECD). It describes AI literacy as knowledge, skills and attitudes that help young people understand how AI systems work, critically evaluate their outputs, and use them ethically and creatively.
PISA 2029 will add a Media and AI Literacy (MAIL) domain for 15-year-olds (OECD project page). ACER’s Goran Lazendic describes it as the first time PISA will show how education systems prepare young people to exercise judgement in digital environments (Teacher Magazine). The draft framework, as that article reports it, lists five competences: Reflect and Act Ethically and Responsibly; Access and Use; Analyse and Evaluate; Participate and Collaborate; Create.
“Analyse and Evaluate” and “Reflect and Act Ethically” sit beside “Access and Use”. The ambition is no longer only that students can operate a tool.
Limits. PISA samples students to describe education systems. It does not grade a school’s programme. The MAIL framework was still in draft when it was reported, so details may change.
Our recommendation. Do not wait. Define your own evidence list now.
If students can use an AI system but cannot explain why it produces errors, what exactly have they learned?
Four things a school can do this term
- Write outcomes as verbs: explain, calculate, evaluate, defend.
- Add one “why did it fail?” task to every unit.
- Mark method, not only the answer.
- Keep a simple portfolio so evidence builds across years.
A portfolio pairs well with an AI capstone. Error-analysis tasks can start from the confusion matrix as assessment evidence.
FAQs
Can multiple-choice quizzes assess AI understanding?
For terms and basic facts, yes. For understanding, they need partners such as calculations, error analysis and a defended position.
How do you assess AI understanding in young children?
Through observation, picture sorts, spoken answers and drawings. APEX uses these for KG to Grade 2.
Should schools wait for PISA 2029 before designing assessment?
No. PISA gives system-level information, and the framework was in draft when reported. Your own evidence starts now.