Assessing competency in some form or another has been around almost as long as civilisation – and it has come a long way: Long gone are the odd attempts at measuring intelligence by using graphology or measuring skull size. One question that remains at the heart of psychological assessment: How can we best assess psychological traits that are not directly observable? To answer this question in relation to how to best assess students, Luxembourg researchers are developing cognitive psychology-based blueprints for mathematical test tasks.
Educational Assessment is an essential part of today’s society – every school student gets tested many times and we all have a memory of that one unfair teacher.
Assessing people’s competencies and taking decisions based on the results is thought to have been around for as long as humankind. The scientific discipline of psychological assessment on the other hand, is still young at just over 100 years.
“Since then, we overcame techniques, such as graphology, or measuring the skulls of people to determine their intelligence. The central question – how we can assess not directly observable psychological traits – remains at the centre though and is a truly scientific and fascinating one. You need to be creative, to capture what is going on in people’s heads. Psychological traits are shy, you need to lure them by clever tasks to make them visible. ”Philipp Sonnleitner Psychologist and researcher
Helping teachers design better and fairer assessments
“In my work, I strive to optimise the small window we get, when we test, for example, students’ math or reading comprehension skills. Increasing the information we get out of every single student response by clever task design or improved statistical analysis is my daily work.”
“Improving these procedures is extremely rewarding, since assessments are used for many purposes ranging from policy decisions to individual evaluations, which school track would be the best fit. With this experience, I’m helping teachers to design better and fairer assessments through workshops or self-administered diagnostic tools.”
How to measure what you cannot directly observe
A challenge for researchers such as Philipp is that it is not possible to directly observe what he wants to measure: How well a student can handle math can only be inferred by a sample of tasks – but: the choice of tasks and how the sample is designed has an impact, not only on the student’s motivation, but also on what can be deducted from the responses.
“Each psychological measurement is noisy. You have the student’s motivation or maybe a distracting test environment. We can only work out the best way for assessing a certain competence by clean experimental design. It takes a lot of time to know if such a task works. ”Philipp Sonnleitner Psychologist and researcher
Blueprints for mathematical test tasks
In his current CORE Junior research project, funded by the FNR, Philipp & his team are developing blueprints for mathematical test tasks that are grounded in cognitive psychology. The blueprints can be implemented into algorithms that automatically generate thousands of suited test tasks.
“At the same time, due to the theoretical underpinnings, we learn more about the underlying ability from student responses on these tasks. These blueprints need to be evaluated and validated of course if they work as supposed. But if this approach works for a sample of tasks, we could immediately make statements about the potential of each blueprint itself – we are scaling the validation process.”
Generating tasks at large scale reduces cost
The researchers have already been successful in developing several task blueprints for the mathematical domains of numbers & operations, as well as space & form. These enabled the researchers to generate millions of tasks with more or less known characteristics.
“We found out that the older the kids are, the less they are impressed by the context a mathematical problem is presented in. This means that we could assess students with individualized test tasks that provide comparable information, making it impossible to cheat, for example. Or hundreds of these tasks could be used for training purposes while another sample is used for assessment. Generating tasks at this scale reduces the costs of one task dramatically. ”Philipp Sonnleitner Psychologist and researcher
Dr Philipp Sonnleitner is a CORE Junior PI at the Luxembourg Centre for Educational Testing (LUCET) at the University of Luxembourg.
MORE ABOUT PHILIPP SONNLEITNER
Describing his research, in 1 sentence
“I strive to make the world (of assessment) a better and fairer place.”
What drives him as a researcher
“I find it truly intriguing to find ways to capture otherwise not observable psychological abilities. For me, the most interesting scientific frontier is situated in our heads and history of psychological assessment showed that you need to be really creative to expand this frontier. Being able to add a small puzzle piece to this body of knowledge fills me with excitement and I have a lot of fun while doing it too.”
What he loves about research
“It’s hard to find such a broad field of activity! I can be creative, play with words, data, lately artificial intelligence, and spread my ideas via presentations around the globe, in scientific papers or questionnaires and tests. I truly love the combination of intellectual freedom and scientific tools and on top you can help make the world a better place.”
Where he sees himself in 5 years
“Hopefully still in the field where I am now but with many projects and ideas gone from “in progress” to “finished”. It would be fantastic to see the impact of some of our tools on educational assessment in Luxembourg.”
About Spotlight on Young Researchers
Spotlight on Young Researchers is an annual FNR campaign where we shine a Spotlight on early-career researchers across the world with a connection to Luxembourg. Over 100 features have been published since the first edition in 2016.
Discover all FNR Highlights