How do you measure the quality of education?
The Educationalist. By Alexandra Mihai
Welcome to a new edition of “The Educationalist”! I must confess it is getting harder and harder to write this newsletter without making any reference to the current climate in higher education. As a political scientist turned into educationalist, it is quite painful to observe how political decisions impact (mostly negatively) the HE landscape, often attacking its foundations and core principles. And this is happening across the world, at a different scale and in different ways. Because the aim of this space is to provide support and inspiration for teaching and learning, I did (and indeed I do) not want to dwell too much on political commentary. Nevertheless, sometimes the two topics interfere. There has been a lot of talk in the past months about efficiency in higher education. This is an uncomfortable topic for universities, not something we are used to considering on a daily basis. And also, rightfully, I think, not something we should be considering (primarily) when we make our learning design decisions. And still, we may have to. More and more. So, before I turn to post like “how to do good teaching on a shoestring” (and no, I sincerely hope this is not what you will be reading in this newsletter in the foreseeable future!), I would rather start by asking “what is good teaching?”. And more broadly, “what is good quality education?”. And how can we measure it in a proper way? I’ve put down some thoughts, as usual, and added some useful resources, but I very much look forward to your ideas and comments. Have a nice rest of the week!
Last week I met a good friend who works in the corporate sector and, among other more pleasant things, we ended up talking about the impact of the current political climate in our respective fields. I started venting my frustrations about the excessive talk about efficiency, about being asked to do more with less, which, I said, can have a negative impact on the quality of the education we provide. Her next question was simple and yet it hit me with an unexpected force: “But how do you measure the quality of education?”. After quite a long - and uncharacteristic for me- moment of silence, I mumbled sort of an answer. As the conversation moved on, one thought got stuck in my mind: why is this question so difficult to answer? A lot of people in academia (especially outside STEM) are reluctant to using any kind of metrics. After all, education is not (only) about numbers, there are so many nuances that are very hard, if not impossible, to measure. While I totally agree with this view, and in fact I’ve been using it as a shield for a long time, I also realised this is the reason I could not give a clear answer to my friend’s question. And it also may well be the reason we cannot have a strong stance on the efficiency discussion. If we cannot really prove (yes, with hard evidence, if necessary, even numbers!) that the methods we use are working and are most suited, we are making ourselves vulnerable to the internal and external forces that are pushing for “education on a tight budget”.
So, let’s stop for a moment and think of the KPIs (Key Performance Indicators) that we want to consider in order to answer the question? Below I will give my personal perspective, and I think each of us, as educators, students or administrators in HE, needs to think about what good education means for them. Then, of course, we need to have this discussion at department and at institutional level.
Some ways of measuring quality
Don’t get me wrong, there are many formal ways of measuring (and indeed, ensuring) the quality of education. Take accreditation criteria, for instance. Pages and pages of often very abstract and rigid criteria and checkboxes that universities have to fill in on a regular basis. Waves of stress for everyone involved once every 4-5 years. What are we actually measuring there? Sometimes I get the feeling it’s the resilience of people having to prepare these thick files. Of course, it’s also about formal rules and regulations universities have to abide to. But what is labelled as a quality assurance measure all too often end up being a mere bureaucratic exercise (and a costly one, at that), very far removed from actually measuring and safeguarding the actual quality of education.
Then we have the university rankings. Highly visible and flashy, their purpose is to drive up enrolments, in a highly competitive HE landscape. Based on sometimes doubtful metrics, they paint a superficial picture of quality assessment, often biased and highly relative. Again, very little genuine quality evaluation at the core, mostly replaced by marketing efforts to find the best packaging.
Are these accurate way of measuring? What are we missing?
Where do I stand?
My friend’s question and my inability to give a decent answer pushed me to really think how I would measure the quality of education. What is for me quality education and what are some proxies that can help us measure it? This will be a very personal account, but you can find some resources for a more nuanced understanding at the end of this post.
What is quality education?
For me, as an educator and faculty developer, quality education means:
First and foremost, a coherent, meaningful student experience- this does not only translate in high quality courses, but also in aligned, consolidated curricula, personalised, student-centred learning and assessment spaces (like ePortfolios, Problem-Based Learning, Project-Based Learning), collaborative learning opportunities, etc;
An inclusive learning space: seriously thinking about who our students are, what they bring to the classroom and what they need;
A supportive learning environment- making the appropriate resources and infrastructure available to students and educators: various services (career counselling, student support, Centres for Teaching and Learning (CTLs) and IT systems that support the student journey. Often underrated, they are crucial to student success;
Educators that feel empowered and supported to enhance their teaching; and more importantly, an environment where educators regularly talk about teaching, ideally as much and as often as they talk about their research.
Links with society: quality education means education that reaches and impacts different target audiences. This means looking beyond our traditional students, descending from the “Ivory Tower” and trying to engage more broadly and thus respond to different societal demands (we often do in our research, not so much in our teaching).
“The best is the enemy of good”
Never did Voltaire’s words sound more relevant than when I think of an important dilemma regarding education quality, one that comes in the spotlight with the efficiency discussion. As universities want to thrive in a highly competitive environment, they strive, at least rhetorically, to provide excellence in education. This race sometimes leads to innovation for the sake of innovation, often a flashy facade to attract students, very weakly related, if at all, with genuine quality.
And then the calls for efficiency force us to think where we can compromise and what is non-negotiable, where compromising would actually mean we are not standing by our educational principles and quality standards. It is precisely to answer this question that we need a good understanding of, and possible ways of measuring, what is excellent education, what is high quality education and what is good enough education (the minimum we can accept). And, of course, what resources go into each of them. Only when we have some of these answers can we draw the line and make some informed decisions. (Disclaimer: this is a process that takes time and strategic thinking, so it will not happen in weeks or months.)
Let’s take the example of Problem-Based Learning (PBL) at Maastricht University (UM). It is a student centred, active learning method that is generally conducted in small groups (max 16 students) and thus requires a high amount of tutors and other resources. The benefits are clear, and PBL is also the signature pedagogy and an important USP for the university. Every method needs to adapt in time to respond to the changes in student needs and other external forces (like for instance the Covid-19 pandemic). The key question is how far can we go in order to preserve the core principles while being mindful of the costs? Using educational technologies can, for instance, contribute to a more flexible and inclusive learning environment, but we need to seriously consider to what extent and in which ways technology can support learning in a PBL context and not see it simply as a cost-cutting exercise in terms of reducing the number of tutors. When implemented well, educational technology requires, at least in the beginning, quite a high investment (both in terms of infrastructure and in terms of support), something that educational leaders often choose to turn a blind eye to.
At an individual level
Moving away from the institutional level, as academics and educators we also have to deal with our own dilemmas. A recent post by Karen Costa on LinkedIn about what is a “good enough professor” brings out some of these thoughts. Our growth mindset, and also in my case the fact that my job is to support faculty to grow, makes it very difficult for us to settle for less. Even the sound of it makes some of us cringe. But how about framing it differently: it is not about settling for less or not being ambitious, it is about seriously considering the trade-offs we need to make in order to preserve our well-being.
It is not about becoming selfish and not thinking about our students. It is about reflecting on our role as educators and the ways we can bring most added value to the education process and to the learning experience of our students (with limited resources- alas, the day has only 24h even if some of us would like to think differently, and not all of them need to go into work). And based on that, decide on how we use our time best. Perhaps we will not be creating new educational materials but will dedicate more time to feedback. Or we will decrease the number of assignments but do more active learning in class. We need to understand when and how our students learn and prioritise those moments and modalities, while giving up on the illusion we can do everything perfectly at all times.
But how can we measure all this?
I’ve come up with quite a list of what quality education means for me. So I need to take a step back and see how we could possibly measure all those things. Here are some thoughts:
Measuring the student experience
Student evaluations: Let’s think for a moment: do the evaluations we currently use capture the entire student experience? They often focus on the evaluation of teaching, and even then, the questions and the metrics used are debatable. So, what would I like to try to evaluate (quantitatively and/ or qualitatively)?
the types of interaction, not only at “contact hours”;
forms of assessment and their impact on student learning;
the alignment at programme/ curriculum level, connections between various courses;
interdisciplinarity & transdisciplinarity (involving stakeholders from outside academia);
what skills / competences do our students acquire?
Measuring the teaching staff experience
This perspective is often missed, unfortunately. How do we capture this? Self-reflection, discussions with colleagues, engagement with professional development activities, these are just a few ways in which we can get an insight into what educators think and do;
How about the links to the “outside world”?
Employability: What do our students end up doing? And (to what extent) has their education helped them perform in the labour market (do we provide them with what they need or not quite?). Evaluating this requires longitudinal studies, which are time and resource intensive. The downside here is that this angle, seen in a very rigid way, places universities in the position of degree factories. Is this what we want? Some food for thought.
Resources
“Measuring Quality as if Quality Mattered”, by Alex Usher;
“Measuring the Quality of University Education: Beyond the Nonsense of University Rankings”, by Paul Ashwin;
“Measuring teaching quality in higher education: assessing selection bias in course evaluations”, by Maarten Goos & Anna Salomons;
“Measuring Quality in Higher Education: A competency approach”, by James Warn & Tranter Paul.
