Unpolished

Evening all,

No, the title isn't about my "colourful" turns of phrase, or proclivity for swearing in the grand antipodean manner. What I'm referring to here is things I've read, thought about, and a panel I participated in last week. This combo has resonated with me in the days since, and I wanted to unpack it a bit more, and add a few unpolished views of my own.

Last week I was honoured to be invited to be on a panel for CRADLE, the Centre for Research in Assessment and Digital Learning at Deakin uni in Melbourne. The panel included some big names like Danny Liu (of 2 lanes and Cogniti fame); Phill Dawson (he who coined assessment security); fellow cheating expert, absolute clever clogs and friend of the blog Cath Ellis, and yours truly.

Nominally, we were supposed to talk about assessment tasks, although I'm not sure how on topic we were. More particularly though, I think it was hoped that we would address a question something like "if all tasks are fallible [to AI or other forms of learning avoidance] in one way or another, is there any point to programmatic assessment or similar approaches to securing programs?" In truth I'm not sure we ever really got near this question on the panel so I wanted to give it a go myself (note- I'm speaking for me, not Cath, Danny or Phill, or indeed the wonderfully hirsute Tom Corbin who MC'd the panel in marvellous style- like an academic Chopper Read in my mind).

On the panel there was a general acknowledgement that assessment as we knew it is broken, and cannot be repaired using the same approaches as previous. As Cath said during the panel session, education is experiencing a "paradigm shift", and I strongly agree. Of course, as is my fashion, I described it more like students having knocked the walls down with a bus, and they are currently driving off into the distance, while many are still discussing what colour to paint the walls. Or something like that. Essentially, the past decade or so has shown me that not only are more students prone to taking "shortcuts" than anyone imagined, but the shortcuts are more numerous and systematic than anyone also imagined. Generative AI has simply stolen away the fig leaf of plausible deniability from education, and turbo charged the shortcuts. And by that I mean this- at least as a starting point for discussion or negotiation- online assessment is dead. It is deceased. It is no more.

The point which appears to have flown above the heads of many since 2020 is that education didn't really "pivot" to online during Covid, did it? Teaching did, sure. But teaching isn't the entirety of education, and teachers are not the central figures in this melodrama. Students are the main actors, and (shocking to many I know), some of them don't much care for director's instructions. So teaching changed, but assessment on the other hand was substantially, and often completely, conducted online well prior to the pandemic. I feel like a moron pointing this out, but it is a fact that it was entirely possible to get a degree in a large and reputable university without ever lifting a finger years before Covid and ChatGPT. This fact is enabled by not only online assessment, but other things we consider positives- like flexibility, and progressive andragogy (we are teaching adults people, not children, get it right). Those assessment tasks that academia puts such stock in, such as the essay, were so broken as a valid indicator of learning that the essay mills arose to fill a giant industrial void. Incidentally, if you haven't seen The Shadow Scholars, I highly recommend it. But let's not stop there, how about online quizzes, forum posts, reports submitted online, online exams. Every one of these assessment types is invalid, from an assurance of learning perspective. Demonstrably, we can have no assurance that a given piece of work in front of a marker is the named student's work.

It's not that all students will avoid learning, it's that some definitely will (despite every positive effort to encourage them otherwise), and you don't know who they are, do you? Our collective job isn't to assure learning for the good students. It's to assure learning for every single student. No exceptions. That's the barest minimum society expects of us.

So now, I'm going to ask you a question- how do you propose to assure online learning for every single student? But before you jump to answers, let me tell you just some of the ways that I have seen and/or proven students avoid individual learning:

• Chat groups of up to 300 students, taking online exams in a timed exam window, sharing questions and answers to the chat group, before another student takes the exam in sequence, assisting each subsequent student;

• Essay mills (duh);

• A 3rd party posing as multiple students (dozens in some cases) in the LMS for entire subjects across a degree;

• A 3rd party posing as a student during an online invigilated exam, tutorials, and any other necessary subject components;

• Various usages of AI (side note, take a look at this);

• Students have a small camera on their lapel, and an earpiece, both connecting to a phone in their bag across the room, or to a person outside the room.

• and more...

Now, go back to the question- which of your students are doing these things, and how are you going to find out? The straightforward (but correct) answer is that, as a teacher, you almost certainly don't know, and you are almost certainly not going to be able to find out. So then, what do you do about it? I've had a lot of conversations, and read an awful lot about various things people are trying (R/professors is a gold mine, like a giant game of Rock 'em Sock em' Robots). In the gold mine there is an endless array of ideas that will do almost literally nothing to improve assurance of learning, the problem that "cheating" creates.

I wanted to start with the worst idea, the one that was thrown up first and loudest after generative AI exploded- AI detection. Along with colleagues from across the sector here in the great brown land, we set about destroying the arguments for AI detection, and I think we did a bang-up job. I'm not going to rehash it here, have a read, but after reading if you still believe detectors are a valid approach, do take a look at yourself. Using detectors will not assure learning, it will not be a fair process, and by definition you cannot care for your students if you use them. Spare me the excuses and self-justifications.

The next approach, if not pervasive then widely popular, is to rely on trust, or an honour system. I wouldn't disagree at all that this is part of the recipe, but it isn't the whole cake. I often ask the question of people who lean more heavily into this approach what they will do when if students decline to act how they might hope. Most commonly I get no answer at all, but otherwise I get "Oh well, I'll refer them for academic misconduct!" To which I ask, "based on what?" This entire approach is not geared toward actively or skeptically seeking evidence of unlearning, or necessarily even learning. It's an approach based on creating an environment of trust which the teacher assumes will be successful but leaves them flapping their arms about when it isn't. Worse, sometimes they will choose to "other" a student based on vibes, reacting like a toddler when a student doesn't get with the program.

As Cath and I said in our Pyramid paper, no one singular approach will be successful. I would say that adopting a singular, "silver bullet" approach almost guarantees the opposite. Moreover, given more time since then to roll these questions and conundrums around in my head, I have concluded that approaches devised by individual teachers cannot succeed. One thing that I think is often forgotten, or simply ignored, in considering what to do about the crisis we are experiencing in education (it is a crisis as well as a paradigm shift) is that we in educational institutions have more than just the obligations to our students to attend to. A term which has been thrown around a lot recently in Australia refers to the "social license" of universities to operate. This means that we (and students) are not laws unto ourselves, but rather have to maintain the trust of other key elements on society- government, employers, and the tax paying public.

And here's where we get to the tricky bit. Various people have told me variations on the "faculty will explode [if XYZ is even suggested]." What this says to me is that academic freedom is being abused to shelter and protect the worst kind of reactionary. Academic freedom is not the freedom to be shit at your job. Imagine if a scientist decided that gravity doesn't exist, or World War 2 didn't occur? They would be laughed at and scorned by their colleagues. But somehow, many academics retain the freedom to decide that failing to assure learning is somehow not their job. This doesn't include just their own singular approaches in their subjects, but includes the right to resist change which is absolutely necessary. Recently Australia's higher ed regulator published a piece developed by a collective of staff from across our sector that outlined three structural models for assuring learning in the age of AI. But, interestingly enough, I'm hearing from various angles of a lack of buy in from some academics, who continue to do what they want, safely hidden under the moist towelette of "academic freedom." I'll repeat, academic freedom is not the freedom to be shit at your job.

*Bonus Simpsons content

Just to close, I'm going to add a few links of interesting things I've read that I think everyone should read and allow to meander around their heads:

• AI disclosure? Maybe it's nunya- Kelly Webb-Davies

• The Verification-Value Paradox: A Normative Critique of Gen AI in Legal Practice- Joshua Yuvaraj

• On the essay in a time of GenAI- Tom Corbin and colleagues

I think I've written enough here for now, but just to seed my next post, I'm going to pose two questions

• Do we know what's worth learning during a degree in 2025?

• Is it time to reintroduce doubt, reduce the assessment scaffolding for students, and thereby (hopefully) reintroduce intrinsic motivation?
  

Until next time,

KM