On Bondage

Evening all,

You might be thinking "Guerilla Warfare turned into a hardcore sex blog so gradually I didn't notice" but, thankfully, no. It's not for me to kinkshame what consenting adults do on their own time, but I have something very different in mind. What brings me here, like a thousand monkeys at a thousand typewriters, is a conversation I had recently with an academic colleague at an institution I won't mention. I hope he doesn't mind me unpacking our discussion. As much as I appreciate the odd dunk on academics, I also want to discuss my appreciation and admiration for those who are wrestling and struggling with the issues that I also see from my spot down in the slop. So, at the peril of restating things or belaboring previously made points, let me lay out where my head is after the conversation.

So, this gentleman is an academic in a humanities discipline, and has also been an Academic Integrity Officer in the past. While we were talking he said "We basically haven't had a [plagiarism-type] case in a year." He and I both understood that this stands to reason. Word generators being available and trained on millions of the same type of thing will produce generic, but distinct, versions of the thing. He also told me that fail rates have collapsed, and pass rates have skyrocketed. This too is unsurprising. If there are no effective methods to detect AI (there are not, shameless paper plug), and no structural impediment to avoiding accountability for learning, this is what's going to happen.

Now, what the Australian higher ed sector has landed on three major approaches to (nominally) assuring learning:

1. Programmatic Assessment- assessing and assuring learning across the entirety of a program of study, rather than subject by subject;

2. Subject-by-Subject- Assuring (or securing) every subject individually across a program of study, or;

3. A hybrid approach- for example, securing a core set of mandatory subjects, and then securing all others (like electives) subject-by-subject.

Now, that's all fine and dandy but, as always, the devil is in the detail. I'm less concerned with Programmatic assessment (less concerned, not unconcerned), and I'm going to focus on subject-by-subject, where the vast majority of summative assessment has always lived, and continues to live today.

So where is this devilish detail? In my mind it's not in the assessment tasks themselves, it's in the structure and governance of the overall assessment package for a subject. Let me give you an example. If a subject has 3 assessments, one of them must be in-person, the others being "open AI", it is the in-person assessment that provides the assurance, right? Well, maybe. If we factor in the skyrocketing pass rates, this tells us that students are somehow a lot better than they were a few years ago, or that the marks being awarded to "open AI" assessments are not necessarily a reflection of the student's capability or learning. That's where the in-person assessment comes in to put a check on rampant AI use. But let's also say a student gets 35/50 in the open assessments, and 15/50 in the invigilated, in-person assessments. They've passed the subject, but what does this tell us about their learning? Not much. I can hear the lowing of the cattle from here "students get stressed."

That's one interpretation. The other is that student's lack of learning is being exposed where they cannot rely on Claude. But when the in-person assessments are not hurdles, that exposure goes unremarked, and the student is credited with a pass regardless. You see the problem, right? It's not how good or bad, how supported or not, the assessment is, it's the policy that allows failure to masquerade as accomplishment. Repeat this many times across a degree and you have a dubiously educated student graduating. This is what I mean by bondage: each subject convenor must pass students who may or may not have learned anything much, because they have their hands tied behind their back, from a policy perspective.

Now, let's remind ourselves (at least us Australians) of the standards expected of universities. The Higher Education Standards Framework (the law of the land) requires, under section 1.4.4, that "On completion of a course of study, students have demonstrated the learning outcomes specified for the course of study, whether assessed at unit level, course level, or in combination." It is our job in higher ed to assure the public of that fact, for each and every student. But how can do this satisfactorily when hands are tied?

I've expressed this concern before, but I saw a nice quote that expresses my broader fears: "If the public loses faith in what they hear & see & truth becomes a matter of opinion, then power flows to those whose opinions are most prominent." If the example that I described above becomes common place, it seems entirely probably that the public will lose faith, if it hasn't already. We shouldn't be creating our own fig leaves, it seems to me the least the public can expect of us.

Until next time,

KM