Exams and AI: Take-home exams may be a thing of the past
Just a few years ago, chatbots failed exam assignments. Now they can produce answers worthy of top grades. AU Associate Professor Carsten Bergenholtz has followed developments closely and is now advising against classic take-home exams.
What is GAI?
GAI stands for generative artificial intelligence and is commonly known as chatbots that can generate responses based on the questions or prompts you provide them with. GAI technology is able to do so because it’s based on large language models that draw on existing text, patterns, or structures.
When Carsten Bergenholtz first tested a chatbot on the exam syllabus in his Bachelor's course and scientific methodological course, the conclusion was clear.
"They can't do anything, they fail," says Carsten Bergenholtz, who is an associate professor at the Department of Management at Aarhus BSS. That was in March 2022, and the conclusion was sent to the department's head of studies administration, Christian Waldstrøm. But a few months later, he found himself having to put pen to paper once again.
"You have to ignore everything I've written in the past. Now it has actually started to pass some of my exam questions," says Carsten Bergenholtz, who has also been part of the working group, which has been involved in drawing up AU's official guidelines for the use of GAI. He has also followed research in the field closely and has helped publish a scientific article on the impact of GAI on student performance in exam situations.
Today, the technology has advanced to the point where the best chatbots can consistently produce exam answers at the equivalent of a grade 10 or 12 in many standard subjects using only a single prompt, he says.
"You have to be very careful about saying that GAI can't X by definition," says Carsten Bergenholtz
"You have to be very careful about saying that GAI can't do X by definition. The last few years have shown that the boundary is constantly being moved," Carsten Bergenholtz points out, and continues:
"At this time, we are not at the point where it is possible to write a Bachelor's project with a single prompt. But we are at a point where skilled people can write a Master’s thesis in a matter of hours or a day,” he says.
Medium-length take-home exams may be a thing of the past
The arrival of GAI has already had consequences at the Department of Management, where Carsten Bergenholtz and his colleagues teach students in business administration. Here, they have scaled back—and almost entirely phased out—longer take-home exams lasting between three and seven days.
"You can create a four-day take-home exam that a chatbot can't solve today. But the question is whether it can in a year and a half. It's not a risk I would dare to take myself," Carsten Bergenholtz stresses, referring to the fact that teachers typically plan courses and exam formats 1 to 1.5 years in advance. He also passes this recommendation on to his colleagues in other departments.
However, the solution is not to go back to pen-and-paper exams and traditional memory tests.
According to the associate professor, there are still a number of adjustments that can be made.
"We have different types of exam formats, and if you have a diverse exam portfolio on the given degree programme, then I believe that we can reasonably test what the students can do both with and without the use of AI," he points out.
This would mean short on-site exams where the students have access to their notes, but not to AI. Larger project-based assignments linked more closely to teaching through supervision and ongoing activities. And oral defences of assignments, where the students can demonstrate that they understand the work they have submitted.
Bachelor's projects and Master's theses in particular give rise to considerations for the future, according to Carsten Bergenholtz.
"Three quarters of an hour to defend a Bachelor's project is no longer quite enough," says Carsten Bergenholtz
"I am somewhat worried about our Bachelor's projects and Master's theses, considering that a very large part of the assignment can be solved and written by GAI. But at our department, we have a principle that we have an oral defence. They may have to be longer, as three-quarters of an hour to defend a Bachelor's project is no longer quite sufficient," he says.
Chatbots don't know if they’re good or not
Although it may seem like AI can solve almost any task, it is Carsten Bergenholtz's experience that this is far from the case. AI is still best in areas where it can draw on large amounts of existing knowledge, he says.
"It is good at mathematics because it has read a lot of mathematical texts, and this is also an area similar to programming, where it can immediately find out whether something works or not," says Carsten Bergenholtz.
Conversely, the results from chatbots quickly become more uncertain when the technology is to be used on new phenomena or complex issues, where there is not much previous research or clear criteria for what is considered to be a good solution.
"If there is no previous literature, and if there is no standardised way to assess quality, then a chatbot is just generally worse," says Carsten Bergenholtz, mentioning the humanities and other areas where lesser-known phenomena are investigated, or where phenomena are investigated from a new angle that has not been written about before.
At the same time, he emphasizes that AI is also not good at assessing its own limitations.
“An AI is not good at judging for itself when it has crossed the line. So when I say that it can do a great many things, that claim should definitely be treated with caution,” he points out.
GAI must not replace learning
Although Carsten Bergenholtz is generally optimistic about GAI, he warns against confusing technology with learning. Because reading a chatbot-generated text and copying it into an exam paper is not the same as gaining an understanding of the material yourself, he emphasizes.
"That a GAI has produced a page of text, after which you look at the text a bit, skim it and copy-paste it – and afterwards think that it makes sense and that you have understood it," he says, and continues:
"It’s not at all the same kind of learning and building of internal mental models. The degree programmes must ensure that the students know what it means to learn and what impact GAI can have on learning," he concludes.
I’m allowed to use GAI in my studies and for my exams - but should I?
This text is machine translated and post-edited by Mie Skov Jeppesen.