In Greek mythology, Iris is the goddess in charge of conveying important messages from Zeus, father of all gods. Today, Iris answers questions raised by students at TUM Campus Heilbronn on behalf of Professor of Software Engineering Stephan Krusche. Prof. Krusche and his team have developed a chatbot for the field of education, in particular for programming courses.
One of the requirements for Iris to come into existence was met as far back as 2016, when Prof. Krusche followed his passion to support all students and developed a learning platform for Technical University of Munich which he named Artemis after the Greek goddess of hunting. “We are continuously exploring ways to further develop the platform to improve teaching,” he says. Currently, students’ questions are answered predominantly by human tutors, but “they are not available 24/7.”
Help at all hours
Iris can help students solve problems and answer questions at any time. While availability is one factor, students’ diverse personality features also play a crucial role in the development of the chatbot. Prof. Krusche says: “We want to enable students to get feedback even if they have difficulty communicating.” One dilemma: “We should actually teach them to leave their comfort zone and have the courage to ask questions.” Preliminary results of the team’s research indicate that the chatbot lowers students’ common inhibitions to ask for help. For young, slightly insecure academics to realize their concerns are not banal could boost their self-confidence and improve communication in their study routine.
Solutions from the all-knowing goddess
Iris took on this role in October 2023. “Students receive messages from the all-knowing goddess,” Prof. Krusche says, laughing. When creating a didactic chatbot, the greatest challenge is ensuring that it does not merely offer solutions to problems but that it gives food for thought, leading users in the right direction just as human tutors would. This means the chatbot first must determine whether a question makes sense; then, it can offer a didactically sensible response. This may sound easy, but it is technically complex and calls for intelligent prompt engineering. At the interface with the large language models (LLMs), giving clear instructions is the be-all and end-all. Questions are processed in three steps: “We use the initial prompt to determine whether the question asked makes sense, that is, whether it serves to achieve the goal. Then, the question is answered based on our instructions, and the response is evaluated to ensure it makes sense from a didactical point of view and instructions have been followed.” Another great challenge is that some LLMs tend to convey inaccurate information convincingly, a feature also referred to as hallucinating.
A parrot with hallucinations
How do chatbots actually work? Prof. Krusche uses the example of ChatGPT: “LLMs acquire large amounts of information from texts on the internet, from books and documents available online, and even from large companies’ source codes.” This means that they are trained using several billions of lines of text. Trained chatbots can answer questions by determining with statistical probability which answer most likely matches the respective question. However, the system is not flawless. Prof. Krusche says: “According to some critics, chatbots are statistical parrots that can only repeat things written somewhere in a similar manner, potentially generating false facts.” Chatbots are not able to generate truly new knowledge; rather, they draw their information from the past. To increase the probability of answers being correct, chatbots must be trained more and fine-tuned for specific fields of application, and the information provided for context must be selected diligently. Prof. Krusche and his team currently are working on these tasks.
Visions of the future
While the large number of students using Iris since its inception testify to the direction in which Prof. Krusche is headed, he recognizes the potential for optimization. “We want to enhance data protection and also address users’ needs and knowledge in an even more individualized manner.” The plan is to employ LLMs to be able to generate suitable task variants for diverse levels of knowledge a few years from now and, thus, cater to users’ strengths and weaknesses in a targeted way. “We are working to enable Iris to boost students’ motivation,” says Prof. Krusche. Personalization can help: Individualized positive messages are sent to students who have achieved specific learning goals. If a student has been inactive on the platform for an extended period, it could mean he or she is overtaxed. In this case, the system would ask the user what is wrong. Going forward, in addition to solving problems Iris could contribute to individualizing teaching and, thus, increase motivation and learning success.
Source of text: Mindshift, 8th edition
Publisher: TUM Campus Heilbronn, Bildungscampus 9, 74076 Heilbronn