This report holds a subset of an integrated thesis involving the development and testing of an educational tool named ”Playlist,” aimed at improving the learning experience of Computer Science & Engineering bachelor students at the TU Delft. The Playlist tool was developed as a feature for the existing tool called Skill Circuits, supporting students in creating personalized sequences of learning activities for their study sessions. The integrated research contributes to the complexities of modern digital learning environments and the challenges students face in navigating vast amounts of educational content.  The study uses a design-based research approach, which involves iterative development and research cycles and including students in the evaluation phases. This report focuses on the description and design of these phases. For each phase, it then reports on how the design of the Playlist tool utilized user feedback gathered through surveys and focus group sessions with Computer Science students.  Key findings in the integrated thesis indicate that playlists can improve student engagement, motivation, and overall learning experience by supporting students in finding learning activities aligned with their learning preferences and goals. The research highlights the importance of providing students with tools that improve navigating the learning environment and support them in self-directed learning. The Playlist feature not only aids students in organizing their learning activities but also promotes a holistic approach to education by incorporating supporting activities such as reflection and taking breaks.  The integrated thesis makes two contributions: it serves as an example of the design and implementation of personalization tools in higher education and provides insights into integrating such tools into an existing educational system. The thesis aims to demonstrate the potential of designing tools in an iterative manner to personalize learning and support students in achieving their learning goals.

This thesis involves the development and testing of an educational tool named ”Playlist,” aimed at improving the learning experience of Computer Science & Engineering bachelor students at the TU Delft. The Playlist tool was developed as a feature for the existing tool called Skill Circuits, supporting students in creating personalized sequences of learning activities for their study sessions. This research contributes to the complexities of modern digital learning environments and the challenges students face in navigating vast amounts of educational content.

This report holds the full description of the integrated thesis. The first part including the chapters of ”Fostering a personal sequence of learning activities in Skill Circuits”.
The study uses a design-based research approach, which involves iterative development and research cycles and including students in the evaluation phases. By drawing on educational theories and addressing practical needs identified at TU Delft, the Playlist tool was designed to seamlessly integrate into the existing learning environment. The tool’s functionalities were prioritized based on user feedback provided by Computer Science students and gathered through surveys and focus group sessions.

Key findings indicate that playlists can improve student engagement, motivation, and overall learning experience by supporting students in finding learning activities aligned with their learning preferences and goals. The research highlights the importance of providing students with tools that improve navigating the learning environment and support them in self-directed learning. The Playlist feature not only aids students in organizing their learning activities but also promotes a holistic approach to education by incorporating supporting activities such as reflection and taking breaks.

The integrated thesis hopes to demonstrate the potential and importance of designing tools to personalize learning and support students in achieving their learning goals. It also explains why personalization tools in higher education should be implemented in an iterative manner.

The rising number of students in computer science presents a challenge for educators to analyse student work and identify problem areas at scale. Learning analytics systems often fall short in offering detailed, actionable insights that educators can use to enhance their teaching. To address this, we propose a system leveraging Large Language Models to analyse programming submissions and generate actionable analytics. The study focused on two primary research questions regarding the accuracy of LLMs in classifying programming submissions and identifying common issues, and educators' perceptions of the usefulness of the system. Through a system evaluation and focus group, we found that LLMs can analyse SQL assignment submissions with reasonable accuracy and can identify common issues. Educators found the insights potentially useful but noted areas for improvement such as the need for concrete statistics, accurate lists of submissions with specific issues, and more readable reports. Future research should address these implications for the system, evaluate the system in other courses with different programming languages, and involve stakeholders directly related to the course.

Learning a Foreign Language is crucial in today’s interconnected world, as it enhances cross-cultural communication and fosters personal growth. Despite its many benefits, learners often face significant challenges, primarily due to Anxiety. To address these challenges, we introduce ELLA, a Dutch tutoring system powered by GPT-4. ELLA features an innovative Episodic Memory architecture, enabling learners to review and delete system-generated episodes from past interactions, thereby giving them a sense of control. Through this thesis, we explore the impact of Episodic Memory on Foreign Language Learners' Speaking Anxiety and the associated learning benefits. The investigation is divided into four sub-questions: the relationship between Foreign Language Anxiety and Language Learning, the optimization of the Episodic Memory framework for Learning, the influence of Episodic Memory on speaking-related State Anxiety, and the perceived benefits for learners with Foreign Language Anxiety. Our experimental study demonstrated that ELLA has the potential to reduce speaking-related State Anxiety in Dutch learners and offers significant learning benefits as perceived by users. However, areas for improvement are identified, such as the need for an Automatic Speech Recognition system that accurately recognizes foreign accents and a Large Language Model fine-tuned for the Language Learning task. Future research should focus on these enhancements, evaluate the system in different languages using standardized pre-and post-tests, and involve longer interactions with a larger and more diverse user base over time.

With the fast integration of Machine Learning(ML) into several industries, the motivation to develop effective pedagogical strategies for teaching this complex and evolving field has become critical. Machine Learning, once mainly a topic in Computer Science Bachelor programs, is now widely integrated into various majors and introduced at earlier educational stages, including high school and secondary school. However, the reduced research focus on ML pedagogy results in a lack of standard teaching methods compared to other science-related subjects, which have established norms for topic introduction, teaching tools, and assessment methods. With inspiration from other fields, this research aims to look into using interactive visualizations in teaching ML topics, more specifically in teaching Gradient Descent and Principal Component Analysis (PCA). The research includes the development and evaluation of Jupyter Notebooks for introducing these visualizations to students. The targeted student population is composed of Computer Science and Engineering Bachelor students who have not yet followed any Machine Learning courses but have the necessary background knowledge, namely calculus, linear algebra, and statistics knowledge. The evaluation of this teaching method measures the knowledge gained and the motivation of the students, compared to a static version of the materials. The results of the evaluation have shown a significant effect of interactive visualizations on knowledge gained related to PCA. The evaluation has not identified any difference in knowledge gain for gradient descent and learning motivation for both topics. With these results, we contribute to the body of evidence-based teaching methods in Machine Learning and identify further research needed to generalize the effect of interactive visualizations as a teaching method for teaching ML basic concepts.

Memorizing vocabulary is a key part of second language acquisition; however, many people rely on rote memorization. Despite the proven effectiveness of the mnemonic keyword method for learning vocabulary, its usage remains limited because coming up with keywords can be time-consuming and creatively demanding. Previous solutions for automatically generating mnemonic keywords are inflexible and outdated, given the advancements in the field of Natural Language Processing driven by large language models (LLMs) in recent years. This study's research questions focus on how LLMs can be used to generate personalized mnemonic keywords and how these personalized mnemonics impact the learning experience and outcome compared to non-personalized approaches. By designing Keymagine, an LLM-powered system for keyword generation, we show that LLMs can effectively generate keywords through In-Context Learning and be personalized through user feedback. In an experimental evaluation, students (N = 22) used both Keymagine-generated and other automatically generated keywords to learn 36 German words. Results demonstrated a significantly higher perceived helpfulness of Keymagine-generated keywords and a significantly higher rate of recall.

Within the field of computer science (CS), women are under-represented in the workforce and education settings. As Massive Open Online Courses (MOOCs) grow in popularity, understanding the gender differences in reasons for enrolment and engagement remains crucial to improving learner outcomes. This study investigates why men and women enrol in introductory CS MOOCs and how they interact with these courses. This is done with data from four MOOCs offered by TU Delft between 2015 and 2022.

Using survey data for the learners' reasons for enrolment and clickstream data for their behavioural engagement, we applied k-means clustering to identify engagement patterns. Our analysis reveals that the three most important reasons for men and women are career-related, interest-related, and degree-related, in that order. Women are more likely to enrol for career-related reasons than men, while men are more driven by interest in the topic than women. Women also tend to show lower engagement levels compared to men, who are more likely to complete the courses. We found no significant association between reasons for enrolment and engagement for men and women.

These findings highlight the need for gender-sensitive course design strategies to enhance engagement and completion rates. Providing mentorship opportunities, fostering peer interaction platforms, and highlighting role models in the field could also help create a more inclusive learning environment. Future research should explore specific learner challenges and incorporate a more comprehensive engagement model.

The online website Skill Circuits is a tool developed by teachers at the Delft University of Technology. Skill Circuits is an online learning tool that presents students with a node-link (i.e. a tree) structure where each node represents a skill, containing tasks that aim to teach the skill. The website aims to benefit teachers, by letting them think how all the 'skills' in their course relate and whether or not they have sufficient material to teach these skills. At the same time, the website aims to benefit students by giving them a learning path to follow, where each step is a small task, and each skill is visually connected to previous skills, showing how the learning material is connected.
This thesis aimed to help new teachers understand the concept of these node-link structures --- called skill circuits --- and also expand the website with features to help them evaluate their skill circuits, both before and during the course.
A documentation help page was developed, which explained the concept and intended usage of the website and was accompanied by a proof-of-concept skill circuit explaining the same topics. Tools to help evaluate a skill circuit were also developed, which included tools that could be used before the course was held, and tools that used analytics generated by students. One noteworthy tool took inspiration from the Constructive Alignment principle, where a teacher can label parts of their skill circuit with learning objectives to see if there is alignment with their course material.
All of these features were evaluated with a small focus group consisting of teachers, which showed enthusiasm toward the proposed features.

Background
This research explores the role of visualization tools in enhancing collaborative learning in virtual reality environments.
Methodology
An experiment was conducted, involving two groups of participants, each tasked with collaboratively solving a complex maze in a virtual reality environment. Both groups experienced two conditions: the experimental condition, where participants had access to visualization tools such as visual cones and highlighting capabilities, and the
controlled condition, where these tools were not available. The participants’ situational awareness were assessed using both the Situation Awareness Rating Technique questionnaire and the Situational Awareness Linked Indicators Adapted to Novel Tasks framework.Results
The Situational Awareness Linked Indicators Adapted to Novel Tasks scores were higher for the experimental session for Group 2, while being lower for Group 1. However, the Situation Awareness Rating Technique scores were generally lowerin the experimental condition for both groups.
Conclusion
These results demonstrate the potential virtual reality has as a platform for collaborative learning and highlight the need for further research into the design of effective visualization tools and the measurement of situational awareness in virtual reality. Furthermore, this research opens up new possibilities into the design of virtual reality environments for collaborative learning and the potential of virtual reality as a tool for enhancing collaborative learning.

Background
Learning is a core part of how we grow as humans. Over the last few years it has been shown increasingly that learning in groups tends to be more effective than learning individually. This research aims to show a link between an individual’s Situational awareness and the degree of collaboration.
Method
An experiment was conducted where groups of participants navigated a maze. The maze was designed to require collaboration to reach the end. Data was gathered about both the situational awareness of the participants and their levels ofSocial Modes of Co-Construction.
Conclusion
The current data suggests no strong correlation between Situational Awareness and Levels of Social Modes of Co-Construction. However, more data will need to be gathered to increase the reliability of this conclusion.

Background: The advancements in Virtual Reality (VR) technology have opened up new possibilities for studying human dynamics and conducting experiments in immersive environments. To gain insights into collaborative learning and how it can be enhanced, an experiment was conducted to investigate the effects of visualizations of activities in VR on social modes of co-construction, which is to what extent learners refer to their partner's contribution.

Methods: A maze, specifically designed for collaboration, has been chosen to use for this study and visualization tools such as laser pointing and vision cones were made available for certain sessions.

Results.: The findings from this maze experiment did not provide conclusive evidence regarding the impact of visualization tools on social modes of co-construction, mainly due to the limited comparison material available. An interesting finding however, is that sessions with visualization tools tend to have more distracted participants compared to the absence of them.

Conclusion: Further research is needed to examine the relationship between visualization tools and social modes of co-construction, as well as to explore whether the observed distractions are specific to the participants in this study or representative of a bigger spectrum. By addressing these aspects, we can gain a better understanding of the role of visualization tools in collaborative learning and uncover strategies to mitigate any potential distractions they may introduce.

Background
Virtual Reality is an increasingly hot topic these days. As a user, you can play more and more high-quality games and strides are being made to use this medium for education and new kinds of workspaces. Previous research on Virtual Reality shows that it could improve teamwork by letting users share information easier. Users in Virtual Reality can communicate in ways exclusive to the medium, like highlighting exact locations, so this paper analyzed how effective verbal communication is in this situation.

Methods
To analyze how Virtual Reality influences group interaction, an experiment was set up that ascertained in the presence and absence of visualizations of user actions whether a group being more aware influences how they communicate in Virtual Reality. To approach this, this paper used a maze in Virtual Reality where participants need to communicate hints of their color.

Results
The data encoding of the experiment showed that when a group's Shared Situational Awareness decreased between sessions, their level of Social Modes of Co-Construction did as well to a similar degree. When the awareness remained around the same level, only decreasing slightly, the level of social modes remained as well, going slightly up or down depending on the reading of the data.

Conclusions
From the research experiment, this paper observes there is a positive correlation between group Shared Situational Awareness and their Social Modes of Co-Construction.
Because of limited time and data sources, it is suggested that this experiment is reproduced over a longer period on a larger scale to support these findings.

Previous studies have found that there is a strong correlation between the social presence of a teacher and perceived performance of students, and that an increased cognitive load decreases the effectiveness in problem solving. The objective of this study was to find out to what extend holograms can be used to improve the social presence of the teacher and reduce the cognitive load within distance learning. This was done by recording a lecture and using that recording to represent the teacher in different ways: a zoom lecture with a small video of the teacher on the slides as a control group, slides with the head of the teacher on a telepresence robot, slides with the teacher on a holographic screen and a lecture in a VR environment. 22 participants filled in questionnaires about the social presence and cognitive load after attending one of the 4 lecture types. The small number of participants made it hard to find any significant differences between the lecture groups while using non-parametric tests but the medians hinted that the VR lecture improved the social presence of the teacher compared to the zoom lecture and the telepresence robot lecture reduced the cognitive load compared to the zoom lecture.

Deficiencies in online communication software inspired the HoloLearn project that proposes to aid hybrid education via a hologram. Yet the epistemic and psychological effects of this medium remain largely uncharted. To help integrate the hologram into a hybrid education system, the present study conducted experiments which explored the nature of the hologram itself. With a lecture on Japanese History until 1603, this research compared the hologram's twofold effects on students with Zoom, an already popular option for online lecturing. The study concludes that there is no statistically significant difference between the two teaching modalities, and also suggests that the hologram, depending on the viewer, can be either engaging or distracting. Flaws in experimental design and other external constraints might have exerted some influence on these findings.

Instructional Design is a discipline and a science that has existed for decades. There has been research done into the most effective instructional designs for different study disciplines, the same cannot be said about machine learning. As ML is a relatively young discipline, no research has been done into instructional designs and teaching methods of machine learning. This paper investigates the existing instructional de- signs of ML by analysing various bachelor ML courses at different universities, the compare the results with the existing instructional designs at the TU Delft using the help of interviews with the education experts at TU Delft in order to draw conclusions regarding the differences and how to improve on the course at the TU Delft.

Machine learning is becoming more and more applied within business and academia alike. This has led researchers to look inwards and discuss whether the current way of teaching and learning machine learning is the right way. Within this train of thought, one must investigate the characteristics of teaching and learning. Namely, what are the competences required to learn, and in what order should these be learned. This research paper focuses on realizing the answer to that question by combining past research papers on how to define competences, how to establish an order of competences, and how to establish competences through the help of questionnaires answered by academics within the machine learning sector teaching Computer Science students. The results from the academic analysis and the resulting data from the questionnaires are combined to organize a result, established through two different methods used by researchers. To conclude, the paper is reflected upon, and its merits, demerits, and trade-offs are summarized, presenting a final recommendation for any future work done on this topic.

Even though machine learning field is growing rapidly, research on education of machine learning is scarce. In this paper a research about creating assessments in the machine learning’s context is presented. The aim of the research is to answer how to design assessments that reliably show progress on a module in machine learning. Learning outcomes and Bloom’s taxonomy are used to make the research reproducible, and draw conclusions. One of the main conclusions drawn in this paper is that verbs that are used in learning outcomes can also be used to find the appropriate question type (e.g. open ended, multiple-choice) to assess that learning outcome. Additionally, this paper concludes there is no strict procedure of creating assessment questions. Therefore, a guideline is created by the researcher and presented in the paper. Lastly, four questions are created using this guideline and evaluated with interviews with three machine learning professors.

Distance learning brings all sorts of advantages. The ability to follow lectures at home can save people transportation costs and time. Teaching through videoconferencing software such as Zoom is one of the methods to learn remotely. To explore new and better teaching methods, the possibility of using a telepresence robot is investigated. To determine its worth, an experiment is done to compare the learning outcomes of telepresence robot lectures to Zoom lectures. Differences in risk of eye-strain is also compared, as most distance learning methods are digital. An increase in risk of eye-strain may negatively impact the choice of adopting the learning method. Due to the small sample size, non-parametric tests were conducted. Concluding from the research, neither Zoom nor telepresence robot lectures had better learning outcomes compared to the other. The learning outcomes of both lectures were similar. Risks of eye-strain also did not differ between the two. Further analysis was done comparing the telepresence robot lecture with a hologram and a virtual reality lecture, as these were also part of the experiment. Similarly to zoom, there was no significant evidence that telepresence robot lectures had different learning outcomes or risks of eye-strain than holograms and virtual reality lectures.

This study examines the effects of holograms when used for online learning. The research question is: "How does a holographic teacher projection affect engagement of the students with the learning material?". The engagement of students was compared in different lectures. The base lecture was a zoom lecture. The holographic lectures were a VR lecture, a HoloDisplay lecture and a robot lecture. A big experiment was done where different people attended one of those 4 lectures. After the participants were done watching the lectures they had to fill in questionnaires. The data of these questionnaires was then used to compare the lectures. The main conclusions are that Zoom lectures definitely take more attention than holographic lectures. Another conclusion was that the HoloDisplay and the Robot lecture are more engaging than a normal zoom lecture, this conclusion was backed up with statistical tests. There was no evidence that either the HoloDisplay or the Robot lecture was more engaging than the other one. Both of them need to be looked into further and be developed further. These ways of teaching can definitely be the future of online learning.