While digital education technologies have improved to make educational resources more available, the modes of interaction they implement remain largely unnatural for the learner. Modern sensor-enabled computer systems allow extending human-computer interfaces for multimodal communication. Advances in Artificial Intelligence allow interpreting the data collected from multimodal and multi-sensor devices. These insights can be used to support deliberate practice with personalised feedback and adaptation through Multimodal Learning Experiences (MLX). This chapter elaborates on the approaches, architectures, and methodologies in five different use cases that use multimodal learning analytics applications for deliberate practice.@en

Beginner table-tennis players require constant real-time feedback while learning the funda-mental techniques. However, due to various constraints such as the mentor’s inability to be around all the time, expensive sensors and equipment for sports training, beginners are unable to get the immediate real-time feedback they need during training. Sensors have been widely used to train beginners and novices for various skills development, including psychomotor skills. Sensors enable the collection of multimodal data which can be utilised with machine learning to classify training mistakes, give feedback, and further improve the learning outcomes. In this paper, we introduce the Table Tennis Tutor (T3), a multi-sensor system consisting of a smartphone device with its built-in sensors for collecting motion data and a Microsoft Kinect for tracking body position. We focused on the forehand stroke mistake detection. We collected a dataset recording an experienced table tennis player performing 260 short forehand strokes (correct) and mimicking 250 long forehand strokes (mistake). We analysed and annotated the multimodal data for training a recurrent neural network that classifies correct and incorrect strokes. To investigate the accuracy level of the afore-mentioned sensors, three combinations were validated in this study: smartphone sensors only, the Kinect only, and both devices combined. The results of the study show that smartphone sensors alone perform sub-par than the Kinect, but similar with better precision together with the Kinect. To further strengthen T3’s potential for training, an expert interview session was held virtually with a table tennis coach to investigate the coach’s perception of having a real-time feedback system to assist beginners during training sessions. The outcome of the interview shows positive expectations and provided more inputs that can be beneficial for the future implementations of the T3.

@en

Gamification aims at addressing problems in various fields such as the high dropout rates, the lack of engagement, isolation, or the lack of personalisation faced by Massive Open Online Courses (MOOC). Even though gamification is widely applied, not only in MOOCs, only few cases are meaningfully designed and empirically tested. The Gamification Design Process (GaDeP) aims to cover this gap. This article first briefly introduces GaDeP, presents the concept of meaningful gamification, and derives how it motivates the need for the Gamifire platform (as a scalable and platform-independent reference infrastructure for MOOC). Secondly, it defines the requirements for platformindependent gamification and describes the development of the Gamifire infrastructure. Thirdly we describe how Gamifire was successfully applied in four different cases. Finally, the applicability of GaDeP beyond MOOC is presented by reporting on a case study where GaDeP has been successfully applied by four student research and development projects. From both, the Gamifire cases and the GaDeP cases we derive the key contribution of this article: insights in the strengths and weaknesses of the Gamifire infrastructure as well as lessons learned about the applicability and limitations of the GaDeP framework. The paper ends detailing our future works and planned development activities.

@en

Sensors can monitor physical attributes and record multimodal data in order to provide feedback. The application calligraphy trainer, exploits these affordances in the context of handwriting learning. It records the expert’s handwriting performance to compute an expert model. The application then uses the expert model to provide guidance and feedback to the learners. However, new learners can be overwhelmed by the feedback as handwriting learning is a tedious task. This paper presents the pilot study done with the calligraphy trainer to evaluate the mental effort induced by various types of feedback provided by the application. Ten participants, five in the control group and five in the treatment group, who were Ph.D. students in the technology-enhanced learning domain, took part in the study. The participants used the application to learn three characters from the Devanagari script. The results show higher mental effort in the treatment group when all types of feedback are provided simultaneously. The mental efforts for individual feedback were similar to the control group. In conclusion, the feedback provided by the calligraphy trainer does not impose high mental effort and, therefore, the design considerations of the calligraphy trainer can be insightful for multimodal feedback designers.

@en

This paper introduces the Visual Inspection Tool (VIT) which supports researchers in the annotation of multimodal data as well as the processing and exploitation for learning purposes. While most of the existing Multimodal Learning Analytics (MMLA) solutions are tailor-made for specific learning tasks and sensors, the VIT addresses the data annotation for different types of learning tasks that can be captured with a customisable set of sensors in a flexible way. The VIT supports MMLA researchers in 1) triangulating multimodal data with video recordings; 2) segmenting the multimodal data into time-intervals and adding annotations to the time-intervals; 3) downloading the annotated dataset and using it for multimodal data analysis. The VIT is a crucial component that was so far missing in the available tools for MMLA research. By filling this gap we also identified an integrated workflow that characterises current MMLA research. We call this workflow the Multimodal Learning Analytics Pipeline, a toolkit for orchestration, the use and application of various MMLA tools.

@en

Goal achievement is a measure of success; this could be particularly true in Massive Online Open Courses (MOOC), which are approached by a massive audience with an enormous variety of needs. Despite the huge number of MOOC users, it is unlikely to find solutions that allow them to pursue their individual goals. To overcome this issue, we have developed a first prototype of the Stimulated Planning (SP) game element, inspired by the implementation intention theory and by our experience with strategy games. In this study, the SP prototype is presented and its performance assessed via a new combination of three methodologies: usability test, eye tracking and the retrospective think-aloud technique. The results are promising and contribute to the field of gamification of MOOC at three levels: at conceptual level, by introducing and interpreting a new theory for gamification design; at design level, by demonstrating that is possible to design advanced gamification for MOOCs; and at assessment level, by applying a new methodology for MOOC gamification design assessment.

@en

Experts are imperative for supporting expertise development in apprentices but learning from them is difficult. In many cases, there are shortages of experts to train apprentices. To address this issue, we use wearable sensors and augmented reality to record expert performance for supporting the training of apprentices. In this context, we present the conceptual framework which outlines different instructional design methodologies for training various attributes of a task. These instructional design methodologies are characterized by their dependencies on expert performance and experts as model for training. In addition, they exploit the affordances of modern wearable sensors and augmented reality. The framework also outlines a training workflow based on the 4C/ID model, a pedagogic model for complex learning, which ensures that all aspects of conventional training are considered. The paper concludes with application guidelines and examples along with reflection of the authors.@en

Gamification aims at addressing inherent problems of massive open online courses (MOOC): high dropouts, lack of engagement, isolation, lack of individualization. However, each MOOC platform offers different features and technical interfaces. Also, each platform collects different sets of data about user interaction, learning progress, or completion and success rates. This is an obstacle to the theoretically sound application of gamification in a vendor independent way and to the evaluation of the impact of gamification. We define our understanding of meaningful gamification, introduce requirements for platform-independent gamification, present the resulting Gamifire infrastructure, and describe application cases. We also point out planned development activities.

@en

Body-worn sensors can be used to capture, analyze, and replay human performance for training purposes. The key challenge to any such approach is to establish validity that the captured expert experience is actually suitable for training. In this paper, to evaluate this, we apply a questionnaire-based expert assessment and a complementary trainee knowledge assessment to study the approach adopted and the models generated with the WEKIT solution, a hardware and software application that complements Augmented Reality glasses with wearable sensor-actuator experience. This solution was developed using the ID4AR framework which as also developed within the WEKIT project. ID4AR framework is a domain agnostic framework which can be used to design augmented reality and sensor based applications for training. The study presented triangulates validity across three independent test-beds in the professional domains of aircraft maintenance, medical imaging, and astronaut training, with 61 experts completing the expert survey and 337 students completing the trainee knowledge test. Results show that the captured expert models were positively received in all three domains and the identified level of acceptance suggests that the solution is capable of capturing models for training purposes at large.

@en

Augmented reality and sensor technologies have been analysed extensively in several domains including education and training. Although, varieties of use cases and applications exist, these studies were conducted in controlled laboratory environments. This paper reports on the first user study of augmented reality prototype developed to support students to learn from trainers in professional domains using augmented reality and sensors. The prototype records the performance of trainers in the first phase to support students by making it available during practice in the second phase. The performance data is made available to both the students and trainers in the third phase for reflection. A total of 142 participants which included trainers and students from three professional domains, namely 1) aircraft maintenance 2) medical imaging and 3) astronaut training, evaluated the prototype. The trainers used the prototype to record their performance while the students used the prototype to learn from the recorded performance. Participants from the three professional domains evaluated the usability of the prototype by means of a questionnaire. Randomly selected participants were also interviewed to collect their opinions and suggestion for further usability improvement. Furthermore, they also evaluated the implementation of the instructional design methods, which were identified prior in a literature review, with a brief questionnaire. The questionnaire was designed to measure the acceptance of the implementation of instructional design methods and to evaluate its adherence to the authors definition. The results of this study show that the usability of the prototype is below expected standard acceptable level. The results of the questionnaire on the implementation of the instructional design methods varied show above average acceptance levels by both the trainers and the students in the three professional domains. To conclude, the prototype shows potential to be used in different domains to support expertise development.@en

Experts are imperative for training apprentices, but learning from experts is difficult. Experts often struggle to explicate and/or verbalize their knowledge or simply overlook important details due to internalization of their skills, which may make it more difficult for apprentices to learn from experts. In addition, the shortage of experts to support apprentices in one-to-one settings during trainings limits the development of apprentices. In this review, we investigate how augmented reality and sensor technology can be used to capture expert performance in such a way that the captured performance can be used to train apprentices without increasing the workload on experts. To this end, we have analysed 78 studies that have implemented augmented reality and sensor technology for training purposes. We explored how sensors have been used to capture expert performance with the intention of supporting apprentice training. Furthermore, we classified the instructional methods used by the studies according to the 4C/ID framework to understand how augmented reality and sensor technology have been used to support training. The results of this review show that augmented reality and sensor technology have the potential to capture expert performance for training purposes. The results also outline a methodological approach to how sensors and augmented reality learning environments can be designed for training using recorded expert performance.

@en

Gamification is well known as a design strategy used to generate a change in users’ behaviour, such as motivation. However, while in recent years interest in it has been growing, empirical evidence on the effects that the application of game elements can generate on users’ behaviour is still lacking. We present the results of a study as a step towards designing gamification with better understanding of the possible effects that each game element could generate on end users. By involving three groups of experts: game designers, learning scientists and specialists in technology-enhanced learning (TEL), we assessed a selected number of 21 game design patterns in relation to the effects these could generate on learning performance, goal achievement and engagement of learners if implemented in a Massive Online Open Course (MOOC). Based on quantitative and qualitative data collected, 9 game elements have been selected to be further investigated.

@en

Gamification has gained great interest recently in several fields. However, while the literature reports that a gamification design relying on external motivation only can lead users to cognitive dissonance, most gamification approaches use points, badges and leaderboards as dominant game elements. We present our developed testable predictions with the aim of investigating additional motivational theories (flow and persuasion) to argue for a deeper integration of gamification and the learning content at hand. Relying on expert selected game elements, we consequently derive design considerations to create gMOOCs, gamified massive online open courses, designed according to the principles of flow and persuasion. Our findings are the basis of our experiment and a contribution to the development of a new theoretical design for gamification.

@en

The WEKIT.one prototype is a platform for immersive procedural training with wearable sensors and Augmented Reality. Focusing on capture and re-enactment of human expertise, this work looks at the unique affordances of suitable hard- and software technologies. The practical challenges of interpreting expertise, using suitable sensors for its capture and specifying the means to describe and display to the novice are of central significance here. We link affordances with hardware devices, discussing their alternatives, including Microsoft Hololens, Thalmic Labs MYO, Alex Posture sensor, MyndPlay EEG headband, and a heart rate sensor. Following the selection of sensors, we describe integration and communication requirements for the prototype. We close with thoughts on the wider possibilities for implementation and next steps.

@en