Background: Readability metrics provide us with an objective and efficient way to assess the quality of educational texts. We can use the readability measures for finding assessment items that are difficult to read for a given grade level. Hard-to-read math word problems can put some students at a disadvantage if they are behind in their literacy learning. Despite their math abilities, these students can perform poorly on difficult-to-read word problems because of their poor reading skills. Less readable math tests can create equity issues for students who are relatively new to the language of assessment. Less readable test items can also affect the assessment's construct validity by partially measuring reading comprehension. Objectives: This study shows how large language models help us improve the readability of math assessment items. Methods: We analysed 250 test items from grades 3 to 5 of EngageNY, an open-source curriculum. We used the GPT-3 AI system to simplify the text of these math word problems. We used text prompts and the few-shot learning method for the simplification task. Results and Conclusions: On average, GPT-3 AI produced output passages that showed improvements in readability metrics, but the outputs had a large amount of noise and were often unrelated to the input. We used thresholds over text similarity metrics and changes in readability measures to filter out the noise. We found meaningful simplifications that can be given to item authors as suggestions for improvement. Takeaways: GPT-3 AI is capable of simplifying hard-to-read math word problems. The model generates noisy simplifications using text prompts or few-shot learning methods. The noise can be filtered using text similarity and readability measures. The meaningful simplifications AI produces are sound but not ready to be used as a direct replacement for the original items. To improve test quality, simplifications can be suggested to item authors at the time of digital question authoring.

@en

Intelligent Tutoring Systems (ITS) can only respond adaptively to the digital learning activities of the students. If students are learning offline without any digital devices, they have little or no means to receive personalized learning materials with the help of intelligent systems. This paper proposes a Paper-Digital Integration System that can provide offline learners equitable access to ITS capabilities by looking at their work on paper and giving personalized printable feedback. We analyzed data from a paper algebra assessment of N = 17 students and found mistakes that may generalize and help us offer adaptive paper-based recommendations to students. Our analysis showed us some specific algebra mistakes that may help in providing intelligent feedback.

@en

Technology aided learning is becoming increasingly popular. In some of the countries, online learning has taken over for traditional classroom-based learning. With this, educational data is being generated in vast amounts. Knowing the potential of this data, many education stakeholders have turned to evidence-based decision making to improve the learning outcomes of the students. EdOptimize platform provides extensive actionable insights for a range of stakeholders through a suite of 3 data dashboards, each one intended for a certain type of stakeholder. We have designed a conceptual model and data architecture that can generalize across many different edtech implementation scenarios. Our source code is available at https://github.com/PlaypowerLabs/EdOptimize @en

This paper explores several approaches for designing data-informed intelligent systems to create a positive impact. Two contrasting case studies in K12 education are used to illustrate design methods, questions and recommendations. The first case study addresses the poverty achievement gap in America and shows how product data can be used to identify areas of inequity in digital education. The second case study looks at the unintended consequences of automating data-driven optimization in the context of a digital math game. Together, the two case studies reveal generalizable knowledge that supports the design of intelligent feedback loops to create a positive impact. Further, this paper considers both the benefits and limitations of data feedback in complex social-technical systems.@en

Process Analysis is an emerging approach to discover meaningful knowledge from temporal educational data. The study presented in this paper shows how we used Process Analysis methods on the National Assessment of Educational Progress (NAEP) test data for modeling and predicting student test-taking behavior. Our process-oriented data exploration gave us insightful findings of how students were interacting with the digital assessment system over time. To discover what processes students were following during the NAEP Digital Assessment, we first developed an innovative set of research questions. Then, we used Process Analysis methods to answer these questions and created a set of features that described student behavior over time. These features were used to create an ensemble model that aimed to accurately predict the digital test-taking efficiency of the students taking NAEP. Our model emerged as one of the most successful models in the 2019 NAEP Data Mining Competition, scoring second place out of 89 teams.

@en

This paper examines the use of “pacing plots” to represent variations in student learning sequences within a digital curriculum. Pacing plots are an intuitive and flexible data visualizations that have a potential for revealing the diversity of blended classroom instructional models. By using curriculum pacing plots, we identified several common implementation patterns in real-world classrooms. After analyzing two years’ worth of data from over 150,000 students in a digital math curriculum, we found that a PCA and K-Means clustering approach was able to discover pedagogically relevant instructional practices.

@en

Introduction Currently, the learning science community is exploring the use of data-driven design to improve K12 educational systems. These “continuous-improvement systems” aim to align strategic goals, outcome metrics and human-computer system processes to support improved learning outcomes. However, the learning science community has only begun to apply systemic design to practical implementation of these systems. In this paper, we present several examples of data-driven design in K12 educational systems in order to identify aspects that can benefit from systemic design. Through these case studies, we focus on three concepts: 1) systemic designers can ensure that the system is capable of measuring successful outcomes; 2) systemic designers can ensure that system optimization will improve intended outcomes while minimizing unintended consequences; and 3) systemic designers can portray what a future with these continuous improvement systems will be like to the educational community, before any resources are committed to building the technology. Example #1: Ensure that the system is capable of measuring successful outcomes Data can be used to inform system stakeholders about the success of designed systems; that is, how well outcome measures align with system intentions. For instance, after providing an instructional activity (lecture, small group, video, etc) in class, a teacher might assign their students an “exit ticket” quiz to assess whether the instructional activity was successful. These quizzes support data-driven decisions about how to spend time and effort in the classroom. Variations in student performance give teachers an understanding of the students who need greater attention and the learning objectives that need greater attention. Further, digital data from exit tickets or other formative assessments can be aggregated across teachers to provide school administrators with continuous insight into the areas of need, such as students or teachers who need additional help or learning objectives that are posing special challenges. Providers of digital instruction can then aggregate usage and performance across many schools in order to identify successful and unsuccessful usage patterns. Data-driven continuous improvement can occur at multiple levels (i.e., teacher, school & software provider) when systems are designed to generate valid outcome metrics of success (goal achievement). Example #2: Ensure that system optimization will improve intended outcomes while minimizing unintended consequence Success metrics can be used by human teams and AI systems to drive continuous improvement. However, the optimization of metrics can produce unintended consequences when chosen metrics are not fully aligned to intended outcomes and when feedback loops about metric suitability are impoverished. In this case study, an online educational game is designed with the goal of motivating students to practice math problems. After being deployed online, the game attracts several thousand students a day; these players are randomly assigned to different game design variations to observe how the effects of different designs on key outcome metrics (e.g., duration of voluntary play). To investigate the role of AI in system design optimization, we implemented a UCB multi-armed bandit (a reinforcement learning AL/ML algorithm) to automatically test variations in the existing game parameter space (e.g., time limits, etc). The algorithm is designed to optimally balance the exploration of potential game designs with exploitation of the most successful designs; sometimes it will randomly search the game design space for configurations that maximize metrics (duration of voluntary play time) and sometimes it will deploy the most successful variations. While the algorithm worked as intended, the system “spun out of control” and primarily deployed malformed game designs that were maximizing the outcome metric but were misaligned with the original educational intent: the game variations were likely played for long periods of time because they were absurdly easy. This shows the pitfalls of having AI systems engage in automatic optimization without humans in the loop as a governing feedback system. Systemic designers need to design feedback systems to monitor system AI to ensure that outputs are meaningfully aligned to system intentions. Example #3: Portray what the future will be like Artificial intelligence has the potential to facilitate the work of teachers by reducing the effort required to use data to inform personalized instruction. However, AI can be intimidating or off-putting to teachers who do not understand its operation or intentions. In this case study, we deployed a teacher-facing recommendation system that uses reinforcement learning to continuously improve recommendation usefulness to teachers. To design a reinforcement learning AI system, there must be data representations of the system state, the space of possible actions and a reward signal tied to a success metric. In our case, the system state is student digital performance on learning activities, the action possibilities are the different digital items that teachers can next assign to a student and the reward signal occurs when teachers act upon a recommendation (i.e., when they assign those digital activities recommended by the system). This system embodies two key elements that diverge from most existing work in “adaptive learning” or “intelligent tutoring systems.” First, the system emphasizes human-technology teamwork, in contrast to human replacement, so that teachers are empowered by the assistance of the AI. Secondly, the artificial intelligence is deliberately constructed as an aggregation of human intelligence: the system learns from the activity-assignment decisions that are made by thousands of other human teachers and aggregates them into artificially intelligent recommendations. To promote adoption of this system, a key role for systemic design is making the intended future vision accessible and attractive to teachers and other stakeholders. Systemic designers can help to engage humans to participate in the decision making by presenting a glimpse of what a data-driven future might be like in the classroom. Conclusion Across these case studies, we show how systemic design can aid diverse participants in the implementation of data-driven design and optimization. Systemic design insight can contribute to the negotiation of meaningful and robust metrics of success, to the construction of human-in-the-loop governance of AI systems and to the representation of potential futures. We expect designers to play a crucial role in taming the complexity of practical AI-human systems and aligning system outcomes to sustainable, humanistic values. @en

Many game designers aim to optimize difficulty to make games that are "not too hard, not too easy." However, recent experiments have shown that even moderate difficulty can reduce player engagement. The present work investigates other design factors that may account for the purported benefits of difficulty, such as choice, novelty and suspense. These factors were manipulated in three design experiments involving over 20,000 play sessions of an online educational game. The first experiment (n=10,472) randomly assigned some players to a particular level of difficulty but allowed other players to freely choose their difficulty. Moderately difficult levels were most motivating when self-selected; yet, when difficulty was blindly assigned, the easiest games were most motivating. The second experiment (n=5,065) randomly assigned players to differing degrees of novelty. Moderate novelty was optimal, while too much or too little novelty reduced intrinsic motivation. A final experiment (n=6,511) investigated the role of suspense in "close games", where it was found to be beneficial. If difficulty decreases motivation while novelty and suspense increase it, then an implication for educational game designers is to make easy, interesting games that are "not too hard, not too boring".

@en

"Multi-armed bandits" offer a new paradigm for the AIassisted design of user interfaces. To help designers understand the potential, we present the results of two experimental comparisons between bandit algorithms and random assignment. Our studies are intended to show designers how bandits algorithms are able to rapidly explore an experimental design space and automatically select the optimal design configuration. Our present focus is on the optimization of a game design space. The results of our experiments show that bandits can make data-driven design more efficient and accessible to interface designers, but that human participation is essential to ensure that AI systems optimize for the right metric. Based on our results, we introduce several design lessons that help keep human design judgment in the loop. We also consider the future of human-technology teamwork in AI-assisted design and scientific inquiry. Finally, as bandits deploy fewer lowperforming conditions than typical experiments, we discuss ethical implications for bandits in large-scale experiments in education.@en