Many game designers aim to optimize difficulty to make games that are "not too hard, not too easy." However, recent experiments have shown that even moderate difficulty can reduce player engagement. The present work investigates other design factors that may account for the purported benefits of difficulty, such as choice, novelty and suspense. These factors were manipulated in three design experiments involving over 20,000 play sessions of an online educational game. The first experiment (n=10,472) randomly assigned some players to a particular level of difficulty but allowed other players to freely choose their difficulty. Moderately difficult levels were most motivating when self-selected; yet, when difficulty was blindly assigned, the easiest games were most motivating. The second experiment (n=5,065) randomly assigned players to differing degrees of novelty. Moderate novelty was optimal, while too much or too little novelty reduced intrinsic motivation. A final experiment (n=6,511) investigated the role of suspense in "close games", where it was found to be beneficial. If difficulty decreases motivation while novelty and suspense increase it, then an implication for educational game designers is to make easy, interesting games that are "not too hard, not too boring".

@en

"Multi-armed bandits" offer a new paradigm for the AIassisted design of user interfaces. To help designers understand the potential, we present the results of two experimental comparisons between bandit algorithms and random assignment. Our studies are intended to show designers how bandits algorithms are able to rapidly explore an experimental design space and automatically select the optimal design configuration. Our present focus is on the optimization of a game design space. The results of our experiments show that bandits can make data-driven design more efficient and accessible to interface designers, but that human participation is essential to ensure that AI systems optimize for the right metric. Based on our results, we introduce several design lessons that help keep human design judgment in the loop. We also consider the future of human-technology teamwork in AI-assisted design and scientific inquiry. Finally, as bandits deploy fewer lowperforming conditions than typical experiments, we discuss ethical implications for bandits in large-scale experiments in education.@en