While reinforcement learning (RL) has made great advances in scalability, exploration and partial observability are still active research topics. In contrast, Bayesian RL (BRL) provides a principled answer to both state estimation and the exploration-exploitation trade-off, but struggles to scale. To tackle this challenge, BRL frameworks with various prior assumptions have been proposed, with varied success. This work presents a representation-agnostic formulation of BRL under partially observability, unifying the previous models under one theoretical umbrella. To demonstrate its practical significance we also propose a novel derivation, Bayes-Adaptive Deep Dropout rl (BADDr), based on dropout networks. Under this parameterization, in contrast to previous work, the belief over the state and dynamics is a more scalable inference problem. We choose actions through Monte-Carlo tree search and empirically show that our method is competitive with state-of-the-art BRL methods on small domains while being able to solve much larger ones.

@en

Model-based Bayesian Reinforcement Learning (BRL) provides a principled solution to dealing with the exploration-exploitation trade-off, but such methods typically assume a fully observable environments. The few Bayesian RL methods that are applicable in partially observable domains, such as the Bayes-Adaptive POMDP (BA-POMDP), scale poorly. To address this issue, we introduce the Factored BA-POMDP model (FBA-POMDP), a framework that is able to learn a compact model of the dynamics by exploiting the underlying structure of a POMDP. The FBA-POMDP framework casts the problem as a planning task, for which we adapt the Monte-Carlo Tree Search planning algorithm and develop a belief tracking method to approximate the joint posterior over the state and model variables. Our empirical results show that this method outperforms a number of BRL baselines and is able to learn efficiently when the factorization is known, as well as learn both the factorization and the model parameters simultaneously.@en

While the POMDP has proven to be a powerful framework to model and solve partially observable stochastic problems, it assumes ac- curate and complete knowledge of the environment. When such information is not available, as is the case in many real world appli- cations, one must learn such a model. The BA-POMDP considers the model as part of the hidden state and explicitly considers the uncertainty over it, and as a result transforms the learning problem into a planning problem. This model, however, grows exponentially with the underlying POMDP size, and becomes intractable for non- trivial problems. In this article we propose a factored framework, the FBA-POMDP that represents the model as a Bayes-Net, dras- tically decreasing the number of parameters required to describe the dynamics of the environment. We demonstrate that the our ap- proach allows solvers to tackle problems much larger than possible in the BA-POMDP.@en

The Association for the Advancement of Artifcial Intelligence, in cooperation with Stanford University's Department of Computer Science, presented the 2018 Spring Symposium Series, held March 26-28, 2018, on the campus of Stanford University. The seven symposia held were AI and Society: Ethics, Safety, and Trustworthiness in Intelligent Agents; Artifcial Intelligence for the Internet of Everything; Beyond Machine Intelligence: Understanding Cognitive Bias and Humanity for WellBeing AI; Data-Effcient Reinforcement Learning; The Design of the User Experience for Artifcial Intelligence (the UX of AI); Integrated Representation, Reasoning, Learning, and Execution for Goal-Directed Autonomy; Learning, Inference, and Control of Multiagent Systems. This report, compiled from organizers of the symposia, summarizes the research of the symposia that took place.

@en