In the field of Artificial Intelligence (AI), techniques like Reinforcement Learning (RL) and Decision Transformer (DT) are utilized by machines to learn from experiences and solve problems. The distinction between offline and online learning determines whether the machine learns from a live environment or simply observes pre-recorded actions. The difference between single-task and multi-task settings indicates whether the machine can handle similar but not identical tasks. Multi-task, offline learning, the focus of this paper, allows machines to address a variety of related tasks, based on a pre-recorded set of experiences. This approach is particularly valuable in situations where traditional training methods are costly or challenging. For instance, in robotics, multi-task, offline learning enables robots to use experiences from various tasks, such as picking up objects, to solve new problems like placing them down. This research paper explores the effectiveness of Decision Transformers in multi-task environments through theoretical discussions and practical examples. It also tries to answer the question, of how introducing sub-optimal training data, affects the performance or generalisation ability of the model.

Recent work has shown that offline reinforcement learning (RL) does not generalize well to new environments compared to behavioral cloning (BC). We propose WSAC-N, an ensemble model of soft actor-critics with weights to de-emphasize actions with high variance. We compare the zero-shot generalization abilities of WSAC-N with the baseline BC in a four-room maze-like environment, testing on unseen tasks. Our findings indicate that WSAC-N has worse zero-shot generalization compared to BC, aligning with previous work. Additionally, we investigate the impact of dataset characteristics on generalization, finding that dataset size has a negligible impact, while the quality of trajectories generally has a positive effect. These results are consistent with prior research.

This paper examines the generalization capabilities of the Soft Actor-Critic (SAC) algorithm when combined with Behavioral Cloning (BC) in a MiniGrid Four-Room Environment. Reinforcement learning (RL), particularly offline, is important for tasks where interactions with the environments are risky or costly, and this research focuses on multi-task environments where generalizability to new tasks is crucial. Our findings indicate that SAC+BC can achieve generalization performance close to BC. Notably, while BC shows robustness across various dataset characteristics (quality, diversity, size), SAC alone struggles without integrating BC, highlighting the enhancement in generalization brought by this hybrid approach. Furthermore, an increased data size only enhances generalizability when introducing greater diversity. However, these results are constrained by hardware limitations, suggesting that further hyperparameter optimization and using more seeds could validate and possibly enhance our findings, demonstrating that SAC+BC is even more effective
than shown. The implementation details and the source code for this study are available on GitHub at https://github.com/AxelGeist/multi-task-offlinereinforcement-learning.

Reinforcement learning (RL) is a type of machine learning where a model learns by
making an observation of the current state it is in, picking out an action to execute, and
observing the reward of said action, after which it receives the next state and repeats the
cycle until it reaches its goal. The traditional online training approach allows the agent to
directly interact with the live environment, but that is not always possible due to the live
environment possibly being too dangerous or costly to train in. In cases like these offline
training, which instead trains the agent on already pre-collected datasets of previously
mentioned interactions and tries to learn a better policy than the one used for collection,
offers a viable alternative by employing Q-Learning methods like CQL. However, prior
studies, such as Mediratta et al., have suggested that Behavior Cloning (BC), a
type of imitation cloning, may outperform modern offline RL methods in the multi-task
setting, where model generalization is tested on new or similar tasks rather than the ones
trained on. Considering these results, it begs the question whether it is worthwhile to
employ modern Q-Learning methods designed to derive a better policy than the one used
to collect the data, especially when they are unable to outperform standard imitation
learning.
This study seeks to reproduce and extend these findings within a custom environment.
The results reveal that, contrary to the aforementioned report, BC does not consistently
outperform CQL. Both machine learning methods exhibit comparable performance across
datasets varying in diversity and size. Additionally, incorporating more diverse data
significantly enhances generalization performance.