Recent developments in applying reinforcement learning to cooperative environments, like negotiation, have brought forward an important question: how well can a negotiating agent be trained through self-play? Previous research has seen successful application of self-play to other settings, like the games of chess and Go. This paper explores the usage of self-play within the training of a negotiating agent and determines if it is possible to successfully train an agent purely through self-play. The results of the experimentation show that a training stage using self-play can match or even exceed an approach using a set of training opponents. By using multiple self-play opponents, the average utility can be further improved by introducing more variance during training. In addition, using a combination of both self-play and training opponents leads to a hybrid approach that performs better than either of the two techniques separately.

Automated negotiation agents can highly benefit from learning their opponent’s preferences. Multiple algorithms have been developed with the two main categories being: heuristic techniques and machine learning techniques. Historically, heuristic techniques have dominated the field, but with the recent development in the field of machine learning, this is no longer true. The main goal of the paper is to compare these two techniques quantitatively using the Pearson correlation of bids. The models that were chosen as the heuristic and machine learning baseline are the Smith and the Perceptron models, respectively. Our results show that the two baselines have similar performance. This leads us to conclude that machine learning algorithms have caught up with their heuristic counterparts. Furthermore, we have also found a statistically significant correlation between the Perceptron model’s accuracy and the seen bid space.

With the prospects of decentralized multi-agent systems becoming more prevalent in daily life, automated negotiation agents have made their place in these collaborative settings. They are an approach to promote communication between the agents in reaching solutions that are better for all involved.

Recent literature has shown great potential in using machine learning, particularly model-free deep reinforcement learning like Proximal Policy Optimization (PPO), to develop more performant automated negotiation strategies. This work focuses on using information from the opponent's sequence of offers in a bilateral negotiation to further improve a baseline PPO agent. This involves extracting and representing information from the opponent's sequence of offers into a state vector with a fixed dimension to modify the input to the agent's policy, and then comparing the utilities this modified agent achieves to the baseline PPO agent. Since there is a large variety of numerical measures to represent a sequence of offers, an ablation study is conducted to investigate the effectiveness of each.

The modified agents consistently reached solutions that had higher social welfare, although the agent's own utility did not improve or diminish significantly in comparison to the base PPO agent.

This paper introduces a strategy for learning opponent parameters in automated negotiation and using them for future negotiation sessions. The goal is to maximize the agent’s utility while being consistent in its performance over various negotiation scenarios. While a number of reinforcement learning approaches in the field have used Q-learning, this paper uses the newer Proximal Policy Optimization algorithm. Machine learning has been used in opponent modeling, classifying opponents, and learning strategies, but there have been few attempts to store and re-use this information. In an experimental setup, it is shown that this approach outperforms a baseline in terms of individual utility.

The domains of the negotiation can vary significantly. It is possible that a domain is very cooperative, where both agents can receive a high utility; the opposite is also possible, where the domain is very competitive and the agents cannot both get a high utility. In the same manner, the agents can have different strategies leading to a complicated problem with no obvious solution.

This research seeks to represent the differences in negotiation domains to improve a machine learning based agent to help the agent generalize these domains. To achieve this several ways of representing the domain have been explored.
First is the shared domain information. With this representation, the agent uses information about the amount of issues, values and possible bids there are. Second is the private domain information, in this representation, the agent uses different calculations to get a view of how favorable the domain is in terms of utility. Last is the derived information, this is the representation where the agent learns about the domain by interaction with the environment or the opposing agent.

From the experiments, a conclusion could be made that a part of these representations had a positive impact on the final utility of the agent. The shared domain information had a considerable improvement over the base agent with the features having a non-negligible impact on the negotiation. The derived information also had a considerable impact on the final outcome.