In order to do something useful, you should know what to do and how to do it. The same goes for robots or other machines, which are also referred to as agents. Advances in e.g. technology and science have made such agents more and more sophisticated and capable. This opens up a plethora of possibilities for solving problems or automating processes.
However, as agents and the problems they solve become more and more sophisticated, the software that controls these agents naturally also becomes more complex. Especially when considering systems of multiple cooperating agents. With rapidly increasing amounts of agents, observations and capabilities to take into account, the plans used to come to decisions become increasingly infeasible to create by hand. Thus automated methods are necessary to create such plans.
The approach this thesis uses is inspired by the ability of human intelligence to abstract away pedantic details when planning. We do not for example include which muscles we will utilize, when we make a plan for picking up groceries. Deciding to get groceries is on a completely different level than deciding how many centimeters we want to move a limb in a particular direction. Abstracting away such details greatly reduces the size of any potential plan. Which makes plans easier to construct, while the scope of the problem remains the same.
Thus, utilizing this notion of different levels of abstraction during planning, is the approach this thesis uses to improve planning for agents. This is done by extending the Macro Decentralized Partially Observable Markov Decision Process (MacDecPOMDP) framework to allow for an arbitrary number of levels of abstraction.

Identifying the most efficient exploration approach for deep reinforcement learning in traffic light control is not a trivial task, and can be a critical step in the development of reinforcement learning solutions that can effectively reduce traffic congestion. It is common to use baseline dithering methods such as $\epsilon$-greedy. However, the value of more evolved exploration approaches in this setting has not yet been determined. This paper addresses this concern by comparing the performance of the popular deep Q-learning algorithm using one baseline and two state of the art exploration approaches, and their combination. Specifically, $\epsilon$-greedy is used as a baseline, and compared to the exploration approaches Bootstrapped DQN, randomized prior functions, and their combination. This is done in three different traffic scenarios, capturing different traffic profiles. The results obtained suggest that the higher the complexity of the traffic scenario, and the larger the size of the observation space of the agent, the larger the gain from efficient exploration. This is illustrated by the improved performance observed in the agents using efficient exploration and enjoying a larger observation space in the complex traffic scenarios.

Previous research has in reinforcement learning for traffic control has used various state abstractions. Some use feature vectors while others use matrices of car positions. This paper first compares a simple feature vector consisting of only queue sizes per incoming lane to a matrix of car positions. Then it investigates if knowledge can be transferred from a simple agent using the feature vector abstraction to a more complex agent that uses the position matrix abstraction.We find that training cannot be sped up by first training an agent with the feature vector abstraction and then reusing this Q-function to train an agent with the position matrix abstraction. The simple agent does not take considerably fewer samples to converge, and the total time needed to first train the simple agent and then transfer exceeds the time needed to train the complex agent from scratch.