Circular Image

M.T.J. Spaan

info

Please Note

68 records found

An offshore wind installation campaign is executed once: on a single timeline, against a single budget. Sequencing it well is hard in two distinct ways. It is structurally hard—hundreds of coupled decisions under hard capacity, storage, and precedence constraints, over event-driven dynamics that industrial practice captures in a high-fidelity discrete-event simulator rather than in tractable algebra. It is also valuationally hard—project duration, operational cost, and emissions are measured in incommensurable units and pull against one another, so what counts as a good schedule is a non-linear judgment owned by the campaign's stakeholders rather than a constant a modeller may assume.

This thesis presents EOS, a framework that learns a scheduling policy directly against such a simulator and takes its objective from the stakeholders rather than from hand-tuned reward weights. Preference Function Modelling is embedded in the reward pipeline: each realised metric is mapped through a stakeholder-specified preference curve before aggregation, and the result is evaluated per realised trajectory under the expected-scalarised-return criterion—which requires augmenting the state with the accumulated outcomes and stabilising a normalisation that must be estimated online from a shifting policy. Feasibility is structural rather than learned: entities commit sequentially under a frozen simulation clock, in an order the policy itself learns, with each action mask regenerated from an isolated constraint projection after every commitment, so no conflict is ever repaired after the fact. A potential-based shaping signal, extended to variable-duration semi-Markov transitions, makes the sparse terminal objective reachable within a realistic training budget.

On two benchmark campaigns, EOS improved median makespan over a rule-based industrial heuristic by 10.7% and 25.2%, evaluated from a single deterministic rollout against the heuristic's best of 10,000 stochastic ones. Re-weighting the stakeholder preferences alone—with no change to the reward code and no manual penalty tuning—moved the selected schedule across a genuine physical trade-off, reducing accumulated storage cost by 94.8% for a 23.1% increase in makespan, with every selected schedule feasible. The trade-off is thereby stated where it belongs, by the parties who own it, and remains auditable against the schedule it produced. ...

Introducing Interactive and Spatially Grounded Decision Support to Bayesian Preference Elicitation

Master thesis (2026) - O.A.H. Wezenaar, M.T.J. Spaan, L. Cavalcante Siebert, Erwin Walraven
Urban planning problems typically involve multiple conflicting objectives and a decision maker whose preferences are difficult to articulate up front. Bayesian preference elicitation (BPE) addresses this by learning the decision maker’s utility through pairwise comparisons of candidate solutions on the Pareto front. Existing BPE methods operate purely in objective space, however, leaving out an important source of planner knowledge: two solutions with identical objective values can place infrastructure in very different parts of the city, and the planner’s preference between them cannot be recovered from pairwise queries alone.

This thesis introduces spatially grounded BPE and investigates whether allowing planners to indicate preferred locations on a map makes pairwise elicitation more informative, and what a single early grouping question contributes relative to the same budget of sequential queries. The existing objective-only elicitors are extended with six Gaussian process architectures that incorporate spatial features, a map-click injection mechanism that converts each click into a small batch of synthetic pairwise observations, and a cluster warm start that explores the Pareto front through a single interaction. The methods are evaluated on two Dutch case studies: Rotterdam electric vehicle charger placement at the postal-code level and Amsterdam shared mobility hub placement. Nine experiments and four appendix studies compare the proposed methods with an objective-only elicitor without spatial information and a hard zone filter that imposes geographic constraints outside the preference model.

Four findings emerge. First, non-linear flexibility is not always beneficial. A simple Bayesian linear utility baseline is statistically indistinguishable from the objective-only Gaussian process when utility is approximately linear in the objectives, and performs substantially better when the linear model is correctly specified. Added complexity is beneficial only when the utility contains structure that linear features cannot capture. Second, the value of map-based interaction depends on whether the elicitor learns the relative importance of spatial and objective information from the data or fixes it through the injection mechanism. Only the Additive Kernel architecture, which learns a shared output-scale ratio between objective and spatial sub-kernels through marginal likelihood, reliably benefits from map clicks. It generalises to held-out zones, scales positively with objective-spatial conflict, and remains robust as the optimum moves on and off preferred zones. However, a follow-up experiment shows that this apparent generalisation reflects preferred zones being activated together on the tested Pareto front rather than true extrapolation: no architecture provides information about a zone that the planner never clicked.

Third, the hard zone filter, initially expected to be a robust default, misrepresents coverage-style spatial preferences and is dominated by calibrated soft models when the optimum lies in preferred zones. The filter remains appropriate for genuinely hard geographic requirements. Fourth, the cluster warm start does not reduce final regret in these case studies but produces multi-cluster selections that can capture multimodality in the decision-maker’s utility.

Overall, the calibrated Additive Kernel with map-click injection before sequential queries provides the strongest extension when a map interface is available and coverage-style spatial preferences are plausible. The study identifies objective-spatial conflict as a useful measure for determining whether spatially grounded BPE provides additional value. Key limitations are the use of simulated decision makers under a probit response model, the expressiveness of map clicks, and evaluation on fixed Pareto fronts. A user study with real planners is the most immediate next step for validating the proposed interaction design.
...
Bachelor thesis (2026) - A. Durmaz, S. Feld, A. Kundu, M.T.J. Spaan, A. Lukina
Running a quantum circuit on hardware with limited qubit connectivity requires inserting SWAP gates, each of which adds depth and exposes qubits to decoherence, so that two qubits that must interact become physically adjacent on the chip. Reinforcement learning (RL) is an increasingly used adaptive alternative to hand-engineered routing heuristics, but how an RL agent’s observation and action encodings should be designed has received little attention, even though environments such as qgym leave both choices to the user.

We study these two encodings one at a time. The first is observation reach, or how many upcoming gates the agent can see; the second is action-space granularity, ranging from single SWAPs through a heuristically pruned set to multi-SWAP macro actions. Holding the reward, algorithm, hyperparameters, and hardware constant, we evaluate both on the coupling graphs of a 7-qubit and a 16-qubit IBM device.

Observation reach has little effect on routing, and the full-circuit view performs poorly on the larger device. Action-space granularity matters much more: macro actions solve substantially more circuits and route them with fewer SWAPs, and this advantage grows with device size. Injecting the same routing heuristic as a soft prior on the action distribution, rather than as a hard mask, preserves completeness while roughly halving SWAP overhead. A disjoint held-out evaluation suggests these results reflect transferable routing skill rather than memorized training circuits. ...

From Reward–Metric Alignment to Compiled-Circuit Quality

Bachelor thesis (2026) - F. LENTINI, S. Feld, A. Kundu, M.T.J. Spaan, A. Lukina
Quantum compilers are necessary to adapt quantum circuits to the connectivity and noise constraints of Noisy Intermediate-Scale Quantum devices. A key step in the compilation process is initial qubit mapping, where logical qubits are assigned to physical qubits. This assignment affects the overall reliability of the circuit, as mappings that place interacting qubits on non-adjacent physical qubits can require additional SWAP gates during routing. Recently, reinforcement learning techniques have been employed to tackle the initial qubit mapping problem. In this setting, the choice of reward function is particularly important, since it defines the objective the RL agent learns to optimize. This raises a central question for RL-based initial mapping: whether graph-level rewards can serve as useful proxies for downstream compiled-circuit quality. This work addresses this question by engineering reward functions based on hardware-distance and hardware-fidelity signals, including a hybrid reward that combines both, and evaluating how well they predict two downstream metrics: compiled SWAP count and estimated success probability (ESP). The same rewards are used to train action-masked PPO agents with terminal, shaped and (n)-step shaped variants on 5-qubit hardware topologies. Overall, the results show that graph-level rewards provide useful learning signals to guide RL-based initial mapping, but remain limited as proxies for full-compilation performance. This highlights the need for reward design that includes more routing relevant information, such as repeated interactions and gate ordering, to better approximate the effects of routing on final compiled-circuit quality. ...
Bachelor thesis (2026) - A. Govenko, S. Feld, A. Kundu, M.T.J. Spaan, A. Lukina
Compiling quantum circuits for physical hardware requires an initial mapping step that assigns virtual qubits to physical qubits such that interacting pairs are placed on connected hardware locations. Current approaches train a separate agent per device topology, requiring significant compute for each new hardware generation and transferring no knowledge across devices. This work investigates whether curriculum learning --- progressively training a reinforcement learning agent on hardware topologies of increasing size --- can produce a single agent that generalises to unseen topologies. We evaluate three curriculum variants differing in replay ratio and warmup length, alongside three non-curriculum baselines, in the QGym InitialMapping environment using MaskablePPO. Results show that curriculum agents outperform single-topology and single-size training on held-out topologies, reaching strong frontier performance with greater sample efficiency than direct training. Against unordered exposure to the same topology distribution, however, curriculum ordering's advantage holds on the target topology size but not on generalisation to unseen topologies. While absolute performance remains modest and variance across seeds is substantial, the findings establish curriculum learning as a viable approach to topology-general qubit mapping and provide a proof of concept for training a single model that transfers across hardware topologies, reducing the computational cost of re-training for each new device. ...
Quantum gate scheduling assigns start cycles to quantum-circuit operations while respecting precedence, resource, and hardware constraints. Although schedules are commonly evaluated by makespan, it is only an indirect proxy for execution reliability, since schedules of equal duration may differ in gate errors, idle-time decoherence, and crosstalk exposure. This thesis investigates whether reinforcement learning benefits from domain knowledge in quantum gate scheduling. Building on qgym’s scheduling environment, we evaluate Maskable Proximal Policy Optimization against greedy ASAP and ALAP baselines on Random, GHZ, QFT, and QAOA circuit families using IBM calibration data. We study commutation-awareness, which relaxes unnecessary ordering constraints between commuting gates, and hardware-awareness, which injects calibration data through extended observations and/or a log-ESP-based reward. The main finding is that commutation-awareness is the most reliable improvement: it reduces makespan by approximately 20% for QAOA and Random circuits, while giving little benefit for GHZ and QFT circuits. Furthermore, noise-aware observation space proves promising for further research. ...

Closing the Proxy Gap Between SWAP Count and Schedule Length

Bachelor thesis (2026) - D. Cernatinschi, S. Feld, A. Kundu, M.T.J. Spaan, A. Lukina
To run a quantum program on real hardware, a compiler must rewrite it so that every interacting pair of qubits is physically adjacent on the chip, which it does by inserting extra SWAP operations. How these SWAPs are chosen determines how reliably the program runs: each one adds gates and lengthens the schedule, and on today’s noisy devices both effects make a wrong answer more likely. Reliability thus depends on two costs at once, the program’s size and its running time, yet the routing pass that quantum compilers deploy, SABRE, optimises only the first: it minimises the number of SWAPs. The SWAP count stands in for size but not for running time, even though running time is the cost that limits reliability most on current hardware. Optimising it alone is a case of a pass tuning a convenient proxy rather than the quantity that ultimately matters.

We show this proxy gap is real and then close it. SABRE-MS keeps SABRE’s objective and adds the missing one, running time, so the router balances the two costs instead of ignoring one. A single tunable weight sets how much each of the two metrics, the program’s size and its running time, counts in the balance. It cuts a compiled program’s number of cycles by about 20% on average compared with the SABRE version that Qiskit ships. A standard reliability model shows the trade-off pays off despite the added operations, the gain holds as circuits grow to tens of qubits, and on a real 156-qubit IBM processor it raises the measured circuit fidelity by about 2.5×. The same idea, applied to a very different reinforcement-learning router, helps it in the same way, providing evidence that the benefit belongs to the objective rather than to SABRE itself. ...
Master thesis (2026) - Y. Wu, P.R. van der Vaart, M.T.J. Spaan, D.M.J. Tax
Efficient exploration is a central challenge in reinforcement learning, especially under sparse rewards, large state-action spaces, or adversarial environment dynamics. A principled approach is exploration guided by epistemic uncertainty. Under certain conditions, this can be formulated in the Bayesian framework as exploration based on the posterior variance. In practice, the predictive variance of a neural network is often used as a surrogate for the posterior variance by techniques such as ensembles, randomized priors, and random network distillation. However, for wide neural networks, the theory of neural tangent kernels reveals a discrepancy between the predictive variance of a wide neural network trained by gradient descent and its posterior variance. This is a known issue in supervised learning and there exist methods to reconcile this discrepancy. In this thesis, we show that, in contrast to the supervised setting, reconciliation is not possible for semi-gradient temporal difference learning, which is a commonly used algorithm in deep reinforcement learning. Instead, we derive the upper confidence bound of the value function in semi-gradient temporal difference learning directly and show that this upper confidence bound can be estimated by a neural network. To aid our analysis, we extend the theory of neural tangent kernels to regularized semi-gradient temporal difference learning. ...
Doctoral thesis (2026) - P.R. van der Vaart, M.T.J. Spaan, N. Yorke-Smith
The goal of reinforcement learning is to train agents to perform tasks under little supervision. Tasks are specified by a reward function and transition function, which state how much reward the agent gets for its action in a state, and how the environment state changes based on the action the agent took. Typically reinforcement learning assumes no prior knowledge over the reward and transition function,meaning that agents need to explore the environment and learn essentially through trial and error. Model-free methods attempt to learn which actions lead to good outcomes without modeling the reward or environments itself. Efficiently selecting that actions are promising is an active research direction which can greatly reduce the number of total interactions needed for an agent to learn the task, potentially opening the door to new applications where trials or simulations are expensive or compute is limited.
Uncertainty quantification is a central mechanism in such efficient exploration methods. Provided with an estimate of how certain the agent is about the outcome of an action, it can intelligently weigh whether it is worth exploring. The Bayesian paradigm is one method to quantify uncertainty in machine learning. It models the uncertainty with a probability distributions over models, specifying how likely a model is based on the data the agent has collected.
We adopt a Bayesian point of view in model-free reinforcement learning, and develop a deeper understanding on when Bayesian reinforcement learning methods can be expected to work well and challenges that remain. To this end, in Chapter 2 we propose training ensembles through Sequential Monte Carlo, obtaining a sample from the posterior distribution of a deep Q-learning agent. We observe that agents are able to perform directed exploration, although not necessarily more efficiently than standard ensembles in every environment. Furthermore, in Chapter 3 we theoretically analyze existing Bayesian Deep model-Free Reinforcement Learning methods, and unify them into a single theoretical framework we call Epistemic Bellman Operators. We prove that these operators are contractions, establishing convergence of derived algorithms in a simplified setting. Finally, in Chapter 4 we analyze the likelihood and prior assumptions
in existing Bayesian deep model-free reinforcement learning methods, and find through statistical tests that the standard likelihood assumptions are violated on every benchmark we tested. We also find that we can improve performance of Bayesian model free reinforcement learning methods by picking different priors based on empirical data from unrelated tasks, which transfer to new environments.
This dissertation establishes several desirable properties of Bayesian Deep model free reinforcement learning, but also raises some key issues, most notably misspecification in Chapter 4. We hope our findings convince other Bayesian reinforcement learning researchers to give more attention to assumptions about priors and likelihoods. ...
Doctoral thesis (2026) - J.A. de Vries, M.T.J. Spaan, M.M. de Weerdt
This dissertation studies the optimal decision-making under uncertainty problem from the context of sequential experiment planning in autonomous laboratories. Our focus is on deep reinforcement learning methods, which we first motivate by building up from the challenges involved in conventional optimal experiment design methods and black-box numerical optimization. Secondly, we frame deep reinforcement learning in a unifying view of commonplace Bayesian optimization methods to conceptually streamline extensions or adoption of reinforcement learning methods in existing infrastructure. Each chapter investigates challenges and questions that emerge from our framing, which more specifically relate to uncertainty quantification and proper algorithm-hardware tailoring for improving performance scaling.

Bayesian inference of large complex models is typically computationally infeasible unless stringent, but possibly unrealistic, assumptions are placed on the model of the physical system that we’re trying to capture. Deep learning has since emerged as a powerful way to approximate this computationally infeasible routine through pattern recognition based on an abundance of data. This principle underpins most of the success stories of recent artificial intelligence breakthroughs, like agentic chatbots or game-playing agents. This dissertation investigates this principle from a more fundamental basis to improve our algorithmic tools for the overarching experiment-design use case.

The field of meta-reinforcement learning, similarly, has often used deep learning to capture various sources of uncertainty for decision-making agents, such as unknown rewards or world dynamics. This approximates what is known as Bayes-optimal control, an agent that optimally trades off between information-seeking actions and actions that maximize a (subjective) value. In Chapter 2 we show how common deep learning approaches, like recurrent neural networks, for meta-reinforcement learning do not appropriately capture true uncertainty from the Bayesian viewpoint. Rather, we found that uncertainty is obfuscated in the activations of the neural networks. We present the Laplace Variational Recurrent Neural Network to extract this implicit uncertainty as a means to enable practitioners to deal with uncertainty explicitly.

From Chapter 2 we built the insight that one of the roles that deep learning plays in deep reinforcement learning is to amortize an exponentially costly policy optimization problem. A well-known consequence of amortization is that we always have to deal with an amortization gap (approximation gap) between the true values and the predicted values. Improving the performance of an algorithm, thus, requires a choice of tools that minimize the runtime and data needed to reduce this gap. Among the most successful approaches in this space are those that use adaptive computation at inference time to reduce the amortization gap and complement the neural network. Monte-Carlo tree search, for instance, achieved strong data-efficiency and enabled breakthroughs like defeating the human world champion in Go. However, Monte-Carlo tree search-based methods do not favorably scale in runtime. Thus, in Chapter 3 we improve the state-of-the-art of sequential Monte-Carlo planners, which permit much more efficient parallelization on modern GPU hardware. Our newly proposed algorithm demonstrated improved scaling in performance given additional planning budget, not only in data-efficiency but also in runtime-efficiency compared to prior approaches.

The final Chapter 4 exploits our finding that sequential Monte-Carlo-based planning can improve runtime complexity by shifting computational budget from neural network learning to additional planning budget to get higher quality data. Since in meta-learning settings we typically deal with learning from trajectory-based data, we often find that when trajectories increase in length, any updates to a neural network architecture will become more expensive. This is because the computational cost of gradient updates grows with trajectory length, which causes bottlenecks in scaling. For instance, updates can start to exceed hardware memory limits, forcing sequential processing of smaller batches and reducing throughput. Thus, we again find that the performance scaling to additional planning compute in deep meta-reinforcement learning can be much improved when equipped with our designed sequential Monte-Carlo planner. In contrast to prior approaches, our planner explicitly accounts for how uncertainty evolves in light of new data. As a result, our method directly targets an optimal policy that trades off exploration with exploitation, without ad hoc exploration heuristics.

In summary, this dissertation advances the state of probabilistic reinforcement learning for uncertainty-based decision-making. We show that Bayesian optimization and deep reinforcement learning can be formulated very similarly from a probabilistic viewpoint, and that only (relatively) minor assumptions and choice of approximation tools shift us from one setting to the other. This is a particularly useful insight for our use case of self-driving laboratories, since at the time of writing, Bayesian optimization is a more established approach in practice. Our findings establish a stronger common ground between the seemingly separate Bayesian optimization and reinforcement learning fields, and provide a principled basis for transferring approximation tools between these communities.
...
Doctoral thesis (2026) - M.A. Zanger, M.T.J. Spaan, J.W. Böhmer
This dissertation concerns the efficient quantification of uncertainty in the field of deep reinforcement learning. At the time of this writing, artificial intelligence is being adopted rapidly into the critical pipelines of numerous scientific and societal domains — from autonomous driving and medical diagnostics to scientific discovery. A particular class of machine learning models, deep neural networks, has been pivotal in this recent development due to their extraordinary scalability and expressive power. Such models learn by optimizing vast sets of parameters to shape predictions according to previous measurements, captured in large datasets. When we deploy such learned models for practical applications, however, they are asked to make predictions for novel inputs not represented in their training data. Such predictions are the result of inductive generalization — deriving insights about future situations from past experience — and are inherently subject to uncertainty. For these predictions to be actionable, they must often be accompanied by a reliable measure of confidence. This need to know what one does not know is addressed by the quantification of epistemic uncertainty, which arises from the imperfection of a learned model, often due to a lack of sufficient relevant data. This stands in contrast to aleatoric uncertainty — the irreducible, inherent randomness in a process — and it is this reducible, model-centric epistemic uncertainty that forms the central object of inquiry for this dissertation.

The challenge of epistemic uncertainty estimation becomes especially tangible in the context of sequential decision-making problems. In such settings, an agent’s actions can have long-term consequences that compound over time, shaping downstream outcomes and choices. Reinforcement learning, a paradigm in which agents learn such decision-making strategies through direct interaction with an environment, faces several fundamental challenges that hinge on reliable uncertainty estimation. Underpinning both efficient exploration and safe decision-making is a principled understanding of an agent’s own epistemic uncertainty — the central topic of this thesis.

Examining the current research landscape of uncertainty quantification in deep learning, we observe a persistent tension between theoretically well-motivated yet computationally expensive techniques on one hand, and computationally efficient yet less understood methods on the other. Bayesian inference, widely regarded as the gold standard for reasoning about epistemic uncertainty, is generally intractable for modern, large-scale neural networks. This has led to a spectrum of approximate methods — including deep ensembles, advanced sampling techniques, and variational inference — that navigate this trade-off to varying degrees. More pragmatic solutions often offer substantial computational savings but lack a deeper theoretical understanding of what their uncertainty estimates represent, or how they behave in practice. From this landscape, we derive the research mission for this dissertation: to develop and analyze uncertainty quantification methods that are both computationally tractable and theoretically well-motivated.

Our first line of inquiry investigates deep neural network ensembles. We hypothesize that their efficacy is constrained not merely by the number of constituent models but by the quality of their diversity. Focusing on distributional reinforcement learning, we show that architectural components such as projection operators can induce inductive biases that shape generalization. We develop diverse projection ensembles, which induce diversity through architecturally distinct members, and demonstrate empirically that this approach yields more robust uncertainty signals and improved exploration performance.

Our second line of inquiry pursues the goal of emulating ensemble uncertainty within a single, efficient model. We introduce contextual similarity distillation, a technique enabling epistemic uncertainty estimation with a single model trained via gradient descent. Using neural tangent kernel theory, we reinterpret ensemble variance estimation as a tractable kernel regression problem. Complementarily, we provide a theoretical foundation for random network distillation, showing that, in the infinite-width limit, its uncertainty corresponds to the predictive variance of a deep ensemble. We further develop a Bayesian variant whose error signal matches the posterior predictive variance of an infinitely wide Bayesian neural network.

Finally, we address the estimation of long-term, cumulative uncertainty in reinforcement learning. We propose a novel single-model method — universal value-function uncertainties — that quantifies uncertainty over entire trajectories. Grounded in neural tangent kernel theory, we show that this method yields estimates equivalent to those of a full ensemble, while retaining computational efficiency.

In conclusion, this dissertation advances uncertainty quantification in deep reinforcement learning by bridging the gap between theoretical rigor and computational practicality. It provides both methodological innovations and theoretical insights toward more reliable, uncertainty-aware autonomous agents. ...
Master thesis (2025) - R. Haket, M.T.J. Spaan, J.H. Krijthe, O. Azizi
Model-free deep reinforcement learning has shown remarkable promise in solving highly complex sequential decision-making problems. However, the widespread adoption of reinforcement learning algorithms has not materialized in real-world applications such as robotics. A primary challenge is the general assumption that environments remain stationary at deployment. This problem is exacerbated when agents rely on pixel-based observations, dramatically increasing task complexity. As a result, these algorithms often fail when environments change over time. The perspective that agents should learn a disentangled representation has already been shown to be effective in improving generalization to domain shifts. We extend previous work by introducing Generalized Disentanglement (GED), an auxiliary contrastive learning task that encourages pixel-based deep reinforcement learning algorithms to isolate factors of variation in the data by leveraging temporal structure. We show that our methodology can improve generalization to unseen domains in several environments. ...
Recent advancements in differential simulators offer a promising approach to enhancing the sim2real transfer of reinforcement learning (RL) agents by enabling the computation of gradients of the simulator’s dynamics with respect to its parameters. However, the application of these gradients is often limited to specific scenarios. In this thesis, we address these limitations by proposing methods to obtain accurate gradients through the use of a privileged value function. This approach provides valuable insights into the effectiveness of differential gradients and demonstrates that, in certain cases, it can significantly improve sim2real performance. To illustrate this, we develop an adversary that identifies the worst-case domain parameters for a given policy using local gradients. Our experiments are conducted on the Pendulum swing-up environment. This thesis forms the basis for the exploration of further possibilities of leveraging differential simulator gradients. ...

Experimental Evaluation of the Generalizability of the Soft Actor-Critic + Behavioral Cloning Algorithm

This paper examines the generalization capabilities of the Soft Actor-Critic (SAC) algorithm when combined with Behavioral Cloning (BC) in a MiniGrid Four-Room Environment. Reinforcement learning (RL), particularly offline, is important for tasks where interactions with the environments are risky or costly, and this research focuses on multi-task environments where generalizability to new tasks is crucial. Our findings indicate that SAC+BC can achieve generalization performance close to BC. Notably, while BC shows robustness across various dataset characteristics (quality, diversity, size), SAC alone struggles without integrating BC, highlighting the enhancement in generalization brought by this hybrid approach. Furthermore, an increased data size only enhances generalizability when introducing greater diversity. However, these results are constrained by hardware limitations, suggesting that further hyperparameter optimization and using more seeds could validate and possibly enhance our findings, demonstrating that SAC+BC is even more effective
than shown. The implementation details and the source code for this study are available on GitHub at https://github.com/AxelGeist/multi-task-offlinereinforcement-learning. ...
Offline Reinforcement Learning (Offline RL) involves learning policies from a static dataset without further interactions with the environment, making it suitable for high-stakes scenarios where data collection is costly or risky. This paper investigates the generalization capabilities of Implicit Q-Learning (IQL), an offline RL algorithm, compared to Behavioral Cloning (BC). We adapt the IQL algorithm for discrete control and evaluate both IQL and BC in a four-room environment using training datasets generated from different behavioral policies. Performance is assessed based on average rewards over various test seeds, on reachable and unreachable tasks, as well as the training set. Our results indicate that BC consistently outperforms IQL across all scenarios, although IQL reaches peak performance faster. This study highlights the need for further research into offline RL algorithms for better generalization and more robust performance in diverse environments. Full code available on GitHub. ...

How does the introduction of sub-optimal data affect the performance of the model?

In the field of Artificial Intelligence (AI), techniques like Reinforcement Learning (RL) and Decision Transformer (DT) are utilized by machines to learn from experiences and solve problems. The distinction between offline and online learning determines whether the machine learns from a live environment or simply observes pre-recorded actions. The difference between single-task and multi-task settings indicates whether the machine can handle similar but not identical tasks. Multi-task, offline learning, the focus of this paper, allows machines to address a variety of related tasks, based on a pre-recorded set of experiences. This approach is particularly valuable in situations where traditional training methods are costly or challenging. For instance, in robotics, multi-task, offline learning enables robots to use experiences from various tasks, such as picking up objects, to solve new problems like placing them down. This research paper explores the effectiveness of Decision Transformers in multi-task environments through theoretical discussions and practical examples. It also tries to answer the question, of how introducing sub-optimal training data, affects the performance or generalisation ability of the model. ...
Recent work has shown that offline reinforcement learning (RL) does not generalize well to new environments compared to behavioral cloning (BC). We propose WSAC-N, an ensemble model of soft actor-critics with weights to de-emphasize actions with high variance. We compare the zero-shot generalization abilities of WSAC-N with the baseline BC in a four-room maze-like environment, testing on unseen tasks. Our findings indicate that WSAC-N has worse zero-shot generalization compared to BC, aligning with previous work. Additionally, we investigate the impact of dataset characteristics on generalization, finding that dataset size has a negligible impact, while the quality of trajectories generally has a positive effect. These results are consistent with prior research. ...

Teacher-Student Curriculum Learning for Efficient Meta-Learning

We investigate whether a teacher-student curriculum learning approach using a teacher network with a simpler structure than the student network can achieve better results at meta-learning. The goal of meta-learning is to learn from a set of tasks, and then perform well on a new, structurally similar but unseen task with minimal retraining. Instead of sampling uniformly from all data to create the training batches, the curriculum-learning approach aims to create a sequence of mini-batches that enhances the training process, also known as a curriculum. During teacher-student curriculum learning a "teacher" network is trained in the standard manner, and then its outputs are used to order the training samples by difficulty and categorise them into mini-batches. This curriculum is then used to train the "student" network. Previous teacher-student models either had pre-trained more complex teachers, or teachers with the same structure as the student network. We investigate whether a teacher network with a simpler structure can also increase accuracy, while preserving computational resources. We find that using such a curriculum worsens performance compared to not using any curriculum at all. ...
Meta-Learning is an emerging field where the main challenge is to develop models capable of distilling previous experiences to efficiently learn new tasks. Curriculum Learning, a group of optimization strategies, structures data in a meaningful order which aids learning. However, the extent to which curriculum strategies can optimize the performance of meta-learners remains unclear. Here we study the separate and joint effects of a model-based (ScreenerNet) and a statistics-based (Active Bias) curriculum strategy on the training of a meta-learning model (Neural Processes) which solves 1-D function regression tasks. The findings show that ScreenerNet increases in-task accuracy and accelerates convergence, but decreases the generalization performance. Active Bias achieves mixed generalization results and significantly decreases training efficiency when trained on noisy data-sets. Combining them partially mitigates ScreenerNet's overfitting and stabilizes Active Bias' susceptibility to noise, but further research is necessary in order to achieve consistent improvements to the baseline. ...
Meta-learning is an important emerging paradigm in machine learning, aimed at improving data-efficiency and generalization performance across learning tasks. Challenges caused by noisy data has been extensively researched in traditional learning settings. However, its impact in the context of meta-learning, especially concerning label noise in meta-training, remains under-explored. Curriculum Learning (CL), is an approach where training data is ordered from easy to complex, and models learn from easier to harder samples. A type of CL , Self-Paced learning (SPL) offers adaptive data curriculum, where ordering is based on per sample model performance during training. Self-Paced Learning (SPL) has proven effective in enhancing model robustness and convergence under noisy data scenarios. However, its application in meta-learning under these conditions remains limited. In this paper, we use a Neural Process model on 1D sinusoidal function regression tasks, with different ratio of clean / noisy training data scenarios to empirically observe the same benefits SPL can potentially offer for noisy meta-learning. In line with findings in traditional learning settings, SPL improved overall training convergence, also lead to increase in generalization thus noise robustness. Furthermore SPL lead the model to be more robust to increasing scale of noise for tasks within the training data distribution.
...