M.T.J. Spaan
Please Note
68 records found
1
This thesis presents EOS, a framework that learns a scheduling policy directly against such a simulator and takes its objective from the stakeholders rather than from hand-tuned reward weights. Preference Function Modelling is embedded in the reward pipeline: each realised metric is mapped through a stakeholder-specified preference curve before aggregation, and the result is evaluated per realised trajectory under the expected-scalarised-return criterion—which requires augmenting the state with the accumulated outcomes and stabilising a normalisation that must be estimated online from a shifting policy. Feasibility is structural rather than learned: entities commit sequentially under a frozen simulation clock, in an order the policy itself learns, with each action mask regenerated from an isolated constraint projection after every commitment, so no conflict is ever repaired after the fact. A potential-based shaping signal, extended to variable-duration semi-Markov transitions, makes the sparse terminal objective reachable within a realistic training budget.
On two benchmark campaigns, EOS improved median makespan over a rule-based industrial heuristic by 10.7% and 25.2%, evaluated from a single deterministic rollout against the heuristic's best of 10,000 stochastic ones. Re-weighting the stakeholder preferences alone—with no change to the reward code and no manual penalty tuning—moved the selected schedule across a genuine physical trade-off, reducing accumulated storage cost by 94.8% for a 23.1% increase in makespan, with every selected schedule feasible. The trade-off is thereby stated where it belongs, by the parties who own it, and remains auditable against the schedule it produced. ...
This thesis presents EOS, a framework that learns a scheduling policy directly against such a simulator and takes its objective from the stakeholders rather than from hand-tuned reward weights. Preference Function Modelling is embedded in the reward pipeline: each realised metric is mapped through a stakeholder-specified preference curve before aggregation, and the result is evaluated per realised trajectory under the expected-scalarised-return criterion—which requires augmenting the state with the accumulated outcomes and stabilising a normalisation that must be estimated online from a shifting policy. Feasibility is structural rather than learned: entities commit sequentially under a frozen simulation clock, in an order the policy itself learns, with each action mask regenerated from an isolated constraint projection after every commitment, so no conflict is ever repaired after the fact. A potential-based shaping signal, extended to variable-duration semi-Markov transitions, makes the sparse terminal objective reachable within a realistic training budget.
On two benchmark campaigns, EOS improved median makespan over a rule-based industrial heuristic by 10.7% and 25.2%, evaluated from a single deterministic rollout against the heuristic's best of 10,000 stochastic ones. Re-weighting the stakeholder preferences alone—with no change to the reward code and no manual penalty tuning—moved the selected schedule across a genuine physical trade-off, reducing accumulated storage cost by 94.8% for a 23.1% increase in makespan, with every selected schedule feasible. The trade-off is thereby stated where it belongs, by the parties who own it, and remains auditable against the schedule it produced.
Multi Objective Preference Elicitation in Urban Planning
Introducing Interactive and Spatially Grounded Decision Support to Bayesian Preference Elicitation
This thesis introduces spatially grounded BPE and investigates whether allowing planners to indicate preferred locations on a map makes pairwise elicitation more informative, and what a single early grouping question contributes relative to the same budget of sequential queries. The existing objective-only elicitors are extended with six Gaussian process architectures that incorporate spatial features, a map-click injection mechanism that converts each click into a small batch of synthetic pairwise observations, and a cluster warm start that explores the Pareto front through a single interaction. The methods are evaluated on two Dutch case studies: Rotterdam electric vehicle charger placement at the postal-code level and Amsterdam shared mobility hub placement. Nine experiments and four appendix studies compare the proposed methods with an objective-only elicitor without spatial information and a hard zone filter that imposes geographic constraints outside the preference model.
Four findings emerge. First, non-linear flexibility is not always beneficial. A simple Bayesian linear utility baseline is statistically indistinguishable from the objective-only Gaussian process when utility is approximately linear in the objectives, and performs substantially better when the linear model is correctly specified. Added complexity is beneficial only when the utility contains structure that linear features cannot capture. Second, the value of map-based interaction depends on whether the elicitor learns the relative importance of spatial and objective information from the data or fixes it through the injection mechanism. Only the Additive Kernel architecture, which learns a shared output-scale ratio between objective and spatial sub-kernels through marginal likelihood, reliably benefits from map clicks. It generalises to held-out zones, scales positively with objective-spatial conflict, and remains robust as the optimum moves on and off preferred zones. However, a follow-up experiment shows that this apparent generalisation reflects preferred zones being activated together on the tested Pareto front rather than true extrapolation: no architecture provides information about a zone that the planner never clicked.
Third, the hard zone filter, initially expected to be a robust default, misrepresents coverage-style spatial preferences and is dominated by calibrated soft models when the optimum lies in preferred zones. The filter remains appropriate for genuinely hard geographic requirements. Fourth, the cluster warm start does not reduce final regret in these case studies but produces multi-cluster selections that can capture multimodality in the decision-maker’s utility.
Overall, the calibrated Additive Kernel with map-click injection before sequential queries provides the strongest extension when a map interface is available and coverage-style spatial preferences are plausible. The study identifies objective-spatial conflict as a useful measure for determining whether spatially grounded BPE provides additional value. Key limitations are the use of simulated decision makers under a probit response model, the expressiveness of map clicks, and evaluation on fixed Pareto fronts. A user study with real planners is the most immediate next step for validating the proposed interaction design.
...
This thesis introduces spatially grounded BPE and investigates whether allowing planners to indicate preferred locations on a map makes pairwise elicitation more informative, and what a single early grouping question contributes relative to the same budget of sequential queries. The existing objective-only elicitors are extended with six Gaussian process architectures that incorporate spatial features, a map-click injection mechanism that converts each click into a small batch of synthetic pairwise observations, and a cluster warm start that explores the Pareto front through a single interaction. The methods are evaluated on two Dutch case studies: Rotterdam electric vehicle charger placement at the postal-code level and Amsterdam shared mobility hub placement. Nine experiments and four appendix studies compare the proposed methods with an objective-only elicitor without spatial information and a hard zone filter that imposes geographic constraints outside the preference model.
Four findings emerge. First, non-linear flexibility is not always beneficial. A simple Bayesian linear utility baseline is statistically indistinguishable from the objective-only Gaussian process when utility is approximately linear in the objectives, and performs substantially better when the linear model is correctly specified. Added complexity is beneficial only when the utility contains structure that linear features cannot capture. Second, the value of map-based interaction depends on whether the elicitor learns the relative importance of spatial and objective information from the data or fixes it through the injection mechanism. Only the Additive Kernel architecture, which learns a shared output-scale ratio between objective and spatial sub-kernels through marginal likelihood, reliably benefits from map clicks. It generalises to held-out zones, scales positively with objective-spatial conflict, and remains robust as the optimum moves on and off preferred zones. However, a follow-up experiment shows that this apparent generalisation reflects preferred zones being activated together on the tested Pareto front rather than true extrapolation: no architecture provides information about a zone that the planner never clicked.
Third, the hard zone filter, initially expected to be a robust default, misrepresents coverage-style spatial preferences and is dominated by calibrated soft models when the optimum lies in preferred zones. The filter remains appropriate for genuinely hard geographic requirements. Fourth, the cluster warm start does not reduce final regret in these case studies but produces multi-cluster selections that can capture multimodality in the decision-maker’s utility.
Overall, the calibrated Additive Kernel with map-click injection before sequential queries provides the strongest extension when a map interface is available and coverage-style spatial preferences are plausible. The study identifies objective-spatial conflict as a useful measure for determining whether spatially grounded BPE provides additional value. Key limitations are the use of simulated decision makers under a probit response model, the expressiveness of map clicks, and evaluation on fixed Pareto fronts. A user study with real planners is the most immediate next step for validating the proposed interaction design.
Observation and Action Encodings for Reinforcement Learning–Based Qubit Routing
A Controlled Ablation Study in qgym
We study these two encodings one at a time. The first is observation reach, or how many upcoming gates the agent can see; the second is action-space granularity, ranging from single SWAPs through a heuristically pruned set to multi-SWAP macro actions. Holding the reward, algorithm, hyperparameters, and hardware constant, we evaluate both on the coupling graphs of a 7-qubit and a 16-qubit IBM device.
Observation reach has little effect on routing, and the full-circuit view performs poorly on the larger device. Action-space granularity matters much more: macro actions solve substantially more circuits and route them with fewer SWAPs, and this advantage grows with device size. Injecting the same routing heuristic as a soft prior on the action distribution, rather than as a hard mask, preserves completeness while roughly halving SWAP overhead. A disjoint held-out evaluation suggests these results reflect transferable routing skill rather than memorized training circuits. ...
We study these two encodings one at a time. The first is observation reach, or how many upcoming gates the agent can see; the second is action-space granularity, ranging from single SWAPs through a heuristically pruned set to multi-SWAP macro actions. Holding the reward, algorithm, hyperparameters, and hardware constant, we evaluate both on the coupling graphs of a 7-qubit and a 16-qubit IBM device.
Observation reach has little effect on routing, and the full-circuit view performs poorly on the larger device. Action-space granularity matters much more: macro actions solve substantially more circuits and route them with fewer SWAPs, and this advantage grows with device size. Injecting the same routing heuristic as a soft prior on the action distribution, rather than as a hard mask, preserves completeness while roughly halving SWAP overhead. A disjoint held-out evaluation suggests these results reflect transferable routing skill rather than memorized training circuits.
Reward Engineering for RL-Based Initial Qubit Mapping
From Reward–Metric Alignment to Compiled-Circuit Quality
Quantum Circuit Routing Optimises the Wrong Metric
Closing the Proxy Gap Between SWAP Count and Schedule Length
We show this proxy gap is real and then close it. SABRE-MS keeps SABRE’s objective and adds the missing one, running time, so the router balances the two costs instead of ignoring one. A single tunable weight sets how much each of the two metrics, the program’s size and its running time, counts in the balance. It cuts a compiled program’s number of cycles by about 20% on average compared with the SABRE version that Qiskit ships. A standard reliability model shows the trade-off pays off despite the added operations, the gain holds as circuits grow to tens of qubits, and on a real 156-qubit IBM processor it raises the measured circuit fidelity by about 2.5×. The same idea, applied to a very different reinforcement-learning router, helps it in the same way, providing evidence that the benefit belongs to the objective rather than to SABRE itself. ...
We show this proxy gap is real and then close it. SABRE-MS keeps SABRE’s objective and adds the missing one, running time, so the router balances the two costs instead of ignoring one. A single tunable weight sets how much each of the two metrics, the program’s size and its running time, counts in the balance. It cuts a compiled program’s number of cycles by about 20% on average compared with the SABRE version that Qiskit ships. A standard reliability model shows the trade-off pays off despite the added operations, the gain holds as circuits grow to tens of qubits, and on a real 156-qubit IBM processor it raises the measured circuit fidelity by about 2.5×. The same idea, applied to a very different reinforcement-learning router, helps it in the same way, providing evidence that the benefit belongs to the objective rather than to SABRE itself.
Uncertainty quantification is a central mechanism in such efficient exploration methods. Provided with an estimate of how certain the agent is about the outcome of an action, it can intelligently weigh whether it is worth exploring. The Bayesian paradigm is one method to quantify uncertainty in machine learning. It models the uncertainty with a probability distributions over models, specifying how likely a model is based on the data the agent has collected.
We adopt a Bayesian point of view in model-free reinforcement learning, and develop a deeper understanding on when Bayesian reinforcement learning methods can be expected to work well and challenges that remain. To this end, in Chapter 2 we propose training ensembles through Sequential Monte Carlo, obtaining a sample from the posterior distribution of a deep Q-learning agent. We observe that agents are able to perform directed exploration, although not necessarily more efficiently than standard ensembles in every environment. Furthermore, in Chapter 3 we theoretically analyze existing Bayesian Deep model-Free Reinforcement Learning methods, and unify them into a single theoretical framework we call Epistemic Bellman Operators. We prove that these operators are contractions, establishing convergence of derived algorithms in a simplified setting. Finally, in Chapter 4 we analyze the likelihood and prior assumptions
in existing Bayesian deep model-free reinforcement learning methods, and find through statistical tests that the standard likelihood assumptions are violated on every benchmark we tested. We also find that we can improve performance of Bayesian model free reinforcement learning methods by picking different priors based on empirical data from unrelated tasks, which transfer to new environments.
This dissertation establishes several desirable properties of Bayesian Deep model free reinforcement learning, but also raises some key issues, most notably misspecification in Chapter 4. We hope our findings convince other Bayesian reinforcement learning researchers to give more attention to assumptions about priors and likelihoods. ...
Uncertainty quantification is a central mechanism in such efficient exploration methods. Provided with an estimate of how certain the agent is about the outcome of an action, it can intelligently weigh whether it is worth exploring. The Bayesian paradigm is one method to quantify uncertainty in machine learning. It models the uncertainty with a probability distributions over models, specifying how likely a model is based on the data the agent has collected.
We adopt a Bayesian point of view in model-free reinforcement learning, and develop a deeper understanding on when Bayesian reinforcement learning methods can be expected to work well and challenges that remain. To this end, in Chapter 2 we propose training ensembles through Sequential Monte Carlo, obtaining a sample from the posterior distribution of a deep Q-learning agent. We observe that agents are able to perform directed exploration, although not necessarily more efficiently than standard ensembles in every environment. Furthermore, in Chapter 3 we theoretically analyze existing Bayesian Deep model-Free Reinforcement Learning methods, and unify them into a single theoretical framework we call Epistemic Bellman Operators. We prove that these operators are contractions, establishing convergence of derived algorithms in a simplified setting. Finally, in Chapter 4 we analyze the likelihood and prior assumptions
in existing Bayesian deep model-free reinforcement learning methods, and find through statistical tests that the standard likelihood assumptions are violated on every benchmark we tested. We also find that we can improve performance of Bayesian model free reinforcement learning methods by picking different priors based on empirical data from unrelated tasks, which transfer to new environments.
This dissertation establishes several desirable properties of Bayesian Deep model free reinforcement learning, but also raises some key issues, most notably misspecification in Chapter 4. We hope our findings convince other Bayesian reinforcement learning researchers to give more attention to assumptions about priors and likelihoods.
Bayesian inference of large complex models is typically computationally infeasible unless stringent, but possibly unrealistic, assumptions are placed on the model of the physical system that we’re trying to capture. Deep learning has since emerged as a powerful way to approximate this computationally infeasible routine through pattern recognition based on an abundance of data. This principle underpins most of the success stories of recent artificial intelligence breakthroughs, like agentic chatbots or game-playing agents. This dissertation investigates this principle from a more fundamental basis to improve our algorithmic tools for the overarching experiment-design use case.
The field of meta-reinforcement learning, similarly, has often used deep learning to capture various sources of uncertainty for decision-making agents, such as unknown rewards or world dynamics. This approximates what is known as Bayes-optimal control, an agent that optimally trades off between information-seeking actions and actions that maximize a (subjective) value. In Chapter 2 we show how common deep learning approaches, like recurrent neural networks, for meta-reinforcement learning do not appropriately capture true uncertainty from the Bayesian viewpoint. Rather, we found that uncertainty is obfuscated in the activations of the neural networks. We present the Laplace Variational Recurrent Neural Network to extract this implicit uncertainty as a means to enable practitioners to deal with uncertainty explicitly.
From Chapter 2 we built the insight that one of the roles that deep learning plays in deep reinforcement learning is to amortize an exponentially costly policy optimization problem. A well-known consequence of amortization is that we always have to deal with an amortization gap (approximation gap) between the true values and the predicted values. Improving the performance of an algorithm, thus, requires a choice of tools that minimize the runtime and data needed to reduce this gap. Among the most successful approaches in this space are those that use adaptive computation at inference time to reduce the amortization gap and complement the neural network. Monte-Carlo tree search, for instance, achieved strong data-efficiency and enabled breakthroughs like defeating the human world champion in Go. However, Monte-Carlo tree search-based methods do not favorably scale in runtime. Thus, in Chapter 3 we improve the state-of-the-art of sequential Monte-Carlo planners, which permit much more efficient parallelization on modern GPU hardware. Our newly proposed algorithm demonstrated improved scaling in performance given additional planning budget, not only in data-efficiency but also in runtime-efficiency compared to prior approaches.
The final Chapter 4 exploits our finding that sequential Monte-Carlo-based planning can improve runtime complexity by shifting computational budget from neural network learning to additional planning budget to get higher quality data. Since in meta-learning settings we typically deal with learning from trajectory-based data, we often find that when trajectories increase in length, any updates to a neural network architecture will become more expensive. This is because the computational cost of gradient updates grows with trajectory length, which causes bottlenecks in scaling. For instance, updates can start to exceed hardware memory limits, forcing sequential processing of smaller batches and reducing throughput. Thus, we again find that the performance scaling to additional planning compute in deep meta-reinforcement learning can be much improved when equipped with our designed sequential Monte-Carlo planner. In contrast to prior approaches, our planner explicitly accounts for how uncertainty evolves in light of new data. As a result, our method directly targets an optimal policy that trades off exploration with exploitation, without ad hoc exploration heuristics.
In summary, this dissertation advances the state of probabilistic reinforcement learning for uncertainty-based decision-making. We show that Bayesian optimization and deep reinforcement learning can be formulated very similarly from a probabilistic viewpoint, and that only (relatively) minor assumptions and choice of approximation tools shift us from one setting to the other. This is a particularly useful insight for our use case of self-driving laboratories, since at the time of writing, Bayesian optimization is a more established approach in practice. Our findings establish a stronger common ground between the seemingly separate Bayesian optimization and reinforcement learning fields, and provide a principled basis for transferring approximation tools between these communities.
...
Bayesian inference of large complex models is typically computationally infeasible unless stringent, but possibly unrealistic, assumptions are placed on the model of the physical system that we’re trying to capture. Deep learning has since emerged as a powerful way to approximate this computationally infeasible routine through pattern recognition based on an abundance of data. This principle underpins most of the success stories of recent artificial intelligence breakthroughs, like agentic chatbots or game-playing agents. This dissertation investigates this principle from a more fundamental basis to improve our algorithmic tools for the overarching experiment-design use case.
The field of meta-reinforcement learning, similarly, has often used deep learning to capture various sources of uncertainty for decision-making agents, such as unknown rewards or world dynamics. This approximates what is known as Bayes-optimal control, an agent that optimally trades off between information-seeking actions and actions that maximize a (subjective) value. In Chapter 2 we show how common deep learning approaches, like recurrent neural networks, for meta-reinforcement learning do not appropriately capture true uncertainty from the Bayesian viewpoint. Rather, we found that uncertainty is obfuscated in the activations of the neural networks. We present the Laplace Variational Recurrent Neural Network to extract this implicit uncertainty as a means to enable practitioners to deal with uncertainty explicitly.
From Chapter 2 we built the insight that one of the roles that deep learning plays in deep reinforcement learning is to amortize an exponentially costly policy optimization problem. A well-known consequence of amortization is that we always have to deal with an amortization gap (approximation gap) between the true values and the predicted values. Improving the performance of an algorithm, thus, requires a choice of tools that minimize the runtime and data needed to reduce this gap. Among the most successful approaches in this space are those that use adaptive computation at inference time to reduce the amortization gap and complement the neural network. Monte-Carlo tree search, for instance, achieved strong data-efficiency and enabled breakthroughs like defeating the human world champion in Go. However, Monte-Carlo tree search-based methods do not favorably scale in runtime. Thus, in Chapter 3 we improve the state-of-the-art of sequential Monte-Carlo planners, which permit much more efficient parallelization on modern GPU hardware. Our newly proposed algorithm demonstrated improved scaling in performance given additional planning budget, not only in data-efficiency but also in runtime-efficiency compared to prior approaches.
The final Chapter 4 exploits our finding that sequential Monte-Carlo-based planning can improve runtime complexity by shifting computational budget from neural network learning to additional planning budget to get higher quality data. Since in meta-learning settings we typically deal with learning from trajectory-based data, we often find that when trajectories increase in length, any updates to a neural network architecture will become more expensive. This is because the computational cost of gradient updates grows with trajectory length, which causes bottlenecks in scaling. For instance, updates can start to exceed hardware memory limits, forcing sequential processing of smaller batches and reducing throughput. Thus, we again find that the performance scaling to additional planning compute in deep meta-reinforcement learning can be much improved when equipped with our designed sequential Monte-Carlo planner. In contrast to prior approaches, our planner explicitly accounts for how uncertainty evolves in light of new data. As a result, our method directly targets an optimal policy that trades off exploration with exploitation, without ad hoc exploration heuristics.
In summary, this dissertation advances the state of probabilistic reinforcement learning for uncertainty-based decision-making. We show that Bayesian optimization and deep reinforcement learning can be formulated very similarly from a probabilistic viewpoint, and that only (relatively) minor assumptions and choice of approximation tools shift us from one setting to the other. This is a particularly useful insight for our use case of self-driving laboratories, since at the time of writing, Bayesian optimization is a more established approach in practice. Our findings establish a stronger common ground between the seemingly separate Bayesian optimization and reinforcement learning fields, and provide a principled basis for transferring approximation tools between these communities.
The challenge of epistemic uncertainty estimation becomes especially tangible in the context of sequential decision-making problems. In such settings, an agent’s actions can have long-term consequences that compound over time, shaping downstream outcomes and choices. Reinforcement learning, a paradigm in which agents learn such decision-making strategies through direct interaction with an environment, faces several fundamental challenges that hinge on reliable uncertainty estimation. Underpinning both efficient exploration and safe decision-making is a principled understanding of an agent’s own epistemic uncertainty — the central topic of this thesis.
Examining the current research landscape of uncertainty quantification in deep learning, we observe a persistent tension between theoretically well-motivated yet computationally expensive techniques on one hand, and computationally efficient yet less understood methods on the other. Bayesian inference, widely regarded as the gold standard for reasoning about epistemic uncertainty, is generally intractable for modern, large-scale neural networks. This has led to a spectrum of approximate methods — including deep ensembles, advanced sampling techniques, and variational inference — that navigate this trade-off to varying degrees. More pragmatic solutions often offer substantial computational savings but lack a deeper theoretical understanding of what their uncertainty estimates represent, or how they behave in practice. From this landscape, we derive the research mission for this dissertation: to develop and analyze uncertainty quantification methods that are both computationally tractable and theoretically well-motivated.
Our first line of inquiry investigates deep neural network ensembles. We hypothesize that their efficacy is constrained not merely by the number of constituent models but by the quality of their diversity. Focusing on distributional reinforcement learning, we show that architectural components such as projection operators can induce inductive biases that shape generalization. We develop diverse projection ensembles, which induce diversity through architecturally distinct members, and demonstrate empirically that this approach yields more robust uncertainty signals and improved exploration performance.
Our second line of inquiry pursues the goal of emulating ensemble uncertainty within a single, efficient model. We introduce contextual similarity distillation, a technique enabling epistemic uncertainty estimation with a single model trained via gradient descent. Using neural tangent kernel theory, we reinterpret ensemble variance estimation as a tractable kernel regression problem. Complementarily, we provide a theoretical foundation for random network distillation, showing that, in the infinite-width limit, its uncertainty corresponds to the predictive variance of a deep ensemble. We further develop a Bayesian variant whose error signal matches the posterior predictive variance of an infinitely wide Bayesian neural network.
Finally, we address the estimation of long-term, cumulative uncertainty in reinforcement learning. We propose a novel single-model method — universal value-function uncertainties — that quantifies uncertainty over entire trajectories. Grounded in neural tangent kernel theory, we show that this method yields estimates equivalent to those of a full ensemble, while retaining computational efficiency.
In conclusion, this dissertation advances uncertainty quantification in deep reinforcement learning by bridging the gap between theoretical rigor and computational practicality. It provides both methodological innovations and theoretical insights toward more reliable, uncertainty-aware autonomous agents. ...
The challenge of epistemic uncertainty estimation becomes especially tangible in the context of sequential decision-making problems. In such settings, an agent’s actions can have long-term consequences that compound over time, shaping downstream outcomes and choices. Reinforcement learning, a paradigm in which agents learn such decision-making strategies through direct interaction with an environment, faces several fundamental challenges that hinge on reliable uncertainty estimation. Underpinning both efficient exploration and safe decision-making is a principled understanding of an agent’s own epistemic uncertainty — the central topic of this thesis.
Examining the current research landscape of uncertainty quantification in deep learning, we observe a persistent tension between theoretically well-motivated yet computationally expensive techniques on one hand, and computationally efficient yet less understood methods on the other. Bayesian inference, widely regarded as the gold standard for reasoning about epistemic uncertainty, is generally intractable for modern, large-scale neural networks. This has led to a spectrum of approximate methods — including deep ensembles, advanced sampling techniques, and variational inference — that navigate this trade-off to varying degrees. More pragmatic solutions often offer substantial computational savings but lack a deeper theoretical understanding of what their uncertainty estimates represent, or how they behave in practice. From this landscape, we derive the research mission for this dissertation: to develop and analyze uncertainty quantification methods that are both computationally tractable and theoretically well-motivated.
Our first line of inquiry investigates deep neural network ensembles. We hypothesize that their efficacy is constrained not merely by the number of constituent models but by the quality of their diversity. Focusing on distributional reinforcement learning, we show that architectural components such as projection operators can induce inductive biases that shape generalization. We develop diverse projection ensembles, which induce diversity through architecturally distinct members, and demonstrate empirically that this approach yields more robust uncertainty signals and improved exploration performance.
Our second line of inquiry pursues the goal of emulating ensemble uncertainty within a single, efficient model. We introduce contextual similarity distillation, a technique enabling epistemic uncertainty estimation with a single model trained via gradient descent. Using neural tangent kernel theory, we reinterpret ensemble variance estimation as a tractable kernel regression problem. Complementarily, we provide a theoretical foundation for random network distillation, showing that, in the infinite-width limit, its uncertainty corresponds to the predictive variance of a deep ensemble. We further develop a Bayesian variant whose error signal matches the posterior predictive variance of an infinitely wide Bayesian neural network.
Finally, we address the estimation of long-term, cumulative uncertainty in reinforcement learning. We propose a novel single-model method — universal value-function uncertainties — that quantifies uncertainty over entire trajectories. Grounded in neural tangent kernel theory, we show that this method yields estimates equivalent to those of a full ensemble, while retaining computational efficiency.
In conclusion, this dissertation advances uncertainty quantification in deep reinforcement learning by bridging the gap between theoretical rigor and computational practicality. It provides both methodological innovations and theoretical insights toward more reliable, uncertainty-aware autonomous agents.
Multi-Task Offline Reinforcement Learning
Experimental Evaluation of the Generalizability of the Soft Actor-Critic + Behavioral Cloning Algorithm
than shown. The implementation details and the source code for this study are available on GitHub at https://github.com/AxelGeist/multi-task-offlinereinforcement-learning. ...
than shown. The implementation details and the source code for this study are available on GitHub at https://github.com/AxelGeist/multi-task-offlinereinforcement-learning.
Performance of Decision Transformer in multi-task offline reinforcement learning
How does the introduction of sub-optimal data affect the performance of the model?
Teaching How to Learn to Learn
Teacher-Student Curriculum Learning for Efficient Meta-Learning
Comparative Analysis of Curriculum Strategies in training Meta-Learning
Curriculum Strategies for Faster Meta-Learning
...