Over the last two decades, the machine learning (ML) field has witnessed a dramatic expansion, propelled by burgeoning data volumes and the advancement of computational technologies. Deep learning (DL) in particular has demonstrated remarkable success across a wide range of domains, including healthcare, mobility, life sciences, and energy systems. This success has been further accelerated by the availability and efficiency of open-source ML frameworks like TensorFlow and PyTorch, making ML methodologies more accessible than ever.

However, this rapid growth has brought its own set of challenges. The proliferation of ML models and related artifacts, such as datasets, have brought abundant information during the ML lifecycle. The descriptive and property information of these artifacts is referred as metadata. Yet current practices, such as model cards used in public model zoos and tools to track metadata within scripts, cannot fully captured the metadata of these artifacts, let alone a standardized approach for their management, and access. In addition, the prevailing practice of managing ML/DL scripts via traditional software repositories, while adequate for software engineering, falls short in addressing the unique needs of ML workflows, such as model reuse and comparative analysis. These practices hinder the effective use of structured and comprehensive metadata representation. This disconnect points to a pressing need for improved methodologies and tools in the ML field.

In response to these challenges, this thesis delves into the development and exploitation of structured metadata representations within ML model zoos. In Chapter 2, we first propose a metamodel that represent different types of metadata, thus transforming the metadata from being merely descriptive to being queryable and machine-readable. The structured nature of our metamodel allows for more efficient querying and retrieval of information, which is a substantial improvement over the traditional, text-based descriptions.

Additionally, the thesis explores the use of metadata to optimize various ML processes, particularly in the selection of appropriate models for specific tasks, i.e., model inference and fine-tuning. In Chapter 3, we investigate the optimization of ML inference queries in heterogeneous model zoos using a Mixed-Integer-Programming-based optimizer. This optimizer, which considers multiple objectives such as accuracy and inference speed, provides a robust framework for model selection and execution planning. In Chapter 4, the research extends to model selection for fine-tuning. We investigate on predicting model performance, particularly accuracy, in scenarios where data domains shift, thus negating the need for constant model fine-tuning. By selectively choosing only the most promising candidates, this method substantially lowers the computational burden and associated costs of extensive model fine-tuning.

Overall, this thesis investigates the representation and application of metadata. The insights and methodologies presented not only improve the efficiency and effectiveness of ML workflows but also pave the way for further exploration in the integration of metadata within ML practices, highlighting the continual development and potential for advancements in ML.
@en

Search engines are used to gather and collect information. This interaction sometimes influences the user and changes their attitude towards a topic after such interaction. Prior work has shown that it is a complex endeavour to understand attitude change, as there are many things that can influence a user during a search session. One of the aspects that have not been researched is the influence of informedness. Therefore we examine the role of informedness in attitude change. To do that, we first defined what a well-informed user in the context of web search is. Then a user study is conducted to understand informedness’s role in attitude change. From the platform Prolific, we recruited participants (N=320) for the main user study. The experiment is a 3 x 3 (Informedness level x SERP viewpoint bias) factorial between-subjects study and it requires interaction with the provided search results. The participants were placed in one of three categories, ”Well Informed”, ”Mildly Informed”, and ”Uninformed”, based on their results on a knowledge questionnaire. We created 3 different conditions for the search engine results page (SERP). The SERP had a viewpoint-biased ranking that was either supportive, opposing or balanced towards the topic. There were two topics, abortion and obesity, to which the participants were assigned. Our findings showed that 37% of the participants with a valid submission had changed attitudes. The findings from the user study suggest that the level of informedness does not play a role in facilitating attitude change among users during a web search. This has implications for the design of information retrieval systems and web search experiences in general.

Several input types have been developed in different technological landscapes like crowdsourcing and conversational agents. However, sign language remains one of the input types that has not been looked upon. Although numerous amount of people around the world use sign language as their primary language, there have not been many efforts to include them in these technological landscapes. In this thesis, we hope to draw attention to and take a step towards the inclusion of deaf and mute people in microtask crowdsourcing.
We identify some of the existing technical and research gaps in the current architectures for Sign Language Recognition/Translation in a real-time setting. Next, we determine various microtasks which can be adapted to use sign language as input, keeping in mind the challenges it introduces. We, then, investigate the effectiveness of a system that uses sign language as input by building a web application - SignUpCrowd - for microtask crowdsourcing, namely Visual Question Answering and Tweet Sentiment Analysis tasks, and comparing it with already prevalent input types such as text and click. This comparison with different popular input types will help understand how much of a difference there is for sign language as input. In addition, it will also show the preference of input types for the particular microtasks. For this, we developed three web applications with different input types and conducted a between-subject experimental study on Prolific wherein a number of workers (N=240) were asked to perform the above-mentioned tasks using sign language, text, and click input. Our results indicate that, in terms of task completion time and task accuracy, sign language as an input modality in microtask crowdsourcing is not significantly different from other, commonly used, input types. We also noticed that people's input type preference for the given microtasks for sign language was more than text input. Although people with no knowledge of sign language found it difficult, this input modality aims at a different target audience. This shows us that there is scope for sign language as an input type for microtask crowdsourcing among people, and paves the way for more efforts for the introduction of sign language in real-world applications.

This thesis mainly studies the causality in natural language processing. Understanding causality is key to the success of NLP applications, especially in high-stakes domains. Causality comes in various perspectives such as enable and prevent that, despite their importance, have been largely ignored in the literature. In view of the lack of a dataset that can be used for causality-related research, in this thesis, we first build a first-of-its-kind, fine-grained causal reasoning dataset - FineCR, that contains new causality relations such as enable and prevent, with the help of human annotators. Our dataset contains human annotations of 25K cause-effect event pairs and 24K question-answering pairs within multi-sentence samples, where each can contain multiple causal relationships. To study current NLP models' ability to deal with the causality-related dataset and to figure out the problems that still exist, we define a series of NLP tasks based on FineCR, including causality detection, causality event extraction and causality question answering. Our experimental results with state-of-the-art deep learning models prove that there is still much room for improvement on those causal reasoning tasks. We found that those models have different shortcomings for different tasks. For the causality detection task, current classification models are easily affected by keywords, while the model cannot accurately extract the events for the causality event extraction task. And for the causality question answering task, it is sometimes difficult for the model to find the corresponding answer due to its inability to understand the semantics well. Those discoveries indicate the need to design better solutions to event causality research. In conclusion, our novel datasets and tasks provide a challenging benchmark for evaluating models' causal ability, and the experimental results shed light on future directions for improving neural language models.

Interpretability of ML models and image recognition models specifaclly, is a increasing problem. In this thesis, the design and implementation of Brickroutine: a system that used a trained model, is presented. Using human annotations, semantic interpretations are given to image classification problems. By giving an iterative approach in terms of workflows and technically designing it in a modular, salable way using Docker, the authors aim for bridging the gap between performance and interpretability of AI by combining human and computational intelligence.

As the world continues to embrace cloud computing, more applications are being scaled elastically. Elastic scaling allows applications to add or remove computing resources based on the load experienced by the application. When the load is high more resources are provisioned enabling the application to keep up with the load. When the load is low resources are removed ensuring that no resources are sitting idle. When implemented correctly elastic scaling allows applications to use fewer resources while maintaining application performance. One type of application that can benefit greatly from elastic scaling is a distributed stream processing application. Distributed stream processing applications are suited well for elastic scaling because the data coming through the data stream can be dynamic. This dynamic data stream makes it difficult to provision the right amount of computing power. One way to solve this problem is by using an auto-scaler that elastically scales the stream processing application. In this thesis, we compare different auto-scaling techniques for the distributed stream processing application Apache Flink. We implement a modern version of DS2 using metrics native to Apache Flink. An auto-scaler designed specifically for Apache Flink by Varga et al.. A modified version of the Dhalion, and a simple CPU based Kubernetes Horizontal Pod Auto-scaler (HPA). We compare the auto-scalers on the average number of resources used, the average latency, and the number of re-scale operations. Our results show the importance of a cooldown period between scaling events. The benefits of incorporating metrics from the message queue into the scaling decision, and that throughput based evaluation methods work well for determining by how much to scale.

Natural Language Interfaces for Databases (NLIDBs) offer a way for users to reason about data. It does not require the user to know the data structure, its relations, or familiarity with a query language like SQL. It only requires the use of Natural Language. This thesis focuses on a subset of NLIDBs, namely those with 'plain English' sentences as input and SQL queries as output.

Study 1 recruits participants from multiple origins (i.e. academia, a crowdsourcing platform, banking industry) without selection based on their query language capabilities. Next, participants are segmented based on query language capabilities to distinguish between non-experts and experts. A common way to retrieve information from databases is by using SQL. Thus knowledge of SQL is assumed to be a proxy for participants' skill level (i.e. SQL proficient, non-SQL proficient). We create an approach that uses an automated near semantic equivalence evaluation for user-generated queries against a predefined gold-standard SQL query and thus segment participants. We find that 70 out of 242 participants are identified as SQL proficient. To differentiate between the segmentations, we define 42 requirements often implemented for NLIDB systems, from which both segmentations pick a selection as their preferred requirements. We are unable to find statistically significant differences between the segmentations' preferences. However, exploratory findings reveal the importance of origin, namely the banking industry, which prefers explanation over answer accuracy, different from other segmentations.

Study 2 is inspired by the exploratory findings of Study 1 and uses requirements from Study 1 to create an application that tests two conditions, one with an explanation by using color-coding (i.e. to show the relations between the natural language question asked and the models' output columns) and another without. NLIDBs make it hard for users to verify if the answer provided by its model is correct. Therefore, Study 2 uses these two conditions above to test if color-coding improves performance for the participants. Our findings suggest that color-coding only improves performance for non-aggregate selection queries with multiple columns.

In the use of Machine Learning systems, attaining the trust of those that are the end-users can often be difficult. Many of the current state-of-the-art systems operate as Black-Boxes. Errors produced by these Black-Box systems, without further explanation as to why these decisions were made, will deteriorate trust. This effect is especially strong when these erroneous decisions are generated with high confidence. This thesis presents both a data-driven as well as a human-in-the-Loop based methodology to characterize and mitigate high-confidence errors. We propose an Iterative Expert Session based methodology. By engaging domain experts through a series of interaction sessions, we aim to reduce the disconnect and knowledge gap between data scientists and domain experts, and to ultimately increase trust in the model. A practical approach was taken working in close connection with the practice of the data scientists of the ILT, helping them in improving their model and providing a direct contribution. We study the problem in the context of Road and Transportation law violations, by engaging inspectors (i.e., domain experts) in day-in-the-life and in-house interview sessions.
A thorough analysis is performed of the most important features for data instances that were in error with a high degree of confidence. A method is presented that helps in characterizing these errors by predicting errors.
We show that by careful removal of biased data features, proper data selection and by bridging the knowledge gap between domain experts and data scientists, we can improve the performance of the machine learning model. We show an increase of model Precision from 0.56800 with a baseline of 0.32968 to a Precision of 0.52077 with a baseline of 0.23473. Considering the baseline, this is an increase of 28.9% in Precision. We reduce biases existent in the data by reducing variables that predict on inspector practice. The magnitude of High Confidence Errors in the top 20% errors went from 0.70435 to 0.70465 showing an improvement taking into account the reduced baseline and removal of overfitted variables.

Machine learning inference queries are a type of database query for databases where a model pipeline is needed to evaluate its boolean predicates. Using a model zoo it is possible to select a variety of models to execute in a sequence rather than using a highly specialized model to answer every query predicate. Machine learning models can have multiple measurements for gauging performance however, and the quality of a query plan therefore is not only dependent on the time needed to compute it. Selecting a query plan of models that balances multiple objectives is not a trivial feat however. This work builds upon existing methods that utilize MIPs for model selection and ordering for machine learning inference queries by extending them with multi-objective optimizing capabilities. The opportunity for adding a third objective, namely memory footprint, to that of accuracy and execution cost is explored. Several methods are then considered and compared on their suitability, and the final chosen method, the Archimedean goal method, can generate Pareto optimal query plans that provide gains over naive, greedy methods. In addition, several methods of cutting down runtime on the original optimizer are explored, leading to a program than can generate higher quality solutions in less time.

Artificial Intelligence (AI) is increasingly helping people with all kinds of tasks, due to its promising capabilities. In some tasks, an AI system by itself will take over tasks, but in other tasks, an AI system making decisions on its own would be undesired due to ethical and legal reasons. In those cases, AI can still be of help by forming human-AI teams, in which humans get advice from the AI system helping them with making their final decisions. Human-AI teams are for instance used in the medical and legal fields. One problem arises, in which instances should one trust an AI system and in which not? Trusting the AI system when it is correct and trusting yourself when you are correct, results in a high appropriate reliance. If users appropriately rely on AI systems, it is possible to achieve complementary team performance, which is better than any single teammate. However, as known from previous literature, people struggle with assessing their performance and knowing how well they perform compared to peers. When one overestimates their performance this can be because of a dual burden, due to the lack of skill they also lack the skill to accurately estimate their performance. This phenomenon is called the Dunning-Kruger Effect (DKE). This raises questions about whether the inability to estimate their own capabilities would also reflect on their assessment of the AI system its performance.

In this thesis we look at how the DKE affects (appropriate) reliance on AI systems and if so, how such effects due to the DKE can be mitigated. The effects of the DKE and possible mitigation are being tested through an empirical study (N = 249). The attempt at mitigation is done by including a tutorial intervention, which has been proved in previous research to be useful in decreasing the DKE. The tutorial intervention is aimed at revealing the weaknesses of the participant and making them aware of their miscalibrated self-estimation. Furthermore, in this thesis, the effects of revealing how the AI system makes its decisions through explainable AI (XAI) are explored. The XAI consisted of highlights from logic unit-based explanations, it should allow participants to gain more understanding of the AI advice. This thesis shows how this will affect user self-assessment and reliance on the AI system.

We found that participants who overestimate themselves tend to rely less on the AI system, compared to participants that had an accurate or underestimation of their performance. After the tutorial participants have a better calibration of their self-assessment. While the designed tutorial intervention can help participants calibrate their self-assessment, it fails to promote (appropriate) reliance. Furthermore, the logic units-based explanations did not improve accurate self-assessing or increase user (appropriate) reliance on AI systems.
This thesis shows the importance of considering cognitive biases when dealing with human-AI teams and invites more research on how to handle and mitigate the DKE in human-AI decision making.

Data drift refers to the variation in the production data compare to the training data and sometimes the machine learning model would decay because of it. Some machine learning models face the problem when in production: they receive drift data while there’s no ground truth to evaluate model performance, thus no way of determining the effectiveness of the model. Most of the previous researchers either detect drift in a statistical way and don’t look into the meanings of the drift, or require ground truth when it’s often not possible.

In this research, we work with the inland shipping model from ILT, which is a random forest model that is used to predict whether a ship is likely to violate certain regulations. We are given an out-of-sample set, which contains ships that the inland shipping model can predict but with no labels, so no ground truth to validate. We aim to evaluate the effect of possible data drift of the out-of-sample set on model performance. We propose a novel framework that first takes advantage of expert knowledge to decide drift groups with real meanings; then, we apply pseudo labeling, a mechanism that can create approximate labels for the unlabeled data; finally, we use the pseudo-labeled data to evaluate model performance. Our work shows a possible way when facing the common data drift problem and no ground truth problem for a real dataset in a production environment.

Major advances in the fault tolerance of distributed stream processing systems provided the systems with the capacity to produce strictly consistent results under failures. Consistent fault tolerance has been one of the catalysts fueling the maturity of streaming systems and boosting their widespread adoption not only for analytics use cases, but also as a platform for running applications, such as microservices and stateful functions. The most common fault tolerance strategy that many modern streaming systems adopt is an adaptation of Chandy-Lamport's distributed snapshots protocol that is based on global coordinated checkpoints. Given the importance of fault tolerance and the impact of coordinated checkpoints in the stream processing space, it is surprising that no other fault tolerance algorithm has been considered as an alternative. In this paper we bring together and benchmark three seminal fault tolerance algorithms from the distributed systems literature: the coordinated, uncoordinated, and communication-induced checkpoint algorithms. We evaluate their behavior in terms of runtime performance and failure recovery on a variety of streaming workloads including pipeline, scatter-gather, and cyclical topologies. We benchmark the algorithms on a novel streaming system, FERDiS, built from scratch as an extensible benchmarking framework. The experiment results show that the coordinated approach that state of the art streaming systems adopt is in many cases optimal, but with some exceptions where it is outperformed in both runtime performance and failure recovery by the uncoordinated and communication-induced algorithms. The cause of the differences in performance, both at runtime and during recovery, was found to be strongly related to the characteristics of the input stream(s) and the query graph. On the tested cyclic topology, the coordinated checkpointing algorithm could not be applied as by its design it will deadlock on such query graphs. The natural cycle support of the uncoordinated and communication -induced proved advantageous here and we observed high susceptibility to the unbounded domino effect of the uncoordinated algorithm which the communication-induced was not affected by, indicating the unsuitability of uncoordinated checkpointing in cyclic stream processing.

Search engines play an important role in the provision of information. Recently, researchers have raised their concerns about the potential of search engine providers trying to shift the opinion of their users by manipulating search results. But search results might also be manipulated for financial gain. If an increase in User Engagement (UE) can be achieved by manipulating search results, the manipulation of search results can lead to increased advertisement revenue. In this thesis, a 2 x 3 x 3 factorial user study (n = 376) investigating the effect of biased Search Engine Results Pages (SERPs) on UE in users with strong opinions on the topic is presented. Different search result ranking biases were used: a search result ranking biased to agree with the topic, one that disagrees with the topic and one that is indifferent towards the topic. One of two different topics is assigned to the user. The bias in the SERP is determined by the condition assigned to the user (i.e., mitigate, control or reinforce) and the viewpoint of the user towards the topic (i.e., agree or disagree). Furthermore, one out of three queries is presented at the top of the SERP (i.e., a query that agrees, disagrees or is impartial with the topic). It was found that neither mitigating nor reinforcing bias through the SERP affected UE. The exploratory analysis suggests that confirmation bias did not guide the results clicked by the users. Users were not refrained from clicking search results that do not align with their viewpoint on the topic. Instead, the exploratory analysis suggests that results are clicked based on their position (position bias), regardless of whether bias of the user is mitigated or reinforced.

Blockchains like Bitcoin are known to be victim of scalability issues. The lack in high throughput and low latency form a great bottleneck to its network. A promis- ing solution are layer 2 protocols, more precisely payment channel networks (PCN). Payment success rates are a common metric in these networks. These rates can be in- creased by tweaking the routing of payments in the network. Local routing is a form of routing that allows payments in such networks to be split over multiple paths to reach its receiver. This significantly increases the rate of payment successes, however there is no trivial way to integrate fees in such protocol. This paper focuses on the integration of fees in local routing protocols by proposing a viable solution. Local Routing Fee Protocol (LRFP) is a protocol designed to extend an existing local routing protocol and is proven to be secure. It is a light addition but works as intended. Proofs on se- curity guarantees and a formal description on the protocol form the main contribution of this paper.

Blockchain technology is the underlying mechanism that many cryptocurrencies operate on. It relies on cryptographic techniques that enforce integrity on transaction records. The records (blocks) stored are limited in size and frequency. One well-known issue regarding blockchain technology is the lack of scalability. In order to mitigate this problem, payment channels were introduced. These are considered an "off-chain" solution as communication with the blockchain is not required for executing transactions. Nonetheless, payment channels are debit-based, thus each channel that contributes to forwarding the payment is required to have a sufficient amount of coins to route the money. If this is not the case, the transaction may not succeed. For solving this matter, the node may decide to split the payment and forward it through multiple intermediaries until the destination is reached. This paper studies how fee models can be integrated into splitting protocols. An overview of the current state of the technology is provided, followed by a proposed solution for integrating fees into splitting protocols. It has been observed that splitting the payments and charging lower fees leads to a higher number of successful transactions, an outcome that was expected. The proposed fee model relies on the payment value and a statistic about the distribution of coins owned by the involved parties. The highest success ratio was achieved when combining the proposed design with the SplitIfNecessary splitting protocol. Nonetheless, when using the same fee model, there does not seem to be any pattern between the splitting protocol and the transaction value.

The largest payment channel network, Bitcoin Lightning, shows a potential alternative to cur- rent financial systems, overcoming the scalability limitations of blockchain. Source onion routing is used to route payments, but novel routing protocols claim improved effectiveness by employ- ing strategies not compatible with the current state of affairs. In this paper, we compare the current implementations deployed in the Lightning Network, with a novel routing algorithm with splitting and local routing. We find the latter option to significantly boost the success ratio of payments, while inducing some monetary overhead. Then, we propose future directions of research on Lightning protocols to boost their performance.

The skyline operator has been proposed to bridge the gap between traditional and multimedia database systems by finding the optimal objects according to the notion of Pareto dominance. According to the notion of Pareto dominance an object dominates another if it is better in one attribute and equal in all others. Skyline sets end up being pretty large because of the "curse of dimensionality". Many skyline reduction algorithms have been proposed to choose "interesting" objects from skyline sets in order to reduce their size. The purpose of this master thesis is to propose a new way of using reduction algorithms, that is to summarize datasets. A framework is proposed for interactive query refinement that will give users an overview of their query results provided by a skyline reduction algorithm. On the grounds of summarizing datasets different reduction algorithms will be compared against each other and a new novel reduction algorithm will be proposed that will hopefully summarize better query results.

Training machine learning (ML) models for natural language processing usually requires lots of data that is often acquired through crowdsourcing. In crowdsourcing, crowd workers annotate data samples according to one or more properties, such as the sentiment of a sentence, the violence of a video segment, the aesthetics of an image, ... To ensure quality of the annotations, several workers annotate the same sample, and their annotations are combined into one unique label using aggregation techniques such as majority voting.

When the property to be annotated by the workers is subjective, the workers’ annotations for one same sample might differ, but all be valid. The way the annotations are aggregated can have an effect on the fairness of the outputs of the trained model. For example only accounting for the majority vote leads to ignoring the workers’ opinions which differ from the majority and consequently being discriminative towards certain workers. Also, ML models are not always designed to account for individual opinions, for simplicity's or performance's sake. Finally, to the best of our knowledge, no method exists to assess the fairness of a ML algorithm predicting a subjective property. In this thesis we address such limitations by seeking an answer to the following research question: how can targeted crowdsourcing be used to increase the fairness of ML algorithms trained for subjective properties' prediction?

We investigate how annotation aggregation via majority voting creates a dataset bias towards the majority opinion, and how this dataset bias in combination with the current limits of ML models lead to an algorithmic bias of the ML models trained with this dataset and unfairness in the model’s outputs. We assume that an ML model able to return each annotation of each user is a fair model. We propose a new evaluation method of the ML models' fairness, and a methodology to highlight and mitigate potential unfairness based on the creation of adapted training datasets and ML models.
Although our work is applicable to any kind of label aggregation for any data subject to multiple interpretations, we focus on the effects of the bias introduced by majority voting for the task of predicting sentence toxicity.

Our results show that the fairness evaluation method that we create enables to identify unfair algorithms and compare algorithmic fairness, and the final fairness metric is usable in the training process of ML models. The experiments on the models point out that we can mitigate the biases resulting from majority voting and increase the fairness towards the minority opinions. This is provided that the workers’ individual information and each of their annotations are taken into account when training adapted models, rather than only relying on the aggregated annotations, and that the dataset is resampled on criteria according to the favoured aspect of fairness. We also highlight that more work needs to be done to develop crowdsourcing methods to collect high-quality annotations of subjective properties, possibly at low-cost.