As digitalization advances, cybersecurity departments are increasingly overwhelmed by alerts and potential threats, leading to decision fatigue among security analysts. In response, many are adopting Artificial Intelligence (AI) to automate routine tasks, prioritize alerts, and accelerate incident response. However, the pace of AI adoption in cybersecurity lags behind that of threat actors, revealing underlying challenges. This thesis explores these challenges through a case study of a large European bank's cybersecurity department, using a sociotechnical systems (STS) approach. Semi-structured interviews with security analysts, data scientists, and leadership revealed four key themes: (1) mixed perceptions of AI, (2) the influence of organizational factors on AI adoption, (3) the importance of interdisciplinary collaboration, and (4) the critical role of trust in AI systems. While AI offers potential benefits like improved threat detection and reduced decision fatigue, challenges such as data quality issues and lack of transparency persist. Organizational readiness, leadership support, and effective change management are crucial for successful AI integration. Building trust through transparency and active user involvement is essential for adoption. The thesis proposes a conceptual model that addresses these challenges by integrating technical and social factors, offering practical recommendations for enhancing AI adoption in cybersecurity. Future research should expand on these findings through diverse case studies, human-centered AI frameworks, and longitudinal studies to better understand AI’s impact on trust and collaboration in cybersecurity.

Intrusion detection systems (IDSs) are essential for protecting computer systems and networks from malicious attacks. However, IDSs face challenges in dealing with dynamic and imbalanced data, as well as limited label availability. In this thesis, we propose a novel elastic gradient boosting decision tree algorithm, namely Elastic CatBoost Uncertainty (eCBU), that adapts to concept drifts and copes with label scarcity by using a novel sequential uncertainty estimation method. We compare our method with state-of-the-art techniques on synthetic and real-world datasets and show that it achieves comparable accuracy and higher robustness to limited label availability in intrusion detection tasks.

Phishing attacks are a growing cause of cybersecurity incidents such as data breaches. With these attacks, malicious actors try to gain access to systems by exploiting the vulnerability of employees. Particularly, intruders use different tricks to create convincing phishing emails to force people to behave less securely. Unfortunately, technological applications are insufficient in detecting all possible phishing attempts. As the consequences of a successful breach can be disastrous for a company, especially in the banking sector, pressure is put on employees to recognise and report potential phishing emails. To aid them in this task, phishing training is provided by employers. Employees are taught what phishing emails look like and which actions they should take if an email appears malicious. However, the effectiveness of these training sessions is difficult to evaluate.

To understand the interaction of employees with phishing emails without influencing their behaviour, we study the emails employees report as suspicious to characterise the security culture at a bank. A better understanding of the behaviour provides grounds for recommendations to improve anti-phishing training and create a safer environment. The newfound metrics can provide an alternative to current methods.

This research uses Exploratory Data Analysis (EDA) to evaluate the email reporting behaviour of employees at a bank to answer the Research Question How can email reporting patterns in a large organisation measure the relationship between phishing training, reported emails and employee behaviour? With this case study, we apply EDA to a large dataset containing bank employees' reported emails over 16 months. We analysed the reported emails and related them to the provided phishing training events. Moreover, we did a text analysis of the emails' content using the Term Frequency - Inverse Document Frequency (TF-IDF) method. Additionally, we extract the dominant topics of the emails using topic modelling. Lastly, with the help of interviews, the results are tied to employees' experiences to understand their behaviour.

The major findings of the research are, firstly, the new metrics we identified to measure the security culture of a company. These metrics were found from both the analysis of employee behaviour over time, as well as the analysis of the email content. From the analysis of the reporting behaviour over time, new metrics include the unique reporters in relation to the total reported emails over time. Besides the unique reporters, unique reported emails can uncover the presence of campaigns. For example, a single email can explain the increase from 50 to 350 daily benign reports. The difference between the total reports and unique reports uncovered this. Secondly, topic analysis and content comparison show similarities between benign and malicious reported emails, indicating an increased vigilance of employees on these attributes.

A second finding originates from the analysis of the email components. One of the components was used in all simulation emails, while it was not present in all the benign and malicious reported emails. This shows that the simulation emails can be extended to include different scenarios. Therefore, we recommend the company to extend the phishing simulation emails to contain varied phishing tactics to expose employees to other types of attacks and incorporate all aspects taught during the E-learning.

Lastly, the analysis shows no concrete relation between the number of reported emails and the timing of the simulation wave. Although the reported p-value of benign emails after the simulation is 0.03, this significance can also be explained by external factors.

With these results, we can measure employees' security culture and awareness in real-world circumstances without influencing the employees' behaviour, providing a new approach to investigating phishing behaviour. Adding to the research of Steves et al. (2020),, the click rates can be explained by more than solely the employees' awareness levels, and new explanations come forward to handle phishing threats. Moreover, the absence of a required test environment for the analysis created a solution for existing gaps. For example, as seen in (Hillman et al., 2023).

A limitation of exploratory data analysis is that results are often ambiguous and mainly provide possible directions for future research. Furthermore, external factors influencing the behaviour could provide alternative reasons for the discussed interactions.

To conclude, by using the reported emails to measure security behaviour related to phishing, we found new metrics which do not influence the employees in their daily behaviour while still providing insights to improve the tactics of a company in combating phishing attacks. Reporting behaviour can be used to analyse the current anti-phishing tactics of a company and provide suggestions for improvements.

Future research should explore the differences in applying the method in other companies and across sectors. Overlap and differences can create an understanding of the diversity in security culture and the effect of external factors. Combining these results with a comprehensive understanding of a company's operations can expose directions for improvement in the security approach. Additionally, the effect of the recommendations can be analysed using the metrics we proposed. This can be done with a follow-up analysis of the behaviour to see whether the desired effect can be observed.

With the ever-increasing digitalisation of society and the explosion of internet-enabled devices with the Internet of Things (IoT), keeping services and devices secure is becoming more important. Logs play a critical role in sustaining system reliability. Manual analysis of logs has become increasingly difficult, accelerating the development of automated methods for log-anomaly detection. Despite significant progress in automating log analysis, current state-of-the-art methods face challenges dealing with unstable log data, which means that the content of log messages evolves over time.

We show that LogBERT, a state-of-the-art technique based on Bidirectional Encoder Representations from Transformers (BERT), cannot deal with unstable log data. On the three most prevalent publicly available log datasets, Mathew's Correlation Coefficient (MCC) score (which measures the correlation between a model's output and the correct labels) of LogBERT dropped by 90% after increasing log data instability from 1% to over 80% normal sequences containing logkeys in the test set. Log data instability was increased by only reassigning samples between the train and test set. Furthermore, we show that the high performance of LogBERT reported in the original paper was achieved because the model relied on a simple heuristic that only worked under specific conditions.
To address this issue, we propose a novel sequence anomaly detection technique based on BERT: Vocabulary-Free BERT (VoBERT). VoBERT uses a novel pre-training task we designed specifically for anomaly detection: Vocabulary-Free Masked Language Modeling (VF-MLM). We adapted traditional MLM and removed the fixed vocabulary constraint, which allows VF-MLM to classify out-of-vocabulary logkeys correctly.
We highlight that VoBERT is more stable than LogBERT and outperforms the latter in certain situations where log data is very unstable. For the public datasets, the MCC score of the specific train-test split used in the LogBERT paper dropped by 90% after reassigning the train-test split, increasing log data instability. In addition to sequence-level anomaly predictions, we evaluated all approaches on element level, providing a more granular performance assessment.
To assess the generalisation of the experimental results to real-world scenarios, we conducted a case study evaluating the anomaly detection models on real-world security event data collected at a large bank (50,000+ employees). We found that the simple heuristic did not work for this real-world data, having a negative correlation with the correct results. VoBERT showed performance on par with LogBERT on this real-world security event dataset.
We urge future researchers to evaluate their methods on real-world data, as we showed that the commonly used public datasets do not represent real-world scenarios. Furthermore, it is important to assess how difficult it is to detect anomalies in datasets used for evaluation. When a simple heuristic can perform well, such datasets might not be well suited to evaluate a complex anomaly detection model.
This thesis is a proof of concept for the novel pre-training task VF-MLM and paves the way for future work to refine this technique further, as well as to develop additional robust and adaptable solutions for log and security event anomaly detection.