At the moment we are witnessing the maturation of distributed streaming dataflow systems whose use-cases have departed from the mere analysis of streaming windows and complex-event processing, as they now extend to cloud applications, workflows and even e-commerce. The state of streaming operators has been so far hidden from external applications. In this thesis it is argued that exposing this internal state to outside applications by making it queryable, opens the road for novel use-cases. To this end, we introduce S-Query: a system and reference architecture where the state of stream operators can be queried - either live or through snapshots, achieving different isolation levels. It is shown how this can be implemented in an existing open-source stream processor and how queryable state can affect the performance of such a system. Our experimental evaluation suggests that snapshot configuration adds only up to 5ms in latency in the 99.99th percentile and no increase in 0-90th percentiles.

Intermax Cloudsourcing B.V. designs, implements and manages critical IT-infrastructures for Dutch clients from the medical, public and financial sectors. The information that passes over these IT-infrastructures is highly confidential and privacy-sensitive, therefore it is essential that these infrastructures are secure. To improve the security of their infrastructure, Intermax is developing a Security Operations Center (SOC) which is fed by information from an optical tap which is placed on one of Intermax's core routers. The goal of this project was to extend the SOC with a REST API that analyses and classifies SSL certificates to filter malicious network data. This REST API makes use of models, so in addition to the REST API itself, a Neural Network Framework has been built to create these models. The framework can be used for different sorts of network data, but for this project, a proof of concept using SSL certificates was worked out to provide Intermax with a working product. The SOC is being built upon the security analytics framework Apache Metron and the REST API will be incorporated using Metron's Model as a Service functionality. Apache Metron sends individual data packets to the REST API, which analyses the data packet and returns whether the packet is malicious or not with a certain accuracy. This analysis takes roughly 1 millisecond and is independent from other data packets. This allows Apache Metron to spawn multiple instances of the REST API, making the solution fast and scalable. The Neural Network Framework runs completely separate from Apache Metron and the REST API. A user can configure, train and test a neural network using the framework and their own dataset. The neural network can be stored on a storage medium. Consequently, the REST API can import and apply the neural network on incoming data.