This work seeks to resolve an outstanding problem in the use of reinforcement-learning methods for the simulation of economically-rational agents. We discuss the problem of non-stationarity, and how this subsequently limits market simulation capabilities. After explicating and isolating the source of the problem for a day-ahead electricity market, we demonstrate the application of methods which resolve this problem in simple test-cases, and prove conditions under which similar methods will work in general. Subsequently, we illustrate how these techniques can be used to solve a restricted market-design problem, in the process introducing a framework for discussing adversarial market-design for electricity markets in general. It is hoped that, insofar as they provide a new feedback-loop for market-design, these results will facilitate the design of more complex electricity-markets suitable for the energy transition.

In the electricity system, one barrier to the energy transition is the degradation of frequency stability due to the decrease of system inertia and frequency control ancillary services (FCAS), which is caused by the replacement of inertia-abundant and governor-based conventional power plants with zero-inertia and inverter-based renewable energy sources (RES).

There are already inverter technologies for RES and battery energy storage systems (BESS) to provide virtual inertia (VI) and fast frequency response (FFR) services, which are equivalent to physical inertia and conventional FCAS. Potential providers include wind, solar, battery energy storage systems (BESS), and other types of devices with the feature of energy storage. However, there are non-technical barriers to the actual implementation. For example, in most parts of the world, these services cannot participate in the electricity market and thus there is a lack of incentive for both the provision of and investment in the VI and FFR.

In the literature, there are already proposals of possible market designs for VI and FFR that procure the services, guarantee the frequency-stability requirements, and provide payments to the service provider. The main focal point is on the formulation of security-constrained unit commitment (SCUC) and security-constrained economic dispatch (SCED) problems. The formulation needs to be accurate in modeling, be solvable, and be with reasonable computational burden. Most of them only considered the allocation of ancillary services but did not price them, or price them but only by directly assigning a shadow price. Only a few works considered explicit prices in the bid.

In this project, we consider a market design with explicit bid prices. In the SCUC and SCED problems, we adopt the state-of-the-art formulation of frequency nadir constraint and method of modeling the frequency dynamics based on the linear ramp assumption of the dynamics of frequency response (FR). With these methodologies, we will investigate the features of such a market by identifying inter-dependencies of parameters in the market setups, including the interaction between the bid price and bid amount of FR and VI, and analyzing the underlying mechanisms. The results show that the amount of FR sold depends on the bid price of FR monotonously and the amount of VI sold depends on the bid price of VI monotonously, though with different patterns. The underlying reason for such dependencies is that FR, VI, and the size of the largest unit both help mitigate frequency drop and recover it and they are influencing each other. The price of FR or VI decides the relative worthiness of each option. The size of the largest unit is also related to the amount of FR sold, which is confirmed by the formulation of the analytical nadir and the QSS constraint. However, the way that the amount of available FR influences the amount of VI sold does not show a clear pattern.

These findings provide insights into the interactions between the FR and VI products and thus provide a reference for the design of the FCAS market.