Exascale High Performance Computing (HPC) represents a tremendous opportunity to push the boundaries of Computational Fluid Dynamics (CFD), but despite the consolidated trend towards the use of Graphics Processing Units (GPUs), programmability is still an issue. STREAmS-2 (Bernardini et al. Comput. Phys. Commun. 285 (2023) 108644) is a compressible solver for canonical wall-bounded turbulent flows capable of harvesting the potential of NVIDIA GPUs. Here we extend the already available CUDA Fortran backend with a novel HIP backend targeting AMD GPU architectures. The main implementation strategies are discussed along with a novel Python tool that can generate the HIP and CPU code versions allowing developers to focus their attention only on the CUDA Fortran backend. Single GPU performance is analyzed focusing on NVIDIA A100 and AMD MI250x cards which are currently at the core of several HPC clusters. The gap between peak GPU performance and STREAmS-2 performance is found to be generally smaller for NVIDIA cards. Roofline analysis allows tracing this behavior to unexpectedly different computational intensities of the same kernel using the two cards. Additional single-GPU comparisons are performed to assess the impact of grid size, number of parallelized loops, thread masking and thread divergence. Parallel performance is measured on the two largest EuroHPC pre-exascale systems, LUMI (AMD GPUs) and Leonardo (NVIDIA GPUs). Strong scalability reveals more than 80% efficiency up to 16 nodes for Leonardo and up to 32 for LUMI. Weak scalability shows an impressive efficiency of over 95% up to the maximum number of nodes tested (256 for LUMI and 512 for Leonardo). This analysis shows that STREAmS-2 is the perfect candidate to fully exploit the power of current pre-exascale HPC systems in Europe, allowing users to simulate flows with over a trillion mesh points, thus reducing the gap between the Reynolds numbers achievable in high-fidelity simulations and those of real engineering applications.

We present STREAmS-2.0, an updated version of the flow solver STREAmS, first introduced in Bernardini et al. (2021) [1]. STREAmS-2.0 has an object-oriented design which separates the physics equations from the specific back-end, making the code more suitable for future expansions, such as porting to novel computing architectures or implementation of additional flow physics. Similarly to the previous version, STREAmS-2.0 supports NVIDIA-GPU and CPU back-ends. Additionally, this version features improvements of the input/output data management, new energy and entropy preserving schemes for the discretization of the convective fluxes, recycling/rescaling inflow boundary condition, and a model for thermally perfect gases with variable specific heats. New version program summary: Program Title: STREAmS CPC Library link to program files: https://doi.org/10.17632/hdcgjpzr3y.2 Developer's repository link: https://github.com/STREAmS-CFD/STREAmS-2 Licensing provisions: GPLv3 Programming language: Fortran, CUDA Journal reference of previous version: M. Bernardini, D. Modesti, F. Salvadore, and S. Pirozzoli. STREAmS: a high-fidelity accelerated solver for direct numerical simulation of compressible turbulent flows. Comput. Phys. Commun. 263 (2021) 107906. Does the new version supersede the previous version?: Yes. Reasons for the new version: New code structure and release of new features. Summary of revisions: • The original solver [1] has been rewritten following an object-oriented design implemented through Fortran derived types that include variables and type bound procedures. The new software architecture has been designed to increase modularity and extensibility of the code, allowing users to add new back-ends and physics equations while maintaining the same code structure. This allows users to reuse portions of the code that are independent of the physics equations, the back-end, or both. The layer of computing procedures maintains a lean structure that can be highly optimized with respect to the implemented back-end. • Input handling is now based on the classic.ini format improving both user readability and input data management. • A family of new kinetic energy and entropy preserving schemes (KEEP) are now available and can be selected for stable, non-dissipative and accurate spatial discretization of the convective terms of the Navier–Stokes equations in smooth flow regions [2]. Concerning the shock-capturing flux, the improved low-dissipative WENO-Z scheme proposed by [3] is now available. • New inflow boundary conditions based on the recycling/rescale approach [4] have been implemented for the simulation of spatially evolving compressible turbulent boundary layers. Moreover, a new inflow condition based on the solution of the compressible Blasius equation is available to take into account the case of laminar boundary layers. • The constitutive relations have been generalized to take into account thermally perfect gases with variable specific heats, approximated with polynomial functions of the temperature that can be specified by the user [5]. • A new stretching function has been implemented to improve the distribution of grid nodes for the computation of wall-bounded turbulent flows. The formulation blends uniform near-wall spacing with uniform resolution in terms of Kolmogorov units in the outer wall layer, guaranteeing accuracy with higher computational efficiency [6]. Nature of problem: The code solves the compressible Navier–Stokes equations in Cartesian coordinates for a thermally perfect gas. The solver is designed for direct numerical simulation (DNS) of compressible supersonic turbulent boundary layers and various canonical configurations are supported, including turbulent channel flow, laminar and turbulent boundary layer and shock-wave/boundary layer interaction. Solution method: The equations are discretized using high-order finite difference approximations with hybrid low-dissipative/shock-capturing capabilities and the time advancement is performed using a Runge–Kutta scheme. References: [1] M. Bernardini, D. Modesti, F. Salvadore, S. Pirozzoli, STREAmS: A high-fidelity accelerated solver for direct numerical simulation of compressible turbulent flows, Comput. Phys. Commun. 263 (2021) 107906. [2] Y. Tamaki, Y. Kuya, S. Kawai, Comprehensive analysis of entropy conservation property of non-dissipative schemes for compressible flows: KEEP scheme redefined, J. Comput. Phys. 468 (2022) 111494. [3] R. Borges, M. Carmona, B. Costa, W. Don, An improved weighted essentially non-oscillatory scheme for hyperbolic conservation laws, J. Comput. Phys. 227 (6) (2008) 3191–3211, https://doi.org/10.1016/j.jcp.2007.11.038 [4] S. Pirozzoli, M. Bernardini, F. Grasso, Direct numerical simulation of transonic shock/boundary layer interaction under conditions of incipient separation, J. Fluid Mech. 657 (2010) 361–393. [5] B. J. McBride, M. J. Zehe, S. Gordon, NASA Glenn coefficients for calculating thermodynamic properties of individual species, NASA/TP 211556, NASA, 2002. [6] S. Pirozzoli, P. Orlandi, Natural grid stretching for DNS of wall-bounded flows, J. Comput. Phys. 439 (2021) 110408.

We perform direct numerical simulation of supersonic turbulent channel flow over cubical roughness elements, spanning bulk Mach numbers -, both in the transitional and fully rough regime. We propose a novel definition of roughness Reynolds number which is able to account for the viscosity variations at the roughness crest and should be used to compare rough-wall flows across different Mach numbers. As in the incompressible flow regime, the mean velocity profile shows a downward shift with respect to the baseline smooth wall cases, however, the magnitude of this velocity deficit is largely affected by the Mach number. Compressibility transformations are able to account for this effect, and data show a very good agreement with the incompressible fully rough asymptote, when the relevant roughness Reynolds number is used. Velocity statistics present outer layer similarity with the equivalent smooth wall cases, however, this does not hold for the thermal field, which is substantially affected by the roughness, even in the channel core. We show that this is a direct consequence of the quadratic temperature-velocity relation which is also valid for rough walls. Analysis of the heat transfer shows that the relative drag increase is always larger than the relative heat transfer enhancement, however, increasing the Mach number brings data closer to the Reynolds analogy line due to the rising relevance of the aerodynamic heating.