Exascale High Performance Computing (HPC) represents a tremendous opportunity to push the boundaries of Computational Fluid Dynamics (CFD), but despite the consolidated trend towards the use of Graphics Processing Units (GPUs), programmability is still an issue. STREAmS-2 (Bernardini et al. Comput. Phys. Commun. 285 (2023) 108644) is a compressible solver for canonical wall-bounded turbulent flows capable of harvesting the potential of NVIDIA GPUs. Here we extend the already available CUDA Fortran backend with a novel HIP backend targeting AMD GPU architectures. The main implementation strategies are discussed along with a novel Python tool that can generate the HIP and CPU code versions allowing developers to focus their attention only on the CUDA Fortran backend. Single GPU performance is analyzed focusing on NVIDIA A100 and AMD MI250x cards which are currently at the core of several HPC clusters. The gap between peak GPU performance and STREAmS-2 performance is found to be generally smaller for NVIDIA cards. Roofline analysis allows tracing this behavior to unexpectedly different computational intensities of the same kernel using the two cards. Additional single-GPU comparisons are performed to assess the impact of grid size, number of parallelized loops, thread masking and thread divergence. Parallel performance is measured on the two largest EuroHPC pre-exascale systems, LUMI (AMD GPUs) and Leonardo (NVIDIA GPUs). Strong scalability reveals more than 80% efficiency up to 16 nodes for Leonardo and up to 32 for LUMI. Weak scalability shows an impressive efficiency of over 95% up to the maximum number of nodes tested (256 for LUMI and 512 for Leonardo). This analysis shows that STREAmS-2 is the perfect candidate to fully exploit the power of current pre-exascale HPC systems in Europe, allowing users to simulate flows with over a trillion mesh points, thus reducing the gap between the Reynolds numbers achievable in high-fidelity simulations and those of real engineering applications.

This paper is associated with a video winner of a 2022 American Physical Society's Division of Fluid Dynamics (DFD) Gallery of Fluid Motion (GFM) Award for work presented at the DFD Gallery of Fluid Motion. The original video is available online at the Gallery of Fluid Motion, https://doi.org/10.1103/APS.DFD.2022.GFM.V0037

We present STREAmS-2.0, an updated version of the flow solver STREAmS, first introduced in Bernardini et al. (2021) [1]. STREAmS-2.0 has an object-oriented design which separates the physics equations from the specific back-end, making the code more suitable for future expansions, such as porting to novel computing architectures or implementation of additional flow physics. Similarly to the previous version, STREAmS-2.0 supports NVIDIA-GPU and CPU back-ends. Additionally, this version features improvements of the input/output data management, new energy and entropy preserving schemes for the discretization of the convective fluxes, recycling/rescaling inflow boundary condition, and a model for thermally perfect gases with variable specific heats. New version program summary: Program Title: STREAmS CPC Library link to program files: https://doi.org/10.17632/hdcgjpzr3y.2 Developer's repository link: https://github.com/STREAmS-CFD/STREAmS-2 Licensing provisions: GPLv3 Programming language: Fortran, CUDA Journal reference of previous version: M. Bernardini, D. Modesti, F. Salvadore, and S. Pirozzoli. STREAmS: a high-fidelity accelerated solver for direct numerical simulation of compressible turbulent flows. Comput. Phys. Commun. 263 (2021) 107906. Does the new version supersede the previous version?: Yes. Reasons for the new version: New code structure and release of new features. Summary of revisions: • The original solver [1] has been rewritten following an object-oriented design implemented through Fortran derived types that include variables and type bound procedures. The new software architecture has been designed to increase modularity and extensibility of the code, allowing users to add new back-ends and physics equations while maintaining the same code structure. This allows users to reuse portions of the code that are independent of the physics equations, the back-end, or both. The layer of computing procedures maintains a lean structure that can be highly optimized with respect to the implemented back-end. • Input handling is now based on the classic.ini format improving both user readability and input data management. • A family of new kinetic energy and entropy preserving schemes (KEEP) are now available and can be selected for stable, non-dissipative and accurate spatial discretization of the convective terms of the Navier–Stokes equations in smooth flow regions [2]. Concerning the shock-capturing flux, the improved low-dissipative WENO-Z scheme proposed by [3] is now available. • New inflow boundary conditions based on the recycling/rescale approach [4] have been implemented for the simulation of spatially evolving compressible turbulent boundary layers. Moreover, a new inflow condition based on the solution of the compressible Blasius equation is available to take into account the case of laminar boundary layers. • The constitutive relations have been generalized to take into account thermally perfect gases with variable specific heats, approximated with polynomial functions of the temperature that can be specified by the user [5]. • A new stretching function has been implemented to improve the distribution of grid nodes for the computation of wall-bounded turbulent flows. The formulation blends uniform near-wall spacing with uniform resolution in terms of Kolmogorov units in the outer wall layer, guaranteeing accuracy with higher computational efficiency [6]. Nature of problem: The code solves the compressible Navier–Stokes equations in Cartesian coordinates for a thermally perfect gas. The solver is designed for direct numerical simulation (DNS) of compressible supersonic turbulent boundary layers and various canonical configurations are supported, including turbulent channel flow, laminar and turbulent boundary layer and shock-wave/boundary layer interaction. Solution method: The equations are discretized using high-order finite difference approximations with hybrid low-dissipative/shock-capturing capabilities and the time advancement is performed using a Runge–Kutta scheme. References: [1] M. Bernardini, D. Modesti, F. Salvadore, S. Pirozzoli, STREAmS: A high-fidelity accelerated solver for direct numerical simulation of compressible turbulent flows, Comput. Phys. Commun. 263 (2021) 107906. [2] Y. Tamaki, Y. Kuya, S. Kawai, Comprehensive analysis of entropy conservation property of non-dissipative schemes for compressible flows: KEEP scheme redefined, J. Comput. Phys. 468 (2022) 111494. [3] R. Borges, M. Carmona, B. Costa, W. Don, An improved weighted essentially non-oscillatory scheme for hyperbolic conservation laws, J. Comput. Phys. 227 (6) (2008) 3191–3211, https://doi.org/10.1016/j.jcp.2007.11.038 [4] S. Pirozzoli, M. Bernardini, F. Grasso, Direct numerical simulation of transonic shock/boundary layer interaction under conditions of incipient separation, J. Fluid Mech. 657 (2010) 361–393. [5] B. J. McBride, M. J. Zehe, S. Gordon, NASA Glenn coefficients for calculating thermodynamic properties of individual species, NASA/TP 211556, NASA, 2002. [6] S. Pirozzoli, P. Orlandi, Natural grid stretching for DNS of wall-bounded flows, J. Comput. Phys. 439 (2021) 110408.

We perform direct numerical simulation of supersonic turbulent channel flow over cubical roughness elements, spanning bulk Mach numbers -, both in the transitional and fully rough regime. We propose a novel definition of roughness Reynolds number which is able to account for the viscosity variations at the roughness crest and should be used to compare rough-wall flows across different Mach numbers. As in the incompressible flow regime, the mean velocity profile shows a downward shift with respect to the baseline smooth wall cases, however, the magnitude of this velocity deficit is largely affected by the Mach number. Compressibility transformations are able to account for this effect, and data show a very good agreement with the incompressible fully rough asymptote, when the relevant roughness Reynolds number is used. Velocity statistics present outer layer similarity with the equivalent smooth wall cases, however, this does not hold for the thermal field, which is substantially affected by the roughness, even in the channel core. We show that this is a direct consequence of the quadratic temperature-velocity relation which is also valid for rough walls. Analysis of the heat transfer shows that the relative drag increase is always larger than the relative heat transfer enhancement, however, increasing the Mach number brings data closer to the Reynolds analogy line due to the rising relevance of the aerodynamic heating.

In order to be able to judge the effectiveness of transition induction in WP-2, reference flow cases were planned in WP-1. There are two obvious reference cases—a fully laminar interaction and a fully turbulent interaction. Here it should be explained that the terms “laminar” and “turbulent” interaction refer to the boundary layer state at the beginning of interaction only. There are two basic configurations of shock wave boundary layer interaction and these are a part of the TFAST project. One is the normal shock wave, which typically appears at the transonic wing and on the turbine cascade. The characteristic incipient separation Mach number range is about M = 1.2 in the case of a laminar boundary layer and about M = 1.32 in the case of turbulent boundary layer. The second typical flow case is the oblique shock wave reflection. The most characteristic case in European research is connected to the 6th FP IP HISAC project concerning a supersonic business jet. The design speed of this airplane is M = 1.6. Therefore the TFAST consortium decided to use this Mach number as the basic case. Pressure disturbance at this Mach number is not very high and can be compared to the disturbance of the normal shock at the incipient separation Mach number mentioned earlier. As mentioned earlier, shock reflection at M = 1.6 may be related to incipient separation. Therefore two additional test cases were planned with different Mach numbers. ITAM conducted an M = 1.5 test case, and TUD an M = 1.7 test case. These partners have also previously made very specialized and successful contributions to the UFAST project.

We present STREAmS, an in-house high-fidelity solver for direct numerical simulations (DNS) of canonical compressible wall-bounded flows, namely turbulent plane channel, zero-pressure gradient turbulent boundary layer and supersonic oblique shock-wave/boundary layer interaction. The solver incorporates state-of-the-art numerical algorithms, specifically designed to cope with the challenging problems associated with the solution of high-speed turbulent flows and can be used across a wide range of Mach numbers, extending from the low subsonic up to the hypersonic regime. From the computational viewpoint, STREAmS is oriented to modern HPC platforms thanks to MPI parallelization and the ability to run on multi-GPU architectures. This paper discusses the main implementation strategies, with particular reference to the CUDA paradigm, the management of a single code for traditional and multi-GPU architectures, and the optimization process to take advantage of the latest generation of NVIDIA GPUs. Performance measurements show that single-GPU optimization more than halves the computing time as compared to the baseline version. At the same time, the asynchronous patterns implemented in STREAmS for MPI communications guarantee very good parallel performance especially in the weak scaling spirit, with efficiency exceeding 97% on 1024 GPUs. For overall evaluation of STREAmS with respect to other compressible solvers, comparison with a recent GPU-enabled community solver is presented. It turns out that, although STREAmS is much more limited in terms of flow configurations that can be addressed, the advantage in terms of accuracy, computing time and memory occupation is substantial, which makes it an ideal candidate for large-scale simulations of high-Reynolds number, compressible wall-bounded turbulent flows. The solver is released open source under GPLv3 license. Program summary: Program Title: STREAmS CPC Library link to program files: https://doi.org/10.17632/hdcgjpzr3y.1 Developer's repository link: https://github.com/matteobernardini/STREAmS Code Ocean capsule: https://codeocean.com/capsule/8931507/tree/v2 Licensing provisions: GPLv3 Programming language: Fortran 90, CUDA Fortran, MPI Nature of problem: Solving the three-dimensional compressible Navier–Stokes equations for low and high Mach regimes in a Cartesian domain configured for channel, boundary layer or shock-boundary layer interaction flows. Solution method: The convective terms are discretized using a hybrid energy-conservative shock-capturing scheme in locally conservative form. Shock-capturing capabilities rely on the use of Lax–Friedrichs flux vector splitting and weighted essentially non-oscillatory (WENO) reconstruction. The system is advanced in time using a three-stage, third-order RK scheme. Two-dimensional pencil distributed MPI parallelization is implemented alongside different patterns of GPU (CUDA Fortran) accelerated routines.