CUBENS

Abstract

We present a massively parallel GPU-accelerated solver for direct numerical simulations of transitional and turbulent flat-plate boundary layers and channel flows involving fluids in non-ideal thermodynamic states. While several high-fidelity solvers are currently available as open source, all of them are restricted to the ideal-gas region. In contrast, the CUBic Equation of state Navier-Stokes solver (CUBENS) can accurately model and simulate the non-ideal thermodynamics of single-phase compressible fluids in the vicinity of the vapor-liquid saturation line or the thermodynamic critical point. By employing high-order finite-difference schemes and convective terms in split, kinetic-energy-, and entropy-preserving form, the solver is numerically stable, and robust with minimal numerical dissipation, enabling it to capture the steep variations of non-ideal thermodynamic properties. For cost-effective high-fidelity simulations, in addition to MPI parallelization, CUBENS is GPU-accelerated using OpenACC directives for computation offloading, and asynchronous GPU-aware MPI for efficient GPU-GPU communication. Moreover, CUBENS is compatible with both NVIDIA and AMD GPU architectures, achieving significant performance results while ensuring energy-efficient simulations. For instance, using 64 NVIDIA A100 GPUs compared to 8192 CPUs at the same computational cost results in a speedup of approximately 130×. In multi-node and multi-GPU configurations ranging from 2 to 128 compute nodes (8 to 512 GPUs), a strong scaling efficiency of around 52% and a weak scaling efficiency of 0.88 with 1024³ points per GPU, corresponding to approximately 5 billion degrees of freedom, are achieved. The CUBENS solver is validated against selected cases from the literature, covering transitional to turbulent ideal and non-ideal flows up to the transonic regime. In particular, we demonstrate the solver's suitability and applicability for direct numerical simulations of transitional boundary layers with fluids at supercritical pressure and with buoyancy effects. The development of this high-fidelity solver offers the potential for future fundamental research in non-ideal compressible fluid dynamics. Program summary: Program Title: CUBic Equation of state Navier-Stokes (CUBENS) CPC Library link to program files: https://doi.org/10.17632/6jfy758gyv.1 Developer's repository link: https://github.com/pcboldini/CUBENS Licensing provisions: MIT Programming language: Fortran 90, OpenACC, MPI, Python, MATLAB Nature of problem: This code solves the three-dimensional Navier-Stokes equations for non-ideal gas flows in a Cartesian domain, applicable to boundary layers and channels. Solution method: This code uses high-order central finite-differences with split-convective form, preserving kinetic energy and entropy (KEEP) and pressure-equilibrium-preserving (PEP) property, for spatial discretization. The time advancement is performed with a third-order Total Variation Diminishing low-storage Runge-Kutta scheme. Flow non-ideality is accounted for by cubic equations of state and complex transport-properties models. Alongside MPI parallelization, the solver is GPU-accelerated using OpenACC for computation offloading and CPU-GPU data transfer, along with GPU-aware MPI for GPU-GPU communication.