ESSS Rocky’s 200 million particle simulation on Oracle Cloud pushes DEM analysis to a new height

February 22, 2022 | 5 minute read
David Chen
Master Principal Cloud Architect
Text Size 100%:

Oracle Cloud Infrastructure (OCI) and Engineering Simulation and Scientific Software corporation (ESSS) have broken simulation barriers for modeling particle laden flows. In a recent optimization study of a special purpose gas cyclone separator using computational fluid dynamics (CFD) and discrete element model (DEM) coupled modeling, a simulation of flow including 200 million discrete particles was successfully completed. This model was the largest particle model ever reported using a CFD-DEM analysis.

Refine a gas cyclone separator design on Oracle Cloud’s most powerful NVIDIA GPU shapes

The key to this success was the parallel workload balance algorithm redesigned for the latest V4.5 release of ESSS’ flagship software solution Rocky DEM. The updated implementation reduced average parallel load imbalance to less than 15%, leading to better GPU memory utilization and the expansion of the simulation model size. The Rocky DEM V4.5 release has been publicly available since October of 2021.

Figure 1: Gas Cyclone Separator simulation

A gas cyclone separator is a device commonly used to separate and remove particles from an airflow stream. You can find such devices in your home, like vacuum cleaners, but also in industrial applications, including separating abrasives, pollutants, dust particles, air pollution, toxic media, in many materials, chemicals, power generation, and other process industry plants.

     A special-purpose, high-efficiency cyclone requires a custom design to match a specific type of inlet airflow and particle separation efficiency. ESSS’ Rocky DEM software is the perfect tool to assist this design optimization process. It can accurately capture the challenging physical interactions through its complex airflow and account for hundreds of millions of fast-moving       particles, particle loading, particle size distribution, and interaction with the fluid-mechanics physics, accounting for high swirling flow inside the circular interior wall structure of the cyclone.

Technology enabler

For this study, we created an on-demand cloud computing infrastructure, illustrated in Figure 2. At the center of this solution is the bare metal BM.GPU4.8 server, designated to run the Rocky DEM simulation. It has eight NVIDIA A100 Tensor Core GPU cards and 40 GB of GPU memory in each card, adding up to 320 GB in total GPU memory—perfect to fit in the entire 200 million particle model.

For post-processing of large models, we needed a hardware-accelerated 3D visualization system, so we used VM.GPU3.1, which uses a NVIDIA V100 Tensor Core GPU card. We also used Oracle Linux, an Oracle-optimized general-purpose Linux distribution, on all three servers. It carried all required dependencies to enable the NVIDIA GPU cards and Rocky DEM software.
To effectively handle multiple terabytes of result data, we deployed a network file system (NFS) file server to serve a 10-TB block storage for both the simulation server and post-processing server together. This design provided storage space to receive the simulation output while allowing immediate read access by the post-processing server, avoiding moving output data between BM.GPU4.8 and VM.GPU3.1. The BM.GPU4.8 was deleted when the simulation work was finished, creating a more cost-effective solution. Similarly, you can delete VM.GPU3.1 while waiting for new intermediate output data.

Figure 2: OCI cloud computing architecture deployed for this study

The simulation model

The geometry of the cyclone and CFD mesh is shown in Figure 3. The CFD solution came from Ansys Fluent and was used in Rocky through its one-way steady state coupling approach. Air was injected into the cyclone at 30 m/s (108 km/hour), making the characteristic time scale of the problem of the order of 10-2 seconds. The simulation was computed for 0.12 seconds—five times longer than the particle flow through time (residence time). Typically, articles started leaving the domain at 0.025 seconds after they entered the device. The dust particles carried by the airflow, size 5–50 μm (10-6 meters), were injected into the cyclone at approximately 2 billion particles/second, bringing cyclone mass load to the level of approximately 50%.

                                                                                        a. Model geometry                     b center-plane cut of the CFD mesh             

Figure 3: Cyclone geometry and CFD mesh

The magnitude of this analysis was best reflected by the numbers summarized in the following table:

Number of injected particles 220 million
Number of simultaneous particles in the system 184 million
Number of poly mesh cells for CFD calculation ~675,000
Maximum GPU memory used 213 GB
Percentage of total GPU memory used 66.7%
Simulated time span 0.12 second
Number of time steps 4.72 million
Total Wall Clock Time of simulation 9.8 days

Table 1: Statistics of the gas cyclone separator simulation

Results of the simulation

The results of this analysis are demonstrated in Figure 1. The particle count grows quickly from zero to tens of millions, creating the expected spiral airflow pattern in the cyclone. The injection rate was achieved, and at the end of the animation, a total of 184 million particles were simultaneously swirling in the cyclone. At this point, approximately 213 GB GPU memory was used, leaving more than one-third of total GPU memory unutilized. This portion is the headroom for future Rocky computations to further expand its size and complexity on OCI’s bare metal machines.

Test run Rocky DEM on Oracle Cloud

For more information on Rocky DEM’s recent V4.5 release, check out the online ESSS webinar. To try a hands-on test of Rocky DEM software on Oracle Cloud Infrastructure, use this 30-day free trial. Remember to use the Oracle Cloud Marketplace image to deploy the BM.GPU4.8 shape for the simulation workload.

David Chen

Master Principal Cloud Architect

Previous Post

Threat Intelligence Now Available in Oracle Cloud Infrastructure

Christa Scura | 4 min read

Next Post

New Feature Announcement: On-Demand Capacity

Michael Brey | 7 min read