[EFDC+] Legacy EFDC is single-threaded: one core, however many the machine has. EFDC+ uses two forms of parallel computing together including OpenMP (OMP) multi-threading within a computer, and MPI domain decomposition across many. A code using both is a hybrid parallel code, and EFDC+ is one. The same model runs on a laptop, a high-core-count workstation, or a cloud cluster with no changes to the model itself.

EFDC+ has been multi-threaded with OpenMP since 2008. OMP uses shared memory: at computationally heavy sections of code it spawns threads that work in parallel and then rejoin, with the Intel® OpenMP Runtime Library binding them to physical processing units. EFDC+ with OMP typically runs up to four times faster on a six-core processor than conventional single-threaded EFDC.

Its limit is the machine. Shared memory cannot reach beyond one computer, and on a large model the gains past roughly 16 cores become minor.

To go further, EFDC+ splits the grid into subdomains, each solved by a separate process on its own smaller grid. Every subdomain runs much like a standalone EFDC+ model, except that it swaps information with its neighbors at each time step. That swap uses MPI, which passes messages between processes instead of sharing memory, so the processes can sit on different computers, and the model can run across a cluster.

Ghost cells make the swap work. Two extra rows or columns, copied from the neighboring subdomain, are added around each border. A cell at the edge can then see across the boundary, and calculations that need a neighboring value still work. Those borrowed values are refreshed at the end of every time step.

A couple of things can’t be worked out subdomain by subdomain. The solver for water surface elevation pools residuals from every subdomain so they all converge on one global solution, and when the time step varies dynamically, the smallest step found anywhere is applied everywhere.

Worth stating, because it is the first question modelers ask. In the Chesapeake Bay benchmark, the same model was run on one, two, and sixteen subdomains, and water surface elevation and salinity compared at five locations: the number of subdomains does not influence model output.

Nor does it change your workflow. Subdomain results are mapped back to the global grid and written in the same format as a single-domain run, with no pre- or post-processing to do.

A 3D model of roughly 800,000 computational cells (~204,000 curvilinear cells, four layers) simulating hydrodynamics, temperature, and salinity over a four-day period, run on an AWS cluster. Times include setup and 120 full output writes.

Speedup is the ratio of single-core run time to run time on the given number of cores. The single-core baseline is the expected performance of legacy EFDC versions not developed by DSI, so these figures represent the gain over the conventional code.

Speedup does not scale linearly with cores. Amdahl’s Law caps it: whatever fraction of the code cannot run in parallel eventually dominates. Communication overhead is the other constraint, it accounted for about 5% of total time at 32 cores, then jumped to nearly 20% at 48 cores, where the run first had to span multiple nodes across the network. When specifying a cluster, bandwidth and latency matter more than raw core count.

Larger domains benefit most. Decomposing a small model into many tiny subdomains can cost more in communication than it saves in computation.

Balance the work, not the cell count: Processes synchronize periodically, so the run moves at the pace of the slowest subdomain. EE’s automatic decomposition balances active cells for you, and every run reports timings to TIME.LOG so you can adjust and re-test. The benchmark above used automatic decomposition with no hand-tuning.

Tune the OMP/MPI blend: Both carry overhead, so the best split between subdomains and threads differs by model. Two threads per subdomain suited Chesapeake Bay; models with many constituents may want more.

Watch memory, output frequency, and hyperthreading: Each MPI process allocates memory for its own subdomain, every output snapshot gathers variables from all subdomains, and hyperthreading has not helped in DSI’s testing — allow one thread per physical core.

Full detail on the decomposition approach, the benchmark, and cluster configuration guidance is in the EFDC+ MPI white paper; multi-threading is covered in the EFDC+ OpenMP white paper.