A primary function of EFDC+ is to solve the governing Partial Differential Equations (PDEs) for fluid flow and constituent transport. This is accomplished using a discretized grid representing the geometry of the problem. For large systems, this grid may become quite cumbersome, requiring significant computation and memory usage, all leading to increases in the run time of a model. To alleviate this issue and take advantage of cluster computers, EFDC+ employs the domain decomposition method. 

Domain decomposition refers to a broad set of techniques to solve PDEs based on splitting the spatial domain of a problem into smaller subdomains. The same governing equations are solved on a smaller grid by separate computer cores. Naturally, solving a smaller problem reduces the memory requirements and the solution time on each subdomain. Subdomains are not isolated from one another, however, since water and constituents can flow across the entire domain. Therefore, implementing domain decomposition into a computational solver requires a means to exchange information between subdomains. Within EFDC+, Message Passing Interface (MPI) is used for this purpose. Information is communicated across the boundaries of subdomains. Communication of information in any MPI-based calculation scheme is of considerable importance, since the amount of communication generally dictates the benefits gained from employing MPI, with high communication needs reducing gains in computation, memory usage and run time speeds.

To reduce the amount of communication, additional cells belonging to the respective adjacent process are added to the border of each subdomain. These cells are referred to as ghost or halo cells. It might not be readily clear why this is necessary but consider what the cells on the border of a subdomain might require. Calculation of discretized derivative values at one of these border cells might require the solution value from its neighbor cell, which resides on a different subdomain. One option would be to communicate that value when a border cell needs it. However, that would entail communicating many times during a time step and pose a challenge for syncing up this communication between subdomains. The ghost cells can already provide that solution since they already reside on the adjacent subdomain. These ghost cells contain the values from the previous time step, so that information is already available when, at the end of a time step, the subdomains communicate their updated values in preparation for the next time step. This simplifies the way in which cell boundary information is communicated and reduces the number of times communication is required.

In addition to the communication of boundary cell information, a couple of other aspects of communication of the EFDC+ calculation scheme must be considered. Of importance is how to preserve the quality of the iterative conjugate gradient method to solve the external pressure solution. Communication is required during iterations of the conjugate gradient method so that the total residual from all subdomains can be determined, and thus the correct global search direction can be obtained on all processes. Similarly, when dynamic time stepping is used, each subdomain will compute its own respective time step. Depending on the dynamics of the system, these time steps between subdomains can be substantially different. To ensure the time steps for each subdomain are equivalent, the minimum of all the locally computed values is determined and communicated to each subdomain.

Another area that is significantly modified in the domain decomposition approach is the handling of spatially dependent input and output files. Fundamentally, the decomposition requires mapping of the user inputted global grid to the subdomain grid. This holds for any spatially dependent boundary conditions like inflows, release of particles for transport, etc. The mapping between the starting “global” domain to the subdomains is handled internally within EFDC+. Additionally, the output data must have a global view of the solution for analysis of the results. This means the subdomain solution values are collected, mapped to the correct global values, and written out. The result is that the output files are identical to how they would appear without domain decomposition but are available in a substantially shorter time period. 

Overall, to use domain decomposition with EFDC+ only requires a modeler to specify how many subdomains he or she wants. Unlike some implementations, the domain decomposition implementation in EFDC+ does not require modification of the input files and or post-processing of output files. In EFDC+, this implementation follows similar approaches to those taken in fluid dynamics solvers, while maintaining the ease of use of EEMS. This amounts to a reduction in both model run time and labor to set up a model.