intro story D-Flow FM

 

D-Flow Flexible Mesh

D-Flow Flexible Mesh (D-Flow FM) is the new software engine for hydrodynamical simulations on unstructured grids in 1D-2D-3D. Together with the familiar curvilinear meshes from Delft3D 4, the unstructured grid can consist of triangles, pentagons (etc.) and 1D channel networks, all in one single mesh. It combines proven technology from the hydrodynamic engines of Delft3D 4 and SOBEK 2 and adds flexible administration, resulting in:

  • Easier 1D-2D-3D model coupling, intuitive setup of boundary conditions and meteorological forcings (amongst others).
  • More flexible 2D gridding in delta regions, river junctions, harbours, intertidal flats and more.
  • High performance by smart use of multicore architectures, and grid computing clusters.
An overview of the current developments can be found here.
 
The D-Flow FM - team would be delighted if you would participate in discussions on the generation of meshes, the specification of boundary conditions, the running of computations, and all kinds of other relevant topics. Feel free to share your smart questions and/or brilliant solutions! 

 

=======================================================
We have launched a new website (still under construction so expect continuous improvements) and a new forum dedicated to Delft3D Flexible Mesh.

Please follow this link to the new forum: 
/web/delft3dfm/forum

Post your questions, issues, suggestions, difficulties related to our Delft3D Flexible Mesh Suite on the new forum.

=======================================================

** PLEASE TAG YOUR POST! **

 

 

Sub groups
D-Flow Flexible Mesh
DELWAQ
Cohesive sediments & muddy systems

 


Message Boards

MPI/ domain decomposition on a cluster and windows: Clarification?

Ben Williams, modified 9 Years ago.

MPI/ domain decomposition on a cluster and windows: Clarification?

Jedi Knight Posts: 114 Join Date: 3/23/11 Recent Posts
Hello,

I have a question just to clarify what functionality exists in Delft3D as of 4.00.07.0892 and planned for near-future releases (e.g. v 5.00.xx.xxx). I find the release notes and current developments pages a little confusing, as there is not date associated with any of the 'current developments' so I cannot distinguish between what has already been achieved, what is about to be released/ currentlty released, and what is in the pipeline for various dated in the future.

May I ask in particular about running parallel (non DD) and running with DD.

For the two seperate scenarios of running on windows (multiple cores, single processor) and also on a linux cluster (multiple cores, multiple nodes):

1) Do online waves+flow+morph work with either DD or MPI functionality (or both)?

2) Assuming it does, Using DD in Delft3D for online wave+flow+morph, can one use mpi to run each decomposed grid in parallel?

In the above arguement, I am assming that any cluster simulation is run on a single node (but multiple cores). Therefore for e.g. a 6 core node (or a 6 core processor on a windows machine), assuming there are two decomposed domains, would it be possible to use three cores per domain for the flow and morph part of the simulation?

Also, I have seen reference to an eventual goal of distributed computing where DD can be used with each domain used on a seperate node (and therefore all cores on each node dedicated to its own domain). May I ask if this is something that is being developed at the momemnt, and if this is something that is considered imminent (1-3 months), short-term (3-6 months), medium term (6 - 12 months) or longterm (>12 months)?

I ask because a roadmap helps if one is considering a long-term project strategy over say a two year period. Or maybe if there are particular functionalities that Deltares would be interested in testing 'out of house'.

Best regards,

Ben
Adri Mourits, modified 9 Years ago.

RE: MPI/ domain decomposition on a cluster and windows: Clarification? (Answer)

Yoda Posts: 1221 Join Date: 1/3/11 Recent Posts
Hi Ben,


1) Do online waves+flow+morph work with either DD or MPI functionality (or both)?
Online waves+flow+morph works with DD. It does not work with MPI.

2) Assuming it does, Using DD in Delft3D for online wave+flow+morph, can one use mpi to run each decomposed grid in parallel?
No.

An additional positive point: When running a model with waves+flow+morph+DD on a multicore Windows machine, have a look at the performance in the task manager:
When doing FLOW calculations, Windows divides the work of the subdomains over the cores. It's something between sequential and parallel. This is done automatically; you can not use 1 or only a few cores.
When doing WAVE calculations, SWAN is working and SWAN is fully parallellized, using all cores optimized. You can choose how many parallel jobs it uses by defining environment parameter OMP_NUM_THREADS, see "<your Delft3D installation directory>\w32\lib\swan.bat" for an example.

DD with each subdomain on a node should work on Linux using the trunk, but this is not fully tested and there are some questions about the performance. These problems are planned to be solved on a short-term.

DD with each subdomain on a core/node on Windows is not planned yet.

Regards,

Adri
NN
Nill Ng, modified 5 Years ago.

RE: MPI/ domain decomposition on a cluster and windows: Clarification?

Youngling Posts: 23 Join Date: 4/16/15 Recent Posts
Adri,

You said in the previous reply that, "When doing FLOW calculations, Windows divides the work of the subdomains over the cores. It's something between sequential and parallel. This is done automatically; you can not use 1 or only a few cores."
Does that mean when I am doing FLOW simulation with DD, the optimized arrangement is already provided by starting DD run with the Delft3D FLOW GUI?
I am using a PC built for modelling with 32 cores (also use for non-Delft3D modelling).
But it seems most of the other cores are not involved in the modelling.

I have gone through a couple of the related threads about parallel run in this forum, but just still a bit confused about the execution.

Regards,
Nill
karunakar kintada, modified 5 Years ago.

RE: MPI/ domain decomposition on a cluster and windows: Clarification?

Padawan Posts: 35 Join Date: 7/31/12 Recent Posts
Hello Adri,
Although this is an older post, I've got couple of questions.

You have mentioned about each sub domain on a node on Linux.
Would the normal launch of d_hydro.exe with xml file having DDB tag is sufficient through job scheduler or any other procedure exists for forcing such an arrangement??
Adri Mourits, modified 5 Years ago.

RE: MPI/ domain decomposition on a cluster and windows: Clarification? (Answer)

Yoda Posts: 1221 Join Date: 1/3/11 Recent Posts
Hi Karunakar,

There is only one way to start a DD calculation: execute "d_hydro.exe config.xml". It doesn't matter whether you execute this via a script or via the GUI.

On execution, multiple threads are started: one for each subdomain, one for each DDboundary connecting two subdomains and a few general threads. Most of the calculation power is needed in the subdomain threads.

Both Windows and Linux distribute the threads over the available cores automatically.

Assuming you have a big model, needing a lot of computation power:
If your model consists of two subdomains, the two cores executing the subdomain threads will be very busy, the rest of the cores will be quiet.
If your model consists of 4 subdomains, the 4 cores executing the subdomain threads will be very busy, the rest of the cores will be quiet. If the computation power is the bottle neck, this calculation should be faster than with 2 subdomains.
Increasing the number of subdomains will lead to using more computation power, but will also increase the communication needed between the threads. Above a certain number of subdomains, the simulation will become slower: computation power is no longer the bottle neck but the communication between the threads. Further increasing the number of subdomains is not useful. The exact number of subdomains marking this turning point depends on a lot of factors: model size, subdomain sizes, DDboundary sizes, hardware used, operating system etc.

I hope this clarifies it a bit.

Regards,

Adri
U
Paul Branson, modified 4 Years ago.

RE: MPI/ domain decomposition on a cluster and windows: Clarification?

Picking up this discussion, given that MPI based domain decomposition has been implemented along a single dimension as discribed in this document:

www.prace-ri.eu/IMG/pdf/wp100.pdf

with some very good scaling for the idealised grid cases. It also describes a case in Figure 3: Full domain for the Zeedelta benchmark. Each colour re
presents a computational domain, with in this case in total 8 domains.

Does this suggest that the combination of MPI + 'traditional' delft3d domain decomposition has been implemented? (i.e. with mpi sub-tasking for each domain?)

Also does it always pick the longest dimension to split.

Further to the response in 2011 regarding MPI not functioning for hydro+wave+mor, is that still the case? Does it work for hydro+waves only and online coupling?

Cheers!
U
Paul Branson, modified 4 Years ago.

RE: MPI/ domain decomposition on a cluster and windows: Clarification?

This has been addressed in the FAQ! For future reference:

https://oss.deltares.nl/web/delft3d/faq#How%20do%20I%20run%20a%20calculation%20in%20parallel?

And there are details of the MPI functionality provided here:
https://svn.oss.deltares.nl/repos/delft3d/trunk/src/engines_gpl/flow2d3d/doc/parallel_status.txt