intro story D-Flow FM

 

D-Flow Flexible Mesh

D-Flow Flexible Mesh (D-Flow FM) is the new software engine for hydrodynamical simulations on unstructured grids in 1D-2D-3D. Together with the familiar curvilinear meshes from Delft3D 4, the unstructured grid can consist of triangles, pentagons (etc.) and 1D channel networks, all in one single mesh. It combines proven technology from the hydrodynamic engines of Delft3D 4 and SOBEK 2 and adds flexible administration, resulting in:

  • Easier 1D-2D-3D model coupling, intuitive setup of boundary conditions and meteorological forcings (amongst others).
  • More flexible 2D gridding in delta regions, river junctions, harbours, intertidal flats and more.
  • High performance by smart use of multicore architectures, and grid computing clusters.
An overview of the current developments can be found here.
 
The D-Flow FM - team would be delighted if you would participate in discussions on the generation of meshes, the specification of boundary conditions, the running of computations, and all kinds of other relevant topics. Feel free to share your smart questions and/or brilliant solutions! 

 

=======================================================
We have launched a new website (still under construction so expect continuous improvements) and a new forum dedicated to Delft3D Flexible Mesh.

Please follow this link to the new forum: 
/web/delft3dfm/forum

Post your questions, issues, suggestions, difficulties related to our Delft3D Flexible Mesh Suite on the new forum.

=======================================================

** PLEASE TAG YOUR POST! **

 

 

Sub groups
D-Flow Flexible Mesh
DELWAQ
Cohesive sediments & muddy systems

 


Message Boards

Linux flow-wave coupling run terminated with segmentation fault

PL
Ping Luo, modified 4 Years ago.

Linux flow-wave coupling run terminated with segmentation fault

Youngling Posts: 8 Join Date: 10/22/15 Recent Posts
Hello,

We are doing a coupled simulation with flow and wave on a Linux cluster (CentOS 6). The flow model contains two mdf files. The simulation runs for a couple of hours and then hang. What happens is the wave process is still running and waiting for the flow to finished. But the flow process has crashed with segmentation fault. The same input files works well in Windows.

We then tested flow separately. We are able to run flow successfully with a single mdf file. We also tested flow with two mdf files, just like the way in the coupled simulation. This time flow crashed with "double free or corruption" error:

*** glibc detected *** d_hydro.exe: double free or corruption (!prev): 0x00002af39ab8a170 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x75e66)[0x2af379416e66]
/lib64/libc.so.6(+0x789b3)[0x2af3794199b3]
/software/easybuild/software/ifort/2015.3.187-GCC-4.8.4/lib/intel64/libifcore.so.5(for__free_vm+0x2a)[0x2af37ccb8aea]
/software/easybuild/software/ifort/2015.3.187-GCC-4.8.4/lib/intel64/libifcore.so.5(for__release_lun+0x2b4)[0x2af37cc85034]
/software/easybuild/software/ifort/2015.3.187-GCC-4.8.4/lib/intel64/libifcore.so.5(for_write_int_fmt_xmit+0x258e)[0x2af37ccc349e]
/software/tamusc/Delft3D/5.01.00.2163-O2/lib/libflow2d3d.so(setcurrentdatetime_+0x3be)[0x2af379ccd13e]
/software/tamusc/Delft3D/5.01.00.2163-O2/lib/libflow2d3d.so(trisol_+0xeef)[0x2af3797d2f1f]
/software/tamusc/Delft3D/5.01.00.2163-O2/lib/libflow2d3d.so(tricom_step_+0x1905)[0x2af3797ce665]
/software/tamusc/Delft3D/5.01.00.2163-O2/lib/libflow2d3d.so(mod_trisim_mp_trisim_step_+0x11)[0x2af3797d04b1]
/software/tamusc/Delft3D/5.01.00.2163-O2/lib/libflow2d3d.so(trisim_+0xcd)[0x2af3797d041d]
/software/tamusc/Delft3D/5.01.00.2163-O2/lib/libflow2d3d.so(_Z14FlowDD_ProcessP8IteratorPKcP4Blob+0x389)[0x2af3797a04e9]
/software/tamusc/Delft3D/5.01.00.2163-O2/lib/libflow2d3d.so(_Z13IteratorShellPv+0x1a2)[0x2af3797917d2]
/lib64/libpthread.so.0(+0x79d1)[0x2af37918a9d1]
/lib64/libc.so.6(clone+0x6d)[0x2af3794898fd]

We tested several times, and each time d_hydro.exe crashed at different time step.

Please help us on this.

Thanks,

Ping
Adri Mourits, modified 4 Years ago.

RE: Linux flow-wave coupling run terminated with segmentation fault

Yoda Posts: 1224 Join Date: 1/3/11 Recent Posts
Hi Ping,

This error message appeared in old versions, including the version you seem to be using: revision 2163.

Switching to a newer version, preferably the latest tagged one (currently https://svn.oss.deltares.nl/repos/delft3d/tags/5425/), will solve your problem. You will need to prepare the NetCDF library, see this page. Alternatively you can switch to the last version without NetCDF: https://svn.oss.deltares.nl/repos/delft3d/tags/4440/

Regards,

Adri
PL
Ping Luo, modified 4 Years ago.

RE: Linux flow-wave coupling run terminated with segmentation fault

Youngling Posts: 8 Join Date: 10/22/15 Recent Posts
Adri,

Thanks for your reply. We tried with the latest version tagged with 5425. But d_hydra still crashed with memory error:

*** glibc detected *** d_hydro.exe: double free or corruption (out): 0x00007ff1dfa0da70 ***
======= Backtrace: =========
/lib64/libc.so.6[0x3da9075e66]
/lib64/libc.so.6[0x3da90789b3]
/software/tamusc/netCDF-Fortran/4.4.2-intel-2015B/lib/libnetcdff.so.6(for__free_vm+0x2a)[0x7ff2042846ea]
/software/easybuild/software/ifort/2015.3.187-GCC-4.8.4/lib/intel64/libifcore.so.5(for_read_int_lis_xmit+0xa1e)[0x7ff1f9d56bde]
/software/tamusc/Delft3D/5.01.00.5520/lib/libflow2d3d.so(meteo_read_mp_readseries_+0x11c)[0x7ff1fa8c451c]
/software/tamusc/Delft3D/5.01.00.5520/lib/libflow2d3d.so(meteo_mp_meteoupdateitem_+0xb31)[0x7ff1fa8ba2c1]
/software/tamusc/Delft3D/5.01.00.5520/lib/libflow2d3d.so(meteo_mp_meteoupdate_+0x11c)[0x7ff1fa8b975c]
/software/tamusc/Delft3D/5.01.00.5520/lib/libflow2d3d.so(incmeteo_+0x4d8)[0x7ff1fa34fa78]
/software/tamusc/Delft3D/5.01.00.5520/lib/libflow2d3d.so(trisol_+0x13316)[0x7ff1fa2e6cf6]
/software/tamusc/Delft3D/5.01.00.5520/lib/libflow2d3d.so(tricom_step_+0x1736)[0x7ff1fa2cf186]
/software/tamusc/Delft3D/5.01.00.5520/lib/libflow2d3d.so(mod_trisim_mp_trisim_step_+0x11)[0x7ff1fa2d1e41]
/software/tamusc/Delft3D/5.01.00.5520/lib/libflow2d3d.so(trisim_+0xd2)[0x7ff1fa2d2a12]
/software/tamusc/Delft3D/5.01.00.5520/lib/libflow2d3d.so(_Z14FlowDD_ProcessP8IteratorPKcP4Blob+0x389)[0x7ff1fa2a25b9]
/software/tamusc/Delft3D/5.01.00.5520/lib/libflow2d3d.so(_Z13IteratorShellPv+0x1a2)[0x7ff1fa2938f2]
/lib64/libpthread.so.0[0x3da94079d1]
/lib64/libc.so.6(clone+0x6d)[0x3da90e88fd]


If you need additional information, please let me know.


Ping
PL
Ping Luo, modified 4 Years ago.

RE: Linux flow-wave coupling run terminated with segmentation fault

Youngling Posts: 8 Join Date: 10/22/15 Recent Posts
Forgot to mention:

The compiler used to build delft3d is Intel 15.0.3
PL
Ping Luo, modified 4 Years ago.

RE: Linux flow-wave coupling run terminated with segmentation fault

Youngling Posts: 8 Join Date: 10/22/15 Recent Posts
Adri Mourits:
Hi Ping,

This error message appeared in old versions, including the version you seem to be using: revision 2163.

Switching to a newer version, preferably the latest tagged one (currently https://svn.oss.deltares.nl/repos/delft3d/tags/5425/), will solve your problem. You will need to prepare the NetCDF library, see this page. Alternatively you can switch to the last version without NetCDF: https://svn.oss.deltares.nl/repos/delft3d/tags/4440/

Regards,

Adri



Hello Adri,

I installed the latest version with Intel compiler 15.0.3 and 14.0.3. In both cases, d_hydro.exe crashed with memory issues.


Waiting for communication with Delft3D-FLOW ...
*** glibc detected *** /software/tamusc/Delft3D/5.01.00.5520-4440/bin/d_hydro.exe: munmap_chunk(): invalid pointer: 0x00007f587394eb30 ***
======= Backtrace: =========
/lib64/libc.so.6[0x3d1b075e66]
/software/easybuild/software/ifort/2013_sp1.3.174/lib/intel64/libifcore.so.5(for__free_vm+0x1b)[0x7f588f1a7f4b]
/software/easybuild/software/ifort/2013_sp1.3.174/lib/intel64/libifcore.so.5(for_read_int_lis_xmit+0xa88)[0x7f588f186b78]
/software/tamusc/Delft3D/5.01.00.5520-4440/lib/libflow2d3d.so(meteo_read_mp_readseries_+0x14f)[0x7f588ffcc97f]
/software/tamusc/Delft3D/5.01.00.5520-4440/lib/libflow2d3d.so(meteo_mp_meteoupdateitem_+0x88c)[0x7f588ffc261c]
/software/tamusc/Delft3D/5.01.00.5520-4440/lib/libflow2d3d.so(meteo_mp_meteoupdate_+0x130)[0x7f588ffc1d50]
/software/tamusc/Delft3D/5.01.00.5520-4440/lib/libflow2d3d.so(incmeteo_+0x4e1)[0x7f588f9bc3d1]
/software/tamusc/Delft3D/5.01.00.5520-4440/lib/libflow2d3d.so(trisol_+0x12f1e)[0x7f588f953cde]
/software/tamusc/Delft3D/5.01.00.5520-4440/lib/libflow2d3d.so(tricom_step_+0x18c6)[0x7f588f93c166]
/software/tamusc/Delft3D/5.01.00.5520-4440/lib/libflow2d3d.so(mod_trisim_mp_trisim_step_+0x16)[0x7f588f93ef76]
/software/tamusc/Delft3D/5.01.00.5520-4440/lib/libflow2d3d.so(trisim_+0xcf)[0x7f588f93fddf]
/software/tamusc/Delft3D/5.01.00.5520-4440/lib/libflow2d3d.so(_Z14FlowDD_ProcessP8IteratorPKcP4Blob+0x389)[0x7f588f90ddd9]
/software/tamusc/Delft3D/5.01.00.5520-4440/lib/libflow2d3d.so(_Z13IteratorShellPv+0x1a4)[0x7f588f8ff184]
/lib64/libpthread.so.0[0x3d1b4079d1]
/lib64/libc.so.6(clone+0x6d)[0x3d1b0e88fd]

The memory dump when it crashed is attached.

Any idea how to resolve this?

Thanks,

Ping
Adri Mourits, modified 4 Years ago.

RE: Linux flow-wave coupling run terminated with segmentation fault

Yoda Posts: 1224 Join Date: 1/3/11 Recent Posts
Hi Ping,

The problem appears on reading one of the meteo input files, containing a simple time series. Does this happen in one of the example testcases or in your own model? Can you check whether the input file is correct: Linux type of file instead of Windows, each line starting with the time in minutes since the reference date, covering the full simulation period.

Does the problem occur at the initialization phase or somewhere during the simulation?

In case you are still stuck and the problem appears on your own model: can you zip the full set of input files (ideally of an as small as possible model that shows the problem) and attach it to a post on this forum?

Thanks.

Regards,

Adri
PL
Ping Luo, modified 4 Years ago.

RE: Linux flow-wave coupling run terminated with segmentation fault

Youngling Posts: 8 Join Date: 10/22/15 Recent Posts
Adri,

Thank you for your reply.

The error occurs with our user's own input file. It happens during the simulation at random time step. Our user used to run his simulations on his windows machine and has just moved to our cluster recently. I'll ask him to double check his input files. If the problem persists, I'll see if the user can provide a smaller model which I can upload to this forum.



Thanks,

Ping
PL
Ping Luo, modified 4 Years ago.

RE: Linux flow-wave coupling run terminated with segmentation fault

Youngling Posts: 8 Join Date: 10/22/15 Recent Posts
Adri,

The case that caused the segmentation fault is attached. It crashed in meteo_read.f90 at line 424.

#0 0x00002ada74e10200 in for_read_int_lis_xmit () from /software/easybuild/software/ifort/2015.3.187-GCC-4.8.4/lib/intel64/libifcore.so.5
#1 0x00002ada71ff8fd4 in meteo_read::readseries (minp=36, d=..., kx=2, tread=900) at meteo_read.f90:424
#2 0x00002ada71fe7b10 in meteo::meteoupdateitem (meteoitem=0x2ada9b483d30, flow_itdate=20140101, flow_tzone=0, tim=840.25)
at meteo.f90:480
#3 0x00002ada71fe75fc in meteo::meteoupdate (runid='coa0', ' ' <repeats 252 times>, flow_itdate=20140101, flow_tzone=0, time=840.25,
.tmp.RUNID.len_V$856=256) at meteo.f90:382
#4 0x00002ada7180be32 in incmeteo (timhr=14.004166666666666, grdang=0, windu=..., windv=..., patm=..., kcs=..., alfas=..., windsu=...,
windsv=..., w10mag=..., gdp=...) at incmeteo.f90:118
#5 0x00002ada71742c3c in trisol (dischy='cn ', solver='jac ', icreep=0, ithisc=1920, timnow=1680.5, nst=1680, itiwec=-1,
trasol='cyclic-method', forfuv='Y', forfww='N', nfltyp=2, saleqs=31, temeqs=15, sferic=.FALSE., grdang=0, ktemp=0, temint='Y', keva=0,
evaint='Y', anglat=37.168999999999997, anglon=0, rouflo='CHEZ', rouwav='FR84', betac=0, tkemod='\000' <repeats 12 times>,
comfil='com-coa0', ' ' <repeats 248 times>, error=.FALSE., gdp=..., .tmp.DISCHY.len_V$a42a=8, .tmp.SOLVER.len_V$a42c=8,
.tmp.TRASOL.len_V$a433=13, .tmp.FORFUV.len_V$a435=1, .tmp.FORFWW.len_V$a437=1, .tmp.TEMINT.len_V$a43f=1, .tmp.EVAINT.len_V$a442=1,
.tmp.ROUFLO.len_V$a446=4, .tmp.ROUWAV.len_V$a448=4, .tmp.TKEMOD.len_V$a44b=12, .tmp.COMFIL.len_V$a44d=256) at trisol.f90:1252
#6 0x00002ada71733f0b in tricom_step (olv_handle=..., gdp=...) at tricom_step.F90:953
#7 0x00002ada7173b4e2 in mod_trisim::trisim_step (olv_handle=..., gdp=...) at trisim_mod.F90:364
#8 0x00002ada7173bdda in trisim (numdom=2, nummap=3, context_id=-1001, fsm_flags=1, runid='coa0', .tmp.RUNID.len_V$d7d5=4)
at trisim.F90:101
#9 0x00002ada716a685f in FlowDD_Process (self=0xa19260, name=0xa192e0 "coa0", configblob=0x2ada940009e0) at flow_nxtstp.cpp:228
#10 0x00002ada7168a8cd in IteratorShell (argument=0x5) at iterator.cpp:521
#11 0x0000003fee0079d1 in start_thread () from /lib64/libpthread.so.0
#12 0x0000003fedce88fd in clone () from /lib64/libc.so.6

The user is eager to get his simulations done by end of November. So please help him resolve this issue.

Thanks and regards,

Ping
Adri Mourits, modified 4 Years ago.

RE: Linux flow-wave coupling run terminated with segmentation fault

Yoda Posts: 1224 Join Date: 1/3/11 Recent Posts
Hi Ping,

Your model runs fine using Delft3D-FLOW version 6.02.01.5425.

The file dd.xml suggests that you use a FLOW version 5.xx.xx.xxxx. I advice you to switch to a newer version, preferably the latest tagged version, currently https://svn.oss.deltares.nl/repos/delft3d/tags/5425/

Regards,

Adri
PL
Ping Luo, modified 4 Years ago.

RE: Linux flow-wave coupling run terminated with segmentation fault

Youngling Posts: 8 Join Date: 10/22/15 Recent Posts
Adri,

Glad you are able to run the model. In fact, the user was also able to run it on his Windows machine.

Please don't be mislead by the version number in its path. The version I am using was downloaded from https://svn.oss.deltares.nl/repos/delft3d/tags/5425/, which you have advised weeks ago.

[pingluo@ada6 ]$ svn info
Path: .
URL: https://svn.oss.deltares.nl/repos/delft3d/tags/5425
Repository Root: https://svn.oss.deltares.nl/repos/delft3d
Repository UUID: 06f92af0-93f6-47d3-9398-6fc6b9b28e5e
Revision: 5520
Node Kind: directory
Schedule: normal
Last Changed Author: mourits
Last Changed Rev: 5436
Last Changed Date: 2015-09-24 05:00:19 -0500 (Thu, 24 Sep 2015)


But I will download the software from the link and install it again to see if I am able to run the model.

Thanks,

Ping
PL
Ping Luo, modified 4 Years ago.

RE: Linux flow-wave coupling run terminated with segmentation fault

Youngling Posts: 8 Join Date: 10/22/15 Recent Posts
Adri,

I made a fresh installations of Delft3D downloaded from https://svn.oss.deltares.nl/repos/delft3d/tags/5425/. Still flow crashed after 66.9% completion, leaving wave waiting idly.

You mentioned the model works fine. Did you run it to completion? What's the OS of your test bed? What compiler and version did you use to build Delft3D. My OS is CentOS 6.6. The compiler I used is Intel 15.0.3, with Intel MPI library 5.0 update 3 for Linux. The command I used to build Delft3D is

NETCDF_FORTRAN_CFLAGS="-I/software/tamusc/netCDF-Fortran/4.4.2-intel-2015B/include" \
NETCDF_FORTRAN_LIBS="-lnetcdff" \
NETCDF_CFLAGS="-I/software/tamusc/netCDF/4.3.3.1-intel-2015B/include" \
NETCDF_LIBS="-lnetcdf" \
MPIF77=mpiifort MPIFC=mpiifort MPICXX=mpiicpc CC=icc CFLAGS='-O3 -fPIC' CXX=icpc CXXFLAGS='-O3 -fPIC' FC=ifort FFLAGS='-O3 -fPIC' \
./configure --prefix=/software/tamusc/Delft3D6.02.01.5425 LDFLAGS="-L/software/tamusc/netCDF-Fortran/4.4.2-intel-2015B/lib -L/software/tamusc/netCDF/4.3.3.1-intel-2015B/lib"


Best Regards,

Ping
Adri Mourits, modified 4 Years ago.

RE: Linux flow-wave coupling run terminated with segmentation fault

Yoda Posts: 1224 Join Date: 1/3/11 Recent Posts
Hi Ping,

I expect that the related meteo input file is causing the problems. So I ran your model on Windows, without waves and with the flag "dryrun" added. That run finished normally.

I'm sorry, but I don't have time to dive into this problem. The best thing you can do is to simplify the model to pinpoint what is causing the problems.

If you really run into heavy problems, you have the option to buy a service package to get support.

Regards,

Adri