Coast / Estuary

Coastal systems are among the most dynamic physical systems on earth and are subject to a large variety of forces. The morphodynamic changes occurring to coastlines worldwide are of great interest and importance. These changes occur as a result of the erosion of sediments, its subsequent transport as bed load or suspended load, and eventual deposition. 
 
Estuaries are partly enclosed water bodies that have an open connection to the coast. Estuaries generally have one or more branching channels, intertidal mudflats and/or salt marshes. Intertidal areas are of high ecological importance and trap sediments (sands, silts, clays and organic matter).
Within the Delft3D modelling package a large variation of coastal and estuarine physical and chemical processes can be simulated. These include waves, tidal propagation, wind- or wave-induced water level setup, flow induced by salinity or temperature gradients, sand and mud transport, water quality and changing bathymetry (morphology). Delft3D can also be used operationally e.g. storm, surge and algal bloom forecasting. 
 
On this discussion page you can post questions, research discussions or just share your experience about modelling coastal and/or estuarine systems with Delft3D FM. 
 

** PLEASE TAG YOUR POST! **

 

 

Sub groups
D-Flow Flexible Mesh
DELWAQ
Cohesive sediments & muddy systems

 

 

Back

RE: FLOW - WAQ integration consuming memory

JC
José Rafael Cavalcanti, modified 11 Days ago.

FLOW - WAQ integration consuming memory

Youngling Posts: 16 Join Date: 10/21/18 Recent Posts

Hello all,

 

I am trying to export a WAQ input file from my FLOW simulations but the process is consuming too much space in my disk. If I save a comm file for further conversion it only takes 1.7 Gb. When I activate the on-line conversion to WAQ my disk memory is almost completely filled with temporary huge files, in addition to the ones I actually need for WAQ. Is it normal?

 

Cheers,

AM
Arjen Markus, modified 10 Days ago.

RE: FLOW - WAQ integration consuming memory

Jedi Knight Posts: 235 Join Date: 1/26/11 Recent Posts

This is rather unusual - I am assuming you are referring to Delft3D-FLOW (not D-Flow FM, its successor). I just tried it myself on an example, but I see no large temporary files appear - only the ones that are to be expected.

Can you tell us the names of these files?

JC
José Rafael Cavalcanti, modified 9 Days ago.

RE: FLOW - WAQ integration consuming memory

Youngling Posts: 16 Join Date: 10/21/18 Recent Posts

Hi,

Yes, I meant the old version. I am attaching a print of my WAQ export menu and one of the file folder. Do you know what I might be doing wrong? 

AM
Arjen Markus, modified 9 Days ago.

RE: FLOW - WAQ integration consuming memory

Jedi Knight Posts: 235 Join Date: 1/26/11 Recent Posts

From the information I have available (the screenshots with the file sizes and the screenshot with the output parameters for WAQ input) I can make a rough estimation:

- You have some 5250 segments per layer and 10 layers (estimate from the size of the LGA file - very rough - and the screenshot).

- Total number of segments: 52500; number of exchanges: ~ 3*number of segments (three-dimensional model) = 160,000

- Per time that output for WAQ is written, that means: 4 * number of segments for volume, shear stress, ... + 4 * number of exchanges for flow and area (4 bytes per value). Roughly 600,000 values of 4 bytes each => 2.4 MB

- You have a lengthy calculation: 2.5 years. It amounts to 4 times per day, roughly 910 days. So: 3640 times.

If my estimates are reasonably accurate, you should have a total of 9 GB of data. About 8 times less than you actually have. Hm. I would have to know the actual sizes of your model to make a better estimate.

 

The one thing that surprises me is the length of the hydrodynamic calculation: unless the hydrodynamics changes a "lot" over that period of time, you could probably select a much shorter interval and have WAQ repeat the hydrodynamics. Could you tell us a bit more about the system you are modelling?

 

 

JC
José Rafael Cavalcanti, modified 8 Days ago.

RE: FLOW - WAQ integration consuming memory

Youngling Posts: 16 Join Date: 10/21/18 Recent Posts

Hi,

I am modelling a reservoir during a drought period. I have changes in the water level that can reach up to 8m bellow the spillway and then return to it. Unfortunely, the entire process of the reservoir wet- and dry-ing takes over during the entire period (2.5 years).

I can perform the hydrodynamic simulation and I have nice temperature profiles (I am using a z-layer grid to better represent stratification). However, the WAQ simulations are giving me a hard time even using only the continuity variable. I used the off-line coupling between flow and waq to get the .hyd file, so I am trying to export the WAQ file on-line during the hydrodynamic simulation in order to verify if the problem relies on the coupling.

I can feed you with more info if you need. Thanks for the reply!

Cheers

 

 

 

AM
Arjen Markus, modified 8 Days ago.

RE: FLOW - WAQ integration consuming memory

Jedi Knight Posts: 235 Join Date: 1/26/11 Recent Posts

If you could send more information indeed, that would be helpful. The discrepancy between the actual file sizes and my estimates is puzzling. The hyd-file should be enough to refine these estimates.

In the screenshot I saw that you have a timestep of 6 hours for the hydrodynamic results going to WAQ. Is that the maximum timestep (based on the variation in the hydrodynamics)? Or could a timestep of 24 hours suffice?

The online coupling avoids the large communication file you would otherwise get - this file should then be read by the coupling program that can convert the information into a form suitable for WAQ. The online coupling performs this conversion directly, so that only the converted information is stored.

JC
José Rafael Cavalcanti, modified 8 Days ago.

RE: FLOW - WAQ integration consuming memory

Youngling Posts: 16 Join Date: 10/21/18 Recent Posts

Attached you may find the .hyd file. Since I am having problems with the water quality simulations I was chaging the time step in the com-file just to remove it from my list of potential errors. The first time I did the off-line coupling using a 24hrs timestep and now I was trying the built-in coupling... and here we are! :]

By the way, would you mind to clarify how WAQ understands the flow output? I mean, we are saving an output using a rather large timestep but WAQ can (and should) work with a much smaller timestep (please, correct me if I am wrong). Is it just interpolating the results?

 

AM
Arjen Markus, modified 7 Days ago.

RE: FLOW - WAQ integration consuming memory

Jedi Knight Posts: 235 Join Date: 1/26/11 Recent Posts

Thanks for that - I will have a closer look rightaway.

 

As for WAQ dealing with the hydrodynamics: yes, it interpolates the volumes and keeps the flows and areas constant between timesteps. This way you get a consistent (volume-conserving) flow field at arbitrarily small timesteps. Of course that assumes that the flows as calculated by the hydrodynamic model do not change too much over the output timestep.

AM
Arjen Markus, modified 7 Days ago.

RE: FLOW - WAQ integration consuming memory

Jedi Knight Posts: 235 Join Date: 1/26/11 Recent Posts

Hm, the hyd-file gives me 53210 grid cells in total, slightly more than my original estimate. Again I conclude in the order of 9 GB, not 70 GB. Unfortunately the screenshot shows "..." for the size of the largest files ,so it is unclear what the precise unit is.

One thing that does come to mind is, that the big size - which appears unrelated to the current calculation - might be due to a peculiarity of the way these files are written. Could you rerun the calculation in a clean directory? Or alternatively (since the disk space is a clear problem) first throw away the output files and then run the calculation again? (If my hunch is correct, then the output files of this type are not written anew, but rather overwritten, and that would leave the size to be same as any existing or larger if more data are written to them. If this is confusing, just concentrate on the first part - it is a rather technical matter)

AM
Arjen Markus, modified 7 Days ago.

RE: FLOW - WAQ integration consuming memory

Jedi Knight Posts: 235 Join Date: 1/26/11 Recent Posts

No, my hunch is incorrect - I did a small experiment with version 6.03.00.62434. That leads to my next question: what version of Delft3D-FLOW are you using?

JC
José Rafael Cavalcanti, modified 6 Days ago.

RE: FLOW - WAQ integration consuming memory

Youngling Posts: 16 Join Date: 10/21/18 Recent Posts

I am attaching the version of all executables to be more efficient.

Cleaning the disk before using it did not improve my situation.

AM
Arjen Markus, modified 6 Days ago.

RE: FLOW - WAQ integration consuming memory

Jedi Knight Posts: 235 Join Date: 1/26/11 Recent Posts

Hm, this is all recent enough, so that does not give us any clue as to what is going on. Would it be possible for you send me the input for Delft3D-FLOW? Then I can examine the problem on my own system.

JC
José Rafael Cavalcanti, modified 6 Days ago.

RE: FLOW - WAQ integration consuming memory

Youngling Posts: 16 Join Date: 10/21/18 Recent Posts

Yes, sure!

AM
Arjen Markus, modified 3 Days ago.

RE: FLOW - WAQ integration consuming memory

Jedi Knight Posts: 235 Join Date: 1/26/11 Recent Posts

Thanks - I am running the calculation on my machine right now, let's see what the result is.

AM
Arjen Markus, modified 3 Days ago.

RE: FLOW - WAQ integration consuming memory

Jedi Knight Posts: 235 Join Date: 1/26/11 Recent Posts

The calculation is 2/3 completed, the total size of the data files is 2.5 GB. I noticed when setting up the calculation, that the item "Export WAQ input" was off, so I turned it on and then used the button "Edit WAQ input" to set the output interval to 360 minutes. I think that is where you made a mistake. Easy to do by the way.

I should explain this a bit:

- The old method was that we let Delft3D-FLOW write a communication file that could be used by different modules, such as the morphology and the water quality modules. For WAQ you needed to run the coupling program in a step after Delft3D-FLOW. As most of these modules were incorporated into Delft3D-FLOW because they need to feed the results back into Delft3D-FLOW, the only practical "client" for the communication file was WAQ.

- The new method is that FLOW directly writes the information for WAQ. And - here is the important bit - this procedure uses input data that are independent from the input data for the communication file method.

Could you check that the output for WAQ is indeed set to 360 min? It is the screen that pops up after clicking "Edit WAQ input"

JC
José Rafael Cavalcanti, modified 3 Days ago.

RE: FLOW - WAQ integration consuming memory

Youngling Posts: 16 Join Date: 10/21/18 Recent Posts

Hi Arjen,

I was testing the water quality simulation with the off-line coupling instead of the built-in option, so that's why the option was not flagged. I am in fact setting the Export WAQ input option. However, you just gave me an important piece of information. So I don't actually need to save the com- files if I am exporting the results via the built-in option?

I am having problems with my water quality simulations. If I set the boundaries and initial conditions and let WAQ run a simulation, it all goes fine. The problem is when I try to set my water temperature as segment function. In this case, I have a NaN situation after some simulation time. I was trying to check the time step in the com- files, that's why I have a 360 min time step intead of a daily interval (which fits my system). The problem remained...

So my next test regarded the conversion between Z-layer and sigma-coordinates grid. Since I am using a Z-layer grid in the hydrodynamic part and the coupling procedure is not straightforward in this case, I was hoping to solve it by performing the built-in coupling.

I will try it again with the built-in option and I let you know.

AM
Arjen Markus, modified 2 Days ago.

RE: FLOW - WAQ integration consuming memory

Jedi Knight Posts: 235 Join Date: 1/26/11 Recent Posts

The builtin coupling is indeed independent of the offline coupling. So you only need one of the two sets of files.

Hm, these NaNs result in the water quality parameters? One thing you could do is use a constant temperature instead of the temperature from the hydrodynamics. That would exclude one possible source of NaNs. Another option is to see what just continuity is doing. The reason I suggest this, is that it may be that segments are becoming "dry" because of a lowering water level and if this happens too quickly, you can get numerical instabilities.

What integration option are you using?

 

JC
José Rafael Cavalcanti, modified 2 Days ago.

RE: FLOW - WAQ integration consuming memory

Youngling Posts: 16 Join Date: 10/21/18 Recent Posts

Yes, the NaN is resulting from the water quality simulation. I can't identify precisely where is the problem though.

Using just continuity I had problems but it was due to the time step, so I am currently using a small one (30s). Now I have continuity working fine, with some variations occurring precisely at the moments where my wet- and dry-ing of computational cells takes place. My first guess was the water level variation occurring quickly and my coupling time step from flow being unable to maintain physical consistency. I am running a new simulation with the 6hrs time step which we just discussed to see what happens.

I 've made a simulation with water temperature constant (15ºC) and it works. The integration options are showed in the figure.

 

Regarding the waq input file... Working with both the coupling and built-in option I noticed some small diferences. The coupling procedure creates a .hyd file with curvilinear grid whereas the built-in option creates a .unstructured.hyd file with unstructured z-layer grid. Since I am adding the z-layer text to my .hyd file before start the conversion in waq, should I change the entire text to the same text written by the built-in option?