Forum

Imported data not remaining in database ["IMPORT"]

MM
Mason Marchildon, modified 2 Months ago.

Imported data not remaining in database ["IMPORT"]

Rookie Crystal gazer Posts: 11 Join Date: 5/1/17 Recent Posts
As this is my first time submitting to the forum, please let me explain my needs of Delft-FEWS.I'm new to FEWS (v2019.02) to which I'm hoping to adopt as an interface for generating numerical model forcing inputs.  I am NOT using FEWS for forecasting.I've been working on a set of workflows, to (i) import (20 years worth) hourly data (from 40 stations) --> (ii) aggregate the data to 6-hourly time steps --> (iii) interpolate to a set (~2800) of 10km² subwatersheds  --> (iv) export to NetCDF.I have been successful with the first 2 workflows, and have successfully tested (iii) for 10 days of data.  Subsequent to the successful test of (iii), I've gone ahead to interpolate the remaining 20 years.  Once complete, I see it reported in the Database Lister, can view the data in the Data Display, etc. From my perspective the data are in the database.
The problem I'm having is that upon shutting down FEWS and restarting, I am left with the original 10 days of data I used for testing. No sign remains of the rest of the 20 years.  So far, (that I can recall) I've looked into: - the status of the rolling barrel
 - extended my global expiry time to 10,000 days
 - experimented with readWriteMode to both "read only" and "add originals"
 - increased ram allocation
 - run the processed using the Manual Forcast display chunk by chunk with debug output
 - ensured workflow autoApprove="true"
 - banged my head on the table, etc.
 
Any help would be much appreciated. Thanks
IM
Ivo Miltenburg, modified 2 Months ago.

RE: Imported data not remaining in database ["IMPORT"]

Keen Forecaster Posts: 7 Join Date: 3/12/13 Recent Posts
Hi Mason,

when you mention that data from a run is only present in the localDataStore until a restart, I am reminded of the Delft-FEWS feature where you can run workflows as an "IFD run" (local) vs. "Server run" server side. When you are running a stand alone (SA) Delft-FEWS (you do not specify this), this seems not relevant, as there is not "server side", but the concept still is in place. This means that also in a SA setup, if you run a workflow as an IFD, or local run, it will not remain in the database. If you run it as a "server run" it will remain in the database.

Note that the information below plays a role when activating a workflow from the Topology (or "Interactive Forecast Display"). 
Below information is coptied from the Topology section of the Delft-FEWS wiki

IFD runBy default all the workflows started from a leaf node are started in IFD mode. The results of an IFD-run are only temporary available in FEWS (until the next restart) and are only available and visible at the operator client which started the run.IFD runs are typically used to evaluate the effect of modifiers on the results of a forecast. 

Server runA workflow which is started from a group node will by default be started as a server run. A server run will run at a FSS when it is started from an operator client. When it is started from a stand alone it will run locally but the results of the run will be available after a restart of the system.----Now, critically, it is possible for workflows defined in the Topology to actually specify if you want a "local run" or a "server run", by setting the localrun element in Tpology.xml. What I would suggest is setting up a Topology.xml and use a
[b]<localRun>false</localRun>[/b]. 

Again from the wiki:

---

It is possible to change these default settings by using the element localRun.In the example below the runs from leaf node B will run in server mode and the runs started from the group node C will run in IFD mode.
 
<?xml version="1.0" encoding="UTF-8"?>
<topology xmlns="[url=http://www.wldelft.nl/fews]http://www.wldelft.nl/fews[/url]" xmlns:xsi="[url=http://www.w3.org/2001/XMLSchema-instance]http://www.w3.org/2001/XMLSchema-instance[/url]" xsi:schemaLocation="[url=http://www.wldelft.nl/fews]http://www.wldelft.nl/fews[/url] [url=http://fews.wldelft.nl/schemas/version1.0/topology.xsd]http://fews.wldelft.nl/schemas/version1.0/topology.xsd[/url]">
    <nodes id="C" name="C">
        <workflowId>workflowC</workflowId>
        <localRun>true</localRun>
        <node id="A" name="A">
            <workflowId>workflowA</workflowId>
            <localRun>false</localRun>
        </node>
        <node id="B" name="B"/>
    </nodes>nfo
</topology>

---
I hope this helps you finding the issue...and potenitally resolving it.
MM
Mason Marchildon, modified 2 Months ago.

RE: Imported data not remaining in database ["IMPORT"]

Rookie Crystal gazer Posts: 11 Join Date: 5/1/17 Recent Posts
Thank you Ivo, that was the problem; was wondering what the difference between those two run options were.

Good to know that feature is there for testing prior to inserting into the database.

Mason