Forum

RE: ["Export"] Cannot export GFS forecast data to ascii grid

Dmitry Kushnir, modified 14 Days ago.

["Export"] Cannot export GFS forecast data to ascii grid

Keen Forecaster Posts: 4 Join Date: 11/9/12 Recent Posts
Hi all, I've been trying to set up an export module in order to export downloaded GFS forecast data on a rectilinear grid. Unfortunately, I constantly get asc-files filled with NoData values.Here's my export module config file:

And here's log message for the workflow:
[2020-06-23 11:58:12,781] INFO - ExplorerRollingBarrel.runRollingBarrel - Rolling barrel started
[2020-06-23 11:58:12,968] INFO - ExplorerRollingBarrel.runRollingBarrel - Rolling barrel finished in 0s
[2020-06-23 12:21:59,933] INFO - ManualForecastDialog.dispatchTasks - ManualForecast.Started: Manual forecast <23-06-2020 00:00:00 Export_GFS_BS> has been sent to the task scheduler
[2020-06-23 12:22:00,017] INFO - TaskRun.logTaskStartedMessage - TaskRun.Started: Starting Task Export_GFS_BS with ID SA11_31, T0 2020-06-23 00:00:00, default forecast length, available memory 845 MB, config revision unknown[2020-06-23 12:22:00,019] INFO - Workflow.run - Workflow.ActivityStarted: Workflow Export_GFS_BS
[2020-06-23 12:22:00,020] INFO - WorkflowPluginActivity.run - Started Activity Export_GFS_BS
[2020-06-23 12:22:00,020] DEBUG - WorkflowPluginActivity.run - Memory usage 66 MB
[2020-06-23 12:22:00,020] INFO - TimeSeriesExport.run - Started timeseries export Export_GFS_BS
[2020-06-23 12:22:00,021] DEBUG - TimeSeriesExport.exportData - Started exportType asciigrid
[2020-06-23 12:22:00,023] DEBUG - OneToOneIdMap.toExternal - internal id parameter:T.forecast qualifiers:GFS location:GFS_Black_Sea ensemble:main ensembleMemberId:0 was mapped to external id parameter:T.forecast qualifiers:GFS location:GFS_Black_Sea ensemble:main ensembleMember:0 by using one to one mapping
[2020-06-23 12:22:00,024] DEBUG - TimeSeriesView.privateRead - read time series for ;loc=GFS Forecast (GFS_Black_Sea);par=GFS forecasted temperature (T.forecast);qualifierSet=GFS;mod=Import_BS_Forecast;external forecasting;never
[2020-06-23 12:22:00,024] DEBUG - TimeSeriesView.privateRead - Trying to read time series data for a not yet used key combination:-2:GFS_Black_Sea:T.forecast:GFS:SETS180:external forecasting:grid
[2020-06-23 12:22:00,025] DEBUG - FileUtils.newOutputStream - Open file for write C:\Users\Server1\ODYSSEA\FEWS_Basic\Export\tmp202006230922GFS_BS.ASC
[2020-06-23 12:22:00,027] DEBUG - FileUtils.newOutputStream - Start writing C:\Users\Server1\ODYSSEA\FEWS_Basic\Export\tmp202006230922GFS_BS.ASC
[2020-06-23 12:22:00,033] DEBUG - FileUtils$8.close - File written in 0.0s IO 0.0s 7.5 kB  1.4 MB/s C:\Users\Server1\ODYSSEA\FEWS_Basic\Export\tmp202006230922GFS_BS.ASC
[2020-06-23 12:22:00,080] DEBUG - TimeSeriesExport.exportData - Finished exporting file: C:\Users\Server1\ODYSSEA\FEWS_Basic\Export\202006230922GFS_BS.ASC
[2020-06-23 12:22:00,080] INFO - TimeSeriesExport.run - Finished timeSeries export Export_GFS_BS
[2020-06-23 12:22:00,080] INFO - WorkflowPluginActivity.run - Completed Activity 'Export_GFS_BS' completed in 0s
[2020-06-23 12:22:00,080] INFO - Workflow.run - Workflow.ActivityCompleted: Workflow Export_GFS_BS completed in 0s
[2020-06-23 12:22:00,185] INFO - TaskRun.logTaskFinishedMessage - TaskRun.Completed: Task  Export_GFS_BS with ID SA11_31 completed in 0s Start time: 2020-06-23 09:22:00 End time: 2020-06-23 09:22:00 User Id: Server1
TimeSeriesExport 0s 34%  datastore 0s 0%  cache files 0 B   database 0s 1% (2 ms/query, 0 B,  0 B/s, 1 queries, 0 rows)  reloaded 0 B  time series read 1 (unique=0, view period=0s)  time series written 0 (modified=0, write period=0s, modified period=0s)  files 0s 33%  7.7 kB 133 kB/s logging 0s 2%  gc 0s 0%  cpu 71%  max mem. 66 MB  index mem. 35 kB  db con. acquire time 0s 1%  lock. acquire time 0s 0%  
[2020-06-23 12:22:00,185] INFO - TaskRun.run -                 ******************      Workflow Export_GFS_BS Completed   ******************
Any suggestions are highly appreciated.
Regards,
Dmitry 
Dmitry Kushnir, modified 13 Days ago.

RE: ["Export"] Cannot export GFS forecast data to ascii grid

Keen Forecaster Posts: 4 Join Date: 11/9/12 Recent Posts
I suppose the problem is somehow related to this line in the debug log file:
[2020-06-23 12:22:00,024] DEBUG - TimeSeriesView.privateRead - Trying to read time series data for a not yet used key combination:-2:GFS_Black_Sea:T.forecast:GFS:SETS180:external forecasting:grid
Though I've no idea, how to solve it.Cheers, Dmitry
DC
Dave Casson, modified 13 Days ago.

RE: ["Export"] Cannot export GFS forecast data to ascii grid

Keen Forecaster Posts: 4 Join Date: 6/19/19 Recent Posts
Hi Dmitry, this looks to me that the issue is in the IdMapping. Can you check that the IdMapping matches both the internal parameter values and what is defined for the export module?
Dmitry Kushnir, modified 13 Days ago.

RE: ["Export"] Cannot export GFS forecast data to ascii grid

Keen Forecaster Posts: 4 Join Date: 11/9/12 Recent Posts
Hi Dave, Thanks for your reply. But I couldn't find references to the IdMapFile in  the schema for ASCII Grid Export module (https://publicwiki.deltares.nl/display/FEWSDOC/ASCII+Grid+Export ). To investigate whether this is the IdMapping issue, I have changed the export module to the NETCDF-CF_GRID exporter, which uses parameter mapping. Unfortunately, using the NetCDF-exporter, I have had the same issue with the missing data  as in the case of ASCII exporter. This is the part of the log file:

[2020-06-24 14:15:53,772] DEBUG - CompactIdMap.toExternal - internal id parameter:T.forecast qualifiers:none location:GFS_Black_Sea ensemble:main ensembleMemberId:0 was mapped to external id parameter:air_temperature qualifiers:temperature (not a standard name, just as example) location:GFS_Black_Sea ensemble:main ensembleMember:0 by using compact id mapping
[2020-06-24 14:15:53,773] DEBUG - TimeSeriesView.privateRead - read time series for ;loc=GFS Forecast (GFS_Black_Sea);par=GFS forecasted temperature (T.forecast);mod=Export_GFS_BS;temporary external forecasting;never
[2020-06-24 14:15:53,775] WARN - NetcdfGridTimeSeriesSerializer.write - Only missing data found in time series to export to netcdf, this time series will be skipped. Parameter: air_temperature, location: GFS_Black_Sea
[2020-06-24 14:15:53,775] ERROR - NetcdfGridTimeSeriesSerializer.write - Only missing data found to export to netcdf, export will be aborted for file: C:\Users\Server1\ODYSSEA\FEWS_Basic\Export\tmp202006241115.nc
[2020-06-24 14:15:53,776] DEBUG - TimeSeriesExport.exportData - Finished exporting file: C:\Users\Server1\ODYSSEA\FEWS_Basic\Export\tmp202006241115.nc
[2020-06-24 14:15:53,776] INFO - TimeSeriesExport.run - Finished timeSeries export Export_GFS_BS
******************      Workflow Export_GFS_BS Completed with ERRORS   ******************
And here's the screen-grab for the database lister. As it can be seen, the database does actually contain the parameters I wanted to be exported.  I can export them to asc-files only by right-clicking on the parameter grid in the Spatial Data viewer. Still no luck with setting up the export module.

Regards,Dmitry
IM
Ivo Miltenburg, modified 12 Days ago.

RE: ["Export"] Cannot export GFS forecast data to ascii grid (Answer)

Keen Forecaster Posts: 1 Join Date: 3/12/13 Recent Posts
Hi Dmitry,my 2cts: I read the "Trying to read time series data for a not yet used key combination:-2:GFS_Black_Sea:T.forecast:GFS:SETS180:external forecasting:grid" message as: FEWS is not able to find the timeseries (as you defined it in the export module) and thus it will not export. 
Maybe you can verify through the workflow navigator -> right-click -> show grids: you expect the grid to show up.
You can also right-click in the timeseries/database lister on Copy TimeSeriesSet xml to Clipboard and verify that there are no differences.
Dmitry Kushnir, modified 12 Days ago.

RE: ["Export"] Cannot export GFS forecast data to ascii grid

Keen Forecaster Posts: 4 Join Date: 11/9/12 Recent Posts
Hi Ivo, Thank you for all your assistance. The latter suggestion helped to solve the problem.  
The main issue,  as I understand, was in the <readWriteMode> in the <timeSeriesSet> section of the export module. It should've been set to "read complete forecast" instead of "read only". After changing that string, the export module worked smoothly, both for ASCII-files and NetCDF-files.Thanks again. I really appreciate your help.  Regards, Dmitry