Over the past several Delft-FEWS releases, Deltares (DNL) and Deltares USA (DUSA) have engaged in a cyclic, event based Beta testing process with our North American clients.  We will conduct a week of testing focused on specific Test Themes.  At the conclusion of the week, we will review the test results with the test team.  The following week the development team will address the issues for re-delivery to the test team the following Monday. The process is described below.

The Event Testing Pattern

Testing Event Week

Monday AM

Software is available on FTP and SFTP sites

Kickoff Meeting with users (DUSA and Testers)

Issues reported via FogBugz (or equivalent bug tracking system)

Thursday PM

Wrap up meeting review issues (DUSA and Testers)

Data Stores posted by Testers

Friday

DUSA review and reproduction of issues

 

Development Week

Monday AM

Development check-in meeting (DUSA and DNL)

Issues will have been moved to Jira

Thursday PM

Coding and development testing complete

Software available to DUSA

Friday

Development wrap up check in (DUSA and DNL)

DUSA validation of corrections

 

Test Themes

Theme 1 SA Testing:

Imports validated

Pre-processing validated

Exports validated

Models run: state updating, forecasts, ensembles

Workflow Navigator

TimeSeries Display functions (zoom, text …) validated

Plot Thumbnails

Data Editing validated

Spatial Display functions (zoom, thumbnails …) validated

Docking/undocking/layout saving validated

Manual Forecast Display

Edit run options

Forecaster Help display

Archive export and import

Forecast Management display

Thresholds

User Groups and Permissions

Database Lister/Viewer

Explorer Map

 

Theme 2 SA Testing:

Validation of fixes.

Test ALL modifiers

Status icon updating based on model runs and modifier interactions

Misc (Water Coach, PC Raster, Forecast Mixer, Calibration)

Data QA/QC (validations)

Rating display

Correlation display

Statistics on the fly

Snow Updating

Search and Select Forecasts

Tabular ConfigFiles Display

Schematic Status Displays

F-12 Key functions

Theme 3 Live System:

Validation of fixes.

Transition to Live System (1 week install, then testing)

System performance (speed)

Data Synchronization OC/FSS

Scheduled jobs run as expected

System monitor reports as expected

Forecast Manager reports as expected

Admin Interface functions

Config Manager

PI-Service imports, exports, archive access

Event-Action triggered workflows

 

Theme 4 Live System:

 Open and Continuous testing

Initially we delayed the start of live system testing believing we could capture most of the beta issues within standalone testing.  However, recent experience indicated some of the more challenging issues only manifested themselves in a live system environment.  Thus, we consolidated some standalone Test Themes and scheduled live system testing earlier.

We have recently held the North American Beta testing in the fall, commencing in late August and concluding in early November.  It typically spans about 12 weeks.  Dates can vary slightly from year to year. 

Prior to the start of formal Beta testing, DNL and DUSA go through a period of Alpha testing in the spring and early summer.  We have set up Gates to ensure the Delft-FEWS software is ready for client-based testing.  During coordination meetings between DNL and DUSA, we determine if testing actions have successfully passed the Gate measures. 

Typical Schedule

April - June

Gate 0: Planning Review (Purpose – Validate planning is complete and comprehensive)

Gate 1: Alpha Test Readiness Review (Purpose – Validate the process is ready to enter DUSA Alpha testing)

Gate 2: Alpha Testing Check (Purpose – Verify DUSA Alpha testing is complete; standalone and live system)

Summer Holidays (limited capacity for collaborative testing with clients)

Mid-August

Gate 3: Beta Readiness Review (Purpose – Demonstrate to users the system is ready to enter client-based Beta testing)

Late-August through early November

Week 1 –Beta Testing Theme 1 – SA Testing

Week 2 – Development Week 1

Week 3 – Beta Testing Theme 2 – SA Testing + Live System Changes presentation

Week 4 – Development Week 2 + Live System Installs

Week 5 – Beta Testing Theme 3 – Live System Testing

Week 6 – Development Week 3

Weeks 7-11 – Beta Testing Theme 4 – Open and continuous testing

A period of testing and fixing with software updates as needed

Week 12 – End of open testing

                        Individual acceptance to close testing

                       Full Release available