2020-07-03 04:25:45,915 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/nowcast.log
2020-07-03 04:25:45,923 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/nowcast.debug.log
2020-07-03 04:25:45,929 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/checklist.log
2020-07-03 04:25:45,929 INFO [rotate_logs] log files rotated
2020-07-03 04:25:45,936 INFO [manager] checklist updated with [log rotation] items from rotate_logs worker
2020-07-03 09:02:59,346 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-west/12/ to /results/forcing/atmospheric/GEM2.5/GRIB/20200703/12/
2020-07-03 09:02:59,348 INFO [collect_weather] 2020-07-03 2.5km weather forecast 12 collection complete
2020-07-03 09:02:59,351 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2020-07-03 09:02:59,625 INFO [manager] launching NextWorker(module='nowcast.workers.get_NeahBay_ssh', args=['nowcast'], host='localhost')
2020-07-03 09:02:59,631 INFO [manager] launching NextWorker(module='nowcast.workers.grib_to_netcdf', args=['nowcast+'], host='localhost')
2020-07-03 09:02:59,635 INFO [manager] launching NextWorker(module='nowcast.workers.download_live_ocean', args=[], host='localhost')
2020-07-03 09:02:59,639 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['18', '2.5km'], host='localhost')
2020-07-03 09:03:00,842 INFO [collect_weather] running in process 1092
2020-07-03 09:03:00,842 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 09:03:00,843 INFO [collect_weather] publishing log messages to tcp://*:5562
2020-07-03 09:03:00,844 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2020-07-03 09:03:00,902 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-west/18/
2020-07-03 09:03:00,911 INFO [grib_to_netcdf] running in process 1090
2020-07-03 09:03:00,912 INFO [grib_to_netcdf] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 09:03:00,912 INFO [grib_to_netcdf] publishing log messages to tcp://*:5563
2020-07-03 09:03:00,913 INFO [grib_to_netcdf] connected to skookum.eos.ubc.ca port 5555
2020-07-03 09:03:00,984 INFO [download_live_ocean] running in process 1091
2020-07-03 09:03:00,985 INFO [download_live_ocean] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 09:03:00,985 INFO [download_live_ocean] publishing log messages to tcp://*:5564
2020-07-03 09:03:00,986 INFO [download_live_ocean] connected to skookum.eos.ubc.ca port 5555
2020-07-03 09:03:00,987 INFO [download_live_ocean] downloading Salish Sea western boundary daily averaged Live Ocean file for 2020-07-03
2020-07-03 09:03:01,770 INFO [get_NeahBay_ssh] running in process 1089
2020-07-03 09:03:01,771 INFO [get_NeahBay_ssh] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 09:03:01,772 INFO [get_NeahBay_ssh] publishing log messages to tcp://*:5567
2020-07-03 09:03:01,791 INFO [get_NeahBay_ssh] connected to skookum.eos.ubc.ca port 5555
2020-07-03 09:03:03,896 INFO [download_live_ocean] Deflating in up to 6 concurrent sub-processes
2020-07-03 09:03:04,908 INFO [download_live_ocean] netCDF4 deflated /results/forcing/LiveOcean/downloaded/20200703/low_passed_UBC.nc
2020-07-03 09:03:04,909 INFO [download_live_ocean] 2020-07-03 Live Ocean file for Salish Sea western boundary downloaded
2020-07-03 09:03:04,913 INFO [manager] checklist updated with [Live Ocean products] items from download_live_ocean worker
2020-07-03 09:03:05,145 INFO [manager] launching NextWorker(module='nowcast.workers.make_live_ocean_files', args=['--run-date', '2020-07-03'], host='localhost')
2020-07-03 09:03:05,748 INFO [get_NeahBay_ssh] Neah Bay sea surface height web scraping and open boundary file creation for nowcast complete
2020-07-03 09:03:05,753 INFO [manager] checklist updated with [Neah Bay ssh] items from get_NeahBay_ssh worker
2020-07-03 09:03:07,566 INFO [make_live_ocean_files] running in process 1190
2020-07-03 09:03:07,567 INFO [make_live_ocean_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 09:03:07,568 INFO [make_live_ocean_files] publishing log messages to tcp://*:5564
2020-07-03 09:03:07,569 INFO [make_live_ocean_files] connected to skookum.eos.ubc.ca port 5555
2020-07-03 09:03:07,570 INFO [make_live_ocean_files] Creating T&S western boundary conditions file from 2020-07-03 Live Ocean run
2020-07-03 09:04:07,917 INFO [make_live_ocean_files] Stored T&S western boundary conditions file: /
2020-07-03 09:04:07,918 INFO [make_live_ocean_files] 2020-07-03 Live Ocean western boundary conditions files created
2020-07-03 09:04:07,920 INFO [manager] checklist updated with [Live Ocean boundary conditions] items from make_live_ocean_files worker
2020-07-03 09:04:43,592 INFO [grib_to_netcdf] 2020-07-03 NEMO-atmos forcing file for nowcast+ created
2020-07-03 09:04:43,595 INFO [manager] checklist updated with [weather forcing] items from grib_to_netcdf worker
2020-07-03 09:04:43,601 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'nowcast+'], host='localhost')
2020-07-03 09:04:43,607 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'nowcast+'], host='localhost')
2020-07-03 09:04:43,643 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['graham-hindcast', 'nowcast+'], host='localhost')
2020-07-03 09:04:43,646 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'nowcast+'], host='localhost')
2020-07-03 09:04:43,648 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['weather'], host='localhost')
2020-07-03 09:04:44,693 INFO [ping_erddap] running in process 2561
2020-07-03 09:04:44,694 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 09:04:44,695 INFO [ping_erddap] publishing log messages to tcp://*:5563
2020-07-03 09:04:44,696 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2020-07-03 09:04:44,697 INFO [ping_erddap] weather ERDDAP dataset flag file(s) created
2020-07-03 09:04:44,699 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2020-07-03 09:04:44,740 INFO [upload_forcing] running in process 2558
2020-07-03 09:04:44,741 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 09:04:44,742 INFO [upload_forcing] publishing log messages to tcp://*:5564
2020-07-03 09:04:44,743 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2020-07-03 09:04:44,744 INFO [upload_forcing] running in process 2557
2020-07-03 09:04:44,745 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 09:04:44,746 INFO [upload_forcing] publishing log messages to tcp://*:5567
2020-07-03 09:04:44,747 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2020-07-03 09:04:44,748 INFO [upload_forcing] running in process 2559
2020-07-03 09:04:44,749 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 09:04:44,749 INFO [upload_forcing] publishing log messages to tcp://*:5568
2020-07-03 09:04:44,750 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2020-07-03 09:04:44,774 INFO [upload_forcing] running in process 2560
2020-07-03 09:04:44,775 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 09:04:44,776 INFO [upload_forcing] publishing log messages to tcp://*:5569
2020-07-03 09:04:44,777 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2020-07-03 09:04:49,082 INFO [upload_forcing] nowcast+ 2020-07-03 forcing files upload to optimum-hindcast completed
2020-07-03 09:04:49,087 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2020-07-03 09:04:49,531 INFO [upload_forcing] nowcast+ 2020-07-03 forcing files upload to orcinus-nowcast-agrif completed
2020-07-03 09:04:49,536 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2020-07-03 09:04:49,953 INFO [upload_forcing] nowcast+ 2020-07-03 forcing files upload to arbutus.cloud-nowcast completed
2020-07-03 09:04:49,957 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2020-07-03 09:04:49,965 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'nowcast+', '--run-date', '2020-07-03'], host='localhost')
2020-07-03 09:04:50,764 INFO [make_forcing_links] running in process 2589
2020-07-03 09:04:50,765 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 09:04:50,766 INFO [make_forcing_links] publishing log messages to tcp://*:5563
2020-07-03 09:04:50,766 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2020-07-03 09:04:51,981 INFO [make_forcing_links] nowcast+ 2020-07-03 forcing file links on arbutus.cloud-nowcast created
2020-07-03 09:04:51,987 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2020-07-03 09:04:51,997 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2020-07-03'], host='arbutus.cloud-nowcast')
2020-07-03 09:04:54,179 INFO [run_NEMO] running in process 24348
2020-07-03 09:04:54,180 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 09:04:54,181 INFO [run_NEMO] publishing log messages to tcp://*:5556
2020-07-03 09:04:54,188 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2020-07-03 09:04:55,121 INFO [run_NEMO] nowcast: launching /nemoShare/MEOPAR/nowcast-sys/runs/03jul20nowcast_2020-07-03T090454.243920-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2020-07-03 09:04:55,138 INFO [run_NEMO] nowcast NEMO run for 2020-07-03 on arbutus.cloud-nowcast started
2020-07-03 09:04:55,147 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2020-07-03 09:04:55,164 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'nowcast'], host='arbutus.cloud-nowcast')
2020-07-03 09:04:55,395 INFO [upload_forcing] nowcast+ 2020-07-03 forcing files upload to graham-hindcast completed
2020-07-03 09:04:55,402 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2020-07-03 09:04:57,095 INFO [watch_NEMO] running in process 24613
2020-07-03 09:04:57,096 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 09:04:57,098 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2020-07-03 09:04:57,099 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2020-07-03 09:04:57,174 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2020-07-03 09:09:57,276 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 7495633 = 2020-07-03 04:48:00 UTC, 20.0% complete
2020-07-03 09:14:57,379 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 7496145 = 2020-07-03 10:29:20 UTC, 43.7% complete
2020-07-03 09:19:57,480 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 7496659 = 2020-07-03 16:12:00 UTC, 67.5% complete
2020-07-03 09:24:57,581 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 7497172 = 2020-07-03 21:54:00 UTC, 91.3% complete
2020-07-03 09:29:57,686 INFO [watch_NEMO] nowcast NEMO run on arbutus.cloud-nowcast completed
2020-07-03 09:29:57,691 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2020-07-03 09:29:57,967 INFO [manager] launching NextWorker(module='nowcast.workers.get_NeahBay_ssh', args=['forecast'], host='localhost')
2020-07-03 09:29:57,973 INFO [manager] launching NextWorker(module='nowcast.workers.make_fvcom_boundary', args=['arbutus.cloud-nowcast', 'x2', 'nowcast'], host='arbutus.cloud-nowcast')
2020-07-03 09:29:57,980 INFO [manager] launching NextWorker(module='nowcast.workers.make_fvcom_boundary', args=['arbutus.cloud-nowcast', 'r12', 'nowcast'], host='arbutus.cloud-nowcast')
2020-07-03 09:29:57,984 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2020-07-03'], host='localhost')
2020-07-03 09:29:59,026 INFO [download_results] running in process 3327
2020-07-03 09:29:59,027 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 09:29:59,028 INFO [download_results] publishing log messages to tcp://*:5563
2020-07-03 09:29:59,029 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2020-07-03 09:29:59,029 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/nowcast/03jul20 to /results/SalishSea/nowcast-blue.201905
2020-07-03 09:30:00,384 INFO [get_NeahBay_ssh] running in process 3324
2020-07-03 09:30:00,385 INFO [get_NeahBay_ssh] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 09:30:00,387 INFO [get_NeahBay_ssh] publishing log messages to tcp://*:5564
2020-07-03 09:30:00,388 INFO [get_NeahBay_ssh] connected to skookum.eos.ubc.ca port 5555
2020-07-03 09:30:04,220 INFO [make_fvcom_boundary] running in process 25026
2020-07-03 09:30:04,221 INFO [make_fvcom_boundary] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 09:30:04,222 INFO [make_fvcom_boundary] publishing log messages to tcp://*:5580
2020-07-03 09:30:04,224 INFO [make_fvcom_boundary] connected to skookum.eos.ubc.ca port 5555
2020-07-03 09:30:04,225 INFO [make_fvcom_boundary] Creating VHFR FVCOM open boundary file for r12 nowcast run from 2020-07-03 NEMO run
2020-07-03 09:30:04,232 INFO [make_fvcom_boundary] running in process 25027
2020-07-03 09:30:04,233 INFO [make_fvcom_boundary] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 09:30:04,234 INFO [make_fvcom_boundary] publishing log messages to tcp://*:5581
2020-07-03 09:30:04,235 INFO [make_fvcom_boundary] connected to skookum.eos.ubc.ca port 5555
2020-07-03 09:30:04,236 INFO [make_fvcom_boundary] Creating VHFR FVCOM open boundary file for x2 nowcast run from 2020-07-03 NEMO run
2020-07-03 09:30:04,501 INFO [get_NeahBay_ssh] Neah Bay sea surface height web scraping and open boundary file creation for forecast complete
2020-07-03 09:30:04,507 INFO [manager] checklist updated with [Neah Bay ssh] items from get_NeahBay_ssh worker
2020-07-03 09:30:04,518 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'ssh'], host='localhost')
2020-07-03 09:30:05,666 INFO [upload_forcing] running in process 3341
2020-07-03 09:30:05,668 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 09:30:05,669 INFO [upload_forcing] publishing log messages to tcp://*:5564
2020-07-03 09:30:05,671 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2020-07-03 09:30:06,879 INFO [upload_forcing] ssh 2020-07-03 forcing files upload to arbutus.cloud-nowcast completed
2020-07-03 09:30:06,885 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2020-07-03 09:30:06,902 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'ssh', '--run-date', '2020-07-03'], host='localhost')
2020-07-03 09:30:08,002 INFO [make_forcing_links] running in process 3349
2020-07-03 09:30:08,003 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 09:30:08,005 INFO [make_forcing_links] publishing log messages to tcp://*:5564
2020-07-03 09:30:08,006 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2020-07-03 09:30:09,159 INFO [make_forcing_links] ssh 2020-07-03 forcing file links on arbutus.cloud-nowcast created
2020-07-03 09:30:09,165 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2020-07-03 09:30:09,181 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2020-07-03'], host='arbutus.cloud-nowcast')
2020-07-03 09:30:11,689 INFO [run_NEMO] running in process 25380
2020-07-03 09:30:11,690 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 09:30:11,691 INFO [run_NEMO] publishing log messages to tcp://*:5556
2020-07-03 09:30:11,692 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2020-07-03 09:30:12,844 INFO [run_NEMO] forecast: launching /nemoShare/MEOPAR/nowcast-sys/runs/03jul20forecast_2020-07-03T093011.777611-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2020-07-03 09:30:12,865 INFO [run_NEMO] forecast NEMO run for 2020-07-03 on arbutus.cloud-nowcast started
2020-07-03 09:30:12,872 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2020-07-03 09:30:12,888 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'forecast'], host='arbutus.cloud-nowcast')
2020-07-03 09:30:15,091 INFO [watch_NEMO] running in process 25646
2020-07-03 09:30:15,092 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 09:30:15,093 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2020-07-03 09:30:15,095 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2020-07-03 09:30:15,201 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2020-07-03 09:32:00,507 INFO [download_results] nowcast 2020-07-03 results files from arbutus.cloud-nowcast downloaded
2020-07-03 09:32:00,514 INFO [manager] checklist updated with [results files] items from download_results worker
2020-07-03 09:32:00,526 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast', 'research', '--run-date', '2020-07-03'], host='localhost')
2020-07-03 09:32:00,530 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast', 'comparison', '--run-date', '2020-07-02'], host='localhost')
2020-07-03 09:32:00,534 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['nowcast', '--run-date', '2020-07-03'], host='localhost')
2020-07-03 09:32:02,915 INFO [make_CHS_currents_file] running in process 3423
2020-07-03 09:32:02,916 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 09:32:02,917 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5563
2020-07-03 09:32:02,918 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555
2020-07-03 09:32:04,067 INFO [make_plots] running in process 3421
2020-07-03 09:32:04,069 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 09:32:04,070 INFO [make_plots] publishing log messages to tcp://*:5567
2020-07-03 09:32:04,071 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2020-07-03 09:32:04,099 INFO [make_plots] running in process 3422
2020-07-03 09:32:04,140 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 09:32:04,141 INFO [make_plots] publishing log messages to tcp://*:5568
2020-07-03 09:32:04,146 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2020-07-03 09:32:04,617 INFO [make_plots] preparing render list for 2020-07-02 NEMO nowcast-blue comparison figures
2020-07-03 09:32:04,635 INFO [make_plots] preparing render list for 2020-07-03 NEMO nowcast-blue research figures
2020-07-03 09:32:06,369 INFO [make_plots] added salinity_thalweg_and_surface figures to 2020-07-03 NEMO nowcast-blue research render list
2020-07-03 09:32:07,849 INFO [make_plots] added temperature_thalweg_and_surface figures to 2020-07-03 NEMO nowcast-blue research render list
2020-07-03 09:32:07,851 INFO [make_plots] added Currents_sections_and_surface figure to 2020-07-03 NEMO nowcast-blue research render list
2020-07-03 09:32:07,853 INFO [make_plots] added Currents_at_VENUS_Central figure to 2020-07-03 NEMO nowcast-blue research render list
2020-07-03 09:32:07,854 INFO [make_plots] added Currents_at_VENUS_East figure to 2020-07-03 NEMO nowcast-blue research render list
2020-07-03 09:32:07,856 INFO [make_plots] starting to render nemo nowcast research 03jul20 figures
2020-07-03 09:32:15,804 INFO [make_plots] added SH_wind figure to 2020-07-02 NEMO nowcast-blue comparison render list
2020-07-03 09:32:15,805 INFO [make_plots] added Compare_VENUS_East figure to 2020-07-02 NEMO nowcast-blue comparison render list
2020-07-03 09:32:15,816 INFO [make_plots] added Compare_VENUS_Central figure to 2020-07-02 NEMO nowcast-blue comparison render list
2020-07-03 09:32:15,817 INFO [make_plots] added Compare_VENUS_Delta_DDL figure to 2020-07-02 NEMO nowcast-blue comparison render list
2020-07-03 09:32:15,818 INFO [make_plots] added Central_ADCP figure to 2020-07-02 NEMO nowcast-blue comparison render list
2020-07-03 09:32:15,819 INFO [make_plots] added Central_depavADCP figure to 2020-07-02 NEMO nowcast-blue comparison render list
2020-07-03 09:32:15,820 INFO [make_plots] added Central_timeavADCP figure to 2020-07-02 NEMO nowcast-blue comparison render list
2020-07-03 09:32:15,820 INFO [make_plots] added East_ADCP figure to 2020-07-02 NEMO nowcast-blue comparison render list
2020-07-03 09:32:15,821 INFO [make_plots] added East_depavADCP figure to 2020-07-02 NEMO nowcast-blue comparison render list
2020-07-03 09:32:15,822 INFO [make_plots] added East_timeavADCP figure to 2020-07-02 NEMO nowcast-blue comparison render list
2020-07-03 09:32:15,823 INFO [make_plots] added ddl_ADCP figure to 2020-07-02 NEMO nowcast-blue comparison render list
2020-07-03 09:32:15,824 INFO [make_plots] added ddl_depavADCP figure to 2020-07-02 NEMO nowcast-blue comparison render list
2020-07-03 09:32:15,824 INFO [make_plots] added ddl_timeavADCP figure to 2020-07-02 NEMO nowcast-blue comparison render list
2020-07-03 09:32:15,825 INFO [make_plots] starting to render nemo nowcast comparison 02jul20 figures
2020-07-03 09:32:24,306 INFO [make_fvcom_boundary] Stored VHFR FVCOM open boundary file: /nemoShare/MEOPAR/nowcast-sys/fvcom-runs/input.x2/bdy_x2_nowcast_brcl_20200703.nc
2020-07-03 09:32:24,315 INFO [make_fvcom_boundary] FVCOM x2 nowcast run boundary condition file for 2020-07-03 created on arbutus.cloud-nowcast
2020-07-03 09:32:24,322 INFO [manager] checklist updated with [FVCOM boundary conditions] items from make_fvcom_boundary worker
2020-07-03 09:32:24,338 INFO [manager] launching NextWorker(module='nowcast.workers.make_fvcom_atmos_forcing', args=['x2', 'nowcast', '--run-date', '2020-07-03'], host='localhost')
2020-07-03 09:32:24,341 INFO [manager] launching NextWorker(module='nowcast.workers.make_fvcom_rivers_forcing', args=['arbutus.cloud-nowcast', 'x2', 'nowcast', '--run-date', '2020-07-03'], host='arbutus.cloud-nowcast')
2020-07-03 09:32:27,740 INFO [make_fvcom_atmos_forcing] running in process 3467
2020-07-03 09:32:27,741 INFO [make_fvcom_atmos_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 09:32:27,743 INFO [make_fvcom_atmos_forcing] publishing log messages to tcp://*:5569
2020-07-03 09:32:27,746 INFO [make_fvcom_atmos_forcing] connected to skookum.eos.ubc.ca port 5555
2020-07-03 09:32:27,747 INFO [make_fvcom_atmos_forcing] Creating VHFR FVCOM atmospheric forcing file for x2 nowcast run from 2020-07-03 HRDPS GRIB files
2020-07-03 09:32:28,374 INFO [make_fvcom_rivers_forcing] running in process 25740
2020-07-03 09:32:28,375 INFO [make_fvcom_rivers_forcing] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 09:32:28,376 INFO [make_fvcom_rivers_forcing] publishing log messages to tcp://*:5582
2020-07-03 09:32:28,377 INFO [make_fvcom_rivers_forcing] connected to skookum.eos.ubc.ca port 5555
2020-07-03 09:32:28,378 INFO [make_fvcom_rivers_forcing] Creating VHFR FVCOM rivers forcing file for x2 nowcast run from 2020-07-03 NEMO runoff forcing files
2020-07-03 09:32:28,611 INFO [make_fvcom_rivers_forcing] Stored VHFR FVCOM rivers forcing file: /nemoShare/MEOPAR/nowcast-sys/fvcom-runs/input.x2/rivers_x2_nowcast_20200703.nc
2020-07-03 09:32:28,612 INFO [make_fvcom_rivers_forcing] FVCOM x2 nowcast run rivers forcing file for 2020-07-03 created on arbutus.cloud-nowcast
2020-07-03 09:32:28,615 INFO [manager] checklist updated with [FVCOM rivers forcing] items from make_fvcom_rivers_forcing worker
2020-07-03 09:32:43,362 INFO [make_fvcom_boundary] Stored VHFR FVCOM open boundary file: /nemoShare/MEOPAR/nowcast-sys/fvcom-runs/input.r12/bdy_r12_nowcast_brcl_20200703.nc
2020-07-03 09:32:43,368 INFO [make_fvcom_boundary] FVCOM r12 nowcast run boundary condition file for 2020-07-03 created on arbutus.cloud-nowcast
2020-07-03 09:32:43,374 INFO [manager] checklist updated with [FVCOM boundary conditions] items from make_fvcom_boundary worker
2020-07-03 09:32:43,387 INFO [manager] launching NextWorker(module='nowcast.workers.make_fvcom_atmos_forcing', args=['r12', 'nowcast', '--run-date', '2020-07-03'], host='localhost')
2020-07-03 09:32:43,393 INFO [manager] launching NextWorker(module='nowcast.workers.make_fvcom_rivers_forcing', args=['arbutus.cloud-nowcast', 'r12', 'nowcast', '--run-date', '2020-07-03'], host='arbutus.cloud-nowcast')
2020-07-03 09:32:44,272 INFO [make_CHS_currents_file] Made CHS currents file for 2020-07-03 for nowcast
2020-07-03 09:32:44,275 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker
2020-07-03 09:32:45,997 INFO [make_fvcom_atmos_forcing] running in process 3476
2020-07-03 09:32:45,998 INFO [make_fvcom_atmos_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 09:32:45,999 INFO [make_fvcom_atmos_forcing] publishing log messages to tcp://*:5563
2020-07-03 09:32:45,999 INFO [make_fvcom_atmos_forcing] connected to skookum.eos.ubc.ca port 5555
2020-07-03 09:32:46,000 INFO [make_fvcom_atmos_forcing] Creating VHFR FVCOM atmospheric forcing file for r12 nowcast run from 2020-07-03 HRDPS GRIB files
2020-07-03 09:32:47,417 INFO [make_fvcom_rivers_forcing] running in process 25831
2020-07-03 09:32:47,418 INFO [make_fvcom_rivers_forcing] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 09:32:47,419 INFO [make_fvcom_rivers_forcing] publishing log messages to tcp://*:5582
2020-07-03 09:32:47,420 INFO [make_fvcom_rivers_forcing] connected to skookum.eos.ubc.ca port 5555
2020-07-03 09:32:47,421 INFO [make_fvcom_rivers_forcing] Creating VHFR FVCOM rivers forcing file for r12 nowcast run from 2020-07-03 NEMO runoff forcing files
2020-07-03 09:32:47,651 INFO [make_fvcom_rivers_forcing] Stored VHFR FVCOM rivers forcing file: /nemoShare/MEOPAR/nowcast-sys/fvcom-runs/input.r12/rivers_r12_nowcast_20200703.nc
2020-07-03 09:32:47,652 INFO [make_fvcom_rivers_forcing] FVCOM r12 nowcast run rivers forcing file for 2020-07-03 created on arbutus.cloud-nowcast
2020-07-03 09:32:47,656 INFO [manager] checklist updated with [FVCOM rivers forcing] items from make_fvcom_rivers_forcing worker
2020-07-03 09:32:51,265 INFO [make_fvcom_atmos_forcing] Stored VHFR FVCOM atmospheric forcing hfx file: /results/forcing/atmospheric/GEM2.5/vhfr-fvcom/atmos_x2_nowcast_hfx_20200703.nc
2020-07-03 09:32:57,946 INFO [make_fvcom_atmos_forcing] Stored VHFR FVCOM atmospheric forcing precip file: /results/forcing/atmospheric/GEM2.5/vhfr-fvcom/atmos_x2_nowcast_precip_20200703.nc
2020-07-03 09:33:07,837 INFO [make_fvcom_atmos_forcing] Stored VHFR FVCOM atmospheric forcing hfx file: /results/forcing/atmospheric/GEM2.5/vhfr-fvcom/atmos_r12_nowcast_hfx_20200703.nc
2020-07-03 09:33:08,112 INFO [make_fvcom_atmos_forcing] Stored VHFR FVCOM atmospheric forcing wnd file: /results/forcing/atmospheric/GEM2.5/vhfr-fvcom/atmos_x2_nowcast_wnd_20200703.nc
2020-07-03 09:33:08,113 INFO [make_fvcom_atmos_forcing] FVCOM x2 nowcast run atmospheric forcing file for 2020-07-03 created
2020-07-03 09:33:08,118 INFO [manager] checklist updated with [FVCOM atmospheric forcing] items from make_fvcom_atmos_forcing worker
2020-07-03 09:33:08,135 INFO [manager] launching NextWorker(module='nowcast.workers.upload_fvcom_atmos_forcing', args=['arbutus.cloud-nowcast', 'x2', 'nowcast', '--run-date', '2020-07-03'], host='localhost')
2020-07-03 09:33:09,110 INFO [upload_fvcom_atmos_forcing] running in process 3505
2020-07-03 09:33:09,111 INFO [upload_fvcom_atmos_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 09:33:09,112 INFO [upload_fvcom_atmos_forcing] publishing log messages to tcp://*:5569
2020-07-03 09:33:09,113 INFO [upload_fvcom_atmos_forcing] connected to skookum.eos.ubc.ca port 5555
2020-07-03 09:33:09,114 INFO [upload_fvcom_atmos_forcing] Uploading VHFR FVCOM atmospheric forcing files for 2020-07-03 x2 nowcast run to arbutus.cloud-nowcast
2020-07-03 09:33:14,215 INFO [upload_fvcom_atmos_forcing] FVCOM x2 nowcast run atmospheric forcing files for 2020-07-03 uploaded to arbutus.cloud-nowcast
2020-07-03 09:33:14,219 INFO [manager] checklist updated with [FVCOM atmospheric forcing upload] items from upload_fvcom_atmos_forcing worker
2020-07-03 09:33:14,239 INFO [manager] launching NextWorker(module='nowcast.workers.run_fvcom', args=['arbutus.cloud-nowcast', 'x2', 'nowcast', '--run-date', '2020-07-03'], host='arbutus.cloud-nowcast')
2020-07-03 09:33:14,507 INFO [make_fvcom_atmos_forcing] Stored VHFR FVCOM atmospheric forcing precip file: /results/forcing/atmospheric/GEM2.5/vhfr-fvcom/atmos_r12_nowcast_precip_20200703.nc
2020-07-03 09:33:16,646 INFO [run_fvcom] running in process 26181
2020-07-03 09:33:16,648 INFO [run_fvcom] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 09:33:16,649 INFO [run_fvcom] publishing log messages to tcp://*:5584
2020-07-03 09:33:16,649 INFO [run_fvcom] connected to skookum.eos.ubc.ca port 5555
2020-07-03 09:33:16,708 INFO [run_fvcom] nowcast: launching /nemoShare/MEOPAR/nowcast-sys/fvcom-runs/20200703-16h33m16.698096s/VHFR_FVCOM.sh on arbutus.cloud-nowcast
2020-07-03 09:33:16,735 INFO [run_fvcom] x2 nowcast FVCOM VH-FR run for 2020-07-03 on arbutus.cloud-nowcast started
2020-07-03 09:33:16,741 INFO [manager] checklist updated with [FVCOM run] items from run_fvcom worker
2020-07-03 09:33:16,762 INFO [manager] launching NextWorker(module='nowcast.workers.watch_fvcom', args=['arbutus.cloud-nowcast', 'x2', 'nowcast'], host='arbutus.cloud-nowcast')
2020-07-03 09:33:18,976 INFO [watch_fvcom] running in process 26283
2020-07-03 09:33:18,977 INFO [watch_fvcom] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 09:33:18,978 INFO [watch_fvcom] publishing log messages to tcp://*:5586
2020-07-03 09:33:18,980 INFO [watch_fvcom] connected to skookum.eos.ubc.ca port 5555
2020-07-03 09:33:19,025 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: no run progress found in fvcom.log ; continuing to watch...
2020-07-03 09:33:24,943 INFO [make_fvcom_atmos_forcing] Stored VHFR FVCOM atmospheric forcing wnd file: /results/forcing/atmospheric/GEM2.5/vhfr-fvcom/atmos_r12_nowcast_wnd_20200703.nc
2020-07-03 09:33:24,943 INFO [make_fvcom_atmos_forcing] FVCOM r12 nowcast run atmospheric forcing file for 2020-07-03 created
2020-07-03 09:33:24,947 INFO [manager] checklist updated with [FVCOM atmospheric forcing] items from make_fvcom_atmos_forcing worker
2020-07-03 09:33:24,967 INFO [manager] launching NextWorker(module='nowcast.workers.upload_fvcom_atmos_forcing', args=['arbutus.cloud-nowcast', 'r12', 'nowcast', '--run-date', '2020-07-03'], host='localhost')
2020-07-03 09:33:25,832 INFO [upload_fvcom_atmos_forcing] running in process 3533
2020-07-03 09:33:25,833 INFO [upload_fvcom_atmos_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 09:33:25,834 INFO [upload_fvcom_atmos_forcing] publishing log messages to tcp://*:5563
2020-07-03 09:33:25,835 INFO [upload_fvcom_atmos_forcing] connected to skookum.eos.ubc.ca port 5555
2020-07-03 09:33:25,836 INFO [upload_fvcom_atmos_forcing] Uploading VHFR FVCOM atmospheric forcing files for 2020-07-03 r12 nowcast run to arbutus.cloud-nowcast
2020-07-03 09:33:31,834 INFO [upload_fvcom_atmos_forcing] FVCOM r12 nowcast run atmospheric forcing files for 2020-07-03 uploaded to arbutus.cloud-nowcast
2020-07-03 09:33:31,838 INFO [manager] checklist updated with [FVCOM atmospheric forcing upload] items from upload_fvcom_atmos_forcing worker
2020-07-03 09:33:31,858 INFO [manager] launching NextWorker(module='nowcast.workers.run_fvcom', args=['arbutus.cloud-nowcast', 'r12', 'nowcast', '--run-date', '2020-07-03'], host='arbutus.cloud-nowcast')
2020-07-03 09:33:34,403 INFO [run_fvcom] running in process 26630
2020-07-03 09:33:34,404 INFO [run_fvcom] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 09:33:34,405 INFO [run_fvcom] publishing log messages to tcp://*:5584
2020-07-03 09:33:34,407 INFO [run_fvcom] connected to skookum.eos.ubc.ca port 5555
2020-07-03 09:33:34,454 INFO [run_fvcom] nowcast: launching /nemoShare/MEOPAR/nowcast-sys/fvcom-runs/20200703-16h33m34.444720s/VHFR_FVCOM.sh on arbutus.cloud-nowcast
2020-07-03 09:33:34,489 INFO [run_fvcom] r12 nowcast FVCOM VH-FR run for 2020-07-03 on arbutus.cloud-nowcast started
2020-07-03 09:33:34,496 INFO [manager] checklist updated with [FVCOM run] items from run_fvcom worker
2020-07-03 09:33:34,517 INFO [manager] launching NextWorker(module='nowcast.workers.watch_fvcom', args=['arbutus.cloud-nowcast', 'r12', 'nowcast'], host='arbutus.cloud-nowcast')
2020-07-03 09:33:36,797 INFO [watch_fvcom] running in process 26735
2020-07-03 09:33:36,798 INFO [watch_fvcom] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 09:33:36,799 INFO [watch_fvcom] publishing log messages to tcp://*:5587
2020-07-03 09:33:36,800 INFO [watch_fvcom] connected to skookum.eos.ubc.ca port 5555
2020-07-03 09:33:36,854 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: fvcom.log not found; continuing to watch...
2020-07-03 09:33:52,312 INFO [make_plots] finished rendering nemo nowcast research 03jul20 figures
2020-07-03 09:33:52,313 INFO [make_plots] nemo research plots for 2020-07-03 nowcast completed
2020-07-03 09:33:52,328 INFO [manager] checklist updated with [plots] items from make_plots worker
2020-07-03 09:34:21,320 WARNING [make_plots] VENUS Delta DDL node CTD comparison figure failed: No observations available
2020-07-03 09:34:22,092 WARNING [make_plots] VENUS Central ADCP comparison figure failed: No observations available
2020-07-03 09:34:22,618 WARNING [make_plots] VENUS Central ADCP comparison figure failed: No observations available
2020-07-03 09:34:23,145 WARNING [make_plots] VENUS Central ADCP comparison figure failed: No observations available
2020-07-03 09:34:23,792 WARNING [make_plots] VENUS East ADCP comparison figure failed: No observations available
2020-07-03 09:34:24,354 WARNING [make_plots] VENUS East ADCP comparison figure failed: No observations available
2020-07-03 09:34:24,896 WARNING [make_plots] VENUS East ADCP comparison figure failed: No observations available
2020-07-03 09:34:25,464 WARNING [make_plots] VENUS ddl ADCP comparison figure failed: No observations available
2020-07-03 09:34:26,003 WARNING [make_plots] VENUS ddl ADCP comparison figure failed: No observations available
2020-07-03 09:34:26,542 WARNING [make_plots] VENUS ddl ADCP comparison figure failed: No observations available
2020-07-03 09:34:26,543 INFO [make_plots] finished rendering nemo nowcast comparison 02jul20 figures
2020-07-03 09:34:26,626 INFO [make_plots] nemo comparison plots for 2020-07-02 nowcast completed
2020-07-03 09:34:26,631 INFO [manager] checklist updated with [plots] items from make_plots worker
2020-07-03 09:35:15,303 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 7497768 = 2020-07-04 04:31:20 UTC, 12.6% complete
2020-07-03 09:38:19,127 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15728099 = 2020-07-03 00:24:59 UTC estimated time to finish: 0000:04:12:29
2020-07-03 09:38:36,890 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39317173 = 2020-07-03 00:04:29 UTC estimated time to finish: 0000:22:47:21
2020-07-03 09:40:15,405 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 7498229 = 2020-07-04 09:38:40 UTC, 26.8% complete
2020-07-03 09:43:19,189 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15729680 = 2020-07-03 00:51:20 UTC estimated time to finish: 0000:04:17:55
2020-07-03 09:43:36,992 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39317965 = 2020-07-03 00:09:46 UTC estimated time to finish: 0000:22:45:32
2020-07-03 09:45:15,507 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 7498673 = 2020-07-04 14:34:40 UTC, 40.5% complete
2020-07-03 09:48:19,292 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15731330 = 2020-07-03 01:18:50 UTC estimated time to finish: 0000:04:13:57
2020-07-03 09:48:37,095 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39318797 = 2020-07-03 00:15:18 UTC estimated time to finish: 0000:22:39:34
2020-07-03 09:50:15,521 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 7499139 = 2020-07-04 19:45:20 UTC, 54.9% complete
2020-07-03 09:53:19,398 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15732870 = 2020-07-03 01:44:30 UTC estimated time to finish: 0000:04:11:45
2020-07-03 09:53:37,198 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39319507 = 2020-07-03 00:20:02 UTC estimated time to finish: 0000:22:31:35
2020-07-03 09:55:15,623 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 7499583 = 2020-07-05 00:41:20 UTC, 68.6% complete
2020-07-03 09:58:19,460 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15734520 = 2020-07-03 02:12:00 UTC estimated time to finish: 0000:04:06:14
2020-07-03 09:58:37,301 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39320327 = 2020-07-03 00:25:30 UTC estimated time to finish: 0000:22:32:44
2020-07-03 10:00:15,728 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 7500030 = 2020-07-05 05:39:20 UTC, 82.4% complete
2020-07-03 10:03:19,564 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15735978 = 2020-07-03 02:36:18 UTC estimated time to finish: 0000:04:02:38
2020-07-03 10:03:37,406 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39321079 = 2020-07-03 00:30:31 UTC estimated time to finish: 0000:22:33:52
2020-07-03 10:05:15,827 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 7500489 = 2020-07-05 10:45:20 UTC, 96.6% complete
2020-07-03 10:08:19,668 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15737559 = 2020-07-03 03:02:39 UTC estimated time to finish: 0000:03:58:35
2020-07-03 10:08:37,508 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39321911 = 2020-07-03 00:36:04 UTC estimated time to finish: 0000:22:21:24
2020-07-03 10:10:15,930 INFO [watch_NEMO] forecast NEMO run on arbutus.cloud-nowcast completed
2020-07-03 10:10:15,935 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2020-07-03 10:10:16,249 INFO [manager] launching NextWorker(module='nowcast.workers.make_turbidity_file', args=['--run-date', '2020-07-03'], host='localhost')
2020-07-03 10:10:16,254 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2020-07-03'], host='localhost')
2020-07-03 10:10:17,259 INFO [download_results] running in process 4352
2020-07-03 10:10:17,260 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 10:10:17,261 INFO [download_results] publishing log messages to tcp://*:5564
2020-07-03 10:10:17,262 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2020-07-03 10:10:17,262 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/forecast/03jul20 to /results/SalishSea/forecast.201905
2020-07-03 10:10:17,529 INFO [make_turbidity_file] running in process 4351
2020-07-03 10:10:17,530 INFO [make_turbidity_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 10:10:17,531 INFO [make_turbidity_file] publishing log messages to tcp://*:5567
2020-07-03 10:10:17,532 INFO [make_turbidity_file] connected to skookum.eos.ubc.ca port 5555
2020-07-03 10:10:17,533 INFO [make_turbidity_file] Creating Fraser River turbidity forcing file for 2020-07-03
2020-07-03 10:10:17,805 INFO [make_turbidity_file] Note: NumExpr detected 12 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
2020-07-03 10:10:17,806 INFO [make_turbidity_file] NumExpr defaulting to 8 threads.
2020-07-03 10:10:19,121 INFO [make_turbidity_file] 2020-07-03 Fraser River turbidity file creation complete
2020-07-03 10:10:19,123 INFO [manager] checklist updated with [Fraser River turbidity] items from make_turbidity_file worker
2020-07-03 10:10:19,149 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'turbidity'], host='localhost')
2020-07-03 10:10:19,153 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'turbidity'], host='localhost')
2020-07-03 10:10:19,157 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['graham-hindcast', 'turbidity'], host='localhost')
2020-07-03 10:10:19,160 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'turbidity'], host='localhost')
2020-07-03 10:10:20,049 INFO [upload_forcing] running in process 4370
2020-07-03 10:10:20,050 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 10:10:20,051 INFO [upload_forcing] publishing log messages to tcp://*:5567
2020-07-03 10:10:20,052 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2020-07-03 10:10:20,083 INFO [upload_forcing] running in process 4373
2020-07-03 10:10:20,085 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 10:10:20,086 INFO [upload_forcing] publishing log messages to tcp://*:5568
2020-07-03 10:10:20,087 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2020-07-03 10:10:20,116 INFO [upload_forcing] running in process 4372
2020-07-03 10:10:20,117 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 10:10:20,118 INFO [upload_forcing] publishing log messages to tcp://*:5569
2020-07-03 10:10:20,119 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2020-07-03 10:10:20,132 INFO [upload_forcing] running in process 4371
2020-07-03 10:10:20,133 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 10:10:20,133 INFO [upload_forcing] publishing log messages to tcp://*:5570
2020-07-03 10:10:20,134 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2020-07-03 10:10:20,595 INFO [upload_forcing] turbidity 2020-07-03 forcing files upload to optimum-hindcast completed
2020-07-03 10:10:20,597 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2020-07-03 10:10:20,648 INFO [upload_forcing] turbidity 2020-07-03 forcing files upload to orcinus-nowcast-agrif completed
2020-07-03 10:10:20,651 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2020-07-03 10:10:20,669 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['orcinus-nowcast-agrif', 'nowcast-agrif', '--run-date', '2020-07-03'], host='localhost')
2020-07-03 10:10:21,374 INFO [upload_forcing] turbidity 2020-07-03 forcing files upload to arbutus.cloud-nowcast completed
2020-07-03 10:10:21,379 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2020-07-03 10:10:21,403 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2020-07-03'], host='localhost')
2020-07-03 10:10:21,649 INFO [make_forcing_links] running in process 4396
2020-07-03 10:10:21,650 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 10:10:21,651 INFO [make_forcing_links] publishing log messages to tcp://*:5568
2020-07-03 10:10:21,652 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2020-07-03 10:10:22,245 INFO [make_forcing_links] nowcast-agrif 2020-07-03 forcing file links on orcinus-nowcast-agrif created
2020-07-03 10:10:22,249 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2020-07-03 10:10:22,273 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_agrif', args=['orcinus-nowcast-agrif', 'nowcast-agrif', '--run-date', '2020-07-03'], host='localhost')
2020-07-03 10:10:22,365 INFO [make_forcing_links] running in process 4400
2020-07-03 10:10:22,366 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 10:10:22,366 INFO [make_forcing_links] publishing log messages to tcp://*:5567
2020-07-03 10:10:22,367 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2020-07-03 10:10:23,194 INFO [run_NEMO_agrif] running in process 4406
2020-07-03 10:10:23,195 INFO [run_NEMO_agrif] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 10:10:23,196 INFO [run_NEMO_agrif] publishing log messages to tcp://*:5568
2020-07-03 10:10:23,198 INFO [run_NEMO_agrif] connected to skookum.eos.ubc.ca port 5555
2020-07-03 10:10:23,298 INFO [upload_forcing] turbidity 2020-07-03 forcing files upload to graham-hindcast completed
2020-07-03 10:10:23,302 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2020-07-03 10:10:23,597 INFO [make_forcing_links] nowcast-green 2020-07-03 forcing file links on arbutus.cloud-nowcast created
2020-07-03 10:10:23,601 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2020-07-03 10:10:23,619 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2020-07-03'], host='arbutus.cloud-nowcast')
2020-07-03 10:10:25,955 INFO [run_NEMO] running in process 27203
2020-07-03 10:10:25,956 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 10:10:25,957 INFO [run_NEMO] publishing log messages to tcp://*:5556
2020-07-03 10:10:25,965 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2020-07-03 10:10:27,328 INFO [run_NEMO] nowcast-green: launching /nemoShare/MEOPAR/nowcast-sys/runs/03jul20nowcast-green_2020-07-03T101026.027970-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2020-07-03 10:10:27,348 INFO [run_NEMO] nowcast-green NEMO run for 2020-07-03 on arbutus.cloud-nowcast started
2020-07-03 10:10:27,353 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2020-07-03 10:10:27,380 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'nowcast-green'], host='arbutus.cloud-nowcast')
2020-07-03 10:10:29,433 INFO [watch_NEMO] running in process 27467
2020-07-03 10:10:29,434 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 10:10:29,435 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2020-07-03 10:10:29,437 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2020-07-03 10:10:29,529 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2020-07-03 10:10:49,432 INFO [run_NEMO_agrif] job id for 03jul20nowcast-agrif: 10846182.orca2.ibb
2020-07-03 10:10:49,433 INFO [run_NEMO_agrif] NEMO nowcast-agrif run for 2020-07-03 queued on orcinus-nowcast-agrif
2020-07-03 10:10:49,437 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_agrif worker
2020-07-03 10:10:49,458 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_agrif', args=['orcinus-nowcast-agrif', '10846182.orca2.ibb'], host='localhost')
2020-07-03 10:10:50,357 INFO [watch_NEMO_agrif] running in process 4421
2020-07-03 10:10:50,359 INFO [watch_NEMO_agrif] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 10:10:50,360 INFO [watch_NEMO_agrif] publishing log messages to tcp://*:5567
2020-07-03 10:10:50,361 INFO [watch_NEMO_agrif] connected to skookum.eos.ubc.ca port 5555
2020-07-03 10:10:50,933 INFO [watch_NEMO_agrif] watching 03jul20nowcast-agrif job 10846182 on orcinus-nowcast-agrif
2020-07-03 10:10:51,118 INFO [watch_NEMO_agrif] 03jul20nowcast-agrif job 10846182 is queued on orcinus-nowcast-agrif
2020-07-03 10:11:51,306 INFO [watch_NEMO_agrif] 03jul20nowcast-agrif job 10846182 is queued on orcinus-nowcast-agrif
2020-07-03 10:12:51,572 INFO [watch_NEMO_agrif] 03jul20nowcast-agrif job 10846182 is queued on orcinus-nowcast-agrif
2020-07-03 10:13:03,093 INFO [download_results] forecast 2020-07-03 results files from arbutus.cloud-nowcast downloaded
2020-07-03 10:13:03,098 INFO [manager] checklist updated with [results files] items from download_results worker
2020-07-03 10:13:03,119 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['forecast', '--run-date', '2020-07-03'], host='localhost')
2020-07-03 10:13:05,302 INFO [make_CHS_currents_file] running in process 4503
2020-07-03 10:13:05,303 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 10:13:05,304 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5564
2020-07-03 10:13:05,304 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555
2020-07-03 10:13:19,777 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15739140 = 2020-07-03 03:29:00 UTC estimated time to finish: 0000:03:53:44
2020-07-03 10:13:37,613 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39322744 = 2020-07-03 00:41:37 UTC estimated time to finish: 0000:21:56:34
2020-07-03 10:13:51,790 INFO [watch_NEMO_agrif] 03jul20nowcast-agrif job 10846182 is queued on orcinus-nowcast-agrif
2020-07-03 10:14:52,002 INFO [watch_NEMO_agrif] 03jul20nowcast-agrif job 10846182 is queued on orcinus-nowcast-agrif
2020-07-03 10:15:03,609 INFO [make_CHS_currents_file] Made CHS currents file for 2020-07-03 for forecast
2020-07-03 10:15:03,612 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker
2020-07-03 10:15:03,629 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['nemo', 'forecast', '--run-date', '2020-07-03'], host='localhost')
2020-07-03 10:15:04,470 INFO [update_forecast_datasets] running in process 4523
2020-07-03 10:15:04,471 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 10:15:04,472 INFO [update_forecast_datasets] publishing log messages to tcp://*:5564
2020-07-03 10:15:04,473 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2020-07-03 10:15:04,474 INFO [update_forecast_datasets] updating nemo forecast directory for forecast/03jul20 run
2020-07-03 10:15:04,798 INFO [update_forecast_datasets] updated nemo forecast directory for forecast/03jul20 run: /results/SalishSea/rolling-forecasts/nemo
2020-07-03 10:15:04,799 INFO [update_forecast_datasets] nemo 2020-07-03 forecast rolling forecast datasets updated
2020-07-03 10:15:04,802 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2020-07-03 10:15:04,830 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nemo-forecast'], host='localhost')
2020-07-03 10:15:04,833 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'forecast', 'publish', '--run-date', '2020-07-03'], host='localhost')
2020-07-03 10:15:04,836 INFO [manager] launching NextWorker(module='nowcast.workers.make_surface_current_tiles', args=['forecast', '--run-date', '2020-07-03'], host='localhost')
2020-07-03 10:15:05,656 INFO [ping_erddap] running in process 4527
2020-07-03 10:15:05,657 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 10:15:05,658 INFO [ping_erddap] publishing log messages to tcp://*:5564
2020-07-03 10:15:05,659 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2020-07-03 10:15:05,702 INFO [ping_erddap] nemo-forecast ERDDAP dataset flag file(s) created
2020-07-03 10:15:05,704 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2020-07-03 10:15:05,997 INFO [make_surface_current_tiles] running in process 4529
2020-07-03 10:15:05,998 INFO [make_surface_current_tiles] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 10:15:05,998 INFO [make_surface_current_tiles] publishing log messages to tcp://*:5564
2020-07-03 10:15:05,999 INFO [make_surface_current_tiles] connected to skookum.eos.ubc.ca port 5555
2020-07-03 10:15:06,034 INFO [make_surface_current_tiles] starting rendering of tiles for 2020-07-03 forecast in 6 processes into /results/nowcast-sys/figures/surface_currents/forecast/03jul20
2020-07-03 10:15:11,014 INFO [make_plots] running in process 4528
2020-07-03 10:15:11,015 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 10:15:11,017 INFO [make_plots] publishing log messages to tcp://*:5568
2020-07-03 10:15:11,018 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2020-07-03 10:15:12,057 INFO [make_plots] preparing render list for 2020-07-03 NEMO forecast publish figures
2020-07-03 10:15:12,518 INFO [make_plots] added Website_thumbnail figure to 2020-07-03 NEMO forecast publish render list
2020-07-03 10:15:12,519 INFO [make_plots] added Threshold_website figure to 2020-07-03 NEMO forecast publish render list
2020-07-03 10:15:12,519 INFO [make_plots] added PA_tidal_predictions figure to 2020-07-03 NEMO forecast publish render list
2020-07-03 10:15:12,520 INFO [make_plots] added Boundary Bay figure to 2020-07-03 NEMO forecast publish render list
2020-07-03 10:15:12,521 INFO [make_plots] added Campbell River figure to 2020-07-03 NEMO forecast publish render list
2020-07-03 10:15:12,521 INFO [make_plots] added Cherry Point figure to 2020-07-03 NEMO forecast publish render list
2020-07-03 10:15:12,522 INFO [make_plots] added Friday Harbor figure to 2020-07-03 NEMO forecast publish render list
2020-07-03 10:15:12,523 INFO [make_plots] added Halfmoon Bay figure to 2020-07-03 NEMO forecast publish render list
2020-07-03 10:15:12,523 INFO [make_plots] added Nanaimo figure to 2020-07-03 NEMO forecast publish render list
2020-07-03 10:15:12,524 INFO [make_plots] added Neah Bay figure to 2020-07-03 NEMO forecast publish render list
2020-07-03 10:15:12,525 INFO [make_plots] added New Westminster figure to 2020-07-03 NEMO forecast publish render list
2020-07-03 10:15:12,526 INFO [make_plots] added Patricia Bay figure to 2020-07-03 NEMO forecast publish render list
2020-07-03 10:15:12,527 INFO [make_plots] added Point Atkinson figure to 2020-07-03 NEMO forecast publish render list
2020-07-03 10:15:12,528 INFO [make_plots] added Port Renfrew figure to 2020-07-03 NEMO forecast publish render list
2020-07-03 10:15:12,529 INFO [make_plots] added Sand Heads figure to 2020-07-03 NEMO forecast publish render list
2020-07-03 10:15:12,530 INFO [make_plots] added Sandy Cove figure to 2020-07-03 NEMO forecast publish render list
2020-07-03 10:15:12,531 INFO [make_plots] added Squamish figure to 2020-07-03 NEMO forecast publish render list
2020-07-03 10:15:12,531 INFO [make_plots] added Victoria figure to 2020-07-03 NEMO forecast publish render list
2020-07-03 10:15:12,532 INFO [make_plots] added Woodwards Landing figure to 2020-07-03 NEMO forecast publish render list
2020-07-03 10:15:12,533 INFO [make_plots] starting to render nemo forecast publish 03jul20 figures
2020-07-03 10:15:29,631 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 7495341 = 2020-07-03 01:33:20 UTC, 6.5% complete
2020-07-03 10:15:43,561 INFO [make_plots] Note: NumExpr detected 12 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
2020-07-03 10:15:43,562 INFO [make_plots] NumExpr defaulting to 8 threads.
2020-07-03 10:15:52,268 INFO [watch_NEMO_agrif] 03jul20nowcast-agrif job 10846182 is queued on orcinus-nowcast-agrif
2020-07-03 10:16:52,478 INFO [watch_NEMO_agrif] 03jul20nowcast-agrif job 10846182 is queued on orcinus-nowcast-agrif
2020-07-03 10:17:52,735 INFO [watch_NEMO_agrif] 03jul20nowcast-agrif job 10846182 is queued on orcinus-nowcast-agrif
2020-07-03 10:18:19,886 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15740434 = 2020-07-03 03:50:34 UTC estimated time to finish: 0000:03:55:17
2020-07-03 10:18:37,718 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39323523 = 2020-07-03 00:46:49 UTC estimated time to finish: 0000:21:57:04
2020-07-03 10:18:52,946 INFO [watch_NEMO_agrif] 03jul20nowcast-agrif job 10846182 is queued on orcinus-nowcast-agrif
2020-07-03 10:19:53,200 INFO [watch_NEMO_agrif] 03jul20nowcast-agrif job 10846182 is queued on orcinus-nowcast-agrif
2020-07-03 10:20:29,732 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 7495502 = 2020-07-03 03:20:40 UTC, 13.9% complete
2020-07-03 10:20:53,407 INFO [watch_NEMO_agrif] 03jul20nowcast-agrif job 10846182 is queued on orcinus-nowcast-agrif
2020-07-03 10:21:53,615 INFO [watch_NEMO_agrif] 03jul20nowcast-agrif job 10846182 is queued on orcinus-nowcast-agrif
2020-07-03 10:22:53,835 INFO [watch_NEMO_agrif] 03jul20nowcast-agrif job 10846182 is queued on orcinus-nowcast-agrif
2020-07-03 10:23:20,001 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15741375 = 2020-07-03 04:06:15 UTC estimated time to finish: 0000:03:59:15
2020-07-03 10:23:37,827 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39324274 = 2020-07-03 00:51:49 UTC estimated time to finish: 0000:21:58:55
2020-07-03 10:23:54,090 INFO [watch_NEMO_agrif] 03jul20nowcast-agrif job 10846182 is queued on orcinus-nowcast-agrif
2020-07-03 10:24:54,343 INFO [watch_NEMO_agrif] 03jul20nowcast-agrif job 10846182 is queued on orcinus-nowcast-agrif
2020-07-03 10:25:29,837 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 7495664 = 2020-07-03 05:08:40 UTC, 21.4% complete
2020-07-03 10:25:54,598 INFO [watch_NEMO_agrif] 03jul20nowcast-agrif job 10846182 is queued on orcinus-nowcast-agrif
2020-07-03 10:26:54,858 INFO [watch_NEMO_agrif] 03jul20nowcast-agrif job 10846182 is queued on orcinus-nowcast-agrif
2020-07-03 10:27:55,121 INFO [watch_NEMO_agrif] 03jul20nowcast-agrif job 10846182 is queued on orcinus-nowcast-agrif
2020-07-03 10:28:20,112 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15742480 = 2020-07-03 04:24:40 UTC estimated time to finish: 0000:04:02:31
2020-07-03 10:28:37,934 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39325066 = 2020-07-03 00:57:06 UTC estimated time to finish: 0000:22:00:21
2020-07-03 10:28:55,376 INFO [watch_NEMO_agrif] 03jul20nowcast-agrif job 10846182 is queued on orcinus-nowcast-agrif
2020-07-03 10:29:55,643 INFO [watch_NEMO_agrif] 03jul20nowcast-agrif job 10846182 is queued on orcinus-nowcast-agrif
2020-07-03 10:30:29,939 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 7495826 = 2020-07-03 06:56:40 UTC, 28.9% complete
2020-07-03 10:30:55,909 INFO [watch_NEMO_agrif] 03jul20nowcast-agrif job 10846182 is queued on orcinus-nowcast-agrif
2020-07-03 10:31:56,160 INFO [watch_NEMO_agrif] 03jul20nowcast-agrif job 10846182 is queued on orcinus-nowcast-agrif
2020-07-03 10:32:22,648 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 08074 Campbell River from 2020-06-28 00:00:00 to 2020-07-05 11:55:00
2020-07-03 10:32:56,360 INFO [watch_NEMO_agrif] 03jul20nowcast-agrif job 10846182 is queued on orcinus-nowcast-agrif
2020-07-03 10:33:20,214 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15743542 = 2020-07-03 04:42:22 UTC estimated time to finish: 0000:04:04:19
2020-07-03 10:33:38,036 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39325817 = 2020-07-03 01:02:06 UTC estimated time to finish: 0000:22:04:46
2020-07-03 10:33:56,622 INFO [watch_NEMO_agrif] 03jul20nowcast-agrif job 10846182 is queued on orcinus-nowcast-agrif
2020-07-03 10:34:56,877 INFO [watch_NEMO_agrif] 03jul20nowcast-agrif job 10846182 is queued on orcinus-nowcast-agrif
2020-07-03 10:35:30,042 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 7495985 = 2020-07-03 08:42:40 UTC, 36.3% complete
2020-07-03 10:35:57,142 INFO [watch_NEMO_agrif] 03jul20nowcast-agrif job 10846182 is queued on orcinus-nowcast-agrif
2020-07-03 10:36:20,127 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 9449424 Cherry Point from 2020-06-28 00:00:00 to 2020-07-05 11:55:00
2020-07-03 10:36:57,358 INFO [watch_NEMO_agrif] 03jul20nowcast-agrif job 10846182 is queued on orcinus-nowcast-agrif
2020-07-03 10:37:24,981 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 9449880 Friday Harbor from 2020-06-28 00:00:00 to 2020-07-05 11:55:00
2020-07-03 10:37:57,607 INFO [watch_NEMO_agrif] 03jul20nowcast-agrif job 10846182 is queued on orcinus-nowcast-agrif
2020-07-03 10:38:20,325 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15744646 = 2020-07-03 05:00:46 UTC estimated time to finish: 0000:04:05:31
2020-07-03 10:38:29,232 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07830 Halfmoon Bay from 2020-06-28 00:00:00 to 2020-07-05 11:55:00
2020-07-03 10:38:38,143 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39326445 = 2020-07-03 01:06:18 UTC estimated time to finish: 0000:22:10:47
2020-07-03 10:38:57,868 INFO [watch_NEMO_agrif] 03jul20nowcast-agrif job 10846182 is queued on orcinus-nowcast-agrif
2020-07-03 10:39:37,112 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07917 Nanaimo from 2020-06-28 00:00:00 to 2020-07-05 11:55:00
2020-07-03 10:39:58,079 INFO [watch_NEMO_agrif] 03jul20nowcast-agrif job 10846182 is queued on orcinus-nowcast-agrif
2020-07-03 10:40:30,143 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 7496145 = 2020-07-03 10:29:20 UTC, 43.7% complete
2020-07-03 10:40:58,345 INFO [watch_NEMO_agrif] 03jul20nowcast-agrif job 10846182 is queued on orcinus-nowcast-agrif
2020-07-03 10:41:58,592 INFO [watch_NEMO_agrif] 03jul20nowcast-agrif job 10846182 is queued on orcinus-nowcast-agrif
2020-07-03 10:42:58,801 INFO [watch_NEMO_agrif] 03jul20nowcast-agrif job 10846182 is queued on orcinus-nowcast-agrif
2020-07-03 10:43:20,440 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15745640 = 2020-07-03 05:17:20 UTC estimated time to finish: 0000:04:05:25
2020-07-03 10:43:38,249 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39327155 = 2020-07-03 01:11:02 UTC estimated time to finish: 0000:22:16:46
2020-07-03 10:43:39,383 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 9443090 Neah Bay from 2020-06-28 00:00:00 to 2020-07-05 11:55:00
2020-07-03 10:43:59,026 INFO [watch_NEMO_agrif] 03jul20nowcast-agrif job 10846182 is queued on orcinus-nowcast-agrif
2020-07-03 10:44:34,628 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07654 New Westminster from 2020-06-28 00:00:00 to 2020-07-05 11:55:00
2020-07-03 10:44:59,288 INFO [watch_NEMO_agrif] 03jul20nowcast-agrif job 10846182 is queued on orcinus-nowcast-agrif
2020-07-03 10:45:17,295 INFO [make_surface_current_tiles] starting PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast/03jul20
2020-07-03 10:45:30,246 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 7496303 = 2020-07-03 12:14:40 UTC, 51.0% complete
2020-07-03 10:45:59,492 INFO [watch_NEMO_agrif] 03jul20nowcast-agrif job 10846182 is queued on orcinus-nowcast-agrif
2020-07-03 10:46:58,817 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07277 Patricia Bay from 2020-06-28 00:00:00 to 2020-07-05 11:55:00
2020-07-03 10:46:59,750 INFO [watch_NEMO_agrif] 03jul20nowcast-agrif job 10846182 is queued on orcinus-nowcast-agrif
2020-07-03 10:48:00,009 INFO [watch_NEMO_agrif] 03jul20nowcast-agrif job 10846182 is queued on orcinus-nowcast-agrif
2020-07-03 10:48:20,552 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15746703 = 2020-07-03 05:35:03 UTC estimated time to finish: 0000:04:05:39
2020-07-03 10:48:38,360 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39327865 = 2020-07-03 01:15:46 UTC estimated time to finish: 0000:22:18:22
2020-07-03 10:49:00,270 INFO [watch_NEMO_agrif] 03jul20nowcast-agrif job 10846182 is queued on orcinus-nowcast-agrif
2020-07-03 10:49:16,503 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07795 Point Atkinson from 2020-06-28 00:00:00 to 2020-07-05 11:55:00
2020-07-03 10:49:36,608 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 08525 Port Renfrew from 2020-06-28 00:00:00 to 2020-07-05 11:55:00
2020-07-03 10:50:00,530 INFO [watch_NEMO_agrif] 03jul20nowcast-agrif job 10846182 is queued on orcinus-nowcast-agrif
2020-07-03 10:50:05,135 INFO [make_surface_current_tiles] finished PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast/03jul20
2020-07-03 10:50:05,156 INFO [make_surface_current_tiles] finished rendering of tiles for 2020-07-03 forecast into /results/nowcast-sys/figures/surface_currents/forecast/03jul20
2020-07-03 10:50:05,158 INFO [make_surface_current_tiles] surface current tile figures for 2020-07-03 forecast completed
2020-07-03 10:50:05,901 INFO [manager] checklist updated with [surface current tiles] items from make_surface_current_tiles worker
2020-07-03 10:50:30,347 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 7496462 = 2020-07-03 14:00:40 UTC, 58.4% complete
2020-07-03 10:51:00,787 INFO [watch_NEMO_agrif] 03jul20nowcast-agrif job 10846182 is queued on orcinus-nowcast-agrif
2020-07-03 10:51:46,764 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07594 Sand Heads from 2020-06-28 00:00:00 to 2020-07-05 11:55:00
2020-07-03 10:52:01,002 INFO [watch_NEMO_agrif] 03jul20nowcast-agrif job 10846182 is queued on orcinus-nowcast-agrif
2020-07-03 10:53:01,222 INFO [watch_NEMO_agrif] 03jul20nowcast-agrif job 10846182 is queued on orcinus-nowcast-agrif
2020-07-03 10:53:20,663 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15747767 = 2020-07-03 05:52:47 UTC estimated time to finish: 0000:04:05:21
2020-07-03 10:53:38,466 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39328616 = 2020-07-03 01:20:46 UTC estimated time to finish: 0000:22:18:28
2020-07-03 10:53:58,737 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07786 Sandy Cove from 2020-06-28 00:00:00 to 2020-07-05 11:55:00
2020-07-03 10:54:01,479 INFO [watch_NEMO_agrif] 03jul20nowcast-agrif job 10846182 is queued on orcinus-nowcast-agrif
2020-07-03 10:55:01,715 INFO [watch_NEMO_agrif] 03jul20nowcast-agrif job 10846182 is queued on orcinus-nowcast-agrif
2020-07-03 10:55:30,449 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 7496621 = 2020-07-03 15:46:40 UTC, 65.8% complete
2020-07-03 10:56:01,980 INFO [watch_NEMO_agrif] 03jul20nowcast-agrif job 10846182 is queued on orcinus-nowcast-agrif
2020-07-03 10:56:10,019 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07811 Squamish from 2020-06-28 00:00:00 to 2020-07-05 11:55:00
2020-07-03 10:56:28,857 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07120 Victoria from 2020-06-28 00:00:00 to 2020-07-05 11:55:00
2020-07-03 10:57:02,139 INFO [watch_NEMO_agrif] 03jul20nowcast-agrif job 10846182 is queued on orcinus-nowcast-agrif
2020-07-03 10:58:02,404 INFO [watch_NEMO_agrif] 03jul20nowcast-agrif job 10846182 is queued on orcinus-nowcast-agrif
2020-07-03 10:58:20,775 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15748830 = 2020-07-03 06:10:30 UTC estimated time to finish: 0000:04:04:41
2020-07-03 10:58:38,573 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39329285 = 2020-07-03 01:25:14 UTC estimated time to finish: 0000:22:19:57
2020-07-03 10:58:39,664 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07610 Woodwards Landing from 2020-06-28 00:00:00 to 2020-07-05 11:55:00
2020-07-03 10:59:02,665 INFO [watch_NEMO_agrif] 03jul20nowcast-agrif job 10846182 is queued on orcinus-nowcast-agrif
2020-07-03 11:00:02,927 INFO [watch_NEMO_agrif] 03jul20nowcast-agrif job 10846182 is queued on orcinus-nowcast-agrif
2020-07-03 11:00:30,552 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 7496776 = 2020-07-03 17:30:00 UTC, 73.0% complete
2020-07-03 11:00:44,480 INFO [make_plots] finished rendering nemo forecast publish 03jul20 figures
2020-07-03 11:00:44,482 INFO [make_plots] nemo publish plots for 2020-07-03 forecast completed
2020-07-03 11:00:44,489 INFO [manager] checklist updated with [plots] items from make_plots worker
2020-07-03 11:00:44,677 INFO [manager] launching NextWorker(module='nowcast.workers.make_feeds', args=['forecast', '--run-date', '2020-07-03'], host='localhost')
2020-07-03 11:00:47,424 INFO [make_feeds] running in process 7462
2020-07-03 11:00:47,426 INFO [make_feeds] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 11:00:47,427 INFO [make_feeds] publishing log messages to tcp://*:5564
2020-07-03 11:00:47,428 INFO [make_feeds] connected to skookum.eos.ubc.ca port 5555
2020-07-03 11:00:56,190 INFO [make_feeds] Note: NumExpr detected 12 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
2020-07-03 11:00:56,191 INFO [make_feeds] NumExpr defaulting to 8 threads.
2020-07-03 11:01:02,664 INFO [make_feeds] ATOM feeds for 2020-07-03 forecast run completed
2020-07-03 11:01:02,669 INFO [manager] checklist updated with [storm surge alert feeds] items from make_feeds worker
2020-07-03 11:01:03,206 INFO [watch_NEMO_agrif] 03jul20nowcast-agrif job 10846182 is queued on orcinus-nowcast-agrif
2020-07-03 11:02:03,437 INFO [watch_NEMO_agrif] 03jul20nowcast-agrif job 10846182 is queued on orcinus-nowcast-agrif
2020-07-03 11:03:03,678 INFO [watch_NEMO_agrif] 03jul20nowcast-agrif job 10846182 is queued on orcinus-nowcast-agrif
2020-07-03 11:03:20,889 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15749934 = 2020-07-03 06:28:54 UTC estimated time to finish: 0000:04:03:04
2020-07-03 11:03:38,679 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39330000 = 2020-07-03 01:30:00 UTC estimated time to finish: 0000:22:19:36
2020-07-03 11:04:03,921 INFO [watch_NEMO_agrif] 03jul20nowcast-agrif job 10846182 is queued on orcinus-nowcast-agrif
2020-07-03 11:05:04,106 INFO [watch_NEMO_agrif] 03jul20nowcast-agrif job 10846182 is queued on orcinus-nowcast-agrif
2020-07-03 11:05:30,653 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 7496932 = 2020-07-03 19:14:00 UTC, 80.2% complete
2020-07-03 11:06:04,365 INFO [watch_NEMO_agrif] 03jul20nowcast-agrif job 10846182 is queued on orcinus-nowcast-agrif
2020-07-03 11:07:04,629 INFO [watch_NEMO_agrif] 03jul20nowcast-agrif job 10846182 is queued on orcinus-nowcast-agrif
2020-07-03 11:08:04,843 INFO [watch_NEMO_agrif] 03jul20nowcast-agrif job 10846182 is queued on orcinus-nowcast-agrif
2020-07-03 11:08:21,003 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15750876 = 2020-07-03 06:44:36 UTC estimated time to finish: 0000:04:01:35
2020-07-03 11:08:38,787 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39330719 = 2020-07-03 01:34:47 UTC estimated time to finish: 0000:22:21:01
2020-07-03 11:09:05,023 INFO [watch_NEMO_agrif] 03jul20nowcast-agrif job 10846182 is queued on orcinus-nowcast-agrif
2020-07-03 11:10:05,266 INFO [watch_NEMO_agrif] 03jul20nowcast-agrif job 10846182 is queued on orcinus-nowcast-agrif
2020-07-03 11:10:30,755 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 7497087 = 2020-07-03 20:57:20 UTC, 87.4% complete
2020-07-03 11:11:05,503 INFO [watch_NEMO_agrif] 03jul20nowcast-agrif job 10846182 is queued on orcinus-nowcast-agrif
2020-07-03 11:12:05,760 INFO [watch_NEMO_agrif] 03jul20nowcast-agrif job 10846182 is queued on orcinus-nowcast-agrif
2020-07-03 11:13:06,007 INFO [watch_NEMO_agrif] 03jul20nowcast-agrif job 10846182 is queued on orcinus-nowcast-agrif
2020-07-03 11:13:21,118 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15751980 = 2020-07-03 07:03:00 UTC estimated time to finish: 0000:03:59:43
2020-07-03 11:13:38,900 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39331429 = 2020-07-03 01:39:31 UTC estimated time to finish: 0000:22:22:27
2020-07-03 11:14:06,265 INFO [watch_NEMO_agrif] 03jul20nowcast-agrif job 10846182 is queued on orcinus-nowcast-agrif
2020-07-03 11:15:06,525 INFO [watch_NEMO_agrif] 03jul20nowcast-agrif job 10846182 is queued on orcinus-nowcast-agrif
2020-07-03 11:15:30,857 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 7497241 = 2020-07-03 22:40:00 UTC, 94.5% complete
2020-07-03 11:16:06,791 INFO [watch_NEMO_agrif] 03jul20nowcast-agrif job 10846182 is queued on orcinus-nowcast-agrif
2020-07-03 11:17:07,003 INFO [watch_NEMO_agrif] 03jul20nowcast-agrif job 10846182 is queued on orcinus-nowcast-agrif
2020-07-03 11:18:07,261 INFO [watch_NEMO_agrif] 03jul20nowcast-agrif job 10846182 is queued on orcinus-nowcast-agrif
2020-07-03 11:18:21,186 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15753042 = 2020-07-03 07:20:42 UTC estimated time to finish: 0000:03:57:41
2020-07-03 11:18:39,011 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39332057 = 2020-07-03 01:43:42 UTC estimated time to finish: 0000:22:21:28
2020-07-03 11:19:07,527 INFO [watch_NEMO_agrif] 03jul20nowcast-agrif job 10846182 is queued on orcinus-nowcast-agrif
2020-07-03 11:20:07,788 INFO [watch_NEMO_agrif] 03jul20nowcast-agrif job 10846182 is queued on orcinus-nowcast-agrif
2020-07-03 11:20:30,958 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 7497360 = 2020-07-03 23:59:20 UTC, 100.0% complete
2020-07-03 11:21:08,053 INFO [watch_NEMO_agrif] 03jul20nowcast-agrif job 10846182 is queued on orcinus-nowcast-agrif
2020-07-03 11:22:08,287 INFO [watch_NEMO_agrif] 03jul20nowcast-agrif job 10846182 is queued on orcinus-nowcast-agrif
2020-07-03 11:23:08,537 INFO [watch_NEMO_agrif] 03jul20nowcast-agrif job 10846182 is queued on orcinus-nowcast-agrif
2020-07-03 11:23:21,294 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15754107 = 2020-07-03 07:38:27 UTC estimated time to finish: 0000:03:55:07
2020-07-03 11:23:39,117 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39332880 = 2020-07-03 01:49:12 UTC estimated time to finish: 0000:22:11:15
2020-07-03 11:24:08,795 INFO [watch_NEMO_agrif] 03jul20nowcast-agrif job 10846182 is queued on orcinus-nowcast-agrif
2020-07-03 11:25:09,060 INFO [watch_NEMO_agrif] 03jul20nowcast-agrif job 10846182 is queued on orcinus-nowcast-agrif
2020-07-03 11:25:31,067 INFO [watch_NEMO] nowcast-green NEMO run on arbutus.cloud-nowcast completed
2020-07-03 11:25:31,071 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2020-07-03 11:25:31,567 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_wind_file', args=['arbutus.cloud-nowcast', 'forecast'], host='arbutus.cloud-nowcast')
2020-07-03 11:25:31,740 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_current_file', args=['arbutus.cloud-nowcast', 'forecast'], host='arbutus.cloud-nowcast')
2020-07-03 11:25:31,742 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['salish-nowcast', 'nowcast+', '--shared-storage'], host='localhost')
2020-07-03 11:25:31,746 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2020-07-03'], host='localhost')
2020-07-03 11:25:32,693 INFO [make_forcing_links] running in process 7979
2020-07-03 11:25:32,693 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 11:25:32,694 INFO [make_forcing_links] publishing log messages to tcp://*:5564
2020-07-03 11:25:32,694 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2020-07-03 11:25:32,760 INFO [download_results] running in process 7980
2020-07-03 11:25:32,761 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 11:25:32,761 INFO [download_results] publishing log messages to tcp://*:5568
2020-07-03 11:25:32,762 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2020-07-03 11:25:32,763 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/nowcast-green/03jul20 to /results2/SalishSea/nowcast-green.201905
2020-07-03 11:25:33,725 INFO [make_forcing_links] nowcast+ 2020-07-03 forcing file links on salish-nowcast created
2020-07-03 11:25:33,729 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2020-07-03 11:25:33,961 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['salish-nowcast', 'nowcast-dev', '--run-date', '2020-07-03'], host='salish-nowcast')
2020-07-03 11:25:35,337 INFO [make_ww3_wind_file] running in process 27888
2020-07-03 11:25:35,338 INFO [make_ww3_wind_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 11:25:35,338 INFO [make_ww3_wind_file] publishing log messages to tcp://*:5570
2020-07-03 11:25:35,346 INFO [make_ww3_wind_file] connected to skookum.eos.ubc.ca port 5555
2020-07-03 11:25:35,347 INFO [make_ww3_wind_file] Creating wwatch3 wind forcing file for 2020-07-03 forecast run
2020-07-03 11:25:35,512 INFO [make_ww3_current_file] running in process 27886
2020-07-03 11:25:35,513 INFO [make_ww3_current_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 11:25:35,514 INFO [make_ww3_current_file] publishing log messages to tcp://*:5571
2020-07-03 11:25:35,515 INFO [make_ww3_current_file] connected to skookum.eos.ubc.ca port 5555
2020-07-03 11:25:35,516 INFO [make_ww3_current_file] Creating wwatch3 currents forcing file for 2020-07-03 forecast run
2020-07-03 11:25:35,532 INFO [run_NEMO] running in process 2480
2020-07-03 11:25:35,533 INFO [run_NEMO] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 11:25:35,534 INFO [run_NEMO] publishing log messages to tcp://*:5556
2020-07-03 11:25:35,535 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2020-07-03 11:25:36,713 WARNING [run_NEMO] There are uncommitted changes in /SalishSeaCast/SalishSeaNowcast
2020-07-03 11:25:37,471 INFO [run_NEMO] nowcast-dev: launching /SalishSeaCast/runs/03jul20nowcast-dev_2020-07-03T112535.707273-0700/SalishSeaNEMO.sh on salish-nowcast
2020-07-03 11:25:37,573 INFO [run_NEMO] nowcast-dev NEMO run for 2020-07-03 on salish-nowcast started
2020-07-03 11:25:37,578 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2020-07-03 11:25:37,838 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['salish-nowcast', 'nowcast-dev'], host='salish-nowcast')
2020-07-03 11:25:38,965 INFO [make_ww3_wind_file] wwatch3 wind forcing file created on arbutus.cloud-nowcast for 2020-07-03 forecast run
2020-07-03 11:25:39,188 INFO [watch_NEMO] running in process 2732
2020-07-03 11:25:39,395 INFO [manager] checklist updated with [WW3 wind forcing] items from make_ww3_wind_file worker
2020-07-03 11:25:39,396 INFO [watch_NEMO] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 11:25:39,397 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2020-07-03 11:25:39,398 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2020-07-03 11:25:39,426 INFO [watch_NEMO] nowcast-dev on salish-nowcast: time.step not found; continuing to watch...
2020-07-03 11:26:09,313 INFO [watch_NEMO_agrif] 03jul20nowcast-agrif job 10846182 is queued on orcinus-nowcast-agrif
2020-07-03 11:26:32,624 INFO [make_ww3_current_file] wwatch3 currents forcing file created on arbutus.cloud-nowcast for 2020-07-03 forecast run
2020-07-03 11:26:32,629 INFO [manager] checklist updated with [WW3 currents forcing] items from make_ww3_current_file worker
2020-07-03 11:26:32,917 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2020-07-03'], host='arbutus.cloud-nowcast')
2020-07-03 11:26:34,859 INFO [run_ww3] running in process 28093
2020-07-03 11:26:34,861 INFO [run_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 11:26:34,861 INFO [run_ww3] publishing log messages to tcp://*:5572
2020-07-03 11:26:34,862 INFO [run_ww3] connected to skookum.eos.ubc.ca port 5555
2020-07-03 11:26:34,864 INFO [run_ww3] Created run directory /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/nowcast_2020-07-03T112634.856208-0700
2020-07-03 11:26:34,865 INFO [run_ww3] nowcast: launching /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/nowcast_2020-07-03T112634.856208-0700/SoGWW3.sh on arbutus.cloud-nowcast
2020-07-03 11:26:34,881 INFO [run_ww3] nowcast WaveWatch3 run for 2020-07-03 on arbutus.cloud-nowcast started
2020-07-03 11:26:34,886 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker
2020-07-03 11:26:35,134 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'nowcast'], host='arbutus.cloud-nowcast')
2020-07-03 11:26:37,199 INFO [watch_ww3] running in process 28197
2020-07-03 11:26:37,200 INFO [watch_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 11:26:37,201 INFO [watch_ww3] publishing log messages to tcp://*:5573
2020-07-03 11:26:37,202 INFO [watch_ww3] connected to skookum.eos.ubc.ca port 5555
2020-07-03 11:26:37,236 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2020-07-03 11:27:09,516 INFO [watch_NEMO_agrif] 03jul20nowcast-agrif job 10846182 is queued on orcinus-nowcast-agrif
2020-07-03 11:28:09,774 INFO [watch_NEMO_agrif] 03jul20nowcast-agrif job 10846182 is queued on orcinus-nowcast-agrif
2020-07-03 11:28:21,404 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15755089 = 2020-07-03 07:54:49 UTC estimated time to finish: 0000:03:52:49
2020-07-03 11:28:39,223 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39333750 = 2020-07-03 01:55:00 UTC estimated time to finish: 0000:21:57:22
2020-07-03 11:29:09,986 INFO [watch_NEMO_agrif] 03jul20nowcast-agrif job 10846182 is queued on orcinus-nowcast-agrif
2020-07-03 11:29:27,472 INFO [download_results] nowcast-green 2020-07-03 results files from arbutus.cloud-nowcast downloaded
2020-07-03 11:29:27,480 INFO [manager] checklist updated with [results files] items from download_results worker
2020-07-03 11:29:27,750 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast-green', 'research', '--run-date', '2020-07-03'], host='localhost')
2020-07-03 11:29:27,754 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nowcast-green'], host='localhost')
2020-07-03 11:29:28,942 INFO [ping_erddap] running in process 8101
2020-07-03 11:29:28,943 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 11:29:28,943 INFO [ping_erddap] publishing log messages to tcp://*:5564
2020-07-03 11:29:28,944 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2020-07-03 11:29:28,948 INFO [ping_erddap] nowcast-green ERDDAP dataset flag file(s) created
2020-07-03 11:29:28,949 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2020-07-03 11:29:31,915 INFO [make_plots] running in process 8100
2020-07-03 11:29:31,916 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 11:29:31,917 INFO [make_plots] publishing log messages to tcp://*:5564
2020-07-03 11:29:31,917 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2020-07-03 11:29:32,311 INFO [make_plots] preparing render list for 2020-07-03 NEMO nowcast-green research figures
2020-07-03 11:29:35,726 INFO [make_plots] added temperature_thalweg_and_surface figures to 2020-07-03 NEMO nowcast-green research render list
2020-07-03 11:29:36,984 INFO [make_plots] added nitrate_thalweg_and_surface figures to 2020-07-03 NEMO nowcast-green research render list
2020-07-03 11:29:38,136 INFO [make_plots] added ammonium_thalweg_and_surface figures to 2020-07-03 NEMO nowcast-green research render list
2020-07-03 11:29:39,314 INFO [make_plots] added silicon_thalweg_and_surface figures to 2020-07-03 NEMO nowcast-green research render list
2020-07-03 11:29:40,696 INFO [make_plots] added dissolved_organic_nitrogen_thalweg_and_surface figures to 2020-07-03 NEMO nowcast-green research render list
2020-07-03 11:29:42,031 INFO [make_plots] added particulate_organic_nitrogen_thalweg_and_surface figures to 2020-07-03 NEMO nowcast-green research render list
2020-07-03 11:29:43,303 INFO [make_plots] added biogenic_silicon_thalweg_and_surface figures to 2020-07-03 NEMO nowcast-green research render list
2020-07-03 11:29:44,708 INFO [make_plots] added diatoms_thalweg_and_surface figures to 2020-07-03 NEMO nowcast-green research render list
2020-07-03 11:29:46,019 INFO [make_plots] added ciliates_thalweg_and_surface figures to 2020-07-03 NEMO nowcast-green research render list
2020-07-03 11:29:47,326 INFO [make_plots] added flagellates_thalweg_and_surface figures to 2020-07-03 NEMO nowcast-green research render list
2020-07-03 11:29:48,745 INFO [make_plots] added microzooplankton_thalweg_and_surface figures to 2020-07-03 NEMO nowcast-green research render list
2020-07-03 11:29:50,104 INFO [make_plots] added mesozooplankton_thalweg_and_surface figures to 2020-07-03 NEMO nowcast-green research render list
2020-07-03 11:29:51,399 INFO [make_plots] added Fraser_tracer_thalweg_and_surface figures to 2020-07-03 NEMO nowcast-green research render list
2020-07-03 11:29:51,526 INFO [make_plots] added temperature_salinity_timeseries figure to 2020-07-03 NEMO nowcast-green research render list
2020-07-03 11:29:51,527 INFO [make_plots] added nitrate_diatoms_timeseries figure to 2020-07-03 NEMO nowcast-green research render list
2020-07-03 11:29:51,528 INFO [make_plots] added mesodinium_flagellates_timeseries figure to 2020-07-03 NEMO nowcast-green research render list
2020-07-03 11:29:51,529 INFO [make_plots] added mesozoo_microzoo_timeseries figure to 2020-07-03 NEMO nowcast-green research render list
2020-07-03 11:29:51,530 INFO [make_plots] starting to render nemo nowcast-green research 03jul20 figures
2020-07-03 11:30:10,249 INFO [watch_NEMO_agrif] 03jul20nowcast-agrif job 10846182 is queued on orcinus-nowcast-agrif
2020-07-03 11:30:39,527 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4624591 = 2020-07-03 00:20:00 UTC, 1.4% complete
2020-07-03 11:31:10,445 INFO [watch_NEMO_agrif] 03jul20nowcast-agrif job 10846182 is queued on orcinus-nowcast-agrif
2020-07-03 11:31:37,337 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 99, 22.9% complete
2020-07-03 11:32:10,708 INFO [watch_NEMO_agrif] 03jul20nowcast-agrif job 10846182 is queued on orcinus-nowcast-agrif
2020-07-03 11:33:10,969 INFO [watch_NEMO_agrif] 03jul20nowcast-agrif job 10846182 is queued on orcinus-nowcast-agrif
2020-07-03 11:33:21,518 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15756120 = 2020-07-03 08:12:00 UTC estimated time to finish: 0000:03:50:21
2020-07-03 11:33:39,255 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39334664 = 2020-07-03 02:01:05 UTC estimated time to finish: 0000:21:43:01
2020-07-03 11:34:11,172 INFO [watch_NEMO_agrif] 03jul20nowcast-agrif job 10846182 is queued on orcinus-nowcast-agrif
2020-07-03 11:35:11,429 INFO [watch_NEMO_agrif] 03jul20nowcast-agrif job 10846182 is queued on orcinus-nowcast-agrif
2020-07-03 11:35:39,560 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4624640 = 2020-07-03 00:52:40 UTC, 3.7% complete
2020-07-03 11:36:11,632 INFO [watch_NEMO_agrif] 03jul20nowcast-agrif job 10846182 is queued on orcinus-nowcast-agrif
2020-07-03 11:36:37,437 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 210, 48.6% complete
2020-07-03 11:37:11,898 INFO [watch_NEMO_agrif] 03jul20nowcast-agrif job 10846182 is queued on orcinus-nowcast-agrif
2020-07-03 11:38:12,156 INFO [watch_NEMO_agrif] 03jul20nowcast-agrif job 10846182 is queued on orcinus-nowcast-agrif
2020-07-03 11:38:21,630 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15757215 = 2020-07-03 08:30:15 UTC estimated time to finish: 0000:03:47:22
2020-07-03 11:38:39,362 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39335525 = 2020-07-03 02:06:50 UTC estimated time to finish: 0000:21:31:39
2020-07-03 11:39:12,414 INFO [watch_NEMO_agrif] 03jul20nowcast-agrif job 10846182 is queued on orcinus-nowcast-agrif
2020-07-03 11:40:12,622 INFO [watch_NEMO_agrif] 03jul20nowcast-agrif job 10846182 is queued on orcinus-nowcast-agrif
2020-07-03 11:40:39,592 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4624693 = 2020-07-03 01:28:00 UTC, 6.1% complete
2020-07-03 11:41:12,888 INFO [watch_NEMO_agrif] 03jul20nowcast-agrif job 10846182 is queued on orcinus-nowcast-agrif
2020-07-03 11:41:37,539 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 318, 73.6% complete
2020-07-03 11:42:13,153 INFO [watch_NEMO_agrif] 03jul20nowcast-agrif job 10846182 is queued on orcinus-nowcast-agrif
2020-07-03 11:43:13,362 INFO [watch_NEMO_agrif] 03jul20nowcast-agrif job 10846182 is queued on orcinus-nowcast-agrif
2020-07-03 11:43:21,740 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15758280 = 2020-07-03 08:48:00 UTC estimated time to finish: 0000:03:44:13
2020-07-03 11:43:39,464 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39336276 = 2020-07-03 02:11:50 UTC estimated time to finish: 0000:21:21:29
2020-07-03 11:44:13,628 INFO [watch_NEMO_agrif] 03jul20nowcast-agrif job 10846182 is queued on orcinus-nowcast-agrif
2020-07-03 11:45:13,887 INFO [watch_NEMO_agrif] 03jul20nowcast-agrif job 10846182 is queued on orcinus-nowcast-agrif
2020-07-03 11:45:39,692 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4624747 = 2020-07-03 02:04:00 UTC, 8.6% complete
2020-07-03 11:46:14,151 INFO [watch_NEMO_agrif] 03jul20nowcast-agrif job 10846182 is queued on orcinus-nowcast-agrif
2020-07-03 11:46:37,639 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 426, 98.6% complete
2020-07-03 11:47:14,408 INFO [watch_NEMO_agrif] 03jul20nowcast-agrif job 10846182 is queued on orcinus-nowcast-agrif
2020-07-03 11:48:14,614 INFO [watch_NEMO_agrif] 03jul20nowcast-agrif job 10846182 is queued on orcinus-nowcast-agrif
2020-07-03 11:48:21,850 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15759232 = 2020-07-03 09:03:52 UTC estimated time to finish: 0000:03:41:31
2020-07-03 11:48:39,571 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39337190 = 2020-07-03 02:17:56 UTC estimated time to finish: 0000:21:09:16
2020-07-03 11:49:14,878 INFO [watch_NEMO_agrif] 03jul20nowcast-agrif job 10846182 is queued on orcinus-nowcast-agrif
2020-07-03 11:50:15,089 INFO [watch_NEMO_agrif] 03jul20nowcast-agrif job 10846182 is queued on orcinus-nowcast-agrif
2020-07-03 11:50:39,713 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4624801 = 2020-07-03 02:40:00 UTC, 11.1% complete
2020-07-03 11:51:15,309 INFO [watch_NEMO_agrif] 03jul20nowcast-agrif job 10846182 is queued on orcinus-nowcast-agrif
2020-07-03 11:51:37,741 INFO [watch_ww3] nowcast WWATCH3 run on arbutus.cloud-nowcast completed
2020-07-03 11:51:37,818 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2020-07-03 11:51:38,408 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2020-07-03'], host='localhost')
2020-07-03 11:51:38,420 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2020-07-03'], host='arbutus.cloud-nowcast')
2020-07-03 11:51:40,258 INFO [download_wwatch3_results] running in process 8644
2020-07-03 11:51:40,274 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 11:51:40,275 INFO [download_wwatch3_results] publishing log messages to tcp://*:5568
2020-07-03 11:51:40,276 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2020-07-03 11:51:40,476 INFO [run_ww3] running in process 28364
2020-07-03 11:51:40,477 INFO [run_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 11:51:40,478 INFO [run_ww3] publishing log messages to tcp://*:5572
2020-07-03 11:51:40,479 INFO [run_ww3] connected to skookum.eos.ubc.ca port 5555
2020-07-03 11:51:40,481 INFO [run_ww3] Created run directory /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast_2020-07-03T115140.473445-0700
2020-07-03 11:51:40,482 INFO [run_ww3] forecast: launching /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast_2020-07-03T115140.473445-0700/SoGWW3.sh on arbutus.cloud-nowcast
2020-07-03 11:51:40,499 INFO [run_ww3] forecast WaveWatch3 run for 2020-07-03 on arbutus.cloud-nowcast started
2020-07-03 11:51:40,504 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker
2020-07-03 11:51:40,764 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'forecast'], host='arbutus.cloud-nowcast')
2020-07-03 11:51:42,666 INFO [watch_ww3] running in process 28553
2020-07-03 11:51:42,756 INFO [watch_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 11:51:42,757 INFO [watch_ww3] publishing log messages to tcp://*:5573
2020-07-03 11:51:42,757 INFO [watch_ww3] connected to skookum.eos.ubc.ca port 5555
2020-07-03 11:51:42,759 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2020-07-03 11:52:15,529 INFO [watch_NEMO_agrif] 03jul20nowcast-agrif job 10846182 is queued on orcinus-nowcast-agrif
2020-07-03 11:52:30,164 INFO [download_wwatch3_results] nowcast 2020-07-03 results files from arbutus.cloud-nowcast downloaded
2020-07-03 11:52:30,168 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2020-07-03 11:53:15,743 INFO [watch_NEMO_agrif] 03jul20nowcast-agrif job 10846182 is queued on orcinus-nowcast-agrif
2020-07-03 11:53:21,960 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15760295 = 2020-07-03 09:21:35 UTC estimated time to finish: 0000:03:38:19
2020-07-03 11:53:39,678 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39338051 = 2020-07-03 02:23:40 UTC estimated time to finish: 0000:20:59:08
2020-07-03 11:54:16,002 INFO [watch_NEMO_agrif] 03jul20nowcast-agrif job 10846182 is queued on orcinus-nowcast-agrif
2020-07-03 11:55:16,200 INFO [watch_NEMO_agrif] 03jul20nowcast-agrif job 10846182 is queued on orcinus-nowcast-agrif
2020-07-03 11:55:39,812 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4624855 = 2020-07-03 03:16:00 UTC, 13.6% complete
2020-07-03 11:56:16,470 INFO [watch_NEMO_agrif] 03jul20nowcast-agrif job 10846182 is queued on orcinus-nowcast-agrif
2020-07-03 11:56:42,803 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 99, 15.3% complete
2020-07-03 11:57:16,734 INFO [watch_NEMO_agrif] 03jul20nowcast-agrif job 10846182 is queued on orcinus-nowcast-agrif
2020-07-03 11:58:16,947 INFO [watch_NEMO_agrif] 03jul20nowcast-agrif job 10846182 is queued on orcinus-nowcast-agrif
2020-07-03 11:58:22,071 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15761359 = 2020-07-03 09:39:19 UTC estimated time to finish: 0000:03:34:51
2020-07-03 11:58:39,786 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39338925 = 2020-07-03 02:29:30 UTC estimated time to finish: 0000:20:48:28
2020-07-03 11:59:17,215 INFO [watch_NEMO_agrif] 03jul20nowcast-agrif job 10846182 is queued on orcinus-nowcast-agrif
2020-07-03 12:00:17,440 INFO [watch_NEMO_agrif] 03jul20nowcast-agrif job 10846182 is queued on orcinus-nowcast-agrif
2020-07-03 12:00:39,912 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4624909 = 2020-07-03 03:52:00 UTC, 16.1% complete
2020-07-03 12:01:18,401 INFO [watch_NEMO_agrif] 03jul20nowcast-agrif on orcinus-nowcast-agrif: time.step not found; continuing to watch...
2020-07-03 12:01:42,869 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 210, 32.4% complete
2020-07-03 12:03:22,182 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15762380 = 2020-07-03 09:56:20 UTC estimated time to finish: 0000:03:31:49
2020-07-03 12:03:39,894 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39339717 = 2020-07-03 02:34:46 UTC estimated time to finish: 0000:20:40:03
2020-07-03 12:05:40,012 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4624964 = 2020-07-03 04:28:40 UTC, 18.7% complete
2020-07-03 12:06:18,953 INFO [watch_NEMO_agrif] 03jul20nowcast-agrif on orcinus-nowcast-agrif: timestep: 4624778 = 2020-07-03 02:24:40 UTC, 10.1% complete
2020-07-03 12:06:42,970 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 315, 48.6% complete
2020-07-03 12:08:22,294 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15763444 = 2020-07-03 10:14:04 UTC estimated time to finish: 0000:03:28:24
2020-07-03 12:08:40,007 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39340577 = 2020-07-03 02:40:30 UTC estimated time to finish: 0000:20:30:45
2020-07-03 12:10:40,113 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4625018 = 2020-07-03 05:04:40 UTC, 21.2% complete
2020-07-03 12:11:19,413 INFO [watch_NEMO_agrif] 03jul20nowcast-agrif on orcinus-nowcast-agrif: timestep: 4625028 = 2020-07-03 05:11:20 UTC, 21.6% complete
2020-07-03 12:11:43,071 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 423, 65.3% complete
2020-07-03 12:13:22,405 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15764398 = 2020-07-03 10:29:58 UTC estimated time to finish: 0000:03:25:10
2020-07-03 12:13:40,117 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39341450 = 2020-07-03 02:46:20 UTC estimated time to finish: 0000:20:21:06
2020-07-03 12:15:40,214 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4625072 = 2020-07-03 05:40:40 UTC, 23.7% complete
2020-07-03 12:16:19,991 INFO [watch_NEMO_agrif] 03jul20nowcast-agrif on orcinus-nowcast-agrif: timestep: 4625280 = 2020-07-03 07:59:20 UTC, 33.3% complete
2020-07-03 12:16:43,172 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 531, 81.9% complete
2020-07-03 12:18:22,519 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15765420 = 2020-07-03 10:47:00 UTC estimated time to finish: 0000:03:21:44
2020-07-03 12:18:40,225 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39342352 = 2020-07-03 02:52:20 UTC estimated time to finish: 0000:20:10:59
2020-07-03 12:20:40,312 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4625127 = 2020-07-03 06:17:20 UTC, 26.2% complete
2020-07-03 12:21:20,532 INFO [watch_NEMO_agrif] 03jul20nowcast-agrif on orcinus-nowcast-agrif: timestep: 4625533 = 2020-07-03 10:48:00 UTC, 45.0% complete
2020-07-03 12:21:43,273 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 639, 98.6% complete
2020-07-03 12:23:22,647 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15766675 = 2020-07-03 11:07:55 UTC estimated time to finish: 0000:03:16:14
2020-07-03 12:23:40,334 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39343226 = 2020-07-03 02:58:10 UTC estimated time to finish: 0000:19:58:59
2020-07-03 12:25:40,412 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4625181 = 2020-07-03 06:53:20 UTC, 28.7% complete
2020-07-03 12:26:43,375 INFO [watch_ww3] forecast WWATCH3 run on arbutus.cloud-nowcast completed
2020-07-03 12:26:43,378 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2020-07-03 12:26:43,902 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2020-07-03'], host='localhost')
2020-07-03 12:26:45,524 INFO [download_wwatch3_results] running in process 9714
2020-07-03 12:26:45,624 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 12:26:45,625 INFO [download_wwatch3_results] publishing log messages to tcp://*:5563
2020-07-03 12:26:45,626 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2020-07-03 12:26:48,728 INFO [watch_NEMO_agrif] 03jul20nowcast-agrif on orcinus-nowcast-agrif: timestep: 4625609 = 2020-07-03 11:38:40 UTC, 48.5% complete
2020-07-03 12:27:31,317 INFO [download_wwatch3_results] forecast 2020-07-03 results files from arbutus.cloud-nowcast downloaded
2020-07-03 12:27:31,322 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2020-07-03 12:27:31,667 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['wwatch3', 'forecast', '--run-date', '2020-07-03'], host='localhost')
2020-07-03 12:27:32,662 INFO [update_forecast_datasets] running in process 9726
2020-07-03 12:27:32,663 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 12:27:32,664 INFO [update_forecast_datasets] publishing log messages to tcp://*:5563
2020-07-03 12:27:32,665 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2020-07-03 12:27:32,665 INFO [update_forecast_datasets] updating wwatch3 most_recent_forecast directory from forecast/03jul20 run
2020-07-03 12:27:32,667 INFO [update_forecast_datasets] updated wwatch3 most_recent_forecast directory from forecast/03jul20 run
2020-07-03 12:27:32,667 INFO [update_forecast_datasets] updating wwatch3 forecast directory for forecast/03jul20 run
2020-07-03 12:27:32,787 INFO [update_forecast_datasets] updated wwatch3 forecast directory for forecast/03jul20 run: /results/SalishSea/rolling-forecasts/wwatch3
2020-07-03 12:27:32,788 INFO [update_forecast_datasets] wwatch3 2020-07-03 forecast rolling forecast datasets updated
2020-07-03 12:27:32,792 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2020-07-03 12:27:33,130 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['wwatch3-forecast'], host='localhost')
2020-07-03 12:27:34,099 INFO [ping_erddap] running in process 9730
2020-07-03 12:27:34,112 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 12:27:34,113 INFO [ping_erddap] publishing log messages to tcp://*:5563
2020-07-03 12:27:34,114 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2020-07-03 12:27:34,115 INFO [ping_erddap] wwatch3-forecast ERDDAP dataset flag file(s) created
2020-07-03 12:27:34,117 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2020-07-03 12:27:34,394 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['wwatch3', 'forecast', 'publish', '--run-date', '2020-07-03'], host='localhost')
2020-07-03 12:27:41,392 INFO [make_plots] running in process 9734
2020-07-03 12:27:41,394 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 12:27:41,395 INFO [make_plots] publishing log messages to tcp://*:5563
2020-07-03 12:27:41,396 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2020-07-03 12:27:41,397 INFO [make_plots] preparing render list for 2020-07-03 WaveWatch3 forecast publish figures
2020-07-03 12:27:41,398 INFO [make_plots] added Halibut Bank figure to 2020-07-03 WaveWatch3 forecast publish render list
2020-07-03 12:27:41,398 INFO [make_plots] added Sentry Shoal figure to 2020-07-03 WaveWatch3 forecast publish render list
2020-07-03 12:27:41,399 INFO [make_plots] starting to render wwatch3 forecast publish 03jul20 figures
2020-07-03 12:28:22,761 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15768052 = 2020-07-03 11:30:52 UTC estimated time to finish: 0000:03:09:39
2020-07-03 12:28:40,443 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39344169 = 2020-07-03 03:04:27 UTC estimated time to finish: 0000:19:47:08
2020-07-03 12:28:48,256 INFO [make_plots] Note: NumExpr detected 12 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
2020-07-03 12:28:48,257 INFO [make_plots] NumExpr defaulting to 8 threads.
2020-07-03 12:28:48,265 INFO [make_plots] retrieving last 45 days of real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46146 Halibut Bank
2020-07-03 12:29:55,099 INFO [make_plots] retrieving last 45 days of real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46131 Sentry Shoal
2020-07-03 12:29:56,437 INFO [make_plots] finished rendering wwatch3 forecast publish 03jul20 figures
2020-07-03 12:29:56,438 INFO [make_plots] wwatch3 publish plots for 2020-07-03 forecast completed
2020-07-03 12:29:56,442 INFO [manager] checklist updated with [plots] items from make_plots worker
2020-07-03 12:30:40,513 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4625235 = 2020-07-03 07:29:20 UTC, 31.2% complete
2020-07-03 12:32:09,054 ERROR [watch_NEMO_agrif] pbs_iff: cannot read reply from pbs_server
2020-07-03 12:32:09,676 ERROR [watch_NEMO_agrif] No Permission.
2020-07-03 12:32:09,681 ERROR [watch_NEMO_agrif] qstat: cannot connect to server orca2.ibb (errno=15007) Unauthorized Request
2020-07-03 12:32:09,686 CRITICAL [watch_NEMO_agrif] NEMO AGRIF run 10846182.orca2.ibb on orcinus-nowcast-agrif watcher failed
2020-07-03 12:33:22,874 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15768965 = 2020-07-03 11:46:05 UTC estimated time to finish: 0000:03:06:38
2020-07-03 12:33:40,552 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39345041 = 2020-07-03 03:10:16 UTC estimated time to finish: 0000:19:38:52
2020-07-03 12:35:40,611 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4625290 = 2020-07-03 08:06:00 UTC, 33.8% complete
2020-07-03 12:38:22,988 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15770070 = 2020-07-03 12:04:30 UTC estimated time to finish: 0000:03:02:29
2020-07-03 12:38:40,618 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39345902 = 2020-07-03 03:16:00 UTC estimated time to finish: 0000:19:31:36
2020-07-03 12:40:40,713 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4625344 = 2020-07-03 08:42:00 UTC, 36.3% complete
2020-07-03 12:43:23,102 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15771091 = 2020-07-03 12:21:31 UTC estimated time to finish: 0000:02:58:47
2020-07-03 12:43:40,727 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39346776 = 2020-07-03 03:21:50 UTC estimated time to finish: 0000:19:23:43
2020-07-03 12:45:40,727 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4625398 = 2020-07-03 09:18:00 UTC, 38.8% complete
2020-07-03 12:48:23,215 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15772156 = 2020-07-03 12:39:16 UTC estimated time to finish: 0000:02:54:42
2020-07-03 12:48:40,837 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39347568 = 2020-07-03 03:27:07 UTC estimated time to finish: 0000:19:16:59
2020-07-03 12:50:40,827 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4625452 = 2020-07-03 09:54:00 UTC, 41.3% complete
2020-07-03 12:53:23,258 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15773178 = 2020-07-03 12:56:18 UTC estimated time to finish: 0000:02:50:52
2020-07-03 12:53:40,946 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39348389 = 2020-07-03 03:32:35 UTC estimated time to finish: 0000:19:10:40
2020-07-03 12:54:01,734 INFO [make_plots] finished rendering nemo nowcast-green research 03jul20 figures
2020-07-03 12:54:01,902 INFO [make_plots] nemo research plots for 2020-07-03 nowcast-green completed
2020-07-03 12:54:02,045 INFO [manager] checklist updated with [plots] items from make_plots worker
2020-07-03 12:55:40,928 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4625506 = 2020-07-03 10:30:00 UTC, 43.8% complete
2020-07-03 12:58:23,371 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15774241 = 2020-07-03 13:14:01 UTC estimated time to finish: 0000:02:46:47
2020-07-03 12:58:41,056 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39349262 = 2020-07-03 03:38:24 UTC estimated time to finish: 0000:19:03:43
2020-07-03 13:00:40,948 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4625561 = 2020-07-03 11:06:40 UTC, 46.3% complete
2020-07-03 13:03:23,470 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15775195 = 2020-07-03 13:29:55 UTC estimated time to finish: 0000:02:43:01
2020-07-03 13:03:41,166 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39350136 = 2020-07-03 03:44:14 UTC estimated time to finish: 0000:18:57:10
2020-07-03 13:05:40,962 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4625616 = 2020-07-03 11:43:20 UTC, 48.9% complete
2020-07-03 13:08:23,585 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15776259 = 2020-07-03 13:47:39 UTC estimated time to finish: 0000:02:38:49
2020-07-03 13:08:41,277 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39350880 = 2020-07-03 03:49:12 UTC estimated time to finish: 0000:18:52:15
2020-07-03 13:10:41,062 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4625670 = 2020-07-03 12:19:20 UTC, 51.4% complete
2020-07-03 13:13:23,699 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15777321 = 2020-07-03 14:05:21 UTC estimated time to finish: 0000:02:34:37
2020-07-03 13:13:41,387 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39351748 = 2020-07-03 03:54:59 UTC estimated time to finish: 0000:18:45:30
2020-07-03 13:15:41,154 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4625725 = 2020-07-03 12:56:00 UTC, 53.9% complete
2020-07-03 13:18:23,815 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15778385 = 2020-07-03 14:23:05 UTC estimated time to finish: 0000:02:30:21
2020-07-03 13:18:41,498 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39352620 = 2020-07-03 04:00:48 UTC estimated time to finish: 0000:18:38:15
2020-07-03 13:20:41,163 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4625779 = 2020-07-03 13:32:00 UTC, 56.4% complete
2020-07-03 13:23:23,930 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15779367 = 2020-07-03 14:39:27 UTC estimated time to finish: 0000:02:26:19
2020-07-03 13:23:41,610 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39353481 = 2020-07-03 04:06:32 UTC estimated time to finish: 0000:18:31:38
2020-07-03 13:25:41,262 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4625834 = 2020-07-03 14:08:40 UTC, 59.0% complete
2020-07-03 13:28:24,047 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15780389 = 2020-07-03 14:56:29 UTC estimated time to finish: 0000:02:22:13
2020-07-03 13:28:41,722 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39354232 = 2020-07-03 04:11:32 UTC estimated time to finish: 0000:18:26:32
2020-07-03 13:30:41,363 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4625888 = 2020-07-03 14:44:40 UTC, 61.5% complete
2020-07-03 13:33:24,164 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15781452 = 2020-07-03 15:14:12 UTC estimated time to finish: 0000:02:17:53
2020-07-03 13:33:41,834 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39355106 = 2020-07-03 04:17:22 UTC estimated time to finish: 0000:18:19:44
2020-07-03 13:35:41,463 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4625943 = 2020-07-03 15:21:20 UTC, 64.0% complete
2020-07-03 13:38:24,281 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15782515 = 2020-07-03 15:31:55 UTC estimated time to finish: 0000:02:13:30
2020-07-03 13:38:41,946 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39356008 = 2020-07-03 04:23:23 UTC estimated time to finish: 0000:18:12:18
2020-07-03 13:40:41,564 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4625997 = 2020-07-03 15:57:20 UTC, 66.5% complete
2020-07-03 13:43:24,397 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15783469 = 2020-07-03 15:47:49 UTC estimated time to finish: 0000:02:09:33
2020-07-03 13:43:42,058 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39356881 = 2020-07-03 04:29:12 UTC estimated time to finish: 0000:18:05:50
2020-07-03 13:45:41,665 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4626052 = 2020-07-03 16:34:00 UTC, 69.1% complete
2020-07-03 13:48:24,514 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15784531 = 2020-07-03 16:05:31 UTC estimated time to finish: 0000:02:05:08
2020-07-03 13:48:42,171 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39357742 = 2020-07-03 04:34:56 UTC estimated time to finish: 0000:17:59:40
2020-07-03 13:50:41,763 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4626106 = 2020-07-03 17:10:00 UTC, 71.6% complete
2020-07-03 13:53:24,631 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15785595 = 2020-07-03 16:23:15 UTC estimated time to finish: 0000:02:00:35
2020-07-03 13:53:42,284 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39358452 = 2020-07-03 04:39:40 UTC estimated time to finish: 0000:17:55:20
2020-07-03 13:55:41,775 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4626161 = 2020-07-03 17:46:40 UTC, 74.1% complete
2020-07-03 13:58:24,749 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15786768 = 2020-07-03 16:42:48 UTC estimated time to finish: 0000:01:55:28
2020-07-03 13:58:42,397 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39359520 = 2020-07-03 04:46:48 UTC estimated time to finish: 0000:17:43:02
2020-07-03 14:00:41,876 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4626215 = 2020-07-03 18:22:40 UTC, 76.6% complete
2020-07-03 14:03:24,866 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15787912 = 2020-07-03 17:01:52 UTC estimated time to finish: 0000:01:50:26
2020-07-03 14:03:42,510 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39360827 = 2020-07-03 04:55:30 UTC estimated time to finish: 0000:17:24:16
2020-07-03 14:05:41,975 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4626270 = 2020-07-03 18:59:20 UTC, 79.2% complete
2020-07-03 14:08:24,903 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15789016 = 2020-07-03 17:20:16 UTC estimated time to finish: 0000:01:45:31
2020-07-03 14:08:42,633 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39362084 = 2020-07-03 05:03:53 UTC estimated time to finish: 0000:17:06:44
2020-07-03 14:10:42,075 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4626325 = 2020-07-03 19:36:00 UTC, 81.7% complete
2020-07-03 14:13:25,020 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15790202 = 2020-07-03 17:40:02 UTC estimated time to finish: 0000:01:40:14
2020-07-03 14:13:42,747 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39363271 = 2020-07-03 05:11:48 UTC estimated time to finish: 0000:16:50:46
2020-07-03 14:15:42,112 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4626379 = 2020-07-03 20:12:00 UTC, 84.2% complete
2020-07-03 14:18:25,139 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15791416 = 2020-07-03 18:00:16 UTC estimated time to finish: 0000:01:34:50
2020-07-03 14:18:42,861 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39364569 = 2020-07-03 05:20:27 UTC estimated time to finish: 0000:16:33:31
2020-07-03 14:20:42,171 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4626434 = 2020-07-03 20:48:40 UTC, 86.8% complete
2020-07-03 14:23:25,258 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15792601 = 2020-07-03 18:20:01 UTC estimated time to finish: 0000:01:29:37
2020-07-03 14:23:42,976 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39365880 = 2020-07-03 05:29:12 UTC estimated time to finish: 0000:16:17:06
2020-07-03 14:25:42,271 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4626489 = 2020-07-03 21:25:20 UTC, 89.3% complete
2020-07-03 14:28:25,378 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15793625 = 2020-07-03 18:37:05 UTC estimated time to finish: 0000:01:25:07
2020-07-03 14:28:43,091 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39367177 = 2020-07-03 05:37:50 UTC estimated time to finish: 0000:16:01:10
2020-07-03 14:30:42,371 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4626544 = 2020-07-03 22:02:00 UTC, 91.8% complete
2020-07-03 14:33:25,498 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15794810 = 2020-07-03 18:56:50 UTC estimated time to finish: 0000:01:19:54
2020-07-03 14:33:43,207 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39368364 = 2020-07-03 05:45:45 UTC estimated time to finish: 0000:15:47:12
2020-07-03 14:35:42,441 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4626599 = 2020-07-03 22:38:40 UTC, 94.4% complete
2020-07-03 14:38:25,618 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15795955 = 2020-07-03 19:15:55 UTC estimated time to finish: 0000:01:14:53
2020-07-03 14:38:43,260 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39369662 = 2020-07-03 05:54:24 UTC estimated time to finish: 0000:15:32:25
2020-07-03 14:40:42,541 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4626654 = 2020-07-03 23:15:20 UTC, 96.9% complete
2020-07-03 14:43:25,737 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15797060 = 2020-07-03 19:34:20 UTC estimated time to finish: 0000:01:10:00
2020-07-03 14:43:43,376 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39370972 = 2020-07-03 06:03:08 UTC estimated time to finish: 0000:15:17:46
2020-07-03 14:45:42,624 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4626708 = 2020-07-03 23:51:20 UTC, 99.4% complete
2020-07-03 14:48:25,859 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15798314 = 2020-07-03 19:55:14 UTC estimated time to finish: 0000:01:04:27
2020-07-03 14:48:43,472 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39372270 = 2020-07-03 06:11:48 UTC estimated time to finish: 0000:15:03:53
2020-07-03 14:50:42,697 INFO [watch_NEMO] nowcast-dev NEMO run on salish-nowcast completed
2020-07-03 14:50:42,704 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2020-07-03 14:53:25,979 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15799540 = 2020-07-03 20:15:40 UTC estimated time to finish: 0000:00:59:01
2020-07-03 14:53:43,587 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39373567 = 2020-07-03 06:20:26 UTC estimated time to finish: 0000:14:50:26
2020-07-03 14:58:26,099 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15800645 = 2020-07-03 20:34:05 UTC estimated time to finish: 0000:00:54:09
2020-07-03 14:58:43,703 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39374755 = 2020-07-03 06:28:22 UTC estimated time to finish: 0000:14:38:29
2020-07-03 15:03:26,220 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15801790 = 2020-07-03 20:53:10 UTC estimated time to finish: 0000:00:49:08
2020-07-03 15:03:43,820 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39376024 = 2020-07-03 06:36:49 UTC estimated time to finish: 0000:14:26:19
2020-07-03 15:08:17,314 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-west/18/ to /results/forcing/atmospheric/GEM2.5/GRIB/20200703/18/
2020-07-03 15:08:17,315 INFO [collect_weather] 2020-07-03 2.5km weather forecast 18 collection complete
2020-07-03 15:08:17,318 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2020-07-03 15:08:17,924 INFO [manager] launching NextWorker(module='nowcast.workers.download_weather', args=['00', '1km', '--no-verify-certs'], host='localhost')
2020-07-03 15:08:17,930 INFO [manager] launching NextWorker(module='nowcast.workers.download_weather', args=['12', '1km', '--no-verify-certs'], host='localhost')
2020-07-03 15:08:17,934 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['00', '2.5km'], host='localhost')
2020-07-03 15:08:19,247 INFO [download_weather] running in process 13430
2020-07-03 15:08:19,248 INFO [download_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 15:08:19,249 INFO [download_weather] publishing log messages to tcp://*:5562
2020-07-03 15:08:19,250 INFO [download_weather] connected to skookum.eos.ubc.ca port 5555
2020-07-03 15:08:19,251 INFO [download_weather] downloading 00 1 km forecast GRIB2 files for 20200703
2020-07-03 15:08:19,252 INFO [download_weather] running in process 13431
2020-07-03 15:08:19,253 INFO [download_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 15:08:19,254 INFO [download_weather] publishing log messages to tcp://*:5563
2020-07-03 15:08:19,255 INFO [download_weather] connected to skookum.eos.ubc.ca port 5555
2020-07-03 15:08:19,255 INFO [download_weather] downloading 12 1 km forecast GRIB2 files for 20200703
2020-07-03 15:08:19,257 INFO [collect_weather] running in process 13432
2020-07-03 15:08:19,258 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 15:08:19,258 INFO [collect_weather] publishing log messages to tcp://*:5564
2020-07-03 15:08:19,259 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2020-07-03 15:08:19,315 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-west/00/
2020-07-03 15:08:26,343 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15803003 = 2020-07-03 21:13:23 UTC estimated time to finish: 0000:00:43:48
2020-07-03 15:08:43,938 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39377321 = 2020-07-03 06:45:28 UTC estimated time to finish: 0000:14:13:38
2020-07-03 15:13:26,465 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15804149 = 2020-07-03 21:32:29 UTC estimated time to finish: 0000:00:38:47
2020-07-03 15:13:44,057 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39378620 = 2020-07-03 06:54:08 UTC estimated time to finish: 0000:14:01:25
2020-07-03 15:18:26,589 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15805334 = 2020-07-03 21:52:14 UTC estimated time to finish: 0000:00:33:35
2020-07-03 15:18:44,177 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39379766 = 2020-07-03 07:01:46 UTC estimated time to finish: 0000:13:51:08
2020-07-03 15:23:26,712 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15806357 = 2020-07-03 22:09:17 UTC estimated time to finish: 0000:00:29:06
2020-07-03 15:23:44,297 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39380872 = 2020-07-03 07:09:08 UTC estimated time to finish: 0000:13:43:01
2020-07-03 15:26:21,146 INFO [download_weather] 2020-07-03 1km weather forecast 00 downloads complete
2020-07-03 15:26:21,151 INFO [manager] checklist updated with [weather forecast] items from download_weather worker
2020-07-03 15:26:25,916 INFO [download_weather] 2020-07-03 1km weather forecast 12 downloads complete
2020-07-03 15:26:25,920 INFO [manager] checklist updated with [weather forecast] items from download_weather worker
2020-07-03 15:28:26,836 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15807420 = 2020-07-03 22:27:00 UTC estimated time to finish: 0000:00:24:29
2020-07-03 15:28:44,416 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39381746 = 2020-07-03 07:14:58 UTC estimated time to finish: 0000:13:39:01
2020-07-03 15:33:26,958 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15808443 = 2020-07-03 22:44:03 UTC estimated time to finish: 0000:00:20:02
2020-07-03 15:33:44,487 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39382606 = 2020-07-03 07:20:42 UTC estimated time to finish: 0000:13:35:29
2020-07-03 15:36:40,601 INFO [download_results] running in process 14443
2020-07-03 15:36:40,601 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 15:36:40,602 INFO [download_results] publishing log messages to tcp://*:5563
2020-07-03 15:36:40,603 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2020-07-03 15:36:40,603 INFO [download_results] downloading results from orcinus-nowcast-agrif:/global/scratch/dlatorne/nowcast-agrif/03jul20 to /results/SalishSea/nowcast-agrif.201702
2020-07-03 15:36:41,687 INFO [download_results] nowcast-agrif 2020-07-03 results files from orcinus-nowcast-agrif downloaded
2020-07-03 15:36:41,690 INFO [manager] checklist updated with [results files] items from download_results worker
2020-07-03 15:36:41,911 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast-agrif', 'research', '--run-date', '2020-07-03'], host='localhost')
2020-07-03 15:36:45,206 INFO [make_plots] running in process 14449
2020-07-03 15:36:45,206 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 15:36:45,207 INFO [make_plots] publishing log messages to tcp://*:5563
2020-07-03 15:36:45,208 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2020-07-03 15:36:45,778 INFO [make_plots] preparing render list for 2020-07-03 NEMO nowcast-agrif research figures
2020-07-03 15:36:45,779 INFO [make_plots] added baynes_sound_surface figure to 2020-07-03 NEMO nowcast-agrif research render list
2020-07-03 15:36:45,780 INFO [make_plots] starting to render nemo nowcast-agrif research 03jul20 figures
2020-07-03 15:36:45,890 ERROR [make_plots] unexpected FileNotFoundError in make_figure:
Traceback (most recent call last):
  File "/SalishSeaCast/nowcast-env/lib/python3.8/site-packages/xarray/backends/file_manager.py", line 198, in _acquire_with_cache_info
    file = self._cache[self._key]
  File "/SalishSeaCast/nowcast-env/lib/python3.8/site-packages/xarray/backends/lru_cache.py", line 53, in __getitem__
    value = self._cache[key]
KeyError: [<class 'netCDF4._netCDF4.Dataset'>, ('/results/SalishSea/nowcast-agrif.201702/03jul20/BaynesSoundSurface_grid_T.nc',), 'r', (('clobber', True), ('diskless', False), ('format', 'NETCDF4'), ('persist', False))]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure
    fig = fig_func(*args, **kwargs)
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/research/baynes_sound_agrif.py", line 87, in make_figure
    plot_data = _prep_plot_data(
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/research/baynes_sound_agrif.py", line 128, in _prep_plot_data
    ss_tracers = xarray.open_dataset(ss_tracers_path)
  File "/SalishSeaCast/nowcast-env/lib/python3.8/site-packages/xarray/backends/api.py", line 501, in open_dataset
    store = backends.NetCDF4DataStore.open(
  File "/SalishSeaCast/nowcast-env/lib/python3.8/site-packages/xarray/backends/netCDF4_.py", line 358, in open
    return cls(manager, group=group, mode=mode, lock=lock, autoclose=autoclose)
  File "/SalishSeaCast/nowcast-env/lib/python3.8/site-packages/xarray/backends/netCDF4_.py", line 314, in __init__
    self.format = self.ds.data_model
  File "/SalishSeaCast/nowcast-env/lib/python3.8/site-packages/xarray/backends/netCDF4_.py", line 367, in ds
    return self._acquire()
  File "/SalishSeaCast/nowcast-env/lib/python3.8/site-packages/xarray/backends/netCDF4_.py", line 361, in _acquire
    with self._manager.acquire_context(needs_lock) as root:
  File "/SalishSeaCast/nowcast-env/lib/python3.8/contextlib.py", line 113, in __enter__
    return next(self.gen)
  File "/SalishSeaCast/nowcast-env/lib/python3.8/site-packages/xarray/backends/file_manager.py", line 186, in acquire_context
    file, cached = self._acquire_with_cache_info(needs_lock)
  File "/SalishSeaCast/nowcast-env/lib/python3.8/site-packages/xarray/backends/file_manager.py", line 204, in _acquire_with_cache_info
    file = self._opener(*self._args, **kwargs)
  File "netCDF4/_netCDF4.pyx", line 2321, in netCDF4._netCDF4.Dataset.__init__
  File "netCDF4/_netCDF4.pyx", line 1885, in netCDF4._netCDF4._ensure_nc_success
FileNotFoundError: [Errno 2] No such file or directory: b'/results/SalishSea/nowcast-agrif.201702/03jul20/BaynesSoundSurface_grid_T.nc'
2020-07-03 15:36:45,953 INFO [make_plots] finished rendering nemo nowcast-agrif research 03jul20 figures
2020-07-03 15:36:45,954 INFO [make_plots] nemo research plots for 2020-07-03 nowcast-agrif completed
2020-07-03 15:36:45,955 INFO [manager] checklist updated with [plots] items from make_plots worker
2020-07-03 15:38:27,081 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15809506 = 2020-07-03 23:01:46 UTC estimated time to finish: 0000:00:15:22
2020-07-03 15:38:44,607 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39383439 = 2020-07-03 07:26:15 UTC estimated time to finish: 0000:13:31:20
2020-07-03 15:43:27,206 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15810529 = 2020-07-03 23:18:49 UTC estimated time to finish: 0000:00:10:53
2020-07-03 15:43:44,727 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39384313 = 2020-07-03 07:32:05 UTC estimated time to finish: 0000:13:27:20
2020-07-03 15:44:25,306 INFO [run_NEMO_agrif] running in process 14687
2020-07-03 15:44:25,307 INFO [run_NEMO_agrif] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 15:44:25,308 INFO [run_NEMO_agrif] publishing log messages to tcp://*:5563
2020-07-03 15:44:25,309 INFO [run_NEMO_agrif] connected to skookum.eos.ubc.ca port 5555
2020-07-03 15:44:50,493 INFO [run_NEMO_agrif] job id for 03jul20nowcast-agrif: 10846935.orca2.ibb
2020-07-03 15:44:50,494 INFO [run_NEMO_agrif] NEMO nowcast-agrif run for 2020-07-03 queued on orcinus-nowcast-agrif
2020-07-03 15:44:50,498 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_agrif worker
2020-07-03 15:44:50,722 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_agrif', args=['orcinus-nowcast-agrif', '10846935.orca2.ibb'], host='localhost')
2020-07-03 15:44:51,685 INFO [watch_NEMO_agrif] running in process 14718
2020-07-03 15:44:51,686 INFO [watch_NEMO_agrif] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 15:44:51,687 INFO [watch_NEMO_agrif] publishing log messages to tcp://*:5563
2020-07-03 15:44:51,688 INFO [watch_NEMO_agrif] connected to skookum.eos.ubc.ca port 5555
2020-07-03 15:44:52,338 INFO [watch_NEMO_agrif] watching 03jul20nowcast-agrif job 10846935 on orcinus-nowcast-agrif
2020-07-03 15:44:52,575 INFO [watch_NEMO_agrif] 03jul20nowcast-agrif job 10846935 is queued on orcinus-nowcast-agrif
2020-07-03 15:45:53,543 INFO [watch_NEMO_agrif] 03jul20nowcast-agrif on orcinus-nowcast-agrif: time.step not found; continuing to watch...
2020-07-03 15:48:27,330 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15811592 = 2020-07-03 23:36:32 UTC estimated time to finish: 0000:00:06:12
2020-07-03 15:48:44,846 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39385174 = 2020-07-03 07:37:49 UTC estimated time to finish: 0000:13:23:20
2020-07-03 15:50:54,075 INFO [watch_NEMO_agrif] 03jul20nowcast-agrif on orcinus-nowcast-agrif: timestep: 4624685 = 2020-07-03 01:22:40 UTC, 5.7% complete
2020-07-03 15:53:27,453 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15812656 = 2020-07-03 23:54:16 UTC estimated time to finish: 0000:00:01:31
2020-07-03 15:53:44,902 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39386048 = 2020-07-03 07:43:39 UTC estimated time to finish: 0000:13:19:11
2020-07-03 15:55:54,534 INFO [watch_NEMO_agrif] 03jul20nowcast-agrif on orcinus-nowcast-agrif: timestep: 4624685 = 2020-07-03 01:22:40 UTC, 5.7% complete
2020-07-03 15:58:27,461 INFO [watch_fvcom] x2 nowcast FVCOM VH-FR run on arbutus.cloud-nowcast completed
2020-07-03 15:58:27,466 INFO [manager] checklist updated with [FVCOM run] items from watch_fvcom worker
2020-07-03 15:58:27,978 INFO [manager] launching NextWorker(module='nowcast.workers.download_fvcom_results', args=['arbutus.cloud-nowcast', 'x2', 'nowcast', '--run-date', '2020-07-03'], host='localhost')
2020-07-03 15:58:27,982 INFO [manager] launching NextWorker(module='nowcast.workers.make_fvcom_boundary', args=['arbutus.cloud-nowcast', 'x2', 'forecast', '--run-date', '2020-07-03'], host='arbutus.cloud-nowcast')
2020-07-03 15:58:28,965 INFO [download_fvcom_results] running in process 14970
2020-07-03 15:58:28,966 INFO [download_fvcom_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 15:58:28,966 INFO [download_fvcom_results] publishing log messages to tcp://*:5567
2020-07-03 15:58:28,967 INFO [download_fvcom_results] connected to skookum.eos.ubc.ca port 5555
2020-07-03 15:58:31,802 INFO [make_fvcom_boundary] running in process 29159
2020-07-03 15:58:31,803 INFO [make_fvcom_boundary] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 15:58:31,803 INFO [make_fvcom_boundary] publishing log messages to tcp://*:5580
2020-07-03 15:58:31,815 INFO [make_fvcom_boundary] connected to skookum.eos.ubc.ca port 5555
2020-07-03 15:58:31,816 INFO [make_fvcom_boundary] Creating VHFR FVCOM open boundary file for x2 forecast run from 2020-07-03 NEMO run
2020-07-03 15:58:45,022 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39387781 = 2020-07-03 07:55:12 UTC estimated time to finish: 0000:13:01:12
2020-07-03 15:59:13,642 INFO [download_fvcom_results] VHFR FVCOM x2 nowcast 2020-07-03 results files from arbutus.cloud-nowcast downloaded
2020-07-03 15:59:13,648 INFO [manager] checklist updated with [VHFR FVCOM results files] items from download_fvcom_results worker
2020-07-03 15:59:13,945 INFO [manager] launching NextWorker(module='nowcast.workers.get_vfpa_hadcp', args=['--data-date', '2020-07-03'], host='localhost')
2020-07-03 15:59:13,948 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['fvcom', 'nowcast-x2', 'research', '--run-date', '2020-07-03'], host='localhost')
2020-07-03 15:59:15,836 INFO [get_vfpa_hadcp] running in process 14980
2020-07-03 15:59:15,837 INFO [get_vfpa_hadcp] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 15:59:15,838 INFO [get_vfpa_hadcp] publishing log messages to tcp://*:5567
2020-07-03 15:59:15,839 INFO [get_vfpa_hadcp] connected to skookum.eos.ubc.ca port 5555
2020-07-03 15:59:15,839 INFO [get_vfpa_hadcp] processing VFPA HADCP data from 2nd Narrows Rail Bridge for 2020-07-03
2020-07-03 15:59:16,779 INFO [get_vfpa_hadcp] added VFPA HADCP data from 2nd Narrows Rail Bridge for 2020-07-03 to /opp/observations/AISDATA/netcdf/VFPA_2ND_NARROWS_HADCP_2s_202007.nc
2020-07-03 15:59:16,780 INFO [get_vfpa_hadcp] VFPA HADCP observations added to 2020-07 netcdf file
2020-07-03 15:59:16,782 INFO [manager] checklist updated with [VFPA HADCP data] items from get_vfpa_hadcp worker
2020-07-03 15:59:17,035 INFO [make_plots] running in process 14981
2020-07-03 15:59:17,036 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 15:59:17,037 INFO [make_plots] publishing log messages to tcp://*:5568
2020-07-03 15:59:17,038 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2020-07-03 15:59:17,071 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['VFPA-HADCP'], host='localhost')
2020-07-03 15:59:17,112 INFO [make_plots] preparing render list for 2020-07-03 VHFR FVCOM nowcast-x2 research figures
2020-07-03 15:59:17,114 INFO [make_plots] added EnglishBay_surface_currents figures to 2020-07-03 VHFR FVCOM nowcast-x2 research render list
2020-07-03 15:59:17,116 INFO [make_plots] added VancouverHarbour_surface_currents figures to 2020-07-03 VHFR FVCOM nowcast-x2 research render list
2020-07-03 15:59:17,118 INFO [make_plots] added IndianArm_surface_currents figures to 2020-07-03 VHFR FVCOM nowcast-x2 research render list
2020-07-03 15:59:17,120 INFO [make_plots] added VancouverHarbour_thalweg figures to 2020-07-03 VHFR FVCOM nowcast-x2 research render list
2020-07-03 15:59:17,123 INFO [make_plots] added PortMoody_thalweg figures to 2020-07-03 VHFR FVCOM nowcast-x2 research render list
2020-07-03 15:59:17,125 INFO [make_plots] added IndianArm_thalweg figures to 2020-07-03 VHFR FVCOM nowcast-x2 research render list
2020-07-03 15:59:17,125 INFO [make_plots] starting to render fvcom nowcast-x2 research 03jul20 figures
2020-07-03 15:59:18,122 INFO [ping_erddap] running in process 14989
2020-07-03 15:59:18,123 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 15:59:18,123 INFO [ping_erddap] publishing log messages to tcp://*:5567
2020-07-03 15:59:18,124 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2020-07-03 15:59:18,125 INFO [ping_erddap] VFPA-HADCP ERDDAP dataset flag file(s) created
2020-07-03 15:59:18,126 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2020-07-03 15:59:18,383 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['fvcom', 'nowcast-x2', 'publish', '--run-date', '2020-07-03'], host='localhost')
2020-07-03 15:59:21,111 INFO [make_plots] running in process 14993
2020-07-03 15:59:21,112 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 15:59:21,113 INFO [make_plots] publishing log messages to tcp://*:5567
2020-07-03 15:59:21,114 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2020-07-03 15:59:22,549 INFO [make_plots] preparing render list for 2020-07-03 VHFR FVCOM nowcast-x2 publish figures
2020-07-03 15:59:22,550 INFO [make_plots] added Calamity Point figure to 2020-07-03 VHFR FVCOM nowcast-x2 publish render list
2020-07-03 15:59:22,551 INFO [make_plots] added Indian Arm Head figure to 2020-07-03 VHFR FVCOM nowcast-x2 publish render list
2020-07-03 15:59:22,552 INFO [make_plots] added New Westminster figure to 2020-07-03 VHFR FVCOM nowcast-x2 publish render list
2020-07-03 15:59:22,552 INFO [make_plots] added Port Moody figure to 2020-07-03 VHFR FVCOM nowcast-x2 publish render list
2020-07-03 15:59:22,553 INFO [make_plots] added Sand Heads figure to 2020-07-03 VHFR FVCOM nowcast-x2 publish render list
2020-07-03 15:59:22,553 INFO [make_plots] added Sandy Cove figure to 2020-07-03 VHFR FVCOM nowcast-x2 publish render list
2020-07-03 15:59:22,554 INFO [make_plots] added Vancouver Harbour figure to 2020-07-03 VHFR FVCOM nowcast-x2 publish render list
2020-07-03 15:59:22,555 INFO [make_plots] added Woodwards Landing figure to 2020-07-03 VHFR FVCOM nowcast-x2 publish render list
2020-07-03 15:59:22,556 INFO [make_plots] added 2ndNarrows_current figure to 2020-07-03 VHFR FVCOM nowcast-x2 publish render list
2020-07-03 15:59:22,562 INFO [make_plots] starting to render fvcom nowcast-x2 publish 03jul20 figures
2020-07-03 15:59:22,604 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07724 Calamity Point from 2020-07-02 23:55:00 to 2020-07-04 00:00:00
2020-07-03 15:59:45,032 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07724 Calamity Point from 2020-07-03 00:00:00 to 2020-07-04 00:00:00
2020-07-03 15:59:46,632 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07774 Indian Arm Head from 2020-07-02 23:55:00 to 2020-07-04 00:00:00
2020-07-03 15:59:47,385 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07774 Indian Arm Head from 2020-07-03 00:00:00 to 2020-07-04 00:00:00
2020-07-03 15:59:49,316 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07654 New Westminster from 2020-07-02 23:55:00 to 2020-07-04 00:00:00
2020-07-03 16:00:09,956 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07654 New Westminster from 2020-07-03 00:00:00 to 2020-07-04 00:00:00
2020-07-03 16:00:12,934 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07755 Port Moody from 2020-07-02 23:55:00 to 2020-07-04 00:00:00
2020-07-03 16:00:34,196 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07755 Port Moody from 2020-07-03 00:00:00 to 2020-07-04 00:00:00
2020-07-03 16:00:37,449 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07594 Sand Heads from 2020-07-02 23:55:00 to 2020-07-04 00:00:00
2020-07-03 16:00:54,764 INFO [watch_NEMO_agrif] NEMO AGRIF run 10846935.orca2.ibb on orcinus-nowcast-agrif completed
2020-07-03 16:00:54,768 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_agrif worker
2020-07-03 16:00:55,800 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['orcinus-nowcast-agrif', 'nowcast-agrif', '--run-date', '2020-07-03'], host='localhost')
2020-07-03 16:00:56,774 INFO [download_results] running in process 15020
2020-07-03 16:00:56,775 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 16:00:56,776 INFO [download_results] publishing log messages to tcp://*:5563
2020-07-03 16:00:56,776 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2020-07-03 16:00:56,777 INFO [download_results] downloading results from orcinus-nowcast-agrif:/global/scratch/dlatorne/nowcast-agrif/03jul20 to /results/SalishSea/nowcast-agrif.201702
2020-07-03 16:00:57,265 INFO [download_results] nowcast-agrif 2020-07-03 results files from orcinus-nowcast-agrif downloaded
2020-07-03 16:00:57,270 INFO [manager] checklist updated with [results files] items from download_results worker
2020-07-03 16:00:57,543 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast-agrif', 'research', '--run-date', '2020-07-03'], host='localhost')
2020-07-03 16:00:58,792 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07594 Sand Heads from 2020-07-03 00:00:00 to 2020-07-04 00:00:00
2020-07-03 16:01:01,029 INFO [make_plots] running in process 15026
2020-07-03 16:01:01,031 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 16:01:01,031 INFO [make_plots] publishing log messages to tcp://*:5563
2020-07-03 16:01:01,032 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2020-07-03 16:01:01,399 INFO [make_plots] preparing render list for 2020-07-03 NEMO nowcast-agrif research figures
2020-07-03 16:01:01,400 INFO [make_plots] added baynes_sound_surface figure to 2020-07-03 NEMO nowcast-agrif research render list
2020-07-03 16:01:01,402 INFO [make_plots] starting to render nemo nowcast-agrif research 03jul20 figures
2020-07-03 16:01:01,454 ERROR [make_plots] unexpected FileNotFoundError in make_figure:
Traceback (most recent call last):
  File "/SalishSeaCast/nowcast-env/lib/python3.8/site-packages/xarray/backends/file_manager.py", line 198, in _acquire_with_cache_info
    file = self._cache[self._key]
  File "/SalishSeaCast/nowcast-env/lib/python3.8/site-packages/xarray/backends/lru_cache.py", line 53, in __getitem__
    value = self._cache[key]
KeyError: [<class 'netCDF4._netCDF4.Dataset'>, ('/results/SalishSea/nowcast-agrif.201702/03jul20/BaynesSoundSurface_grid_T.nc',), 'r', (('clobber', True), ('diskless', False), ('format', 'NETCDF4'), ('persist', False))]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure
    fig = fig_func(*args, **kwargs)
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/research/baynes_sound_agrif.py", line 87, in make_figure
    plot_data = _prep_plot_data(
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/research/baynes_sound_agrif.py", line 128, in _prep_plot_data
    ss_tracers = xarray.open_dataset(ss_tracers_path)
  File "/SalishSeaCast/nowcast-env/lib/python3.8/site-packages/xarray/backends/api.py", line 501, in open_dataset
    store = backends.NetCDF4DataStore.open(
  File "/SalishSeaCast/nowcast-env/lib/python3.8/site-packages/xarray/backends/netCDF4_.py", line 358, in open
    return cls(manager, group=group, mode=mode, lock=lock, autoclose=autoclose)
  File "/SalishSeaCast/nowcast-env/lib/python3.8/site-packages/xarray/backends/netCDF4_.py", line 314, in __init__
    self.format = self.ds.data_model
  File "/SalishSeaCast/nowcast-env/lib/python3.8/site-packages/xarray/backends/netCDF4_.py", line 367, in ds
    return self._acquire()
  File "/SalishSeaCast/nowcast-env/lib/python3.8/site-packages/xarray/backends/netCDF4_.py", line 361, in _acquire
    with self._manager.acquire_context(needs_lock) as root:
  File "/SalishSeaCast/nowcast-env/lib/python3.8/contextlib.py", line 113, in __enter__
    return next(self.gen)
  File "/SalishSeaCast/nowcast-env/lib/python3.8/site-packages/xarray/backends/file_manager.py", line 186, in acquire_context
    file, cached = self._acquire_with_cache_info(needs_lock)
  File "/SalishSeaCast/nowcast-env/lib/python3.8/site-packages/xarray/backends/file_manager.py", line 204, in _acquire_with_cache_info
    file = self._opener(*self._args, **kwargs)
  File "netCDF4/_netCDF4.pyx", line 2321, in netCDF4._netCDF4.Dataset.__init__
  File "netCDF4/_netCDF4.pyx", line 1885, in netCDF4._netCDF4._ensure_nc_success
FileNotFoundError: [Errno 2] No such file or directory: b'/results/SalishSea/nowcast-agrif.201702/03jul20/BaynesSoundSurface_grid_T.nc'
2020-07-03 16:01:01,534 INFO [make_plots] finished rendering nemo nowcast-agrif research 03jul20 figures
2020-07-03 16:01:01,535 INFO [make_plots] nemo research plots for 2020-07-03 nowcast-agrif completed
2020-07-03 16:01:01,538 INFO [manager] checklist updated with [plots] items from make_plots worker
2020-07-03 16:01:01,999 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07786 Sandy Cove from 2020-07-02 23:55:00 to 2020-07-04 00:00:00
2020-07-03 16:01:23,354 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07795 Point Atkinson from 2020-07-03 00:00:00 to 2020-07-04 00:00:00
2020-07-03 16:01:26,331 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07735 Vancouver Harbour from 2020-07-02 23:55:00 to 2020-07-04 00:00:00
2020-07-03 16:01:47,255 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07735 Vancouver Harbour from 2020-07-03 00:00:00 to 2020-07-04 00:00:00
2020-07-03 16:01:50,363 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07610 Woodwards Landing from 2020-07-02 23:55:00 to 2020-07-04 00:00:00
2020-07-03 16:01:55,692 INFO [make_fvcom_boundary] Stored VHFR FVCOM open boundary file: /nemoShare/MEOPAR/nowcast-sys/fvcom-runs/input.x2/bdy_x2_forecast_brcl_20200704.nc
2020-07-03 16:01:55,696 INFO [make_fvcom_boundary] FVCOM x2 forecast run boundary condition file for 2020-07-03 created on arbutus.cloud-nowcast
2020-07-03 16:01:55,701 INFO [manager] checklist updated with [FVCOM boundary conditions] items from make_fvcom_boundary worker
2020-07-03 16:01:56,042 INFO [manager] launching NextWorker(module='nowcast.workers.make_fvcom_atmos_forcing', args=['x2', 'forecast', '--run-date', '2020-07-03'], host='localhost')
2020-07-03 16:01:56,046 INFO [manager] launching NextWorker(module='nowcast.workers.make_fvcom_rivers_forcing', args=['arbutus.cloud-nowcast', 'x2', 'forecast', '--run-date', '2020-07-03'], host='arbutus.cloud-nowcast')
2020-07-03 16:01:59,280 INFO [make_fvcom_rivers_forcing] running in process 29340
2020-07-03 16:01:59,281 INFO [make_fvcom_rivers_forcing] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 16:01:59,282 INFO [make_fvcom_rivers_forcing] publishing log messages to tcp://*:5582
2020-07-03 16:01:59,283 INFO [make_fvcom_rivers_forcing] connected to skookum.eos.ubc.ca port 5555
2020-07-03 16:01:59,283 INFO [make_fvcom_rivers_forcing] Creating VHFR FVCOM rivers forcing file for x2 forecast run from 2020-07-03 NEMO runoff forcing files
2020-07-03 16:01:59,321 INFO [make_fvcom_atmos_forcing] running in process 15089
2020-07-03 16:01:59,322 INFO [make_fvcom_atmos_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 16:01:59,322 INFO [make_fvcom_atmos_forcing] publishing log messages to tcp://*:5563
2020-07-03 16:01:59,323 INFO [make_fvcom_atmos_forcing] connected to skookum.eos.ubc.ca port 5555
2020-07-03 16:01:59,324 INFO [make_fvcom_atmos_forcing] Creating VHFR FVCOM atmospheric forcing file for x2 forecast run from 2020-07-03 HRDPS GRIB files
2020-07-03 16:01:59,473 INFO [make_fvcom_rivers_forcing] Stored VHFR FVCOM rivers forcing file: /nemoShare/MEOPAR/nowcast-sys/fvcom-runs/input.x2/rivers_x2_forecast_20200704.nc
2020-07-03 16:01:59,474 INFO [make_fvcom_rivers_forcing] FVCOM x2 forecast run rivers forcing file for 2020-07-03 created on arbutus.cloud-nowcast
2020-07-03 16:01:59,478 INFO [manager] checklist updated with [FVCOM rivers forcing] items from make_fvcom_rivers_forcing worker
2020-07-03 16:02:11,034 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07610 Woodwards Landing from 2020-07-03 00:00:00 to 2020-07-04 00:00:00
2020-07-03 16:02:16,025 INFO [make_plots] finished rendering fvcom nowcast-x2 publish 03jul20 figures
2020-07-03 16:02:16,658 INFO [make_plots] fvcom publish plots for 2020-07-03 nowcast-x2 completed
2020-07-03 16:02:16,665 INFO [manager] checklist updated with [plots] items from make_plots worker
2020-07-03 16:02:34,167 INFO [make_fvcom_atmos_forcing] Stored VHFR FVCOM atmospheric forcing hfx file: /results/forcing/atmospheric/GEM2.5/vhfr-fvcom/atmos_x2_forecast_hfx_20200704.nc
2020-07-03 16:02:43,218 INFO [make_fvcom_atmos_forcing] Stored VHFR FVCOM atmospheric forcing precip file: /results/forcing/atmospheric/GEM2.5/vhfr-fvcom/atmos_x2_forecast_precip_20200704.nc
2020-07-03 16:02:57,516 INFO [make_fvcom_atmos_forcing] Stored VHFR FVCOM atmospheric forcing wnd file: /results/forcing/atmospheric/GEM2.5/vhfr-fvcom/atmos_x2_forecast_wnd_20200704.nc
2020-07-03 16:02:57,517 INFO [make_fvcom_atmos_forcing] FVCOM x2 forecast run atmospheric forcing file for 2020-07-03 created
2020-07-03 16:02:57,520 INFO [manager] checklist updated with [FVCOM atmospheric forcing] items from make_fvcom_atmos_forcing worker
2020-07-03 16:02:57,766 INFO [manager] launching NextWorker(module='nowcast.workers.upload_fvcom_atmos_forcing', args=['arbutus.cloud-nowcast', 'x2', 'forecast', '--run-date', '2020-07-03'], host='localhost')
2020-07-03 16:02:58,738 INFO [upload_fvcom_atmos_forcing] running in process 15100
2020-07-03 16:02:58,739 INFO [upload_fvcom_atmos_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 16:02:58,740 INFO [upload_fvcom_atmos_forcing] publishing log messages to tcp://*:5563
2020-07-03 16:02:58,741 INFO [upload_fvcom_atmos_forcing] connected to skookum.eos.ubc.ca port 5555
2020-07-03 16:02:58,742 INFO [upload_fvcom_atmos_forcing] Uploading VHFR FVCOM atmospheric forcing files for 2020-07-03 x2 forecast run to arbutus.cloud-nowcast
2020-07-03 16:03:03,587 INFO [upload_fvcom_atmos_forcing] FVCOM x2 forecast run atmospheric forcing files for 2020-07-03 uploaded to arbutus.cloud-nowcast
2020-07-03 16:03:03,593 INFO [manager] checklist updated with [FVCOM atmospheric forcing upload] items from upload_fvcom_atmos_forcing worker
2020-07-03 16:03:03,922 INFO [manager] launching NextWorker(module='nowcast.workers.run_fvcom', args=['arbutus.cloud-nowcast', 'x2', 'forecast', '--run-date', '2020-07-03'], host='arbutus.cloud-nowcast')
2020-07-03 16:03:05,947 INFO [run_fvcom] running in process 29689
2020-07-03 16:03:05,948 INFO [run_fvcom] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 16:03:05,949 INFO [run_fvcom] publishing log messages to tcp://*:5584
2020-07-03 16:03:05,949 INFO [run_fvcom] connected to skookum.eos.ubc.ca port 5555
2020-07-03 16:03:06,105 INFO [run_fvcom] forecast: launching /nemoShare/MEOPAR/nowcast-sys/fvcom-runs/20200703-23h03m06.051919s/VHFR_FVCOM.sh on arbutus.cloud-nowcast
2020-07-03 16:03:06,123 INFO [run_fvcom] x2 forecast FVCOM VH-FR run for 2020-07-03 on arbutus.cloud-nowcast started
2020-07-03 16:03:06,129 INFO [manager] checklist updated with [FVCOM run] items from run_fvcom worker
2020-07-03 16:03:06,451 INFO [manager] launching NextWorker(module='nowcast.workers.watch_fvcom', args=['arbutus.cloud-nowcast', 'x2', 'forecast'], host='arbutus.cloud-nowcast')
2020-07-03 16:03:08,360 INFO [watch_fvcom] running in process 29790
2020-07-03 16:03:08,361 INFO [watch_fvcom] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 16:03:08,362 INFO [watch_fvcom] publishing log messages to tcp://*:5586
2020-07-03 16:03:08,363 INFO [watch_fvcom] connected to skookum.eos.ubc.ca port 5555
2020-07-03 16:03:08,398 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: no run progress found in fvcom.log ; continuing to watch...
2020-07-03 16:03:45,143 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39389639 = 2020-07-03 08:07:35 UTC estimated time to finish: 0000:12:40:44
2020-07-03 16:08:08,500 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15814499 = 2020-07-04 00:24:59 UTC estimated time to finish: 0000:06:24:52
2020-07-03 16:08:45,265 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39390581 = 2020-07-03 08:13:52 UTC estimated time to finish: 0000:12:35:54
2020-07-03 16:13:08,602 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15816161 = 2020-07-04 00:52:41 UTC estimated time to finish: 0000:06:24:01
2020-07-03 16:13:45,386 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39391496 = 2020-07-03 08:19:58 UTC estimated time to finish: 0000:12:31:21
2020-07-03 16:18:08,704 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15817689 = 2020-07-04 01:18:09 UTC estimated time to finish: 0000:06:28:53
2020-07-03 16:18:45,509 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39392357 = 2020-07-03 08:25:42 UTC estimated time to finish: 0000:12:27:36
2020-07-03 16:23:08,808 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15818874 = 2020-07-04 01:37:54 UTC estimated time to finish: 0000:06:58:43
2020-07-03 16:23:45,632 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39393109 = 2020-07-03 08:30:43 UTC estimated time to finish: 0000:12:24:55
2020-07-03 16:24:41,617 INFO [make_plots] finished rendering fvcom nowcast-x2 research 03jul20 figures
2020-07-03 16:24:41,618 INFO [make_plots] fvcom research plots for 2020-07-03 nowcast-x2 completed
2020-07-03 16:24:41,689 INFO [manager] checklist updated with [plots] items from make_plots worker
2020-07-03 16:28:08,912 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15819856 = 2020-07-04 01:54:16 UTC estimated time to finish: 0000:07:16:03
2020-07-03 16:28:45,760 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39393941 = 2020-07-03 08:36:16 UTC estimated time to finish: 0000:12:21:36
2020-07-03 16:33:09,015 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15820960 = 2020-07-04 02:12:40 UTC estimated time to finish: 0000:07:30:43
2020-07-03 16:33:45,882 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39394733 = 2020-07-03 08:41:33 UTC estimated time to finish: 0000:12:18:43
2020-07-03 16:38:09,119 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15822146 = 2020-07-04 02:32:26 UTC estimated time to finish: 0000:07:35:54
2020-07-03 16:38:45,928 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39395553 = 2020-07-03 08:47:01 UTC estimated time to finish: 0000:12:15:43
2020-07-03 16:43:09,224 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15823290 = 2020-07-04 02:51:30 UTC estimated time to finish: 0000:07:40:55
2020-07-03 16:43:46,051 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39396304 = 2020-07-03 08:52:01 UTC estimated time to finish: 0000:12:12:31
2020-07-03 16:48:09,328 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15824313 = 2020-07-04 03:08:33 UTC estimated time to finish: 0000:07:43:03
2020-07-03 16:48:46,175 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39397178 = 2020-07-03 08:57:51 UTC estimated time to finish: 0000:12:08:44
2020-07-03 16:53:09,433 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15825499 = 2020-07-04 03:28:19 UTC estimated time to finish: 0000:07:44:16
2020-07-03 16:53:46,298 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39397970 = 2020-07-03 09:03:08 UTC estimated time to finish: 0000:12:05:42
2020-07-03 16:58:09,539 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15826602 = 2020-07-04 03:46:42 UTC estimated time to finish: 0000:07:45:26
2020-07-03 16:58:46,422 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39398830 = 2020-07-03 09:08:52 UTC estimated time to finish: 0000:12:02:02
2020-07-03 17:03:09,645 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15827747 = 2020-07-04 04:05:47 UTC estimated time to finish: 0000:07:45:31
2020-07-03 17:03:46,546 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39399663 = 2020-07-03 09:14:25 UTC estimated time to finish: 0000:11:58:21
2020-07-03 17:08:09,751 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15828920 = 2020-07-04 04:25:20 UTC estimated time to finish: 0000:07:43:38
2020-07-03 17:08:46,671 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39400455 = 2020-07-03 09:19:42 UTC estimated time to finish: 0000:11:54:40
2020-07-03 17:13:09,857 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15829942 = 2020-07-04 04:42:22 UTC estimated time to finish: 0000:07:41:28
2020-07-03 17:13:46,798 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39401316 = 2020-07-03 09:25:26 UTC estimated time to finish: 0000:11:50:47
2020-07-03 17:18:09,964 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15831128 = 2020-07-04 05:02:08 UTC estimated time to finish: 0000:07:38:18
2020-07-03 17:18:46,921 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39402150 = 2020-07-03 09:31:00 UTC estimated time to finish: 0000:11:46:55
2020-07-03 17:23:10,071 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15832272 = 2020-07-04 05:21:12 UTC estimated time to finish: 0000:07:36:40
2020-07-03 17:23:47,046 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39403064 = 2020-07-03 09:37:05 UTC estimated time to finish: 0000:11:42:27
2020-07-03 17:28:10,178 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15833417 = 2020-07-04 05:40:17 UTC estimated time to finish: 0000:07:33:30
2020-07-03 17:28:47,172 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39403843 = 2020-07-03 09:42:17 UTC estimated time to finish: 0000:11:38:52
2020-07-03 17:33:10,286 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15834399 = 2020-07-04 05:56:39 UTC estimated time to finish: 0000:07:31:44
2020-07-03 17:33:47,297 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39404676 = 2020-07-03 09:47:50 UTC estimated time to finish: 0000:11:34:54
2020-07-03 17:38:10,364 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15835543 = 2020-07-04 06:15:43 UTC estimated time to finish: 0000:07:29:08
2020-07-03 17:38:47,423 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39405590 = 2020-07-03 09:53:56 UTC estimated time to finish: 0000:11:30:23
2020-07-03 17:43:10,472 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15836690 = 2020-07-04 06:34:50 UTC estimated time to finish: 0000:07:25:26
2020-07-03 17:43:47,481 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39406451 = 2020-07-03 09:59:40 UTC estimated time to finish: 0000:11:26:10
2020-07-03 17:48:10,580 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15837834 = 2020-07-04 06:53:54 UTC estimated time to finish: 0000:07:21:53
2020-07-03 17:48:47,607 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39407202 = 2020-07-03 10:04:40 UTC estimated time to finish: 0000:11:22:44
2020-07-03 17:53:10,689 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15838856 = 2020-07-04 07:10:56 UTC estimated time to finish: 0000:07:18:37
2020-07-03 17:53:47,733 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39408075 = 2020-07-03 10:10:30 UTC estimated time to finish: 0000:11:18:31
2020-07-03 17:58:10,798 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15840030 = 2020-07-04 07:30:30 UTC estimated time to finish: 0000:07:14:25
2020-07-03 17:58:47,858 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39408936 = 2020-07-03 10:16:14 UTC estimated time to finish: 0000:11:14:23
2020-07-03 18:03:10,907 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15841134 = 2020-07-04 07:48:54 UTC estimated time to finish: 0000:07:11:21
2020-07-03 18:03:47,984 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39409810 = 2020-07-03 10:22:04 UTC estimated time to finish: 0000:11:10:09
2020-07-03 18:08:11,017 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15842238 = 2020-07-04 08:07:18 UTC estimated time to finish: 0000:07:07:59
2020-07-03 18:08:48,111 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39410712 = 2020-07-03 10:28:04 UTC estimated time to finish: 0000:11:05:37
2020-07-03 18:13:11,127 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15843382 = 2020-07-04 08:26:22 UTC estimated time to finish: 0000:07:04:17
2020-07-03 18:13:48,238 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39411464 = 2020-07-03 10:33:05 UTC estimated time to finish: 0000:11:02:01
2020-07-03 18:18:11,237 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15844406 = 2020-07-04 08:43:26 UTC estimated time to finish: 0000:07:00:40
2020-07-03 18:18:48,366 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39412338 = 2020-07-03 10:38:55 UTC estimated time to finish: 0000:10:57:32
2020-07-03 18:23:11,348 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15845510 = 2020-07-04 09:01:50 UTC estimated time to finish: 0000:06:56:55
2020-07-03 18:23:48,493 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39413240 = 2020-07-03 10:44:56 UTC estimated time to finish: 0000:10:53:03
2020-07-03 18:28:11,459 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15846654 = 2020-07-04 09:20:54 UTC estimated time to finish: 0000:06:52:46
2020-07-03 18:28:48,620 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39414112 = 2020-07-03 10:50:44 UTC estimated time to finish: 0000:10:48:41
2020-07-03 18:33:11,506 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15847759 = 2020-07-04 09:39:19 UTC estimated time to finish: 0000:06:48:49
2020-07-03 18:33:48,748 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39414863 = 2020-07-03 10:55:45 UTC estimated time to finish: 0000:10:45:03
2020-07-03 18:38:11,618 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15848780 = 2020-07-04 09:56:20 UTC estimated time to finish: 0000:06:45:05
2020-07-03 18:38:48,877 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39415724 = 2020-07-03 11:01:29 UTC estimated time to finish: 0000:10:40:55
2020-07-03 18:43:11,730 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15849885 = 2020-07-04 10:14:45 UTC estimated time to finish: 0000:06:40:49
2020-07-03 18:43:49,006 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39416557 = 2020-07-03 11:07:02 UTC estimated time to finish: 0000:10:36:55
2020-07-03 18:48:11,843 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15851071 = 2020-07-04 10:34:31 UTC estimated time to finish: 0000:06:36:01
2020-07-03 18:48:49,135 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39417430 = 2020-07-03 11:12:52 UTC estimated time to finish: 0000:10:32:28
2020-07-03 18:53:11,955 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15852216 = 2020-07-04 10:53:36 UTC estimated time to finish: 0000:06:31:32
2020-07-03 18:53:49,265 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39418210 = 2020-07-03 11:18:04 UTC estimated time to finish: 0000:10:28:28
2020-07-03 18:58:12,069 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15853198 = 2020-07-04 11:09:58 UTC estimated time to finish: 0000:06:27:44
2020-07-03 18:58:49,394 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39419083 = 2020-07-03 11:23:53 UTC estimated time to finish: 0000:10:23:59
2020-07-03 19:03:12,182 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15854342 = 2020-07-04 11:29:02 UTC estimated time to finish: 0000:06:23:13
2020-07-03 19:03:49,523 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39419958 = 2020-07-03 11:29:43 UTC estimated time to finish: 0000:10:19:30
2020-07-03 19:08:12,296 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15855447 = 2020-07-04 11:47:27 UTC estimated time to finish: 0000:06:19:05
2020-07-03 19:08:49,653 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39420860 = 2020-07-03 11:35:44 UTC estimated time to finish: 0000:10:14:54
2020-07-03 19:13:12,409 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15856579 = 2020-07-04 12:06:19 UTC estimated time to finish: 0000:06:14:38
2020-07-03 19:13:49,698 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39421733 = 2020-07-03 11:41:33 UTC estimated time to finish: 0000:10:10:20
2020-07-03 19:18:12,524 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15857683 = 2020-07-04 12:24:43 UTC estimated time to finish: 0000:06:10:16
2020-07-03 19:18:49,829 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39422520 = 2020-07-03 11:46:48 UTC estimated time to finish: 0000:10:06:17
2020-07-03 19:23:12,644 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15858719 = 2020-07-04 12:41:59 UTC estimated time to finish: 0000:06:06:07
2020-07-03 19:23:49,959 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39423386 = 2020-07-03 11:52:34 UTC estimated time to finish: 0000:10:01:53
2020-07-03 19:28:12,761 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15859850 = 2020-07-04 13:00:50 UTC estimated time to finish: 0000:06:01:28
2020-07-03 19:28:50,091 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39424260 = 2020-07-03 11:58:24 UTC estimated time to finish: 0000:09:57:24
2020-07-03 19:33:12,877 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15860955 = 2020-07-04 13:19:15 UTC estimated time to finish: 0000:05:57:01
2020-07-03 19:33:50,222 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39425120 = 2020-07-03 12:04:08 UTC estimated time to finish: 0000:09:52:58
2020-07-03 19:38:12,992 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15862100 = 2020-07-04 13:38:20 UTC estimated time to finish: 0000:05:52:13
2020-07-03 19:38:50,353 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39425912 = 2020-07-03 12:09:24 UTC estimated time to finish: 0000:09:48:53
2020-07-03 19:43:13,107 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15863122 = 2020-07-04 13:55:22 UTC estimated time to finish: 0000:05:48:00
2020-07-03 19:43:50,484 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39426785 = 2020-07-03 12:15:14 UTC estimated time to finish: 0000:09:44:18
2020-07-03 19:48:13,223 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15864268 = 2020-07-04 14:14:28 UTC estimated time to finish: 0000:05:43:16
2020-07-03 19:48:50,614 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39427687 = 2020-07-03 12:21:14 UTC estimated time to finish: 0000:09:39:30
2020-07-03 19:53:13,339 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15865412 = 2020-07-04 14:33:32 UTC estimated time to finish: 0000:05:38:23
2020-07-03 19:53:50,746 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39428561 = 2020-07-03 12:27:04 UTC estimated time to finish: 0000:09:34:48
2020-07-03 19:58:13,454 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15866516 = 2020-07-04 14:51:56 UTC estimated time to finish: 0000:05:33:43
2020-07-03 19:58:50,876 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39429354 = 2020-07-03 12:32:21 UTC estimated time to finish: 0000:09:30:44
2020-07-03 20:03:13,571 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15867539 = 2020-07-04 15:08:59 UTC estimated time to finish: 0000:05:29:25
2020-07-03 20:03:51,008 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39430215 = 2020-07-03 12:38:06 UTC estimated time to finish: 0000:09:26:13
2020-07-03 20:08:13,687 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15868833 = 2020-07-04 15:30:33 UTC estimated time to finish: 0000:05:23:10
2020-07-03 20:08:51,140 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39431130 = 2020-07-03 12:44:12 UTC estimated time to finish: 0000:09:21:11
2020-07-03 20:13:13,804 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15870537 = 2020-07-04 15:58:57 UTC estimated time to finish: 0000:05:12:43
2020-07-03 20:13:51,284 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39432112 = 2020-07-03 12:50:44 UTC estimated time to finish: 0000:09:15:33
2020-07-03 20:18:13,921 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15872186 = 2020-07-04 16:26:26 UTC estimated time to finish: 0000:05:03:10
2020-07-03 20:18:51,422 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39433055 = 2020-07-03 12:57:02 UTC estimated time to finish: 0000:09:10:13
2020-07-03 20:23:14,041 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15873330 = 2020-07-04 16:45:30 UTC estimated time to finish: 0000:04:58:19
2020-07-03 20:23:51,487 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39433847 = 2020-07-03 13:02:18 UTC estimated time to finish: 0000:09:06:04
2020-07-03 20:28:14,158 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15874353 = 2020-07-04 17:02:33 UTC estimated time to finish: 0000:04:54:09
2020-07-03 20:28:51,623 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39434680 = 2020-07-03 13:07:52 UTC estimated time to finish: 0000:09:01:44
2020-07-03 20:33:14,278 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15875539 = 2020-07-04 17:22:19 UTC estimated time to finish: 0000:04:49:12
2020-07-03 20:33:51,756 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39435540 = 2020-07-03 13:13:36 UTC estimated time to finish: 0000:08:57:18
2020-07-03 20:38:14,397 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15876684 = 2020-07-04 17:41:24 UTC estimated time to finish: 0000:04:44:22
2020-07-03 20:38:51,890 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39436414 = 2020-07-03 13:19:25 UTC estimated time to finish: 0000:08:52:42
2020-07-03 20:43:14,516 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15877816 = 2020-07-04 18:00:16 UTC estimated time to finish: 0000:04:39:42
2020-07-03 20:43:52,025 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39437206 = 2020-07-03 13:24:42 UTC estimated time to finish: 0000:08:48:31
2020-07-03 20:48:14,636 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15878880 = 2020-07-04 18:18:00 UTC estimated time to finish: 0000:04:35:14
2020-07-03 20:48:52,159 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39438068 = 2020-07-03 13:30:27 UTC estimated time to finish: 0000:08:43:57
2020-07-03 20:53:14,757 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15879956 = 2020-07-04 18:35:56 UTC estimated time to finish: 0000:04:30:57
2020-07-03 20:53:52,293 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39438941 = 2020-07-03 13:36:16 UTC estimated time to finish: 0000:08:39:20
2020-07-03 20:58:14,877 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15881089 = 2020-07-04 18:54:49 UTC estimated time to finish: 0000:04:26:20
2020-07-03 20:58:52,427 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39439802 = 2020-07-03 13:42:00 UTC estimated time to finish: 0000:08:34:46
2020-07-03 21:03:15,000 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15882192 = 2020-07-04 19:13:12 UTC estimated time to finish: 0000:04:21:42
2020-07-03 21:03:52,563 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39440594 = 2020-07-03 13:47:17 UTC estimated time to finish: 0000:08:30:36
2020-07-03 21:03:52,885 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-west/00/ to /results/forcing/atmospheric/GEM2.5/GRIB/20200704/00/
2020-07-03 21:03:52,886 INFO [collect_weather] 2020-07-04 2.5km weather forecast 00 collection complete
2020-07-03 21:03:52,889 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2020-07-03 21:03:53,527 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['06', '2.5km'], host='localhost')
2020-07-03 21:03:54,913 INFO [collect_weather] running in process 22067
2020-07-03 21:03:54,914 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 21:03:54,915 INFO [collect_weather] publishing log messages to tcp://*:5562
2020-07-03 21:03:54,916 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2020-07-03 21:03:54,932 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-west/06/
2020-07-03 21:08:15,118 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15883256 = 2020-07-04 19:30:56 UTC estimated time to finish: 0000:04:17:12
2020-07-03 21:08:52,700 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39441468 = 2020-07-03 13:53:07 UTC estimated time to finish: 0000:08:25:57
2020-07-03 21:13:15,240 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15884400 = 2020-07-04 19:50:00 UTC estimated time to finish: 0000:04:12:23
2020-07-03 21:13:52,834 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39442329 = 2020-07-03 13:58:51 UTC estimated time to finish: 0000:08:21:16
2020-07-03 21:18:15,361 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15885546 = 2020-07-04 20:09:06 UTC estimated time to finish: 0000:04:07:28
2020-07-03 21:18:52,969 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39443434 = 2020-07-03 14:06:13 UTC estimated time to finish: 0000:08:14:27
2020-07-03 21:23:15,484 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15886800 = 2020-07-04 20:30:00 UTC estimated time to finish: 0000:04:01:45
2020-07-03 21:23:53,102 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39444704 = 2020-07-03 14:14:41 UTC estimated time to finish: 0000:08:06:09
2020-07-03 21:28:15,517 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15888108 = 2020-07-04 20:51:48 UTC estimated time to finish: 0000:03:55:48
2020-07-03 21:28:53,238 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39445850 = 2020-07-03 14:22:20 UTC estimated time to finish: 0000:07:58:37
2020-07-03 21:33:15,639 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15889211 = 2020-07-04 21:10:11 UTC estimated time to finish: 0000:03:50:50
2020-07-03 21:33:53,375 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39447068 = 2020-07-03 14:30:27 UTC estimated time to finish: 0000:07:50:49
2020-07-03 21:38:15,763 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15890508 = 2020-07-04 21:31:48 UTC estimated time to finish: 0000:03:44:58
2020-07-03 21:38:53,512 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39448338 = 2020-07-03 14:38:55 UTC estimated time to finish: 0000:07:42:42
2020-07-03 21:43:15,887 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15891775 = 2020-07-04 21:52:55 UTC estimated time to finish: 0000:03:39:17
2020-07-03 21:43:53,650 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39449553 = 2020-07-03 14:47:01 UTC estimated time to finish: 0000:07:34:59
2020-07-03 21:48:16,011 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15893030 = 2020-07-04 22:13:50 UTC estimated time to finish: 0000:03:33:38
2020-07-03 21:48:53,788 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39450660 = 2020-07-03 14:54:24 UTC estimated time to finish: 0000:07:28:00
2020-07-03 21:53:16,136 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15894052 = 2020-07-04 22:30:52 UTC estimated time to finish: 0000:03:29:20
2020-07-03 21:53:53,926 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39451532 = 2020-07-03 15:00:12 UTC estimated time to finish: 0000:07:23:30
2020-07-03 21:58:16,261 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15895157 = 2020-07-04 22:49:17 UTC estimated time to finish: 0000:03:24:47
2020-07-03 21:58:54,067 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39452400 = 2020-07-03 15:06:00 UTC estimated time to finish: 0000:07:18:58
2020-07-03 22:03:16,387 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15896301 = 2020-07-04 23:08:21 UTC estimated time to finish: 0000:03:19:58
2020-07-03 22:03:54,205 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39453226 = 2020-07-03 15:11:30 UTC estimated time to finish: 0000:07:14:46
2020-07-03 22:08:16,513 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15897487 = 2020-07-04 23:28:07 UTC estimated time to finish: 0000:03:14:52
2020-07-03 22:08:54,345 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39453977 = 2020-07-03 15:16:30 UTC estimated time to finish: 0000:07:10:53
2020-07-03 22:13:16,638 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15898470 = 2020-07-04 23:44:30 UTC estimated time to finish: 0000:03:10:44
2020-07-03 22:13:54,485 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39454891 = 2020-07-03 15:22:36 UTC estimated time to finish: 0000:07:05:58
2020-07-03 22:18:16,764 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15899613 = 2020-07-05 00:03:33 UTC estimated time to finish: 0000:03:05:54
2020-07-03 22:18:54,623 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39455794 = 2020-07-03 15:28:37 UTC estimated time to finish: 0000:07:01:07
2020-07-03 22:23:16,891 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15900799 = 2020-07-05 00:23:19 UTC estimated time to finish: 0000:03:00:47
2020-07-03 22:23:54,765 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39456669 = 2020-07-03 15:34:27 UTC estimated time to finish: 0000:06:56:27
2020-07-03 22:28:17,016 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15901931 = 2020-07-05 00:42:11 UTC estimated time to finish: 0000:02:55:57
2020-07-03 22:28:54,870 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39457529 = 2020-07-03 15:40:11 UTC estimated time to finish: 0000:06:51:56
2020-07-03 22:33:17,143 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15903076 = 2020-07-05 01:01:16 UTC estimated time to finish: 0000:02:51:08
2020-07-03 22:33:54,950 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39458320 = 2020-07-03 15:45:28 UTC estimated time to finish: 0000:06:47:47
2020-07-03 22:38:17,269 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15904099 = 2020-07-05 01:18:19 UTC estimated time to finish: 0000:02:46:47
2020-07-03 22:38:55,091 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39459153 = 2020-07-03 15:51:01 UTC estimated time to finish: 0000:06:43:24
2020-07-03 22:43:17,397 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15905244 = 2020-07-05 01:37:24 UTC estimated time to finish: 0000:02:41:51
2020-07-03 22:43:55,232 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39460096 = 2020-07-03 15:57:18 UTC estimated time to finish: 0000:06:38:13
2020-07-03 22:48:17,524 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15906511 = 2020-07-05 01:58:31 UTC estimated time to finish: 0000:02:36:12
2020-07-03 22:48:55,372 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39461324 = 2020-07-03 16:05:29 UTC estimated time to finish: 0000:06:30:38
2020-07-03 22:53:17,651 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15907847 = 2020-07-05 02:20:47 UTC estimated time to finish: 0000:02:30:13
2020-07-03 22:53:55,498 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39462500 = 2020-07-03 16:13:20 UTC estimated time to finish: 0000:06:23:21
2020-07-03 22:58:17,779 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15908992 = 2020-07-05 02:39:52 UTC estimated time to finish: 0000:02:25:09
2020-07-03 22:58:55,639 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39463769 = 2020-07-03 16:21:47 UTC estimated time to finish: 0000:06:15:32
2020-07-03 23:03:17,908 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15910246 = 2020-07-05 03:00:46 UTC estimated time to finish: 0000:02:19:36
2020-07-03 23:03:55,781 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39464999 = 2020-07-03 16:29:59 UTC estimated time to finish: 0000:06:08:03
2020-07-03 23:08:18,035 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15911554 = 2020-07-05 03:22:34 UTC estimated time to finish: 0000:02:13:48
2020-07-03 23:08:55,921 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39466297 = 2020-07-03 16:38:38 UTC estimated time to finish: 0000:06:00:09
2020-07-03 23:13:18,163 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15912809 = 2020-07-05 03:43:29 UTC estimated time to finish: 0000:02:08:17
2020-07-03 23:13:56,062 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39467443 = 2020-07-03 16:46:17 UTC estimated time to finish: 0000:05:53:11
2020-07-03 23:18:18,291 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15913994 = 2020-07-05 04:03:14 UTC estimated time to finish: 0000:02:03:05
2020-07-03 23:18:56,205 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39468700 = 2020-07-03 16:54:40 UTC estimated time to finish: 0000:05:45:37
2020-07-03 23:23:18,420 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15915249 = 2020-07-05 04:24:09 UTC estimated time to finish: 0000:01:57:37
2020-07-03 23:23:56,348 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39469970 = 2020-07-03 17:03:08 UTC estimated time to finish: 0000:05:38:02
2020-07-03 23:28:18,548 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15916476 = 2020-07-05 04:44:36 UTC estimated time to finish: 0000:01:52:16
2020-07-03 23:28:56,406 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39471116 = 2020-07-03 17:10:46 UTC estimated time to finish: 0000:05:31:25
2020-07-03 23:33:18,678 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15917620 = 2020-07-05 05:03:40 UTC estimated time to finish: 0000:01:47:26
2020-07-03 23:33:56,550 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39471977 = 2020-07-03 17:16:30 UTC estimated time to finish: 0000:05:26:56
2020-07-03 23:38:18,807 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15918765 = 2020-07-05 05:22:45 UTC estimated time to finish: 0000:01:42:32
2020-07-03 23:38:56,692 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39472769 = 2020-07-03 17:21:47 UTC estimated time to finish: 0000:05:22:49
2020-07-03 23:43:18,939 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15919789 = 2020-07-05 05:39:49 UTC estimated time to finish: 0000:01:38:11
2020-07-03 23:43:56,837 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39473640 = 2020-07-03 17:27:36 UTC estimated time to finish: 0000:05:18:16
2020-07-03 23:48:19,069 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15920933 = 2020-07-05 05:58:53 UTC estimated time to finish: 0000:01:33:16
2020-07-03 23:48:56,930 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39474505 = 2020-07-03 17:33:22 UTC estimated time to finish: 0000:05:13:44
2020-07-03 23:53:19,200 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15922106 = 2020-07-05 06:18:26 UTC estimated time to finish: 0000:01:28:14
2020-07-03 23:53:57,074 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39475338 = 2020-07-03 17:38:55 UTC estimated time to finish: 0000:05:09:24
2020-07-03 23:58:19,332 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15923251 = 2020-07-05 06:37:31 UTC estimated time to finish: 0000:01:23:21
2020-07-03 23:58:57,220 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39476160 = 2020-07-03 17:44:24 UTC estimated time to finish: 0000:05:05:04
2020-07-04 00:03:19,464 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15924240 = 2020-07-05 06:54:00 UTC estimated time to finish: 0000:01:19:08
2020-07-04 00:03:57,365 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39476990 = 2020-07-03 17:49:56 UTC estimated time to finish: 0000:05:00:46
2020-07-04 00:08:19,527 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15925378 = 2020-07-05 07:12:58 UTC estimated time to finish: 0000:01:14:16
2020-07-04 00:08:57,512 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39477863 = 2020-07-03 17:55:45 UTC estimated time to finish: 0000:04:56:10
2020-07-04 00:13:19,659 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15927080 = 2020-07-05 07:41:20 UTC estimated time to finish: 0000:01:06:38
2020-07-04 00:13:57,568 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39478847 = 2020-07-03 18:02:18 UTC estimated time to finish: 0000:04:50:48
2020-07-04 00:18:19,791 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15928771 = 2020-07-05 08:09:31 UTC estimated time to finish: 0000:00:59:06
2020-07-04 00:18:57,714 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39479790 = 2020-07-03 18:08:36 UTC estimated time to finish: 0000:04:45:42
2020-07-04 00:23:19,925 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15930352 = 2020-07-05 08:35:52 UTC estimated time to finish: 0000:00:52:08
2020-07-04 00:23:57,860 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39480622 = 2020-07-03 18:14:08 UTC estimated time to finish: 0000:04:41:11
2020-07-04 00:28:20,059 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15932002 = 2020-07-05 09:03:22 UTC estimated time to finish: 0000:00:44:56
2020-07-04 00:28:58,009 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39481605 = 2020-07-03 18:20:42 UTC estimated time to finish: 0000:04:35:48
2020-07-04 00:33:20,196 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15933501 = 2020-07-05 09:28:21 UTC estimated time to finish: 0000:00:38:29
2020-07-04 00:33:58,156 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39482520 = 2020-07-03 18:26:48 UTC estimated time to finish: 0000:04:30:55
2020-07-04 00:38:20,333 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15934606 = 2020-07-05 09:46:46 UTC estimated time to finish: 0000:00:33:50
2020-07-04 00:38:58,305 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39483382 = 2020-07-03 18:32:32 UTC estimated time to finish: 0000:04:26:21
2020-07-04 00:43:20,468 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15935779 = 2020-07-05 10:06:19 UTC estimated time to finish: 0000:00:28:53
2020-07-04 00:43:58,453 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39484174 = 2020-07-03 18:37:49 UTC estimated time to finish: 0000:04:22:09
2020-07-04 00:48:20,604 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15936842 = 2020-07-05 10:24:02 UTC estimated time to finish: 0000:00:24:23
2020-07-04 00:48:58,603 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39485048 = 2020-07-03 18:43:39 UTC estimated time to finish: 0000:04:17:29
2020-07-04 00:53:20,742 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15937947 = 2020-07-05 10:42:27 UTC estimated time to finish: 0000:00:19:43
2020-07-04 00:53:58,752 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39486030 = 2020-07-03 18:50:12 UTC estimated time to finish: 0000:04:12:08
2020-07-04 00:58:20,878 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15939214 = 2020-07-05 11:03:34 UTC estimated time to finish: 0000:00:14:20
2020-07-04 00:58:58,906 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39487329 = 2020-07-03 18:58:51 UTC estimated time to finish: 0000:04:04:35
2020-07-04 01:03:21,015 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15940713 = 2020-07-05 11:28:33 UTC estimated time to finish: 0000:00:07:58
2020-07-04 01:03:59,055 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39488516 = 2020-07-03 19:06:46 UTC estimated time to finish: 0000:03:57:41
2020-07-04 01:08:21,151 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15942322 = 2020-07-05 11:55:22 UTC estimated time to finish: 0000:00:01:10
2020-07-04 01:08:59,226 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39489936 = 2020-07-03 19:16:14 UTC estimated time to finish: 0000:03:49:24
2020-07-04 01:13:21,260 INFO [watch_fvcom] x2 forecast FVCOM VH-FR run on arbutus.cloud-nowcast completed
2020-07-04 01:13:21,327 INFO [manager] checklist updated with [FVCOM run] items from watch_fvcom worker
2020-07-04 01:13:22,030 INFO [manager] launching NextWorker(module='nowcast.workers.download_fvcom_results', args=['arbutus.cloud-nowcast', 'x2', 'forecast', '--run-date', '2020-07-03'], host='localhost')
2020-07-04 01:13:23,634 INFO [download_fvcom_results] running in process 27114
2020-07-04 01:13:23,635 INFO [download_fvcom_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 01:13:23,636 INFO [download_fvcom_results] publishing log messages to tcp://*:5563
2020-07-04 01:13:23,637 INFO [download_fvcom_results] connected to skookum.eos.ubc.ca port 5555
2020-07-04 01:13:59,383 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39492464 = 2020-07-03 19:33:05 UTC estimated time to finish: 0000:03:33:50
2020-07-04 01:14:32,773 INFO [download_fvcom_results] VHFR FVCOM x2 forecast 2020-07-03 results files from arbutus.cloud-nowcast downloaded
2020-07-04 01:14:32,780 INFO [manager] checklist updated with [VHFR FVCOM results files] items from download_fvcom_results worker
2020-07-04 01:14:33,136 INFO [manager] launching NextWorker(module='nowcast.workers.get_vfpa_hadcp', args=['--data-date', '2020-07-03'], host='localhost')
2020-07-04 01:14:33,140 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['fvcom', 'forecast-x2', 'research', '--run-date', '2020-07-03'], host='localhost')
2020-07-04 01:14:33,143 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['fvcom', 'forecast', '--run-date', '2020-07-03'], host='localhost')
2020-07-04 01:14:34,872 INFO [update_forecast_datasets] running in process 27124
2020-07-04 01:14:34,873 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 01:14:34,873 INFO [update_forecast_datasets] publishing log messages to tcp://*:5563
2020-07-04 01:14:34,874 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2020-07-04 01:14:34,875 INFO [update_forecast_datasets] updating fvcom most_recent_forecast directory from forecast/03jul20 run
2020-07-04 01:14:34,877 INFO [update_forecast_datasets] updated fvcom most_recent_forecast directory from forecast/03jul20 run
2020-07-04 01:14:34,877 INFO [update_forecast_datasets] fvcom 2020-07-03 forecast rolling forecast datasets updated
2020-07-04 01:14:34,879 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2020-07-04 01:14:35,221 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['fvcom-forecast'], host='localhost')
2020-07-04 01:14:36,515 INFO [get_vfpa_hadcp] running in process 27122
2020-07-04 01:14:36,516 INFO [get_vfpa_hadcp] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 01:14:36,516 INFO [get_vfpa_hadcp] publishing log messages to tcp://*:5563
2020-07-04 01:14:36,517 INFO [get_vfpa_hadcp] connected to skookum.eos.ubc.ca port 5555
2020-07-04 01:14:36,518 INFO [get_vfpa_hadcp] processing VFPA HADCP data from 2nd Narrows Rail Bridge for 2020-07-03
2020-07-04 01:14:37,932 INFO [get_vfpa_hadcp] added VFPA HADCP data from 2nd Narrows Rail Bridge for 2020-07-03 to /opp/observations/AISDATA/netcdf/VFPA_2ND_NARROWS_HADCP_2s_202007.nc
2020-07-04 01:14:37,933 INFO [get_vfpa_hadcp] VFPA HADCP observations added to 2020-07 netcdf file
2020-07-04 01:14:37,936 INFO [manager] checklist updated with [VFPA HADCP data] items from get_vfpa_hadcp worker
2020-07-04 01:14:38,294 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['VFPA-HADCP'], host='localhost')
2020-07-04 01:14:38,358 INFO [make_plots] running in process 27123
2020-07-04 01:14:38,360 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 01:14:38,361 INFO [make_plots] publishing log messages to tcp://*:5564
2020-07-04 01:14:38,361 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2020-07-04 01:14:43,374 INFO [ping_erddap] running in process 27140
2020-07-04 01:14:43,375 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 01:14:43,376 INFO [ping_erddap] publishing log messages to tcp://*:5563
2020-07-04 01:14:43,377 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2020-07-04 01:14:43,397 INFO [ping_erddap] VFPA-HADCP ERDDAP dataset flag file(s) created
2020-07-04 01:14:43,401 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2020-07-04 01:14:43,760 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['fvcom', 'nowcast-x2', 'publish', '--run-date', '2020-07-03'], host='localhost')
2020-07-04 01:14:43,764 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['fvcom', 'forecast-x2', 'publish', '--run-date', '2020-07-03'], host='localhost')
2020-07-04 01:14:46,612 INFO [make_plots] running in process 27146
2020-07-04 01:14:46,613 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 01:14:46,614 INFO [make_plots] publishing log messages to tcp://*:5563
2020-07-04 01:14:46,614 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2020-07-04 01:14:46,682 INFO [make_plots] running in process 27145
2020-07-04 01:14:46,683 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 01:14:46,684 INFO [make_plots] publishing log messages to tcp://*:5567
2020-07-04 01:14:46,685 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2020-07-04 01:14:50,792 INFO [make_plots] preparing render list for 2020-07-03 VHFR FVCOM forecast-x2 research figures
2020-07-04 01:14:50,801 INFO [make_plots] added EnglishBay_surface_currents figures to 2020-07-03 VHFR FVCOM forecast-x2 research render list
2020-07-04 01:14:50,808 INFO [make_plots] added VancouverHarbour_surface_currents figures to 2020-07-03 VHFR FVCOM forecast-x2 research render list
2020-07-04 01:14:50,815 INFO [make_plots] added IndianArm_surface_currents figures to 2020-07-03 VHFR FVCOM forecast-x2 research render list
2020-07-04 01:14:50,823 INFO [make_plots] added VancouverHarbour_thalweg figures to 2020-07-03 VHFR FVCOM forecast-x2 research render list
2020-07-04 01:14:50,830 INFO [make_plots] added PortMoody_thalweg figures to 2020-07-03 VHFR FVCOM forecast-x2 research render list
2020-07-04 01:14:50,836 INFO [make_plots] added IndianArm_thalweg figures to 2020-07-03 VHFR FVCOM forecast-x2 research render list
2020-07-04 01:14:50,837 INFO [make_plots] starting to render fvcom forecast-x2 research 03jul20 figures
2020-07-04 01:14:51,782 INFO [make_plots] preparing render list for 2020-07-03 VHFR FVCOM forecast-x2 publish figures
2020-07-04 01:14:51,783 INFO [make_plots] added Calamity Point figure to 2020-07-03 VHFR FVCOM forecast-x2 publish render list
2020-07-04 01:14:51,783 INFO [make_plots] added Indian Arm Head figure to 2020-07-03 VHFR FVCOM forecast-x2 publish render list
2020-07-04 01:14:51,784 INFO [make_plots] added New Westminster figure to 2020-07-03 VHFR FVCOM forecast-x2 publish render list
2020-07-04 01:14:51,785 INFO [make_plots] added Port Moody figure to 2020-07-03 VHFR FVCOM forecast-x2 publish render list
2020-07-04 01:14:51,785 INFO [make_plots] added Sand Heads figure to 2020-07-03 VHFR FVCOM forecast-x2 publish render list
2020-07-04 01:14:51,786 INFO [make_plots] added Sandy Cove figure to 2020-07-03 VHFR FVCOM forecast-x2 publish render list
2020-07-04 01:14:51,787 INFO [make_plots] added Vancouver Harbour figure to 2020-07-03 VHFR FVCOM forecast-x2 publish render list
2020-07-04 01:14:51,787 INFO [make_plots] added Woodwards Landing figure to 2020-07-03 VHFR FVCOM forecast-x2 publish render list
2020-07-04 01:14:51,788 INFO [make_plots] added 2ndNarrows_current figure to 2020-07-03 VHFR FVCOM forecast-x2 publish render list
2020-07-04 01:14:51,789 INFO [make_plots] starting to render fvcom forecast-x2 publish 03jul20 figures
2020-07-04 01:14:51,814 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07724 Calamity Point from 2020-07-02 23:55:00 to 2020-07-05 12:00:00
2020-07-04 01:14:51,815 INFO [make_plots] preparing render list for 2020-07-03 VHFR FVCOM nowcast-x2 publish figures
2020-07-04 01:14:51,816 INFO [make_plots] added Calamity Point figure to 2020-07-03 VHFR FVCOM nowcast-x2 publish render list
2020-07-04 01:14:51,816 INFO [make_plots] added Indian Arm Head figure to 2020-07-03 VHFR FVCOM nowcast-x2 publish render list
2020-07-04 01:14:51,817 INFO [make_plots] added New Westminster figure to 2020-07-03 VHFR FVCOM nowcast-x2 publish render list
2020-07-04 01:14:51,817 INFO [make_plots] added Port Moody figure to 2020-07-03 VHFR FVCOM nowcast-x2 publish render list
2020-07-04 01:14:51,818 INFO [make_plots] added Sand Heads figure to 2020-07-03 VHFR FVCOM nowcast-x2 publish render list
2020-07-04 01:14:51,819 INFO [make_plots] added Sandy Cove figure to 2020-07-03 VHFR FVCOM nowcast-x2 publish render list
2020-07-04 01:14:51,819 INFO [make_plots] added Vancouver Harbour figure to 2020-07-03 VHFR FVCOM nowcast-x2 publish render list
2020-07-04 01:14:51,820 INFO [make_plots] added Woodwards Landing figure to 2020-07-03 VHFR FVCOM nowcast-x2 publish render list
2020-07-04 01:14:51,820 INFO [make_plots] added 2ndNarrows_current figure to 2020-07-03 VHFR FVCOM nowcast-x2 publish render list
2020-07-04 01:14:51,821 INFO [make_plots] starting to render fvcom nowcast-x2 publish 03jul20 figures
2020-07-04 01:14:51,837 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07724 Calamity Point from 2020-07-02 23:55:00 to 2020-07-04 00:00:00
2020-07-04 01:15:15,404 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07724 Calamity Point from 2020-07-03 00:00:00 to 2020-07-04 00:00:00
2020-07-04 01:15:16,932 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07774 Indian Arm Head from 2020-07-02 23:55:00 to 2020-07-04 00:00:00
2020-07-04 01:15:17,708 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07774 Indian Arm Head from 2020-07-03 00:00:00 to 2020-07-04 00:00:00
2020-07-04 01:15:19,407 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07654 New Westminster from 2020-07-02 23:55:00 to 2020-07-04 00:00:00
2020-07-04 01:15:22,589 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07724 Calamity Point from 2020-07-03 00:00:00 to 2020-07-05 12:00:00
2020-07-04 01:15:24,004 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07774 Indian Arm Head from 2020-07-02 23:55:00 to 2020-07-05 12:00:00
2020-07-04 01:15:24,555 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07774 Indian Arm Head from 2020-07-03 00:00:00 to 2020-07-05 12:00:00
2020-07-04 01:15:26,145 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07654 New Westminster from 2020-07-02 23:55:00 to 2020-07-05 12:00:00
2020-07-04 01:15:41,260 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07654 New Westminster from 2020-07-03 00:00:00 to 2020-07-04 00:00:00
2020-07-04 01:15:44,410 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07755 Port Moody from 2020-07-02 23:55:00 to 2020-07-04 00:00:00
2020-07-04 01:15:55,246 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07654 New Westminster from 2020-07-03 00:00:00 to 2020-07-05 12:00:00
2020-07-04 01:16:00,463 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07755 Port Moody from 2020-07-02 23:55:00 to 2020-07-05 12:00:00
2020-07-04 01:16:06,385 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07755 Port Moody from 2020-07-03 00:00:00 to 2020-07-04 00:00:00
2020-07-04 01:16:09,288 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07594 Sand Heads from 2020-07-02 23:55:00 to 2020-07-04 00:00:00
2020-07-04 01:16:29,389 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07755 Port Moody from 2020-07-03 00:00:00 to 2020-07-05 12:00:00
2020-07-04 01:16:32,239 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07594 Sand Heads from 2020-07-03 00:00:00 to 2020-07-04 00:00:00
2020-07-04 01:16:34,036 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07594 Sand Heads from 2020-07-02 23:55:00 to 2020-07-05 12:00:00
2020-07-04 01:16:35,239 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07786 Sandy Cove from 2020-07-02 23:55:00 to 2020-07-04 00:00:00
2020-07-04 01:16:57,110 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07795 Point Atkinson from 2020-07-03 00:00:00 to 2020-07-04 00:00:00
2020-07-04 01:16:59,978 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07735 Vancouver Harbour from 2020-07-02 23:55:00 to 2020-07-04 00:00:00
2020-07-04 01:17:02,374 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07594 Sand Heads from 2020-07-03 00:00:00 to 2020-07-05 12:00:00
2020-07-04 01:17:07,390 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07786 Sandy Cove from 2020-07-02 23:55:00 to 2020-07-05 12:00:00
2020-07-04 01:17:22,430 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07735 Vancouver Harbour from 2020-07-03 00:00:00 to 2020-07-04 00:00:00
2020-07-04 01:17:25,394 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07610 Woodwards Landing from 2020-07-02 23:55:00 to 2020-07-04 00:00:00
2020-07-04 01:17:35,737 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07795 Point Atkinson from 2020-07-03 00:00:00 to 2020-07-05 12:00:00
2020-07-04 01:17:40,594 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07735 Vancouver Harbour from 2020-07-02 23:55:00 to 2020-07-05 12:00:00
2020-07-04 01:17:47,381 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07610 Woodwards Landing from 2020-07-03 00:00:00 to 2020-07-04 00:00:00
2020-07-04 01:17:52,369 INFO [make_plots] finished rendering fvcom nowcast-x2 publish 03jul20 figures
2020-07-04 01:17:52,892 INFO [make_plots] fvcom publish plots for 2020-07-03 nowcast-x2 completed
2020-07-04 01:17:52,900 INFO [manager] checklist updated with [plots] items from make_plots worker
2020-07-04 01:18:09,380 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07735 Vancouver Harbour from 2020-07-03 00:00:00 to 2020-07-05 12:00:00
2020-07-04 01:18:14,275 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07610 Woodwards Landing from 2020-07-02 23:55:00 to 2020-07-05 12:00:00
2020-07-04 01:18:42,475 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07610 Woodwards Landing from 2020-07-03 00:00:00 to 2020-07-05 12:00:00
2020-07-04 01:18:47,379 INFO [make_plots] finished rendering fvcom forecast-x2 publish 03jul20 figures
2020-07-04 01:18:47,991 INFO [make_plots] fvcom publish plots for 2020-07-03 forecast-x2 completed
2020-07-04 01:18:47,997 INFO [manager] checklist updated with [plots] items from make_plots worker
2020-07-04 01:18:59,506 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39494990 = 2020-07-03 19:49:56 UTC estimated time to finish: 0000:03:18:35
2020-07-04 01:23:59,655 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39497516 = 2020-07-03 20:06:46 UTC estimated time to finish: 0000:03:03:37
2020-07-04 01:28:59,808 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39499769 = 2020-07-03 20:21:47 UTC estimated time to finish: 0000:02:50:29
2020-07-04 01:33:59,962 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39502297 = 2020-07-03 20:38:38 UTC estimated time to finish: 0000:02:36:01
2020-07-04 01:39:00,117 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39504822 = 2020-07-03 20:55:28 UTC estimated time to finish: 0000:02:21:49
2020-07-04 01:44:00,273 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39507349 = 2020-07-03 21:12:19 UTC estimated time to finish: 0000:02:07:50
2020-07-04 01:49:00,429 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39509602 = 2020-07-03 21:27:20 UTC estimated time to finish: 0000:01:55:34
2020-07-04 01:54:00,586 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39512130 = 2020-07-03 21:44:12 UTC estimated time to finish: 0000:01:42:01
2020-07-04 01:59:00,743 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39514655 = 2020-07-03 22:01:02 UTC estimated time to finish: 0000:01:28:41
2020-07-04 02:04:00,902 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39517181 = 2020-07-03 22:17:52 UTC estimated time to finish: 0000:01:15:34
2020-07-04 02:09:01,063 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39519464 = 2020-07-03 22:33:05 UTC estimated time to finish: 0000:01:03:53
2020-07-04 02:14:01,222 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39521990 = 2020-07-03 22:49:56 UTC estimated time to finish: 0000:00:51:08
2020-07-04 02:16:38,318 INFO [make_plots] finished rendering fvcom forecast-x2 research 03jul20 figures
2020-07-04 02:16:38,319 INFO [make_plots] fvcom research plots for 2020-07-03 forecast-x2 completed
2020-07-04 02:16:38,550 INFO [manager] checklist updated with [plots] items from make_plots worker
2020-07-04 02:19:01,383 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39524516 = 2020-07-03 23:06:46 UTC estimated time to finish: 0000:00:38:34
2020-07-04 02:24:01,543 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39527042 = 2020-07-03 23:23:36 UTC estimated time to finish: 0000:00:26:10
2020-07-04 02:29:01,705 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39529529 = 2020-07-03 23:40:11 UTC estimated time to finish: 0000:00:14:09
2020-07-04 02:34:01,866 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39531781 = 2020-07-03 23:55:12 UTC estimated time to finish: 0000:00:03:24
2020-07-04 02:39:01,982 INFO [watch_fvcom] r12 nowcast FVCOM VH-FR run on arbutus.cloud-nowcast completed
2020-07-04 02:39:01,989 INFO [manager] checklist updated with [FVCOM run] items from watch_fvcom worker
2020-07-04 02:39:02,688 INFO [manager] launching NextWorker(module='nowcast.workers.download_fvcom_results', args=['arbutus.cloud-nowcast', 'r12', 'nowcast', '--run-date', '2020-07-03'], host='localhost')
2020-07-04 02:39:04,069 INFO [download_fvcom_results] running in process 28826
2020-07-04 02:39:04,070 INFO [download_fvcom_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 02:39:04,071 INFO [download_fvcom_results] publishing log messages to tcp://*:5563
2020-07-04 02:39:04,072 INFO [download_fvcom_results] connected to skookum.eos.ubc.ca port 5555
2020-07-04 02:40:37,321 INFO [download_fvcom_results] VHFR FVCOM r12 nowcast 2020-07-03 results files from arbutus.cloud-nowcast downloaded
2020-07-04 02:40:37,329 INFO [manager] checklist updated with [VHFR FVCOM results files] items from download_fvcom_results worker
2020-07-04 02:40:37,774 INFO [manager] launching NextWorker(module='nowcast.workers.get_vfpa_hadcp', args=['--data-date', '2020-07-03'], host='localhost')
2020-07-04 02:40:37,778 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['fvcom', 'nowcast-r12', 'research', '--run-date', '2020-07-03'], host='localhost')
2020-07-04 02:40:40,213 INFO [get_vfpa_hadcp] running in process 28834
2020-07-04 02:40:40,216 INFO [get_vfpa_hadcp] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 02:40:40,217 INFO [get_vfpa_hadcp] publishing log messages to tcp://*:5563
2020-07-04 02:40:40,218 INFO [get_vfpa_hadcp] connected to skookum.eos.ubc.ca port 5555
2020-07-04 02:40:40,219 INFO [get_vfpa_hadcp] processing VFPA HADCP data from 2nd Narrows Rail Bridge for 2020-07-03
2020-07-04 02:40:41,334 INFO [get_vfpa_hadcp] added VFPA HADCP data from 2nd Narrows Rail Bridge for 2020-07-03 to /opp/observations/AISDATA/netcdf/VFPA_2ND_NARROWS_HADCP_2s_202007.nc
2020-07-04 02:40:41,335 INFO [get_vfpa_hadcp] VFPA HADCP observations added to 2020-07 netcdf file
2020-07-04 02:40:41,338 INFO [manager] checklist updated with [VFPA HADCP data] items from get_vfpa_hadcp worker
2020-07-04 02:40:41,782 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['VFPA-HADCP'], host='localhost')
2020-07-04 02:40:41,951 INFO [make_plots] running in process 28836
2020-07-04 02:40:41,952 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 02:40:41,952 INFO [make_plots] publishing log messages to tcp://*:5564
2020-07-04 02:40:41,953 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2020-07-04 02:40:41,954 INFO [make_plots] preparing render list for 2020-07-03 VHFR FVCOM nowcast-r12 research figures
2020-07-04 02:40:41,956 INFO [make_plots] added EnglishBay_surface_currents figures to 2020-07-03 VHFR FVCOM nowcast-r12 research render list
2020-07-04 02:40:41,959 INFO [make_plots] added VancouverHarbour_surface_currents figures to 2020-07-03 VHFR FVCOM nowcast-r12 research render list
2020-07-04 02:40:41,961 INFO [make_plots] added IndianArm_surface_currents figures to 2020-07-03 VHFR FVCOM nowcast-r12 research render list
2020-07-04 02:40:41,963 INFO [make_plots] added VancouverHarbour_thalweg figures to 2020-07-03 VHFR FVCOM nowcast-r12 research render list
2020-07-04 02:40:41,965 INFO [make_plots] added PortMoody_thalweg figures to 2020-07-03 VHFR FVCOM nowcast-r12 research render list
2020-07-04 02:40:41,967 INFO [make_plots] added IndianArm_thalweg figures to 2020-07-03 VHFR FVCOM nowcast-r12 research render list
2020-07-04 02:40:41,967 INFO [make_plots] starting to render fvcom nowcast-r12 research 03jul20 figures
2020-07-04 02:40:42,652 INFO [ping_erddap] running in process 28845
2020-07-04 02:40:42,653 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 02:40:42,654 INFO [ping_erddap] publishing log messages to tcp://*:5563
2020-07-04 02:40:42,654 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2020-07-04 02:40:42,656 INFO [ping_erddap] VFPA-HADCP ERDDAP dataset flag file(s) created
2020-07-04 02:40:42,657 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2020-07-04 02:40:43,003 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['fvcom', 'nowcast-x2', 'publish', '--run-date', '2020-07-03'], host='localhost')
2020-07-04 02:40:43,006 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['fvcom', 'nowcast-r12', 'publish', '--run-date', '2020-07-03'], host='localhost')
2020-07-04 02:40:43,009 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['fvcom', 'forecast-x2', 'publish', '--run-date', '2020-07-03'], host='localhost')
2020-07-04 02:40:45,691 INFO [make_plots] running in process 28851
2020-07-04 02:40:45,692 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 02:40:45,693 INFO [make_plots] publishing log messages to tcp://*:5563
2020-07-04 02:40:45,694 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2020-07-04 02:40:45,783 INFO [make_plots] running in process 28850
2020-07-04 02:40:45,784 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 02:40:45,785 INFO [make_plots] publishing log messages to tcp://*:5567
2020-07-04 02:40:45,786 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2020-07-04 02:40:45,802 INFO [make_plots] running in process 28849
2020-07-04 02:40:45,803 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 02:40:45,803 INFO [make_plots] publishing log messages to tcp://*:5568
2020-07-04 02:40:45,804 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2020-07-04 02:40:46,834 INFO [make_plots] preparing render list for 2020-07-03 VHFR FVCOM nowcast-x2 publish figures
2020-07-04 02:40:46,844 INFO [make_plots] added Calamity Point figure to 2020-07-03 VHFR FVCOM nowcast-x2 publish render list
2020-07-04 02:40:46,845 INFO [make_plots] added Indian Arm Head figure to 2020-07-03 VHFR FVCOM nowcast-x2 publish render list
2020-07-04 02:40:46,846 INFO [make_plots] added New Westminster figure to 2020-07-03 VHFR FVCOM nowcast-x2 publish render list
2020-07-04 02:40:46,847 INFO [make_plots] added Port Moody figure to 2020-07-03 VHFR FVCOM nowcast-x2 publish render list
2020-07-04 02:40:46,847 INFO [make_plots] added Sand Heads figure to 2020-07-03 VHFR FVCOM nowcast-x2 publish render list
2020-07-04 02:40:46,848 INFO [make_plots] added Sandy Cove figure to 2020-07-03 VHFR FVCOM nowcast-x2 publish render list
2020-07-04 02:40:46,848 INFO [make_plots] added Vancouver Harbour figure to 2020-07-03 VHFR FVCOM nowcast-x2 publish render list
2020-07-04 02:40:46,849 INFO [make_plots] added Woodwards Landing figure to 2020-07-03 VHFR FVCOM nowcast-x2 publish render list
2020-07-04 02:40:46,850 INFO [make_plots] added 2ndNarrows_current figure to 2020-07-03 VHFR FVCOM nowcast-x2 publish render list
2020-07-04 02:40:46,850 INFO [make_plots] starting to render fvcom nowcast-x2 publish 03jul20 figures
2020-07-04 02:40:46,852 INFO [make_plots] preparing render list for 2020-07-03 VHFR FVCOM nowcast-r12 publish figures
2020-07-04 02:40:46,853 INFO [make_plots] added Calamity Point figure to 2020-07-03 VHFR FVCOM nowcast-r12 publish render list
2020-07-04 02:40:46,854 INFO [make_plots] added Indian Arm Head figure to 2020-07-03 VHFR FVCOM nowcast-r12 publish render list
2020-07-04 02:40:46,855 INFO [make_plots] added New Westminster figure to 2020-07-03 VHFR FVCOM nowcast-r12 publish render list
2020-07-04 02:40:46,855 INFO [make_plots] added Port Moody figure to 2020-07-03 VHFR FVCOM nowcast-r12 publish render list
2020-07-04 02:40:46,856 INFO [make_plots] added Sand Heads figure to 2020-07-03 VHFR FVCOM nowcast-r12 publish render list
2020-07-04 02:40:46,857 INFO [make_plots] added Sandy Cove figure to 2020-07-03 VHFR FVCOM nowcast-r12 publish render list
2020-07-04 02:40:46,858 INFO [make_plots] added Vancouver Harbour figure to 2020-07-03 VHFR FVCOM nowcast-r12 publish render list
2020-07-04 02:40:46,858 INFO [make_plots] added Woodwards Landing figure to 2020-07-03 VHFR FVCOM nowcast-r12 publish render list
2020-07-04 02:40:46,859 INFO [make_plots] added 2ndNarrows_current figure to 2020-07-03 VHFR FVCOM nowcast-r12 publish render list
2020-07-04 02:40:46,860 INFO [make_plots] starting to render fvcom nowcast-r12 publish 03jul20 figures
2020-07-04 02:40:46,860 ERROR [make_plots] unexpected IndexError in make_figure:
Traceback (most recent call last):
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure
    fig = fig_func(*args, **kwargs)
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/publish/tide_stn_water_level.py", line 78, in make_figure
    plot_data = _prep_plot_data(place, fvcom_ssh_datasets, nemo_ssh_dataset_url_tmpl)
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/publish/tide_stn_water_level.py", line 92, in _prep_plot_data
    station=[
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/publish/tide_stn_water_level.py", line 93, in <listcomp>
    name.decode().strip().split(maxsplit=1)[1]
IndexError: list index out of range
2020-07-04 02:40:46,861 ERROR [make_plots] unexpected IndexError in make_figure:
Traceback (most recent call last):
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure
    fig = fig_func(*args, **kwargs)
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/publish/tide_stn_water_level.py", line 78, in make_figure
    plot_data = _prep_plot_data(place, fvcom_ssh_datasets, nemo_ssh_dataset_url_tmpl)
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/publish/tide_stn_water_level.py", line 92, in _prep_plot_data
    station=[
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/publish/tide_stn_water_level.py", line 93, in <listcomp>
    name.decode().strip().split(maxsplit=1)[1]
IndexError: list index out of range
2020-07-04 02:40:46,889 INFO [make_plots] preparing render list for 2020-07-03 VHFR FVCOM forecast-x2 publish figures
2020-07-04 02:40:46,890 INFO [make_plots] added Calamity Point figure to 2020-07-03 VHFR FVCOM forecast-x2 publish render list
2020-07-04 02:40:46,891 INFO [make_plots] added Indian Arm Head figure to 2020-07-03 VHFR FVCOM forecast-x2 publish render list
2020-07-04 02:40:46,892 INFO [make_plots] added New Westminster figure to 2020-07-03 VHFR FVCOM forecast-x2 publish render list
2020-07-04 02:40:46,892 INFO [make_plots] added Port Moody figure to 2020-07-03 VHFR FVCOM forecast-x2 publish render list
2020-07-04 02:40:46,893 INFO [make_plots] added Sand Heads figure to 2020-07-03 VHFR FVCOM forecast-x2 publish render list
2020-07-04 02:40:46,894 INFO [make_plots] added Sandy Cove figure to 2020-07-03 VHFR FVCOM forecast-x2 publish render list
2020-07-04 02:40:46,894 INFO [make_plots] added Vancouver Harbour figure to 2020-07-03 VHFR FVCOM forecast-x2 publish render list
2020-07-04 02:40:46,895 INFO [make_plots] added Woodwards Landing figure to 2020-07-03 VHFR FVCOM forecast-x2 publish render list
2020-07-04 02:40:46,895 INFO [make_plots] added 2ndNarrows_current figure to 2020-07-03 VHFR FVCOM forecast-x2 publish render list
2020-07-04 02:40:46,896 INFO [make_plots] starting to render fvcom forecast-x2 publish 03jul20 figures
2020-07-04 02:40:46,916 ERROR [make_plots] unexpected IndexError in make_figure:
Traceback (most recent call last):
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure
    fig = fig_func(*args, **kwargs)
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/publish/tide_stn_water_level.py", line 78, in make_figure
    plot_data = _prep_plot_data(place, fvcom_ssh_datasets, nemo_ssh_dataset_url_tmpl)
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/publish/tide_stn_water_level.py", line 92, in _prep_plot_data
    station=[
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/publish/tide_stn_water_level.py", line 93, in <listcomp>
    name.decode().strip().split(maxsplit=1)[1]
IndexError: list index out of range
2020-07-04 02:40:46,922 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07724 Calamity Point from 2020-07-02 23:55:00 to 2020-07-05 12:00:00
2020-07-04 02:40:46,965 ERROR [make_plots] unexpected IndexError in make_figure:
Traceback (most recent call last):
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure
    fig = fig_func(*args, **kwargs)
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/publish/tide_stn_water_level.py", line 78, in make_figure
    plot_data = _prep_plot_data(place, fvcom_ssh_datasets, nemo_ssh_dataset_url_tmpl)
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/publish/tide_stn_water_level.py", line 92, in _prep_plot_data
    station=[
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/publish/tide_stn_water_level.py", line 93, in <listcomp>
    name.decode().strip().split(maxsplit=1)[1]
IndexError: list index out of range
2020-07-04 02:40:46,968 ERROR [make_plots] unexpected IndexError in make_figure:
Traceback (most recent call last):
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure
    fig = fig_func(*args, **kwargs)
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/publish/tide_stn_water_level.py", line 78, in make_figure
    plot_data = _prep_plot_data(place, fvcom_ssh_datasets, nemo_ssh_dataset_url_tmpl)
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/publish/tide_stn_water_level.py", line 92, in _prep_plot_data
    station=[
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/publish/tide_stn_water_level.py", line 93, in <listcomp>
    name.decode().strip().split(maxsplit=1)[1]
IndexError: list index out of range
2020-07-04 02:40:47,018 ERROR [make_plots] unexpected IndexError in make_figure:
Traceback (most recent call last):
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure
    fig = fig_func(*args, **kwargs)
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/publish/tide_stn_water_level.py", line 78, in make_figure
    plot_data = _prep_plot_data(place, fvcom_ssh_datasets, nemo_ssh_dataset_url_tmpl)
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/publish/tide_stn_water_level.py", line 92, in _prep_plot_data
    station=[
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/publish/tide_stn_water_level.py", line 93, in <listcomp>
    name.decode().strip().split(maxsplit=1)[1]
IndexError: list index out of range
2020-07-04 02:40:47,052 ERROR [make_plots] unexpected IndexError in make_figure:
Traceback (most recent call last):
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure
    fig = fig_func(*args, **kwargs)
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/publish/tide_stn_water_level.py", line 78, in make_figure
    plot_data = _prep_plot_data(place, fvcom_ssh_datasets, nemo_ssh_dataset_url_tmpl)
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/publish/tide_stn_water_level.py", line 92, in _prep_plot_data
    station=[
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/publish/tide_stn_water_level.py", line 93, in <listcomp>
    name.decode().strip().split(maxsplit=1)[1]
IndexError: list index out of range
2020-07-04 02:40:47,070 ERROR [make_plots] unexpected IndexError in make_figure:
Traceback (most recent call last):
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure
    fig = fig_func(*args, **kwargs)
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/publish/tide_stn_water_level.py", line 78, in make_figure
    plot_data = _prep_plot_data(place, fvcom_ssh_datasets, nemo_ssh_dataset_url_tmpl)
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/publish/tide_stn_water_level.py", line 92, in _prep_plot_data
    station=[
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/publish/tide_stn_water_level.py", line 93, in <listcomp>
    name.decode().strip().split(maxsplit=1)[1]
IndexError: list index out of range
2020-07-04 02:40:47,123 ERROR [make_plots] unexpected IndexError in make_figure:
Traceback (most recent call last):
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure
    fig = fig_func(*args, **kwargs)
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/publish/tide_stn_water_level.py", line 78, in make_figure
    plot_data = _prep_plot_data(place, fvcom_ssh_datasets, nemo_ssh_dataset_url_tmpl)
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/publish/tide_stn_water_level.py", line 92, in _prep_plot_data
    station=[
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/publish/tide_stn_water_level.py", line 93, in <listcomp>
    name.decode().strip().split(maxsplit=1)[1]
IndexError: list index out of range
2020-07-04 02:40:47,140 ERROR [make_plots] unexpected IndexError in make_figure:
Traceback (most recent call last):
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure
    fig = fig_func(*args, **kwargs)
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/publish/tide_stn_water_level.py", line 78, in make_figure
    plot_data = _prep_plot_data(place, fvcom_ssh_datasets, nemo_ssh_dataset_url_tmpl)
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/publish/tide_stn_water_level.py", line 92, in _prep_plot_data
    station=[
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/publish/tide_stn_water_level.py", line 93, in <listcomp>
    name.decode().strip().split(maxsplit=1)[1]
IndexError: list index out of range
2020-07-04 02:40:47,175 ERROR [make_plots] unexpected IndexError in make_figure:
Traceback (most recent call last):
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure
    fig = fig_func(*args, **kwargs)
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/publish/tide_stn_water_level.py", line 78, in make_figure
    plot_data = _prep_plot_data(place, fvcom_ssh_datasets, nemo_ssh_dataset_url_tmpl)
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/publish/tide_stn_water_level.py", line 92, in _prep_plot_data
    station=[
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/publish/tide_stn_water_level.py", line 93, in <listcomp>
    name.decode().strip().split(maxsplit=1)[1]
IndexError: list index out of range
2020-07-04 02:40:47,229 ERROR [make_plots] unexpected IndexError in make_figure:
Traceback (most recent call last):
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure
    fig = fig_func(*args, **kwargs)
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/publish/tide_stn_water_level.py", line 78, in make_figure
    plot_data = _prep_plot_data(place, fvcom_ssh_datasets, nemo_ssh_dataset_url_tmpl)
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/publish/tide_stn_water_level.py", line 92, in _prep_plot_data
    station=[
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/publish/tide_stn_water_level.py", line 93, in <listcomp>
    name.decode().strip().split(maxsplit=1)[1]
IndexError: list index out of range
2020-07-04 02:40:47,241 ERROR [make_plots] unexpected IndexError in make_figure:
Traceback (most recent call last):
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure
    fig = fig_func(*args, **kwargs)
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/publish/tide_stn_water_level.py", line 78, in make_figure
    plot_data = _prep_plot_data(place, fvcom_ssh_datasets, nemo_ssh_dataset_url_tmpl)
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/publish/tide_stn_water_level.py", line 92, in _prep_plot_data
    station=[
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/publish/tide_stn_water_level.py", line 93, in <listcomp>
    name.decode().strip().split(maxsplit=1)[1]
IndexError: list index out of range
2020-07-04 02:40:47,326 ERROR [make_plots] unexpected IndexError in make_figure:
Traceback (most recent call last):
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure
    fig = fig_func(*args, **kwargs)
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/publish/second_narrows_current.py", line 68, in make_figure
    plot_data = _prep_plot_data(place, fvcom_stns_datasets, obs_dataset)
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/publish/second_narrows_current.py", line 81, in _prep_plot_data
    stations = [
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/publish/second_narrows_current.py", line 82, in <listcomp>
    name.decode().strip().split(maxsplit=1)[1]
IndexError: list index out of range
2020-07-04 02:40:47,375 ERROR [make_plots] unexpected IndexError in make_figure:
Traceback (most recent call last):
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure
    fig = fig_func(*args, **kwargs)
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/publish/tide_stn_water_level.py", line 78, in make_figure
    plot_data = _prep_plot_data(place, fvcom_ssh_datasets, nemo_ssh_dataset_url_tmpl)
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/publish/tide_stn_water_level.py", line 92, in _prep_plot_data
    station=[
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/publish/tide_stn_water_level.py", line 93, in <listcomp>
    name.decode().strip().split(maxsplit=1)[1]
IndexError: list index out of range
2020-07-04 02:40:47,391 INFO [make_plots] finished rendering fvcom nowcast-x2 publish 03jul20 figures
2020-07-04 02:40:47,392 INFO [make_plots] fvcom publish plots for 2020-07-03 nowcast-x2 completed
2020-07-04 02:40:47,395 INFO [manager] checklist updated with [plots] items from make_plots worker
2020-07-04 02:40:47,461 ERROR [make_plots] unexpected IndexError in make_figure:
Traceback (most recent call last):
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure
    fig = fig_func(*args, **kwargs)
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/publish/tide_stn_water_level.py", line 78, in make_figure
    plot_data = _prep_plot_data(place, fvcom_ssh_datasets, nemo_ssh_dataset_url_tmpl)
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/publish/tide_stn_water_level.py", line 92, in _prep_plot_data
    station=[
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/publish/tide_stn_water_level.py", line 93, in <listcomp>
    name.decode().strip().split(maxsplit=1)[1]
IndexError: list index out of range
2020-07-04 02:40:47,546 ERROR [make_plots] unexpected IndexError in make_figure:
Traceback (most recent call last):
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure
    fig = fig_func(*args, **kwargs)
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/publish/tide_stn_water_level.py", line 78, in make_figure
    plot_data = _prep_plot_data(place, fvcom_ssh_datasets, nemo_ssh_dataset_url_tmpl)
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/publish/tide_stn_water_level.py", line 92, in _prep_plot_data
    station=[
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/publish/tide_stn_water_level.py", line 93, in <listcomp>
    name.decode().strip().split(maxsplit=1)[1]
IndexError: list index out of range
2020-07-04 02:40:47,644 ERROR [make_plots] unexpected IndexError in make_figure:
Traceback (most recent call last):
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure
    fig = fig_func(*args, **kwargs)
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/publish/second_narrows_current.py", line 68, in make_figure
    plot_data = _prep_plot_data(place, fvcom_stns_datasets, obs_dataset)
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/publish/second_narrows_current.py", line 81, in _prep_plot_data
    stations = [
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/publish/second_narrows_current.py", line 82, in <listcomp>
    name.decode().strip().split(maxsplit=1)[1]
IndexError: list index out of range
2020-07-04 02:40:47,774 INFO [make_plots] finished rendering fvcom nowcast-r12 publish 03jul20 figures
2020-07-04 02:40:47,775 INFO [make_plots] fvcom publish plots for 2020-07-03 nowcast-r12 completed
2020-07-04 02:40:47,887 INFO [manager] checklist updated with [plots] items from make_plots worker
2020-07-04 02:41:18,944 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07724 Calamity Point from 2020-07-03 00:00:00 to 2020-07-05 12:00:00
2020-07-04 02:41:20,515 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07774 Indian Arm Head from 2020-07-02 23:55:00 to 2020-07-05 12:00:00
2020-07-04 02:41:21,068 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07774 Indian Arm Head from 2020-07-03 00:00:00 to 2020-07-05 12:00:00
2020-07-04 02:41:22,432 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07654 New Westminster from 2020-07-02 23:55:00 to 2020-07-05 12:00:00
2020-07-04 02:41:53,270 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07654 New Westminster from 2020-07-03 00:00:00 to 2020-07-05 12:00:00
2020-07-04 02:41:58,033 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07755 Port Moody from 2020-07-02 23:55:00 to 2020-07-05 12:00:00
2020-07-04 02:42:28,539 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07755 Port Moody from 2020-07-03 00:00:00 to 2020-07-05 12:00:00
2020-07-04 02:42:33,751 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07594 Sand Heads from 2020-07-02 23:55:00 to 2020-07-05 12:00:00
2020-07-04 02:43:04,515 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07594 Sand Heads from 2020-07-03 00:00:00 to 2020-07-05 12:00:00
2020-07-04 02:43:09,440 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07786 Sandy Cove from 2020-07-02 23:55:00 to 2020-07-05 12:00:00
2020-07-04 02:43:39,511 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07795 Point Atkinson from 2020-07-03 00:00:00 to 2020-07-05 12:00:00
2020-07-04 02:43:44,520 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07735 Vancouver Harbour from 2020-07-02 23:55:00 to 2020-07-05 12:00:00
2020-07-04 02:44:15,059 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07735 Vancouver Harbour from 2020-07-03 00:00:00 to 2020-07-05 12:00:00
2020-07-04 02:44:20,301 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07610 Woodwards Landing from 2020-07-02 23:55:00 to 2020-07-05 12:00:00
2020-07-04 02:44:50,321 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07610 Woodwards Landing from 2020-07-03 00:00:00 to 2020-07-05 12:00:00
2020-07-04 02:44:54,940 INFO [make_plots] finished rendering fvcom forecast-x2 publish 03jul20 figures
2020-07-04 02:44:55,287 INFO [make_plots] fvcom publish plots for 2020-07-03 forecast-x2 completed
2020-07-04 02:44:55,296 INFO [manager] checklist updated with [plots] items from make_plots worker
2020-07-04 03:04:47,837 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-west/06/ to /results/forcing/atmospheric/GEM2.5/GRIB/20200704/06/
2020-07-04 03:04:47,838 INFO [collect_weather] 2020-07-04 2.5km weather forecast 06 collection complete
2020-07-04 03:04:47,841 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2020-07-04 03:04:48,573 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['Capilano', '--data-date', '2020-07-03'], host='localhost')
2020-07-04 03:04:48,579 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ChilliwackVedder', '--data-date', '2020-07-03'], host='localhost')
2020-07-04 03:04:48,586 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ClowhomClowhomLake', '--data-date', '2020-07-03'], host='localhost')
2020-07-04 03:04:48,592 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['Englishman', '--data-date', '2020-07-03'], host='localhost')
2020-07-04 03:04:48,598 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['Fraser', '--data-date', '2020-07-03'], host='localhost')
2020-07-04 03:04:48,605 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['HomathkoMouth', '--data-date', '2020-07-03'], host='localhost')
2020-07-04 03:04:48,613 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['SalmonSayward', '--data-date', '2020-07-03'], host='localhost')
2020-07-04 03:04:48,618 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['SanJuanPortRenfrew', '--data-date', '2020-07-03'], host='localhost')
2020-07-04 03:04:48,631 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['SquamishBrackendale', '--data-date', '2020-07-03'], host='localhost')
2020-07-04 03:04:48,638 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['TheodosiaScotty', '--data-date', '2020-07-03'], host='localhost')
2020-07-04 03:04:48,644 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['TheodosiaBypass', '--data-date', '2020-07-03'], host='localhost')
2020-07-04 03:04:48,654 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['TheodosiaDiversion', '--data-date', '2020-07-03'], host='localhost')
2020-07-04 03:04:48,661 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ctd', args=['SCVIP'], host='localhost')
2020-07-04 03:04:48,668 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ctd', args=['SEVIP'], host='localhost')
2020-07-04 03:04:48,673 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ferry', args=['TWDP'], host='localhost')
2020-07-04 03:04:48,679 INFO [manager] launching NextWorker(module='nowcast.workers.get_NeahBay_ssh', args=['forecast2'], host='localhost')
2020-07-04 03:04:48,691 INFO [manager] launching NextWorker(module='nowcast.workers.grib_to_netcdf', args=['forecast2'], host='localhost')
2020-07-04 03:04:48,697 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['12', '2.5km'], host='localhost')
2020-07-04 03:04:50,576 INFO [collect_weather] running in process 29367
2020-07-04 03:04:50,577 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 03:04:50,577 INFO [collect_weather] publishing log messages to tcp://*:5562
2020-07-04 03:04:50,580 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2020-07-04 03:04:50,617 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-west/12/
2020-07-04 03:04:51,313 INFO [grib_to_netcdf] running in process 29366
2020-07-04 03:04:51,313 INFO [grib_to_netcdf] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 03:04:51,317 INFO [grib_to_netcdf] publishing log messages to tcp://*:5563
2020-07-04 03:04:51,317 INFO [grib_to_netcdf] connected to skookum.eos.ubc.ca port 5555
2020-07-04 03:04:51,318 INFO [grib_to_netcdf] forecast section: 20200704/06
2020-07-04 03:04:51,319 INFO [grib_to_netcdf] next day forecast section: 20200704/06
2020-07-04 03:04:51,548 INFO [collect_river_data] running in process 29347
2020-07-04 03:04:51,551 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 03:04:51,552 INFO [collect_river_data] publishing log messages to tcp://*:5567
2020-07-04 03:04:51,553 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2020-07-04 03:04:51,592 INFO [collect_river_data] Collecting Capilano river data for 2020-07-03
2020-07-04 03:04:51,593 INFO [collect_river_data] running in process 29348
2020-07-04 03:04:51,594 INFO [collect_river_data] running in process 29351
2020-07-04 03:04:51,595 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 03:04:51,596 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 03:04:51,608 INFO [collect_river_data] publishing log messages to tcp://*:5568
2020-07-04 03:04:51,609 INFO [collect_river_data] publishing log messages to tcp://*:5569
2020-07-04 03:04:51,656 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2020-07-04 03:04:51,660 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2020-07-04 03:04:51,674 INFO [collect_river_data] Collecting ChilliwackVedder river data for 2020-07-03
2020-07-04 03:04:51,674 INFO [collect_river_data] Collecting Fraser river data for 2020-07-03
2020-07-04 03:04:51,676 INFO [collect_river_data] running in process 29349
2020-07-04 03:04:51,677 INFO [collect_river_data] running in process 29359
2020-07-04 03:04:51,677 INFO [collect_river_data] running in process 29354
2020-07-04 03:04:51,678 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 03:04:51,679 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 03:04:51,680 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 03:04:51,681 INFO [collect_river_data] publishing log messages to tcp://*:5570
2020-07-04 03:04:51,681 INFO [collect_river_data] publishing log messages to tcp://*:5571
2020-07-04 03:04:51,682 INFO [collect_river_data] publishing log messages to tcp://*:5572
2020-07-04 03:04:51,683 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2020-07-04 03:04:51,683 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2020-07-04 03:04:51,684 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2020-07-04 03:04:51,685 INFO [collect_river_data] Collecting ClowhomClowhomLake river data for 2020-07-03
2020-07-04 03:04:51,685 INFO [collect_river_data] Collecting TheodosiaDiversion river data for 2020-07-03
2020-07-04 03:04:51,686 INFO [collect_river_data] Collecting SanJuanPortRenfrew river data for 2020-07-03
2020-07-04 03:04:51,690 INFO [collect_river_data] running in process 29357
2020-07-04 03:04:51,692 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 03:04:51,692 INFO [collect_river_data] publishing log messages to tcp://*:5573
2020-07-04 03:04:51,693 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2020-07-04 03:04:51,694 INFO [collect_river_data] Collecting TheodosiaBypass river data for 2020-07-03
2020-07-04 03:04:51,698 INFO [collect_river_data] Appended Capilano river average discharge for 2020-07-03 to: /opp/observations/rivers/Capilano/Caplilano_08GA010_day_avg_flow
2020-07-04 03:04:51,699 INFO [collect_river_data] Capilano river average discharge for 2020-07-03 calculated and appended to Capilano_flow file
2020-07-04 03:04:51,701 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2020-07-04 03:04:51,706 INFO [collect_river_data] Appended Fraser river average discharge for 2020-07-03 to: /data/dlatorne/SOG-projects/SOG-forcing/ECget/Fraser_flow
2020-07-04 03:04:51,707 INFO [collect_river_data] Fraser river average discharge for 2020-07-03 calculated and appended to Fraser_flow file
2020-07-04 03:04:51,720 INFO [collect_river_data] running in process 29356
2020-07-04 03:04:51,721 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 03:04:51,722 INFO [collect_river_data] publishing log messages to tcp://*:5574
2020-07-04 03:04:51,723 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2020-07-04 03:04:51,724 INFO [collect_river_data] Collecting TheodosiaScotty river data for 2020-07-03
2020-07-04 03:04:51,859 INFO [collect_river_data] Appended ClowhomClowhomLake river average discharge for 2020-07-03 to: /results/forcing/rivers/observations/Clowhom_ClowhomLake_flow
2020-07-04 03:04:51,862 INFO [collect_river_data] ClowhomClowhomLake river average discharge for 2020-07-03 calculated and appended to ClowhomClowhomLake_flow file
2020-07-04 03:04:51,863 INFO [collect_river_data] running in process 29352
2020-07-04 03:04:51,866 INFO [collect_river_data] Appended ChilliwackVedder river average discharge for 2020-07-03 to: /results/forcing/rivers/observations/Chilliwack_Vedder_flow
2020-07-04 03:04:51,902 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 03:04:51,940 INFO [collect_river_data] ChilliwackVedder river average discharge for 2020-07-03 calculated and appended to ChilliwackVedder_flow file
2020-07-04 03:04:51,941 INFO [collect_river_data] publishing log messages to tcp://*:5575
2020-07-04 03:04:51,942 INFO [collect_river_data] Appended SanJuanPortRenfrew river average discharge for 2020-07-03 to: /results/forcing/rivers/observations/SanJuan_PortRenfrew_flow
2020-07-04 03:04:51,943 INFO [collect_river_data] Appended TheodosiaDiversion river average discharge for 2020-07-03 to: /results/forcing/rivers/observations/Theodosia_Diversion_flow
2020-07-04 03:04:51,945 INFO [collect_river_data] Appended TheodosiaBypass river average discharge for 2020-07-03 to: /results/forcing/rivers/observations/Theodosia_Bypass_flow
2020-07-04 03:04:51,947 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2020-07-04 03:04:51,948 INFO [collect_river_data] SanJuanPortRenfrew river average discharge for 2020-07-03 calculated and appended to SanJuanPortRenfrew_flow file
2020-07-04 03:04:51,948 INFO [collect_river_data] TheodosiaDiversion river average discharge for 2020-07-03 calculated and appended to TheodosiaDiversion_flow file
2020-07-04 03:04:51,949 INFO [collect_river_data] TheodosiaBypass river average discharge for 2020-07-03 calculated and appended to TheodosiaBypass_flow file
2020-07-04 03:04:51,950 INFO [collect_river_data] Collecting HomathkoMouth river data for 2020-07-03
2020-07-04 03:04:51,952 INFO [collect_river_data] Appended HomathkoMouth river average discharge for 2020-07-03 to: /results/forcing/rivers/observations/Homathko_Mouth_flow
2020-07-04 03:04:51,954 INFO [collect_river_data] HomathkoMouth river average discharge for 2020-07-03 calculated and appended to HomathkoMouth_flow file
2020-07-04 03:04:51,956 INFO [collect_river_data] running in process 29353
2020-07-04 03:04:51,957 INFO [collect_river_data] running in process 29350
2020-07-04 03:04:51,958 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 03:04:51,971 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 03:04:51,972 INFO [collect_river_data] Appended TheodosiaScotty river average discharge for 2020-07-03 to: /results/forcing/rivers/observations/Theodosia_Scotty_flow
2020-07-04 03:04:51,977 INFO [collect_river_data] publishing log messages to tcp://*:5576
2020-07-04 03:04:51,978 INFO [collect_river_data] publishing log messages to tcp://*:5577
2020-07-04 03:04:51,979 INFO [collect_river_data] TheodosiaScotty river average discharge for 2020-07-03 calculated and appended to TheodosiaScotty_flow file
2020-07-04 03:04:51,979 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2020-07-04 03:04:51,982 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2020-07-04 03:04:51,984 INFO [collect_river_data] Collecting SalmonSayward river data for 2020-07-03
2020-07-04 03:04:51,985 INFO [collect_river_data] Collecting Englishman river data for 2020-07-03
2020-07-04 03:04:51,986 INFO [collect_river_data] running in process 29355
2020-07-04 03:04:51,986 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 03:04:51,987 INFO [collect_river_data] publishing log messages to tcp://*:5578
2020-07-04 03:04:51,988 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2020-07-04 03:04:51,989 INFO [collect_river_data] Collecting SquamishBrackendale river data for 2020-07-03
2020-07-04 03:04:52,114 INFO [collect_river_data] Appended SalmonSayward river average discharge for 2020-07-03 to: /results/forcing/rivers/observations/Salmon_Sayward_flow
2020-07-04 03:04:52,115 INFO [collect_river_data] SalmonSayward river average discharge for 2020-07-03 calculated and appended to SalmonSayward_flow file
2020-07-04 03:04:52,125 INFO [collect_river_data] Appended Englishman river average discharge for 2020-07-03 to: /data/dlatorne/SOG-projects/SOG-forcing/ECget/Englishman_flow
2020-07-04 03:04:52,126 INFO [collect_river_data] Englishman river average discharge for 2020-07-03 calculated and appended to Englishman_flow file
2020-07-04 03:04:52,130 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2020-07-04 03:04:52,144 INFO [collect_river_data] Appended SquamishBrackendale river average discharge for 2020-07-03 to: /results/forcing/rivers/observations/Squamish_Brackendale_flow
2020-07-04 03:04:52,145 INFO [collect_river_data] SquamishBrackendale river average discharge for 2020-07-03 calculated and appended to SquamishBrackendale_flow file
2020-07-04 03:04:52,551 INFO [manager] launching NextWorker(module='nowcast.workers.make_runoff_file', args=[], host='localhost')
2020-07-04 03:04:52,576 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2020-07-04 03:04:53,099 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2020-07-04 03:04:53,201 INFO [get_NeahBay_ssh] running in process 29363
2020-07-04 03:04:53,202 INFO [get_NeahBay_ssh] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 03:04:53,203 INFO [get_NeahBay_ssh] publishing log messages to tcp://*:5567
2020-07-04 03:04:53,222 INFO [get_NeahBay_ssh] connected to skookum.eos.ubc.ca port 5555
2020-07-04 03:04:53,525 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2020-07-04 03:04:53,574 INFO [get_onc_ctd] running in process 29360
2020-07-04 03:04:53,575 INFO [get_onc_ctd] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 03:04:53,576 INFO [get_onc_ctd] publishing log messages to tcp://*:5569
2020-07-04 03:04:53,577 INFO [get_onc_ctd] connected to skookum.eos.ubc.ca port 5555
2020-07-04 03:04:53,578 INFO [get_onc_ctd] requesting ONC SCVIP CTD T&S data for 2020-07-03
2020-07-04 03:04:53,595 INFO [get_onc_ferry] running in process 29362
2020-07-04 03:04:53,596 INFO [get_onc_ferry] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 03:04:53,597 INFO [get_onc_ferry] publishing log messages to tcp://*:5570
2020-07-04 03:04:53,598 INFO [get_onc_ferry] connected to skookum.eos.ubc.ca port 5555
2020-07-04 03:04:53,599 INFO [get_onc_ferry] requesting ONC TWDP.N1 NAV data for 2020-07-03
2020-07-04 03:04:53,625 INFO [get_onc_ctd] running in process 29361
2020-07-04 03:04:53,626 INFO [get_onc_ctd] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 03:04:53,627 INFO [get_onc_ctd] publishing log messages to tcp://*:5579
2020-07-04 03:04:53,627 INFO [get_onc_ctd] connected to skookum.eos.ubc.ca port 5555
2020-07-04 03:04:53,628 INFO [get_onc_ctd] requesting ONC SEVIP CTD T&S data for 2020-07-03
2020-07-04 03:04:53,793 INFO [make_runoff_file] running in process 29435
2020-07-04 03:04:53,793 INFO [make_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 03:04:53,794 INFO [make_runoff_file] publishing log messages to tcp://*:5568
2020-07-04 03:04:53,795 INFO [make_runoff_file] connected to skookum.eos.ubc.ca port 5555
2020-07-04 03:04:53,896 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2020-07-04 03:04:54,117 INFO [make_runoff_file] calculating runoff forcing for bathymetry type: b201702
2020-07-04 03:04:54,246 INFO [make_runoff_file] 2020-07-04 runoff file creation from Fraser at Hope and climatology elsewhere complete
2020-07-04 03:04:54,298 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2020-07-04 03:04:54,696 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2020-07-04 03:04:55,058 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2020-07-04 03:04:55,435 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2020-07-04 03:04:55,801 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2020-07-04 03:04:56,160 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2020-07-04 03:04:56,547 INFO [manager] checklist updated with [rivers forcing] items from make_runoff_file worker
2020-07-04 03:04:57,380 INFO [get_NeahBay_ssh] Neah Bay sea surface height web scraping and open boundary file creation for forecast2 complete
2020-07-04 03:04:57,384 INFO [manager] checklist updated with [Neah Bay ssh] items from get_NeahBay_ssh worker
2020-07-04 03:05:17,283 INFO [get_onc_ferry] requesting ONC TWDP TSG data for 2020-07-03
2020-07-04 03:05:17,405 INFO [get_onc_ctd] 2020-07-03 ONC SEVIP CTD T&S file created
2020-07-04 03:05:17,407 INFO [manager] checklist updated with [ONC CTD data] items from get_onc_ctd worker
2020-07-04 03:05:17,673 INFO [get_onc_ctd] 2020-07-03 ONC SCVIP CTD T&S file created
2020-07-04 03:05:17,769 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['SEVIP-CTD'], host='localhost')
2020-07-04 03:05:17,775 INFO [manager] checklist updated with [ONC CTD data] items from get_onc_ctd worker
2020-07-04 03:05:18,171 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['SCVIP-CTD'], host='localhost')
2020-07-04 03:05:18,570 INFO [ping_erddap] running in process 29817
2020-07-04 03:05:18,571 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 03:05:18,572 INFO [ping_erddap] publishing log messages to tcp://*:5567
2020-07-04 03:05:18,572 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2020-07-04 03:05:18,574 INFO [ping_erddap] SEVIP-CTD ERDDAP dataset flag file(s) created
2020-07-04 03:05:18,575 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2020-07-04 03:05:18,964 INFO [ping_erddap] running in process 29819
2020-07-04 03:05:18,965 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 03:05:18,966 INFO [ping_erddap] publishing log messages to tcp://*:5568
2020-07-04 03:05:18,966 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2020-07-04 03:05:18,967 INFO [ping_erddap] SCVIP-CTD ERDDAP dataset flag file(s) created
2020-07-04 03:05:19,061 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2020-07-04 03:05:49,128 INFO [grib_to_netcdf] 2020-07-04 NEMO-atmos forcing file for forecast2 created
2020-07-04 03:05:49,132 INFO [manager] checklist updated with [weather forcing] items from grib_to_netcdf worker
2020-07-04 03:05:49,501 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'forecast2'], host='localhost')
2020-07-04 03:05:49,504 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'forecast2'], host='localhost')
2020-07-04 03:05:49,510 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['graham-hindcast', 'forecast2'], host='localhost')
2020-07-04 03:05:49,512 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'forecast2'], host='localhost')
2020-07-04 03:05:50,496 INFO [upload_forcing] running in process 30234
2020-07-04 03:05:50,497 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 03:05:50,498 INFO [upload_forcing] publishing log messages to tcp://*:5563
2020-07-04 03:05:50,498 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2020-07-04 03:05:50,523 INFO [upload_forcing] running in process 30233
2020-07-04 03:05:50,524 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 03:05:50,525 INFO [upload_forcing] publishing log messages to tcp://*:5567
2020-07-04 03:05:50,526 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2020-07-04 03:05:50,527 INFO [upload_forcing] running in process 30235
2020-07-04 03:05:50,528 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 03:05:50,529 INFO [upload_forcing] publishing log messages to tcp://*:5568
2020-07-04 03:05:50,529 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2020-07-04 03:05:50,575 INFO [upload_forcing] running in process 30232
2020-07-04 03:05:50,576 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 03:05:50,577 INFO [upload_forcing] publishing log messages to tcp://*:5569
2020-07-04 03:05:50,578 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2020-07-04 03:05:51,534 INFO [get_onc_ferry] requesting ONC TWDP OXYSENSOR data for 2020-07-03
2020-07-04 03:05:52,684 INFO [upload_forcing] LiveOcean boundary conditions file not found; created symlink to /results/forcing/LiveOcean/boundary_conditions/LiveOcean_v201905_y2020m07d03.nc
2020-07-04 03:05:52,767 INFO [upload_forcing] forecast2 2020-07-04 forcing files upload to optimum-hindcast completed
2020-07-04 03:05:52,771 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2020-07-04 03:05:52,969 INFO [upload_forcing] forecast2 2020-07-04 forcing files upload to orcinus-nowcast-agrif completed
2020-07-04 03:05:53,243 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2020-07-04 03:05:53,916 INFO [upload_forcing] forecast2 2020-07-04 forcing files upload to arbutus.cloud-nowcast completed
2020-07-04 03:05:53,920 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2020-07-04 03:05:54,287 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2020-07-04'], host='localhost')
2020-07-04 03:05:55,188 INFO [make_forcing_links] running in process 30256
2020-07-04 03:05:55,189 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 03:05:55,190 INFO [make_forcing_links] publishing log messages to tcp://*:5567
2020-07-04 03:05:55,190 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2020-07-04 03:05:56,453 INFO [make_forcing_links] forecast2 2020-07-04 forcing file links on arbutus.cloud-nowcast created
2020-07-04 03:05:56,457 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2020-07-04 03:05:56,815 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2020-07-04'], host='arbutus.cloud-nowcast')
2020-07-04 03:05:57,956 INFO [upload_forcing] forecast2 2020-07-04 forcing files upload to graham-hindcast completed
2020-07-04 03:05:57,961 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2020-07-04 03:05:59,198 INFO [run_NEMO] running in process 31209
2020-07-04 03:05:59,200 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 03:05:59,201 INFO [run_NEMO] publishing log messages to tcp://*:5556
2020-07-04 03:05:59,211 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2020-07-04 03:06:00,807 INFO [run_NEMO] forecast2: launching /nemoShare/MEOPAR/nowcast-sys/runs/03jul20forecast2_2020-07-04T030559.334751-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2020-07-04 03:06:00,825 INFO [run_NEMO] forecast2 NEMO run for 2020-07-04 on arbutus.cloud-nowcast started
2020-07-04 03:06:00,831 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2020-07-04 03:06:01,303 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'forecast2'], host='arbutus.cloud-nowcast')
2020-07-04 03:06:03,273 INFO [watch_NEMO] running in process 31474
2020-07-04 03:06:03,274 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 03:06:03,274 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2020-07-04 03:06:03,275 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2020-07-04 03:06:03,371 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2020-07-04 03:06:27,232 INFO [get_onc_ferry] requesting ONC TWDP TURBCHLFL data for 2020-07-03
2020-07-04 03:06:57,714 INFO [get_onc_ferry] requesting ONC TWDP CO2SENSOR data for 2020-07-03
2020-07-04 03:07:11,359 INFO [get_onc_ferry] requesting ONC TWDP TEMPHUMID data for 2020-07-03
2020-07-04 03:07:36,000 INFO [get_onc_ferry] requesting ONC TWDP BARPRESS data for 2020-07-03
2020-07-04 03:07:48,139 INFO [get_onc_ferry] requesting ONC TWDP PYRANOMETER data for 2020-07-03
2020-07-04 03:07:59,135 INFO [get_onc_ferry] requesting ONC TWDP PYRGEOMETER data for 2020-07-03
2020-07-04 03:08:14,111 INFO [get_onc_ferry] 2020-07-03 ONC TWDP ferry data file created
2020-07-04 03:08:14,116 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['TWDP-ferry'], host='localhost')
2020-07-04 03:08:14,996 INFO [ping_erddap] running in process 30379
2020-07-04 03:08:14,998 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 03:08:14,998 INFO [ping_erddap] publishing log messages to tcp://*:5563
2020-07-04 03:08:14,999 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2020-07-04 03:08:15,001 INFO [ping_erddap] TWDP-ferry ERDDAP dataset flag file(s) created
2020-07-04 03:08:15,003 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2020-07-04 03:11:03,383 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 7499980 = 2020-07-05 05:06:00 UTC, 17.0% complete
2020-07-04 03:16:03,484 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 7500497 = 2020-07-05 10:50:40 UTC, 36.2% complete
2020-07-04 03:21:03,585 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 7501017 = 2020-07-05 16:37:20 UTC, 55.4% complete
2020-07-04 03:26:03,686 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 7501541 = 2020-07-05 22:26:40 UTC, 74.8% complete
2020-07-04 03:31:03,787 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 7502046 = 2020-07-06 04:03:20 UTC, 93.6% complete
2020-07-04 03:36:03,891 INFO [watch_NEMO] forecast2 NEMO run on arbutus.cloud-nowcast completed
2020-07-04 03:36:03,894 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2020-07-04 03:36:04,511 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_wind_file', args=['arbutus.cloud-nowcast', 'forecast2'], host='arbutus.cloud-nowcast')
2020-07-04 03:36:04,520 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_current_file', args=['arbutus.cloud-nowcast', 'forecast2'], host='arbutus.cloud-nowcast')
2020-07-04 03:36:04,523 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2020-07-03'], host='localhost')
2020-07-04 03:36:05,555 INFO [download_results] running in process 31165
2020-07-04 03:36:05,556 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 03:36:05,557 INFO [download_results] publishing log messages to tcp://*:5563
2020-07-04 03:36:05,558 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2020-07-04 03:36:05,559 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/forecast2/03jul20 to /results/SalishSea/forecast2.201905
2020-07-04 03:36:07,655 INFO [make_ww3_current_file] running in process 31713
2020-07-04 03:36:07,656 INFO [make_ww3_current_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 03:36:07,656 INFO [make_ww3_current_file] publishing log messages to tcp://*:5571
2020-07-04 03:36:07,657 INFO [make_ww3_current_file] connected to skookum.eos.ubc.ca port 5555
2020-07-04 03:36:07,658 INFO [make_ww3_current_file] Creating wwatch3 currents forcing file for 2020-07-04 forecast2 run
2020-07-04 03:36:07,758 INFO [make_ww3_wind_file] running in process 31711
2020-07-04 03:36:07,759 INFO [make_ww3_wind_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 03:36:07,760 INFO [make_ww3_wind_file] publishing log messages to tcp://*:5570
2020-07-04 03:36:07,761 INFO [make_ww3_wind_file] connected to skookum.eos.ubc.ca port 5555
2020-07-04 03:36:07,761 INFO [make_ww3_wind_file] Creating wwatch3 wind forcing file for 2020-07-04 forecast2 run
2020-07-04 03:36:11,544 INFO [make_ww3_wind_file] wwatch3 wind forcing file created on arbutus.cloud-nowcast for 2020-07-04 forecast2 run
2020-07-04 03:36:11,548 INFO [manager] checklist updated with [WW3 wind forcing] items from make_ww3_wind_file worker
2020-07-04 03:37:59,586 INFO [make_ww3_current_file] wwatch3 currents forcing file created on arbutus.cloud-nowcast for 2020-07-04 forecast2 run
2020-07-04 03:37:59,591 INFO [manager] checklist updated with [WW3 currents forcing] items from make_ww3_current_file worker
2020-07-04 03:38:00,041 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2020-07-04'], host='arbutus.cloud-nowcast')
2020-07-04 03:38:02,066 INFO [run_ww3] running in process 31939
2020-07-04 03:38:02,067 INFO [run_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 03:38:02,068 INFO [run_ww3] publishing log messages to tcp://*:5572
2020-07-04 03:38:02,069 INFO [run_ww3] connected to skookum.eos.ubc.ca port 5555
2020-07-04 03:38:02,073 INFO [run_ww3] Created run directory /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast2_2020-07-04T033802.062705-0700
2020-07-04 03:38:02,075 INFO [run_ww3] forecast2: launching /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast2_2020-07-04T033802.062705-0700/SoGWW3.sh on arbutus.cloud-nowcast
2020-07-04 03:38:02,088 INFO [run_ww3] forecast2 WaveWatch3 run for 2020-07-04 on arbutus.cloud-nowcast started
2020-07-04 03:38:02,095 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker
2020-07-04 03:38:02,543 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'forecast2'], host='arbutus.cloud-nowcast')
2020-07-04 03:38:04,454 INFO [watch_ww3] running in process 32042
2020-07-04 03:38:04,455 INFO [watch_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 03:38:04,456 INFO [watch_ww3] publishing log messages to tcp://*:5573
2020-07-04 03:38:04,457 INFO [watch_ww3] connected to skookum.eos.ubc.ca port 5555
2020-07-04 03:38:04,492 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2020-07-04 03:38:43,009 INFO [download_results] forecast2 2020-07-03 results files from arbutus.cloud-nowcast downloaded
2020-07-04 03:38:43,014 INFO [manager] checklist updated with [results files] items from download_results worker
2020-07-04 03:38:43,378 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['forecast2', '--run-date', '2020-07-03'], host='localhost')
2020-07-04 03:38:45,963 INFO [make_CHS_currents_file] running in process 31293
2020-07-04 03:38:45,964 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 03:38:45,965 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5563
2020-07-04 03:38:45,965 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555
2020-07-04 03:41:05,591 INFO [make_CHS_currents_file] Made CHS currents file for 2020-07-03 for forecast2
2020-07-04 03:41:05,594 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker
2020-07-04 03:41:05,923 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['nemo', 'forecast2', '--run-date', '2020-07-03'], host='localhost')
2020-07-04 03:41:06,807 INFO [update_forecast_datasets] running in process 31300
2020-07-04 03:41:06,808 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 03:41:06,809 INFO [update_forecast_datasets] publishing log messages to tcp://*:5563
2020-07-04 03:41:06,810 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2020-07-04 03:41:06,810 INFO [update_forecast_datasets] updating nemo forecast directory for forecast2/03jul20 run
2020-07-04 03:43:04,593 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 90, 16.7% complete
2020-07-04 03:45:17,318 INFO [update_forecast_datasets] updated nemo forecast directory for forecast2/03jul20 run: /results/SalishSea/rolling-forecasts/nemo
2020-07-04 03:45:17,319 INFO [update_forecast_datasets] nemo 2020-07-03 forecast2 rolling forecast datasets updated
2020-07-04 03:45:17,321 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2020-07-04 03:45:17,737 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nemo-forecast'], host='localhost')
2020-07-04 03:45:17,740 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'forecast2', 'publish', '--run-date', '2020-07-03'], host='localhost')
2020-07-04 03:45:17,743 INFO [manager] launching NextWorker(module='nowcast.workers.make_surface_current_tiles', args=['forecast2', '--run-date', '2020-07-03'], host='localhost')
2020-07-04 03:45:18,864 INFO [ping_erddap] running in process 31458
2020-07-04 03:45:18,865 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 03:45:18,865 INFO [ping_erddap] publishing log messages to tcp://*:5563
2020-07-04 03:45:18,866 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2020-07-04 03:45:18,891 INFO [ping_erddap] nemo-forecast ERDDAP dataset flag file(s) created
2020-07-04 03:45:18,895 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2020-07-04 03:45:19,238 INFO [make_surface_current_tiles] running in process 31460
2020-07-04 03:45:19,239 INFO [make_surface_current_tiles] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 03:45:19,239 INFO [make_surface_current_tiles] publishing log messages to tcp://*:5567
2020-07-04 03:45:19,240 INFO [make_surface_current_tiles] connected to skookum.eos.ubc.ca port 5555
2020-07-04 03:45:19,316 INFO [make_surface_current_tiles] starting rendering of tiles for 2020-07-03 forecast2 in 6 processes into /results/nowcast-sys/figures/surface_currents/forecast2/03jul20
2020-07-04 03:45:24,698 INFO [make_plots] running in process 31459
2020-07-04 03:45:24,699 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 03:45:24,699 INFO [make_plots] publishing log messages to tcp://*:5563
2020-07-04 03:45:24,700 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2020-07-04 03:45:26,138 INFO [make_plots] preparing render list for 2020-07-03 NEMO forecast2 publish figures
2020-07-04 03:45:26,397 INFO [make_plots] added Website_thumbnail figure to 2020-07-03 NEMO forecast2 publish render list
2020-07-04 03:45:26,511 INFO [make_plots] added Threshold_website figure to 2020-07-03 NEMO forecast2 publish render list
2020-07-04 03:45:26,512 INFO [make_plots] added PA_tidal_predictions figure to 2020-07-03 NEMO forecast2 publish render list
2020-07-04 03:45:26,513 INFO [make_plots] added Boundary Bay figure to 2020-07-03 NEMO forecast2 publish render list
2020-07-04 03:45:26,513 INFO [make_plots] added Campbell River figure to 2020-07-03 NEMO forecast2 publish render list
2020-07-04 03:45:26,514 INFO [make_plots] added Cherry Point figure to 2020-07-03 NEMO forecast2 publish render list
2020-07-04 03:45:26,515 INFO [make_plots] added Friday Harbor figure to 2020-07-03 NEMO forecast2 publish render list
2020-07-04 03:45:26,515 INFO [make_plots] added Halfmoon Bay figure to 2020-07-03 NEMO forecast2 publish render list
2020-07-04 03:45:26,516 INFO [make_plots] added Nanaimo figure to 2020-07-03 NEMO forecast2 publish render list
2020-07-04 03:45:26,517 INFO [make_plots] added Neah Bay figure to 2020-07-03 NEMO forecast2 publish render list
2020-07-04 03:45:26,517 INFO [make_plots] added New Westminster figure to 2020-07-03 NEMO forecast2 publish render list
2020-07-04 03:45:26,518 INFO [make_plots] added Patricia Bay figure to 2020-07-03 NEMO forecast2 publish render list
2020-07-04 03:45:26,519 INFO [make_plots] added Point Atkinson figure to 2020-07-03 NEMO forecast2 publish render list
2020-07-04 03:45:26,520 INFO [make_plots] added Port Renfrew figure to 2020-07-03 NEMO forecast2 publish render list
2020-07-04 03:45:26,520 INFO [make_plots] added Sand Heads figure to 2020-07-03 NEMO forecast2 publish render list
2020-07-04 03:45:26,521 INFO [make_plots] added Sandy Cove figure to 2020-07-03 NEMO forecast2 publish render list
2020-07-04 03:45:26,522 INFO [make_plots] added Squamish figure to 2020-07-03 NEMO forecast2 publish render list
2020-07-04 03:45:26,523 INFO [make_plots] added Victoria figure to 2020-07-03 NEMO forecast2 publish render list
2020-07-04 03:45:26,524 INFO [make_plots] added Woodwards Landing figure to 2020-07-03 NEMO forecast2 publish render list
2020-07-04 03:45:26,524 INFO [make_plots] starting to render nemo forecast2 publish 03jul20 figures
2020-07-04 03:45:55,663 INFO [make_plots] Note: NumExpr detected 12 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
2020-07-04 03:45:55,664 INFO [make_plots] NumExpr defaulting to 8 threads.
2020-07-04 03:48:04,694 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 192, 35.6% complete
2020-07-04 03:53:04,793 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 297, 55.0% complete
2020-07-04 03:58:04,895 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 399, 73.9% complete
2020-07-04 04:01:32,843 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 08074 Campbell River from 2020-06-29 00:00:00 to 2020-07-06 05:55:00
2020-07-04 04:03:04,995 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 507, 93.9% complete
2020-07-04 04:05:15,488 INFO [make_plots] finished rendering fvcom nowcast-r12 research 03jul20 figures
2020-07-04 04:05:15,489 INFO [make_plots] fvcom research plots for 2020-07-03 nowcast-r12 completed
2020-07-04 04:05:15,806 INFO [manager] checklist updated with [plots] items from make_plots worker
2020-07-04 04:05:27,893 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 9449424 Cherry Point from 2020-06-29 00:00:00 to 2020-07-06 05:55:00
2020-07-04 04:06:24,399 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 9449880 Friday Harbor from 2020-06-29 00:00:00 to 2020-07-06 05:55:00
2020-07-04 04:07:19,265 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07830 Halfmoon Bay from 2020-06-29 00:00:00 to 2020-07-06 05:55:00
2020-07-04 04:08:05,095 INFO [watch_ww3] forecast2 WWATCH3 run on arbutus.cloud-nowcast completed
2020-07-04 04:08:05,100 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2020-07-04 04:08:07,017 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2020-07-04'], host='localhost')
2020-07-04 04:08:09,292 INFO [download_wwatch3_results] running in process 1241
2020-07-04 04:08:09,293 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 04:08:09,293 INFO [download_wwatch3_results] publishing log messages to tcp://*:5564
2020-07-04 04:08:09,295 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2020-07-04 04:08:14,968 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07917 Nanaimo from 2020-06-29 00:00:00 to 2020-07-06 05:55:00
2020-07-04 04:08:42,425 INFO [download_wwatch3_results] forecast2 2020-07-04 results files from arbutus.cloud-nowcast downloaded
2020-07-04 04:08:42,430 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2020-07-04 04:08:43,457 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['wwatch3', 'forecast2', '--run-date', '2020-07-04'], host='localhost')
2020-07-04 04:08:46,325 INFO [update_forecast_datasets] running in process 1261
2020-07-04 04:08:46,326 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 04:08:46,327 INFO [update_forecast_datasets] publishing log messages to tcp://*:5564
2020-07-04 04:08:46,328 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2020-07-04 04:08:46,329 INFO [update_forecast_datasets] updating wwatch3 most_recent_forecast directory from forecast2/04jul20 run
2020-07-04 04:08:46,331 INFO [update_forecast_datasets] updated wwatch3 most_recent_forecast directory from forecast2/04jul20 run
2020-07-04 04:08:46,332 INFO [update_forecast_datasets] updating wwatch3 forecast directory for forecast2/04jul20 run
2020-07-04 04:09:44,293 INFO [update_forecast_datasets] updated wwatch3 forecast directory for forecast2/04jul20 run: /results/SalishSea/rolling-forecasts/wwatch3
2020-07-04 04:09:44,293 INFO [update_forecast_datasets] wwatch3 2020-07-04 forecast2 rolling forecast datasets updated
2020-07-04 04:09:44,296 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2020-07-04 04:09:45,905 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['wwatch3-forecast'], host='localhost')
2020-07-04 04:09:47,517 INFO [ping_erddap] running in process 1394
2020-07-04 04:09:47,518 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 04:09:47,519 INFO [ping_erddap] publishing log messages to tcp://*:5564
2020-07-04 04:09:47,520 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2020-07-04 04:09:47,546 INFO [ping_erddap] wwatch3-forecast ERDDAP dataset flag file(s) created
2020-07-04 04:09:47,550 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2020-07-04 04:09:48,793 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['wwatch3', 'forecast2', 'publish', '--run-date', '2020-07-04'], host='localhost')
2020-07-04 04:09:58,031 INFO [make_plots] running in process 1401
2020-07-04 04:09:58,032 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 04:09:58,033 INFO [make_plots] publishing log messages to tcp://*:5564
2020-07-04 04:09:58,033 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2020-07-04 04:09:58,034 INFO [make_plots] preparing render list for 2020-07-04 WaveWatch3 forecast2 publish figures
2020-07-04 04:09:58,035 INFO [make_plots] added Halibut Bank figure to 2020-07-04 WaveWatch3 forecast2 publish render list
2020-07-04 04:09:58,036 INFO [make_plots] added Sentry Shoal figure to 2020-07-04 WaveWatch3 forecast2 publish render list
2020-07-04 04:09:58,036 INFO [make_plots] starting to render wwatch3 forecast2 publish 04jul20 figures
2020-07-04 04:11:05,416 INFO [make_plots] Note: NumExpr detected 12 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
2020-07-04 04:11:05,417 INFO [make_plots] NumExpr defaulting to 8 threads.
2020-07-04 04:11:05,427 INFO [make_plots] retrieving last 45 days of real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46146 Halibut Bank
2020-07-04 04:12:03,769 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 9443090 Neah Bay from 2020-06-29 00:00:00 to 2020-07-06 05:55:00
2020-07-04 04:12:15,271 INFO [make_plots] retrieving last 45 days of real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46131 Sentry Shoal
2020-07-04 04:12:17,476 INFO [make_plots] finished rendering wwatch3 forecast2 publish 04jul20 figures
2020-07-04 04:12:17,480 INFO [make_plots] wwatch3 publish plots for 2020-07-04 forecast2 completed
2020-07-04 04:12:17,483 INFO [manager] checklist updated with [plots] items from make_plots worker
2020-07-04 04:13:02,661 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07654 New Westminster from 2020-06-29 00:00:00 to 2020-07-06 05:55:00
2020-07-04 04:16:42,735 INFO [make_surface_current_tiles] starting PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast2/03jul20
2020-07-04 04:21:07,842 INFO [make_surface_current_tiles] finished PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast2/03jul20
2020-07-04 04:21:07,863 INFO [make_surface_current_tiles] finished rendering of tiles for 2020-07-03 forecast2 into /results/nowcast-sys/figures/surface_currents/forecast2/03jul20
2020-07-04 04:21:07,864 INFO [make_surface_current_tiles] surface current tile figures for 2020-07-03 forecast2 completed
2020-07-04 04:21:08,393 INFO [manager] checklist updated with [surface current tiles] items from make_surface_current_tiles worker
2020-07-04 09:07:13,842 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-west/12/ to /results/forcing/atmospheric/GEM2.5/GRIB/20200704/12/
2020-07-04 09:07:13,843 INFO [collect_weather] 2020-07-04 2.5km weather forecast 12 collection complete
2020-07-04 09:07:13,847 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2020-07-04 09:07:14,830 INFO [manager] launching NextWorker(module='nowcast.workers.get_NeahBay_ssh', args=['nowcast'], host='localhost')
2020-07-04 09:07:15,012 INFO [manager] launching NextWorker(module='nowcast.workers.grib_to_netcdf', args=['nowcast+'], host='localhost')
2020-07-04 09:07:15,103 INFO [manager] launching NextWorker(module='nowcast.workers.download_live_ocean', args=[], host='localhost')
2020-07-04 09:07:15,107 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['18', '2.5km'], host='localhost')
2020-07-04 09:07:16,863 INFO [collect_weather] running in process 15145
2020-07-04 09:07:16,863 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 09:07:16,864 INFO [collect_weather] publishing log messages to tcp://*:5562
2020-07-04 09:07:16,865 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2020-07-04 09:07:16,891 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-west/18/
2020-07-04 09:07:17,148 INFO [grib_to_netcdf] running in process 15143
2020-07-04 09:07:17,149 INFO [grib_to_netcdf] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 09:07:17,150 INFO [grib_to_netcdf] publishing log messages to tcp://*:5564
2020-07-04 09:07:17,151 INFO [grib_to_netcdf] connected to skookum.eos.ubc.ca port 5555
2020-07-04 09:07:17,340 INFO [download_live_ocean] running in process 15144
2020-07-04 09:07:17,343 INFO [download_live_ocean] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 09:07:17,344 INFO [download_live_ocean] publishing log messages to tcp://*:5567
2020-07-04 09:07:17,344 INFO [download_live_ocean] connected to skookum.eos.ubc.ca port 5555
2020-07-04 09:07:17,345 INFO [download_live_ocean] downloading Salish Sea western boundary daily averaged Live Ocean file for 2020-07-04
2020-07-04 09:07:19,314 INFO [get_NeahBay_ssh] running in process 15141
2020-07-04 09:07:19,315 INFO [get_NeahBay_ssh] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 09:07:19,318 INFO [get_NeahBay_ssh] publishing log messages to tcp://*:5568
2020-07-04 09:07:19,319 INFO [get_NeahBay_ssh] connected to skookum.eos.ubc.ca port 5555
2020-07-04 09:07:22,468 INFO [download_live_ocean] Deflating in up to 6 concurrent sub-processes
2020-07-04 09:07:23,482 INFO [get_NeahBay_ssh] Neah Bay sea surface height web scraping and open boundary file creation for nowcast complete
2020-07-04 09:07:23,486 INFO [manager] checklist updated with [Neah Bay ssh] items from get_NeahBay_ssh worker
2020-07-04 09:07:23,487 INFO [download_live_ocean] netCDF4 deflated /results/forcing/LiveOcean/downloaded/20200704/low_passed_UBC.nc
2020-07-04 09:07:23,488 INFO [download_live_ocean] 2020-07-04 Live Ocean file for Salish Sea western boundary downloaded
2020-07-04 09:07:24,057 INFO [manager] checklist updated with [Live Ocean products] items from download_live_ocean worker
2020-07-04 09:07:24,831 INFO [manager] launching NextWorker(module='nowcast.workers.make_live_ocean_files', args=['--run-date', '2020-07-04'], host='localhost')
2020-07-04 09:07:27,636 INFO [make_live_ocean_files] running in process 15244
2020-07-04 09:07:27,637 INFO [make_live_ocean_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 09:07:27,638 INFO [make_live_ocean_files] publishing log messages to tcp://*:5567
2020-07-04 09:07:27,639 INFO [make_live_ocean_files] connected to skookum.eos.ubc.ca port 5555
2020-07-04 09:07:27,639 INFO [make_live_ocean_files] Creating T&S western boundary conditions file from 2020-07-04 Live Ocean run
2020-07-04 09:08:31,440 INFO [make_live_ocean_files] Stored T&S western boundary conditions file: /
2020-07-04 09:08:31,441 INFO [make_live_ocean_files] 2020-07-04 Live Ocean western boundary conditions files created
2020-07-04 09:08:31,444 INFO [manager] checklist updated with [Live Ocean boundary conditions] items from make_live_ocean_files worker
2020-07-04 09:09:03,613 INFO [grib_to_netcdf] 2020-07-04 NEMO-atmos forcing file for nowcast+ created
2020-07-04 09:09:03,617 INFO [manager] checklist updated with [weather forcing] items from grib_to_netcdf worker
2020-07-04 09:09:04,186 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'nowcast+'], host='localhost')
2020-07-04 09:09:04,190 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'nowcast+'], host='localhost')
2020-07-04 09:09:04,192 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['graham-hindcast', 'nowcast+'], host='localhost')
2020-07-04 09:09:04,195 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'nowcast+'], host='localhost')
2020-07-04 09:09:04,198 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['weather'], host='localhost')
2020-07-04 09:09:05,100 INFO [ping_erddap] running in process 16594
2020-07-04 09:09:05,101 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 09:09:05,102 INFO [ping_erddap] publishing log messages to tcp://*:5564
2020-07-04 09:09:05,103 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2020-07-04 09:09:05,105 INFO [ping_erddap] weather ERDDAP dataset flag file(s) created
2020-07-04 09:09:05,107 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2020-07-04 09:09:05,173 INFO [upload_forcing] running in process 16593
2020-07-04 09:09:05,174 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 09:09:05,175 INFO [upload_forcing] publishing log messages to tcp://*:5567
2020-07-04 09:09:05,176 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2020-07-04 09:09:05,177 INFO [upload_forcing] running in process 16591
2020-07-04 09:09:05,178 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 09:09:05,179 INFO [upload_forcing] publishing log messages to tcp://*:5568
2020-07-04 09:09:05,180 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2020-07-04 09:09:05,181 INFO [upload_forcing] running in process 16590
2020-07-04 09:09:05,181 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 09:09:05,182 INFO [upload_forcing] publishing log messages to tcp://*:5569
2020-07-04 09:09:05,183 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2020-07-04 09:09:05,196 INFO [upload_forcing] running in process 16592
2020-07-04 09:09:05,197 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 09:09:05,198 INFO [upload_forcing] publishing log messages to tcp://*:5570
2020-07-04 09:09:05,199 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2020-07-04 09:09:09,166 INFO [upload_forcing] nowcast+ 2020-07-04 forcing files upload to optimum-hindcast completed
2020-07-04 09:09:09,169 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2020-07-04 09:09:09,574 INFO [upload_forcing] nowcast+ 2020-07-04 forcing files upload to orcinus-nowcast-agrif completed
2020-07-04 09:09:09,858 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2020-07-04 09:09:10,660 INFO [upload_forcing] nowcast+ 2020-07-04 forcing files upload to arbutus.cloud-nowcast completed
2020-07-04 09:09:10,664 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2020-07-04 09:09:11,199 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'nowcast+', '--run-date', '2020-07-04'], host='localhost')
2020-07-04 09:09:12,112 INFO [make_forcing_links] running in process 16622
2020-07-04 09:09:12,113 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 09:09:12,113 INFO [make_forcing_links] publishing log messages to tcp://*:5564
2020-07-04 09:09:12,114 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2020-07-04 09:09:13,348 INFO [make_forcing_links] nowcast+ 2020-07-04 forcing file links on arbutus.cloud-nowcast created
2020-07-04 09:09:13,351 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2020-07-04 09:09:13,913 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2020-07-04'], host='arbutus.cloud-nowcast')
2020-07-04 09:09:15,804 INFO [upload_forcing] nowcast+ 2020-07-04 forcing files upload to graham-hindcast completed
2020-07-04 09:09:15,810 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2020-07-04 09:09:16,302 INFO [run_NEMO] running in process 2632
2020-07-04 09:09:16,303 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 09:09:16,304 INFO [run_NEMO] publishing log messages to tcp://*:5556
2020-07-04 09:09:16,358 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2020-07-04 09:09:17,282 INFO [run_NEMO] nowcast: launching /nemoShare/MEOPAR/nowcast-sys/runs/04jul20nowcast_2020-07-04T090916.367351-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2020-07-04 09:09:17,300 INFO [run_NEMO] nowcast NEMO run for 2020-07-04 on arbutus.cloud-nowcast started
2020-07-04 09:09:17,306 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2020-07-04 09:09:18,006 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'nowcast'], host='arbutus.cloud-nowcast')
2020-07-04 09:09:20,002 INFO [watch_NEMO] running in process 2899
2020-07-04 09:09:20,003 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 09:09:20,004 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2020-07-04 09:09:20,005 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2020-07-04 09:09:20,108 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2020-07-04 09:14:20,210 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 7497811 = 2020-07-04 05:00:00 UTC, 20.8% complete
2020-07-04 09:19:20,311 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 7498324 = 2020-07-04 10:42:00 UTC, 44.6% complete
2020-07-04 09:24:20,412 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 7498829 = 2020-07-04 16:18:40 UTC, 68.0% complete
2020-07-04 09:29:20,514 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 7499319 = 2020-07-04 21:45:20 UTC, 90.7% complete
2020-07-04 09:34:20,618 INFO [watch_NEMO] nowcast NEMO run on arbutus.cloud-nowcast completed
2020-07-04 09:34:20,623 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2020-07-04 09:34:21,589 INFO [manager] launching NextWorker(module='nowcast.workers.get_NeahBay_ssh', args=['forecast'], host='localhost')
2020-07-04 09:34:21,594 INFO [manager] launching NextWorker(module='nowcast.workers.make_fvcom_boundary', args=['arbutus.cloud-nowcast', 'x2', 'nowcast'], host='arbutus.cloud-nowcast')
2020-07-04 09:34:21,601 INFO [manager] launching NextWorker(module='nowcast.workers.make_fvcom_boundary', args=['arbutus.cloud-nowcast', 'r12', 'nowcast'], host='arbutus.cloud-nowcast')
2020-07-04 09:34:21,605 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2020-07-04'], host='localhost')
2020-07-04 09:34:22,630 INFO [download_results] running in process 17104
2020-07-04 09:34:22,631 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 09:34:22,632 INFO [download_results] publishing log messages to tcp://*:5564
2020-07-04 09:34:22,633 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2020-07-04 09:34:22,633 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/nowcast/04jul20 to /results/SalishSea/nowcast-blue.201905
2020-07-04 09:34:23,938 INFO [get_NeahBay_ssh] running in process 17101
2020-07-04 09:34:23,939 INFO [get_NeahBay_ssh] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 09:34:23,940 INFO [get_NeahBay_ssh] publishing log messages to tcp://*:5567
2020-07-04 09:34:23,941 INFO [get_NeahBay_ssh] connected to skookum.eos.ubc.ca port 5555
2020-07-04 09:34:25,260 INFO [make_fvcom_boundary] running in process 3124
2020-07-04 09:34:25,261 INFO [make_fvcom_boundary] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 09:34:25,262 INFO [make_fvcom_boundary] publishing log messages to tcp://*:5580
2020-07-04 09:34:25,263 INFO [make_fvcom_boundary] connected to skookum.eos.ubc.ca port 5555
2020-07-04 09:34:25,264 INFO [make_fvcom_boundary] Creating VHFR FVCOM open boundary file for x2 nowcast run from 2020-07-04 NEMO run
2020-07-04 09:34:25,265 INFO [make_fvcom_boundary] running in process 3125
2020-07-04 09:34:25,266 INFO [make_fvcom_boundary] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 09:34:25,267 INFO [make_fvcom_boundary] publishing log messages to tcp://*:5581
2020-07-04 09:34:25,268 INFO [make_fvcom_boundary] connected to skookum.eos.ubc.ca port 5555
2020-07-04 09:34:25,269 INFO [make_fvcom_boundary] Creating VHFR FVCOM open boundary file for r12 nowcast run from 2020-07-04 NEMO run
2020-07-04 09:34:28,069 INFO [get_NeahBay_ssh] Neah Bay sea surface height web scraping and open boundary file creation for forecast complete
2020-07-04 09:34:28,074 INFO [manager] checklist updated with [Neah Bay ssh] items from get_NeahBay_ssh worker
2020-07-04 09:34:28,705 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'ssh'], host='localhost')
2020-07-04 09:34:29,707 INFO [upload_forcing] running in process 17118
2020-07-04 09:34:29,708 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 09:34:29,710 INFO [upload_forcing] publishing log messages to tcp://*:5567
2020-07-04 09:34:29,711 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2020-07-04 09:34:30,879 INFO [upload_forcing] ssh 2020-07-04 forcing files upload to arbutus.cloud-nowcast completed
2020-07-04 09:34:30,885 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2020-07-04 09:34:31,583 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'ssh', '--run-date', '2020-07-04'], host='localhost')
2020-07-04 09:34:32,701 INFO [make_forcing_links] running in process 17124
2020-07-04 09:34:32,702 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 09:34:32,703 INFO [make_forcing_links] publishing log messages to tcp://*:5567
2020-07-04 09:34:32,704 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2020-07-04 09:34:33,818 INFO [make_forcing_links] ssh 2020-07-04 forcing file links on arbutus.cloud-nowcast created
2020-07-04 09:34:33,823 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2020-07-04 09:34:34,488 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2020-07-04'], host='arbutus.cloud-nowcast')
2020-07-04 09:34:36,736 INFO [run_NEMO] running in process 3480
2020-07-04 09:34:36,737 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 09:34:36,738 INFO [run_NEMO] publishing log messages to tcp://*:5556
2020-07-04 09:34:36,739 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2020-07-04 09:34:37,821 INFO [run_NEMO] forecast: launching /nemoShare/MEOPAR/nowcast-sys/runs/04jul20forecast_2020-07-04T093436.825541-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2020-07-04 09:34:37,840 INFO [run_NEMO] forecast NEMO run for 2020-07-04 on arbutus.cloud-nowcast started
2020-07-04 09:34:37,846 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2020-07-04 09:34:38,529 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'forecast'], host='arbutus.cloud-nowcast')
2020-07-04 09:34:40,701 INFO [watch_NEMO] running in process 3747
2020-07-04 09:34:40,702 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 09:34:40,703 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2020-07-04 09:34:40,704 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2020-07-04 09:34:40,810 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2020-07-04 09:36:26,441 INFO [download_results] nowcast 2020-07-04 results files from arbutus.cloud-nowcast downloaded
2020-07-04 09:36:26,448 INFO [manager] checklist updated with [results files] items from download_results worker
2020-07-04 09:36:27,132 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast', 'research', '--run-date', '2020-07-04'], host='localhost')
2020-07-04 09:36:27,135 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast', 'comparison', '--run-date', '2020-07-03'], host='localhost')
2020-07-04 09:36:27,138 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['nowcast', '--run-date', '2020-07-04'], host='localhost')
2020-07-04 09:36:29,751 INFO [make_CHS_currents_file] running in process 17206
2020-07-04 09:36:29,752 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 09:36:29,753 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5564
2020-07-04 09:36:29,753 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555
2020-07-04 09:36:31,264 INFO [make_plots] running in process 17205
2020-07-04 09:36:31,266 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 09:36:31,267 INFO [make_plots] publishing log messages to tcp://*:5567
2020-07-04 09:36:31,268 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2020-07-04 09:36:31,269 INFO [make_plots] running in process 17204
2020-07-04 09:36:31,270 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 09:36:31,271 INFO [make_plots] publishing log messages to tcp://*:5568
2020-07-04 09:36:31,272 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2020-07-04 09:36:31,704 INFO [make_plots] preparing render list for 2020-07-03 NEMO nowcast-blue comparison figures
2020-07-04 09:36:31,705 INFO [make_plots] preparing render list for 2020-07-04 NEMO nowcast-blue research figures
2020-07-04 09:36:38,768 INFO [make_plots] added salinity_thalweg_and_surface figures to 2020-07-04 NEMO nowcast-blue research render list
2020-07-04 09:36:45,025 INFO [make_plots] added temperature_thalweg_and_surface figures to 2020-07-04 NEMO nowcast-blue research render list
2020-07-04 09:36:45,033 INFO [make_plots] added Currents_sections_and_surface figure to 2020-07-04 NEMO nowcast-blue research render list
2020-07-04 09:36:45,033 INFO [make_plots] added Currents_at_VENUS_Central figure to 2020-07-04 NEMO nowcast-blue research render list
2020-07-04 09:36:45,034 INFO [make_plots] added Currents_at_VENUS_East figure to 2020-07-04 NEMO nowcast-blue research render list
2020-07-04 09:36:45,035 INFO [make_plots] starting to render nemo nowcast research 04jul20 figures
2020-07-04 09:36:45,036 INFO [make_plots] added SH_wind figure to 2020-07-03 NEMO nowcast-blue comparison render list
2020-07-04 09:36:45,036 INFO [make_plots] added Compare_VENUS_East figure to 2020-07-03 NEMO nowcast-blue comparison render list
2020-07-04 09:36:45,037 INFO [make_plots] added Compare_VENUS_Central figure to 2020-07-03 NEMO nowcast-blue comparison render list
2020-07-04 09:36:45,037 INFO [make_plots] added Compare_VENUS_Delta_DDL figure to 2020-07-03 NEMO nowcast-blue comparison render list
2020-07-04 09:36:45,046 INFO [make_plots] added Central_ADCP figure to 2020-07-03 NEMO nowcast-blue comparison render list
2020-07-04 09:36:45,060 INFO [make_plots] added Central_depavADCP figure to 2020-07-03 NEMO nowcast-blue comparison render list
2020-07-04 09:36:45,062 INFO [make_plots] added Central_timeavADCP figure to 2020-07-03 NEMO nowcast-blue comparison render list
2020-07-04 09:36:45,062 INFO [make_plots] added East_ADCP figure to 2020-07-03 NEMO nowcast-blue comparison render list
2020-07-04 09:36:45,063 INFO [make_plots] added East_depavADCP figure to 2020-07-03 NEMO nowcast-blue comparison render list
2020-07-04 09:36:45,063 INFO [make_plots] added East_timeavADCP figure to 2020-07-03 NEMO nowcast-blue comparison render list
2020-07-04 09:36:45,064 INFO [make_plots] added ddl_ADCP figure to 2020-07-03 NEMO nowcast-blue comparison render list
2020-07-04 09:36:45,064 INFO [make_plots] added ddl_depavADCP figure to 2020-07-03 NEMO nowcast-blue comparison render list
2020-07-04 09:36:45,065 INFO [make_plots] added ddl_timeavADCP figure to 2020-07-03 NEMO nowcast-blue comparison render list
2020-07-04 09:36:45,065 INFO [make_plots] starting to render nemo nowcast comparison 03jul20 figures
2020-07-04 09:36:49,000 INFO [make_fvcom_boundary] Stored VHFR FVCOM open boundary file: /nemoShare/MEOPAR/nowcast-sys/fvcom-runs/input.x2/bdy_x2_nowcast_brcl_20200704.nc
2020-07-04 09:36:49,005 INFO [make_fvcom_boundary] FVCOM x2 nowcast run boundary condition file for 2020-07-04 created on arbutus.cloud-nowcast
2020-07-04 09:36:49,011 INFO [manager] checklist updated with [FVCOM boundary conditions] items from make_fvcom_boundary worker
2020-07-04 09:36:49,609 INFO [manager] launching NextWorker(module='nowcast.workers.make_fvcom_atmos_forcing', args=['x2', 'nowcast', '--run-date', '2020-07-04'], host='localhost')
2020-07-04 09:36:49,614 INFO [manager] launching NextWorker(module='nowcast.workers.make_fvcom_rivers_forcing', args=['arbutus.cloud-nowcast', 'x2', 'nowcast', '--run-date', '2020-07-04'], host='arbutus.cloud-nowcast')
2020-07-04 09:36:52,953 INFO [make_fvcom_atmos_forcing] running in process 17256
2020-07-04 09:36:52,955 INFO [make_fvcom_atmos_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 09:36:52,955 INFO [make_fvcom_atmos_forcing] publishing log messages to tcp://*:5569
2020-07-04 09:36:52,956 INFO [make_fvcom_atmos_forcing] connected to skookum.eos.ubc.ca port 5555
2020-07-04 09:36:52,957 INFO [make_fvcom_atmos_forcing] Creating VHFR FVCOM atmospheric forcing file for x2 nowcast run from 2020-07-04 HRDPS GRIB files
2020-07-04 09:36:56,308 INFO [make_fvcom_rivers_forcing] running in process 3838
2020-07-04 09:36:56,309 INFO [make_fvcom_rivers_forcing] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 09:36:56,310 INFO [make_fvcom_rivers_forcing] publishing log messages to tcp://*:5582
2020-07-04 09:36:56,311 INFO [make_fvcom_rivers_forcing] connected to skookum.eos.ubc.ca port 5555
2020-07-04 09:36:56,311 INFO [make_fvcom_rivers_forcing] Creating VHFR FVCOM rivers forcing file for x2 nowcast run from 2020-07-04 NEMO runoff forcing files
2020-07-04 09:36:56,529 INFO [make_fvcom_rivers_forcing] Stored VHFR FVCOM rivers forcing file: /nemoShare/MEOPAR/nowcast-sys/fvcom-runs/input.x2/rivers_x2_nowcast_20200704.nc
2020-07-04 09:36:56,530 INFO [make_fvcom_rivers_forcing] FVCOM x2 nowcast run rivers forcing file for 2020-07-04 created on arbutus.cloud-nowcast
2020-07-04 09:36:56,537 INFO [manager] checklist updated with [FVCOM rivers forcing] items from make_fvcom_rivers_forcing worker
2020-07-04 09:36:58,067 INFO [make_fvcom_boundary] Stored VHFR FVCOM open boundary file: /nemoShare/MEOPAR/nowcast-sys/fvcom-runs/input.r12/bdy_r12_nowcast_brcl_20200704.nc
2020-07-04 09:36:58,071 INFO [make_fvcom_boundary] FVCOM r12 nowcast run boundary condition file for 2020-07-04 created on arbutus.cloud-nowcast
2020-07-04 09:36:58,076 INFO [manager] checklist updated with [FVCOM boundary conditions] items from make_fvcom_boundary worker
2020-07-04 09:36:58,717 INFO [manager] launching NextWorker(module='nowcast.workers.make_fvcom_atmos_forcing', args=['r12', 'nowcast', '--run-date', '2020-07-04'], host='localhost')
2020-07-04 09:36:58,725 INFO [manager] launching NextWorker(module='nowcast.workers.make_fvcom_rivers_forcing', args=['arbutus.cloud-nowcast', 'r12', 'nowcast', '--run-date', '2020-07-04'], host='arbutus.cloud-nowcast')
2020-07-04 09:37:01,469 INFO [make_fvcom_atmos_forcing] running in process 17269
2020-07-04 09:37:01,470 INFO [make_fvcom_atmos_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 09:37:01,471 INFO [make_fvcom_atmos_forcing] publishing log messages to tcp://*:5570
2020-07-04 09:37:01,471 INFO [make_fvcom_atmos_forcing] connected to skookum.eos.ubc.ca port 5555
2020-07-04 09:37:01,472 INFO [make_fvcom_atmos_forcing] Creating VHFR FVCOM atmospheric forcing file for r12 nowcast run from 2020-07-04 HRDPS GRIB files
2020-07-04 09:37:02,803 INFO [make_fvcom_rivers_forcing] running in process 3929
2020-07-04 09:37:02,805 INFO [make_fvcom_rivers_forcing] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 09:37:02,806 INFO [make_fvcom_rivers_forcing] publishing log messages to tcp://*:5582
2020-07-04 09:37:02,807 INFO [make_fvcom_rivers_forcing] connected to skookum.eos.ubc.ca port 5555
2020-07-04 09:37:02,808 INFO [make_fvcom_rivers_forcing] Creating VHFR FVCOM rivers forcing file for r12 nowcast run from 2020-07-04 NEMO runoff forcing files
2020-07-04 09:37:03,098 INFO [make_fvcom_rivers_forcing] Stored VHFR FVCOM rivers forcing file: /nemoShare/MEOPAR/nowcast-sys/fvcom-runs/input.r12/rivers_r12_nowcast_20200704.nc
2020-07-04 09:37:03,100 INFO [make_fvcom_rivers_forcing] FVCOM r12 nowcast run rivers forcing file for 2020-07-04 created on arbutus.cloud-nowcast
2020-07-04 09:37:03,105 INFO [manager] checklist updated with [FVCOM rivers forcing] items from make_fvcom_rivers_forcing worker
2020-07-04 09:37:20,191 INFO [make_fvcom_atmos_forcing] Stored VHFR FVCOM atmospheric forcing hfx file: /results/forcing/atmospheric/GEM2.5/vhfr-fvcom/atmos_x2_nowcast_hfx_20200704.nc
2020-07-04 09:37:27,447 INFO [make_fvcom_atmos_forcing] Stored VHFR FVCOM atmospheric forcing hfx file: /results/forcing/atmospheric/GEM2.5/vhfr-fvcom/atmos_r12_nowcast_hfx_20200704.nc
2020-07-04 09:37:27,806 INFO [make_fvcom_atmos_forcing] Stored VHFR FVCOM atmospheric forcing precip file: /results/forcing/atmospheric/GEM2.5/vhfr-fvcom/atmos_x2_nowcast_precip_20200704.nc
2020-07-04 09:37:34,728 INFO [make_fvcom_atmos_forcing] Stored VHFR FVCOM atmospheric forcing precip file: /results/forcing/atmospheric/GEM2.5/vhfr-fvcom/atmos_r12_nowcast_precip_20200704.nc
2020-07-04 09:37:38,808 INFO [make_fvcom_atmos_forcing] Stored VHFR FVCOM atmospheric forcing wnd file: /results/forcing/atmospheric/GEM2.5/vhfr-fvcom/atmos_x2_nowcast_wnd_20200704.nc
2020-07-04 09:37:38,809 INFO [make_fvcom_atmos_forcing] FVCOM x2 nowcast run atmospheric forcing file for 2020-07-04 created
2020-07-04 09:37:38,814 INFO [manager] checklist updated with [FVCOM atmospheric forcing] items from make_fvcom_atmos_forcing worker
2020-07-04 09:37:39,444 INFO [manager] launching NextWorker(module='nowcast.workers.upload_fvcom_atmos_forcing', args=['arbutus.cloud-nowcast', 'x2', 'nowcast', '--run-date', '2020-07-04'], host='localhost')
2020-07-04 09:37:40,967 INFO [upload_fvcom_atmos_forcing] running in process 17275
2020-07-04 09:37:40,968 INFO [upload_fvcom_atmos_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 09:37:40,969 INFO [upload_fvcom_atmos_forcing] publishing log messages to tcp://*:5569
2020-07-04 09:37:40,971 INFO [upload_fvcom_atmos_forcing] connected to skookum.eos.ubc.ca port 5555
2020-07-04 09:37:40,971 INFO [upload_fvcom_atmos_forcing] Uploading VHFR FVCOM atmospheric forcing files for 2020-07-04 x2 nowcast run to arbutus.cloud-nowcast
2020-07-04 09:37:46,141 INFO [upload_fvcom_atmos_forcing] FVCOM x2 nowcast run atmospheric forcing files for 2020-07-04 uploaded to arbutus.cloud-nowcast
2020-07-04 09:37:46,143 INFO [manager] checklist updated with [FVCOM atmospheric forcing upload] items from upload_fvcom_atmos_forcing worker
2020-07-04 09:37:46,531 INFO [make_fvcom_atmos_forcing] Stored VHFR FVCOM atmospheric forcing wnd file: /results/forcing/atmospheric/GEM2.5/vhfr-fvcom/atmos_r12_nowcast_wnd_20200704.nc
2020-07-04 09:37:46,532 INFO [make_fvcom_atmos_forcing] FVCOM r12 nowcast run atmospheric forcing file for 2020-07-04 created
2020-07-04 09:37:46,625 INFO [manager] launching NextWorker(module='nowcast.workers.run_fvcom', args=['arbutus.cloud-nowcast', 'x2', 'nowcast', '--run-date', '2020-07-04'], host='arbutus.cloud-nowcast')
2020-07-04 09:37:46,639 INFO [manager] checklist updated with [FVCOM atmospheric forcing] items from make_fvcom_atmos_forcing worker
2020-07-04 09:37:47,238 INFO [manager] launching NextWorker(module='nowcast.workers.upload_fvcom_atmos_forcing', args=['arbutus.cloud-nowcast', 'r12', 'nowcast', '--run-date', '2020-07-04'], host='localhost')
2020-07-04 09:37:48,230 INFO [upload_fvcom_atmos_forcing] running in process 17286
2020-07-04 09:37:48,231 INFO [upload_fvcom_atmos_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 09:37:48,232 INFO [upload_fvcom_atmos_forcing] publishing log messages to tcp://*:5569
2020-07-04 09:37:48,233 INFO [upload_fvcom_atmos_forcing] connected to skookum.eos.ubc.ca port 5555
2020-07-04 09:37:48,233 INFO [upload_fvcom_atmos_forcing] Uploading VHFR FVCOM atmospheric forcing files for 2020-07-04 r12 nowcast run to arbutus.cloud-nowcast
2020-07-04 09:37:49,159 INFO [run_fvcom] running in process 4278
2020-07-04 09:37:49,160 INFO [run_fvcom] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 09:37:49,160 INFO [run_fvcom] publishing log messages to tcp://*:5584
2020-07-04 09:37:49,162 INFO [run_fvcom] connected to skookum.eos.ubc.ca port 5555
2020-07-04 09:37:49,219 INFO [run_fvcom] nowcast: launching /nemoShare/MEOPAR/nowcast-sys/fvcom-runs/20200704-16h37m49.207601s/VHFR_FVCOM.sh on arbutus.cloud-nowcast
2020-07-04 09:37:49,255 INFO [run_fvcom] x2 nowcast FVCOM VH-FR run for 2020-07-04 on arbutus.cloud-nowcast started
2020-07-04 09:37:49,260 INFO [manager] checklist updated with [FVCOM run] items from run_fvcom worker
2020-07-04 09:37:49,866 INFO [manager] launching NextWorker(module='nowcast.workers.watch_fvcom', args=['arbutus.cloud-nowcast', 'x2', 'nowcast'], host='arbutus.cloud-nowcast')
2020-07-04 09:37:52,145 INFO [watch_fvcom] running in process 4487
2020-07-04 09:37:52,147 INFO [watch_fvcom] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 09:37:52,148 INFO [watch_fvcom] publishing log messages to tcp://*:5586
2020-07-04 09:37:52,149 INFO [watch_fvcom] connected to skookum.eos.ubc.ca port 5555
2020-07-04 09:37:52,214 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: no run progress found in fvcom.log ; continuing to watch...
2020-07-04 09:37:54,243 INFO [upload_fvcom_atmos_forcing] FVCOM r12 nowcast run atmospheric forcing files for 2020-07-04 uploaded to arbutus.cloud-nowcast
2020-07-04 09:37:54,249 INFO [manager] checklist updated with [FVCOM atmospheric forcing upload] items from upload_fvcom_atmos_forcing worker
2020-07-04 09:37:54,938 INFO [manager] launching NextWorker(module='nowcast.workers.run_fvcom', args=['arbutus.cloud-nowcast', 'r12', 'nowcast', '--run-date', '2020-07-04'], host='arbutus.cloud-nowcast')
2020-07-04 09:37:57,471 INFO [run_fvcom] running in process 4734
2020-07-04 09:37:57,472 INFO [run_fvcom] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 09:37:57,473 INFO [run_fvcom] publishing log messages to tcp://*:5584
2020-07-04 09:37:57,474 INFO [run_fvcom] connected to skookum.eos.ubc.ca port 5555
2020-07-04 09:37:57,557 INFO [run_fvcom] nowcast: launching /nemoShare/MEOPAR/nowcast-sys/fvcom-runs/20200704-16h37m57.544495s/VHFR_FVCOM.sh on arbutus.cloud-nowcast
2020-07-04 09:37:57,582 INFO [run_fvcom] r12 nowcast FVCOM VH-FR run for 2020-07-04 on arbutus.cloud-nowcast started
2020-07-04 09:37:57,588 INFO [manager] checklist updated with [FVCOM run] items from run_fvcom worker
2020-07-04 09:37:58,332 INFO [manager] launching NextWorker(module='nowcast.workers.watch_fvcom', args=['arbutus.cloud-nowcast', 'r12', 'nowcast'], host='arbutus.cloud-nowcast')
2020-07-04 09:38:00,621 INFO [watch_fvcom] running in process 4838
2020-07-04 09:38:00,622 INFO [watch_fvcom] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 09:38:00,623 INFO [watch_fvcom] publishing log messages to tcp://*:5587
2020-07-04 09:38:00,624 INFO [watch_fvcom] connected to skookum.eos.ubc.ca port 5555
2020-07-04 09:38:00,684 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: fvcom.log not found; continuing to watch...
2020-07-04 09:38:42,065 INFO [make_CHS_currents_file] Made CHS currents file for 2020-07-04 for nowcast
2020-07-04 09:38:42,069 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker
2020-07-04 09:39:09,942 WARNING [make_plots] VENUS Delta DDL node CTD comparison figure failed: No observations available
2020-07-04 09:39:10,793 WARNING [make_plots] VENUS Central ADCP comparison figure failed: No observations available
2020-07-04 09:39:11,416 WARNING [make_plots] VENUS Central ADCP comparison figure failed: No observations available
2020-07-04 09:39:12,010 WARNING [make_plots] VENUS Central ADCP comparison figure failed: No observations available
2020-07-04 09:39:12,722 WARNING [make_plots] VENUS East ADCP comparison figure failed: No observations available
2020-07-04 09:39:13,306 WARNING [make_plots] VENUS East ADCP comparison figure failed: No observations available
2020-07-04 09:39:13,905 WARNING [make_plots] VENUS East ADCP comparison figure failed: No observations available
2020-07-04 09:39:14,552 WARNING [make_plots] VENUS ddl ADCP comparison figure failed: No observations available
2020-07-04 09:39:15,174 WARNING [make_plots] VENUS ddl ADCP comparison figure failed: No observations available
2020-07-04 09:39:15,785 WARNING [make_plots] VENUS ddl ADCP comparison figure failed: No observations available
2020-07-04 09:39:15,788 INFO [make_plots] finished rendering nemo nowcast comparison 03jul20 figures
2020-07-04 09:39:15,884 INFO [make_plots] nemo comparison plots for 2020-07-03 nowcast completed
2020-07-04 09:39:15,888 INFO [manager] checklist updated with [plots] items from make_plots worker
2020-07-04 09:39:40,912 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 7499948 = 2020-07-05 04:44:40 UTC, 13.2% complete
2020-07-04 09:40:15,733 INFO [make_plots] finished rendering nemo nowcast research 04jul20 figures
2020-07-04 09:40:15,735 INFO [make_plots] nemo research plots for 2020-07-04 nowcast completed
2020-07-04 09:40:15,763 INFO [manager] checklist updated with [plots] items from make_plots worker
2020-07-04 09:42:52,317 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15814894 = 2020-07-04 00:31:34 UTC estimated time to finish: 0000:03:38:30
2020-07-04 09:43:00,787 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39533446 = 2020-07-04 00:06:18 UTC estimated time to finish: 0000:16:30:40
2020-07-04 09:44:41,014 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 7500427 = 2020-07-05 10:04:00 UTC, 28.0% complete
2020-07-04 09:47:52,421 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15816666 = 2020-07-04 01:01:06 UTC estimated time to finish: 0000:03:36:22
2020-07-04 09:48:00,890 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39534593 = 2020-07-04 00:13:57 UTC estimated time to finish: 0000:16:13:44
2020-07-04 09:49:41,116 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 7500889 = 2020-07-05 15:12:00 UTC, 42.2% complete
2020-07-04 09:52:52,530 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15818601 = 2020-07-04 01:33:21 UTC estimated time to finish: 0000:03:31:20
2020-07-04 09:53:00,994 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39535699 = 2020-07-04 00:21:19 UTC estimated time to finish: 0000:16:11:41
2020-07-04 09:54:41,217 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 7501341 = 2020-07-05 20:13:20 UTC, 56.2% complete
2020-07-04 09:57:52,638 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15820564 = 2020-07-04 02:06:04 UTC estimated time to finish: 0000:03:26:16
2020-07-04 09:58:01,098 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39536723 = 2020-07-04 00:28:09 UTC estimated time to finish: 0000:16:02:55
2020-07-04 09:59:41,320 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 7501796 = 2020-07-06 01:16:40 UTC, 70.2% complete
2020-07-04 10:02:52,745 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15822500 = 2020-07-04 02:38:20 UTC estimated time to finish: 0000:03:21:21
2020-07-04 10:03:01,204 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39537830 = 2020-07-04 00:35:32 UTC estimated time to finish: 0000:16:02:11
2020-07-04 10:04:41,421 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 7502259 = 2020-07-06 06:25:20 UTC, 84.5% complete
2020-07-04 10:07:52,852 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15824272 = 2020-07-04 03:07:52 UTC estimated time to finish: 0000:03:16:49
2020-07-04 10:08:01,276 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39538936 = 2020-07-04 00:42:54 UTC estimated time to finish: 0000:15:59:53
2020-07-04 10:09:41,522 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 7502729 = 2020-07-06 11:38:40 UTC, 99.0% complete
2020-07-04 10:12:52,957 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15826167 = 2020-07-04 03:39:27 UTC estimated time to finish: 0000:03:12:42
2020-07-04 10:13:01,379 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39540164 = 2020-07-04 00:51:05 UTC estimated time to finish: 0000:15:38:49
2020-07-04 10:14:41,627 INFO [watch_NEMO] forecast NEMO run on arbutus.cloud-nowcast completed
2020-07-04 10:14:41,631 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2020-07-04 10:14:42,616 INFO [manager] launching NextWorker(module='nowcast.workers.make_turbidity_file', args=['--run-date', '2020-07-04'], host='localhost')
2020-07-04 10:14:42,622 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2020-07-04'], host='localhost')
2020-07-04 10:14:43,882 INFO [download_results] running in process 18166
2020-07-04 10:14:43,883 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 10:14:43,884 INFO [download_results] publishing log messages to tcp://*:5564
2020-07-04 10:14:43,886 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2020-07-04 10:14:43,887 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/forecast/04jul20 to /results/SalishSea/forecast.201905
2020-07-04 10:14:44,346 INFO [make_turbidity_file] running in process 18165
2020-07-04 10:14:44,347 INFO [make_turbidity_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 10:14:44,348 INFO [make_turbidity_file] publishing log messages to tcp://*:5567
2020-07-04 10:14:44,349 INFO [make_turbidity_file] connected to skookum.eos.ubc.ca port 5555
2020-07-04 10:14:44,350 INFO [make_turbidity_file] Creating Fraser River turbidity forcing file for 2020-07-04
2020-07-04 10:14:44,781 INFO [make_turbidity_file] Note: NumExpr detected 12 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
2020-07-04 10:14:44,782 INFO [make_turbidity_file] NumExpr defaulting to 8 threads.
2020-07-04 10:14:46,685 INFO [make_turbidity_file] 2020-07-04 Fraser River turbidity file creation complete
2020-07-04 10:14:46,687 INFO [manager] checklist updated with [Fraser River turbidity] items from make_turbidity_file worker
2020-07-04 10:14:47,242 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'turbidity'], host='localhost')
2020-07-04 10:14:47,245 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'turbidity'], host='localhost')
2020-07-04 10:14:47,248 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['graham-hindcast', 'turbidity'], host='localhost')
2020-07-04 10:14:47,251 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'turbidity'], host='localhost')
2020-07-04 10:14:48,109 INFO [upload_forcing] running in process 18187
2020-07-04 10:14:48,110 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 10:14:48,111 INFO [upload_forcing] publishing log messages to tcp://*:5567
2020-07-04 10:14:48,111 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2020-07-04 10:14:48,123 INFO [upload_forcing] running in process 18184
2020-07-04 10:14:48,124 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 10:14:48,125 INFO [upload_forcing] publishing log messages to tcp://*:5568
2020-07-04 10:14:48,126 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2020-07-04 10:14:48,127 INFO [upload_forcing] running in process 18186
2020-07-04 10:14:48,128 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 10:14:48,129 INFO [upload_forcing] publishing log messages to tcp://*:5569
2020-07-04 10:14:48,130 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2020-07-04 10:14:48,144 INFO [upload_forcing] running in process 18185
2020-07-04 10:14:48,145 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 10:14:48,146 INFO [upload_forcing] publishing log messages to tcp://*:5570
2020-07-04 10:14:48,147 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2020-07-04 10:14:48,672 INFO [upload_forcing] turbidity 2020-07-04 forcing files upload to optimum-hindcast completed
2020-07-04 10:14:48,677 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2020-07-04 10:14:48,813 INFO [upload_forcing] turbidity 2020-07-04 forcing files upload to orcinus-nowcast-agrif completed
2020-07-04 10:14:49,276 INFO [upload_forcing] turbidity 2020-07-04 forcing files upload to arbutus.cloud-nowcast completed
2020-07-04 10:14:49,360 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2020-07-04 10:14:50,016 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['orcinus-nowcast-agrif', 'nowcast-agrif', '--run-date', '2020-07-04'], host='localhost')
2020-07-04 10:14:50,020 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2020-07-04 10:14:50,611 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2020-07-04'], host='localhost')
2020-07-04 10:14:50,959 INFO [make_forcing_links] running in process 18209
2020-07-04 10:14:50,960 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 10:14:50,961 INFO [make_forcing_links] publishing log messages to tcp://*:5567
2020-07-04 10:14:50,962 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2020-07-04 10:14:51,305 INFO [upload_forcing] turbidity 2020-07-04 forcing files upload to graham-hindcast completed
2020-07-04 10:14:51,310 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2020-07-04 10:14:51,615 INFO [make_forcing_links] nowcast-agrif 2020-07-04 forcing file links on orcinus-nowcast-agrif created
2020-07-04 10:14:51,632 INFO [make_forcing_links] running in process 18211
2020-07-04 10:14:51,633 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 10:14:51,634 INFO [make_forcing_links] publishing log messages to tcp://*:5568
2020-07-04 10:14:51,636 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2020-07-04 10:14:52,005 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2020-07-04 10:14:52,558 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_agrif', args=['orcinus-nowcast-agrif', 'nowcast-agrif', '--run-date', '2020-07-04'], host='localhost')
2020-07-04 10:14:52,858 INFO [make_forcing_links] nowcast-green 2020-07-04 forcing file links on arbutus.cloud-nowcast created
2020-07-04 10:14:52,862 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2020-07-04 10:14:53,459 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2020-07-04'], host='arbutus.cloud-nowcast')
2020-07-04 10:14:53,469 INFO [run_NEMO_agrif] running in process 18221
2020-07-04 10:14:53,470 INFO [run_NEMO_agrif] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 10:14:53,471 INFO [run_NEMO_agrif] publishing log messages to tcp://*:5567
2020-07-04 10:14:53,472 INFO [run_NEMO_agrif] connected to skookum.eos.ubc.ca port 5555
2020-07-04 10:14:53,944 CRITICAL [run_NEMO_agrif] unhandled exception:
Traceback (most recent call last):
  File "/SalishSeaCast/NEMO_Nowcast/nemo_nowcast/worker.py", line 390, in _do_work
    checklist = self.worker_func(
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/run_NEMO_agrif.py", line 110, in run_NEMO_agrif
    prev_run_namelists_info = _get_prev_run_namelists_info(
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/run_NEMO_agrif.py", line 156, in _get_prev_run_namelists_info
    sftp_client.get(os.fspath(prev_namelist_cfg), namelist_cfg.name)
  File "/SalishSeaCast/nowcast-env/lib/python3.8/site-packages/paramiko/sftp_client.py", line 802, in get
    size = self.getfo(remotepath, fl, callback)
  File "/SalishSeaCast/nowcast-env/lib/python3.8/site-packages/paramiko/sftp_client.py", line 778, in getfo
    file_size = self.stat(remotepath).st_size
  File "/SalishSeaCast/nowcast-env/lib/python3.8/site-packages/paramiko/sftp_client.py", line 493, in stat
    t, msg = self._request(CMD_STAT, path)
  File "/SalishSeaCast/nowcast-env/lib/python3.8/site-packages/paramiko/sftp_client.py", line 813, in _request
    return self._read_response(num)
  File "/SalishSeaCast/nowcast-env/lib/python3.8/site-packages/paramiko/sftp_client.py", line 865, in _read_response
    self._convert_status(msg)
  File "/SalishSeaCast/nowcast-env/lib/python3.8/site-packages/paramiko/sftp_client.py", line 894, in _convert_status
    raise IOError(errno.ENOENT, text)
FileNotFoundError: [Errno 2] No such file
2020-07-04 10:14:56,006 INFO [run_NEMO] running in process 5320
2020-07-04 10:14:56,008 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 10:14:56,008 INFO [run_NEMO] publishing log messages to tcp://*:5556
2020-07-04 10:14:56,063 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2020-07-04 10:14:57,696 INFO [run_NEMO] nowcast-green: launching /nemoShare/MEOPAR/nowcast-sys/runs/04jul20nowcast-green_2020-07-04T101456.077018-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2020-07-04 10:14:57,717 INFO [run_NEMO] nowcast-green NEMO run for 2020-07-04 on arbutus.cloud-nowcast started
2020-07-04 10:14:57,723 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2020-07-04 10:14:58,415 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'nowcast-green'], host='arbutus.cloud-nowcast')
2020-07-04 10:15:00,450 INFO [watch_NEMO] running in process 5588
2020-07-04 10:15:00,451 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 10:15:00,452 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2020-07-04 10:15:00,453 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2020-07-04 10:15:00,559 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2020-07-04 10:17:53,063 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15828101 = 2020-07-04 04:11:41 UTC estimated time to finish: 0000:03:07:42
2020-07-04 10:17:53,297 INFO [download_results] forecast 2020-07-04 results files from arbutus.cloud-nowcast downloaded
2020-07-04 10:17:53,304 INFO [manager] checklist updated with [results files] items from download_results worker
2020-07-04 10:17:53,996 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['forecast', '--run-date', '2020-07-04'], host='localhost')
2020-07-04 10:17:57,701 INFO [make_CHS_currents_file] running in process 18326
2020-07-04 10:17:57,703 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 10:17:57,704 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5564
2020-07-04 10:17:57,705 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555
2020-07-04 10:18:01,484 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39541270 = 2020-07-04 00:58:28 UTC estimated time to finish: 0000:15:24:00
2020-07-04 10:20:00,660 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 7497504 = 2020-07-04 01:35:20 UTC, 6.6% complete
2020-07-04 10:20:47,806 INFO [make_CHS_currents_file] Made CHS currents file for 2020-07-04 for forecast
2020-07-04 10:20:47,811 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker
2020-07-04 10:20:48,492 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['nemo', 'forecast', '--run-date', '2020-07-04'], host='localhost')
2020-07-04 10:20:49,477 INFO [update_forecast_datasets] running in process 18367
2020-07-04 10:20:49,478 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 10:20:49,479 INFO [update_forecast_datasets] publishing log messages to tcp://*:5564
2020-07-04 10:20:49,480 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2020-07-04 10:20:49,481 INFO [update_forecast_datasets] updating nemo forecast directory for forecast/04jul20 run
2020-07-04 10:20:49,873 INFO [update_forecast_datasets] updated nemo forecast directory for forecast/04jul20 run: /results/SalishSea/rolling-forecasts/nemo
2020-07-04 10:20:49,874 INFO [update_forecast_datasets] nemo 2020-07-04 forecast rolling forecast datasets updated
2020-07-04 10:20:49,877 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2020-07-04 10:20:50,534 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nemo-forecast'], host='localhost')
2020-07-04 10:20:50,537 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'forecast', 'publish', '--run-date', '2020-07-04'], host='localhost')
2020-07-04 10:20:50,540 INFO [manager] launching NextWorker(module='nowcast.workers.make_surface_current_tiles', args=['forecast', '--run-date', '2020-07-04'], host='localhost')
2020-07-04 10:20:51,375 INFO [ping_erddap] running in process 18371
2020-07-04 10:20:51,376 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 10:20:51,377 INFO [ping_erddap] publishing log messages to tcp://*:5564
2020-07-04 10:20:51,378 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2020-07-04 10:20:51,403 INFO [ping_erddap] nemo-forecast ERDDAP dataset flag file(s) created
2020-07-04 10:20:51,406 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2020-07-04 10:20:51,795 INFO [make_surface_current_tiles] running in process 18373
2020-07-04 10:20:51,796 INFO [make_surface_current_tiles] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 10:20:51,796 INFO [make_surface_current_tiles] publishing log messages to tcp://*:5567
2020-07-04 10:20:51,797 INFO [make_surface_current_tiles] connected to skookum.eos.ubc.ca port 5555
2020-07-04 10:20:51,818 INFO [make_surface_current_tiles] starting rendering of tiles for 2020-07-04 forecast in 6 processes into /results/nowcast-sys/figures/surface_currents/forecast/04jul20
2020-07-04 10:20:58,030 INFO [make_plots] running in process 18372
2020-07-04 10:20:58,031 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 10:20:58,032 INFO [make_plots] publishing log messages to tcp://*:5564
2020-07-04 10:20:58,033 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2020-07-04 10:20:58,789 INFO [make_plots] preparing render list for 2020-07-04 NEMO forecast publish figures
2020-07-04 10:20:59,003 INFO [make_plots] added Website_thumbnail figure to 2020-07-04 NEMO forecast publish render list
2020-07-04 10:20:59,017 INFO [make_plots] added Threshold_website figure to 2020-07-04 NEMO forecast publish render list
2020-07-04 10:20:59,018 INFO [make_plots] added PA_tidal_predictions figure to 2020-07-04 NEMO forecast publish render list
2020-07-04 10:20:59,019 INFO [make_plots] added Boundary Bay figure to 2020-07-04 NEMO forecast publish render list
2020-07-04 10:20:59,020 INFO [make_plots] added Campbell River figure to 2020-07-04 NEMO forecast publish render list
2020-07-04 10:20:59,020 INFO [make_plots] added Cherry Point figure to 2020-07-04 NEMO forecast publish render list
2020-07-04 10:20:59,021 INFO [make_plots] added Friday Harbor figure to 2020-07-04 NEMO forecast publish render list
2020-07-04 10:20:59,022 INFO [make_plots] added Halfmoon Bay figure to 2020-07-04 NEMO forecast publish render list
2020-07-04 10:20:59,022 INFO [make_plots] added Nanaimo figure to 2020-07-04 NEMO forecast publish render list
2020-07-04 10:20:59,023 INFO [make_plots] added Neah Bay figure to 2020-07-04 NEMO forecast publish render list
2020-07-04 10:20:59,024 INFO [make_plots] added New Westminster figure to 2020-07-04 NEMO forecast publish render list
2020-07-04 10:20:59,025 INFO [make_plots] added Patricia Bay figure to 2020-07-04 NEMO forecast publish render list
2020-07-04 10:20:59,026 INFO [make_plots] added Point Atkinson figure to 2020-07-04 NEMO forecast publish render list
2020-07-04 10:20:59,027 INFO [make_plots] added Port Renfrew figure to 2020-07-04 NEMO forecast publish render list
2020-07-04 10:20:59,028 INFO [make_plots] added Sand Heads figure to 2020-07-04 NEMO forecast publish render list
2020-07-04 10:20:59,029 INFO [make_plots] added Sandy Cove figure to 2020-07-04 NEMO forecast publish render list
2020-07-04 10:20:59,029 INFO [make_plots] added Squamish figure to 2020-07-04 NEMO forecast publish render list
2020-07-04 10:20:59,030 INFO [make_plots] added Victoria figure to 2020-07-04 NEMO forecast publish render list
2020-07-04 10:20:59,031 INFO [make_plots] added Woodwards Landing figure to 2020-07-04 NEMO forecast publish render list
2020-07-04 10:20:59,032 INFO [make_plots] starting to render nemo forecast publish 04jul20 figures
2020-07-04 10:21:28,491 INFO [make_plots] Note: NumExpr detected 12 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
2020-07-04 10:21:28,492 INFO [make_plots] NumExpr defaulting to 8 threads.
2020-07-04 10:22:53,099 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15830065 = 2020-07-04 04:44:25 UTC estimated time to finish: 0000:03:02:18
2020-07-04 10:23:01,590 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39542404 = 2020-07-04 01:06:01 UTC estimated time to finish: 0000:15:22:07
2020-07-04 10:25:00,762 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 7497663 = 2020-07-04 03:21:20 UTC, 14.0% complete
2020-07-04 10:27:53,206 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15831959 = 2020-07-04 05:15:59 UTC estimated time to finish: 0000:02:57:25
2020-07-04 10:28:01,696 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39543592 = 2020-07-04 01:13:56 UTC estimated time to finish: 0000:15:13:36
2020-07-04 10:30:00,864 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 7497820 = 2020-07-04 05:06:00 UTC, 21.3% complete
2020-07-04 10:32:53,320 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15833771 = 2020-07-04 05:46:11 UTC estimated time to finish: 0000:02:53:11
2020-07-04 10:33:01,809 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39544698 = 2020-07-04 01:21:19 UTC estimated time to finish: 0000:15:09:57
2020-07-04 10:35:00,965 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 7497975 = 2020-07-04 06:49:20 UTC, 28.4% complete
2020-07-04 10:37:53,435 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15835502 = 2020-07-04 06:15:02 UTC estimated time to finish: 0000:02:48:45
2020-07-04 10:38:01,917 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39545886 = 2020-07-04 01:29:14 UTC estimated time to finish: 0000:15:03:35
2020-07-04 10:39:12,516 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 08074 Campbell River from 2020-06-29 00:00:00 to 2020-07-06 11:55:00
2020-07-04 10:40:01,066 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 7498131 = 2020-07-04 08:33:20 UTC, 35.7% complete
2020-07-04 10:42:53,552 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15837480 = 2020-07-04 06:48:00 UTC estimated time to finish: 0000:02:43:28
2020-07-04 10:43:02,027 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39546950 = 2020-07-04 01:36:20 UTC estimated time to finish: 0000:14:55:32
2020-07-04 10:43:32,921 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 9449424 Cherry Point from 2020-06-29 00:00:00 to 2020-07-06 11:55:00
2020-07-04 10:44:33,131 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 9449880 Friday Harbor from 2020-06-29 00:00:00 to 2020-07-06 11:55:00
2020-07-04 10:45:01,167 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 7498286 = 2020-07-04 10:16:40 UTC, 42.8% complete
2020-07-04 10:45:42,389 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07830 Halfmoon Bay from 2020-06-29 00:00:00 to 2020-07-06 11:55:00
2020-07-04 10:46:38,595 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07917 Nanaimo from 2020-06-29 00:00:00 to 2020-07-06 11:55:00
2020-07-04 10:47:53,663 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15839401 = 2020-07-04 07:20:01 UTC estimated time to finish: 0000:02:38:17
2020-07-04 10:48:02,134 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39548167 = 2020-07-04 01:44:26 UTC estimated time to finish: 0000:14:47:21
2020-07-04 10:48:38,199 INFO [make_surface_current_tiles] starting PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast/04jul20
2020-07-04 10:49:49,325 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 9443090 Neah Bay from 2020-06-29 00:00:00 to 2020-07-06 11:55:00
2020-07-04 10:50:01,268 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 7498440 = 2020-07-04 11:59:20 UTC, 50.0% complete
2020-07-04 10:50:07,006 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07654 New Westminster from 2020-06-29 00:00:00 to 2020-07-06 11:55:00
2020-07-04 10:52:52,026 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07277 Patricia Bay from 2020-06-29 00:00:00 to 2020-07-06 11:55:00
2020-07-04 10:52:53,783 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15841379 = 2020-07-04 07:52:59 UTC estimated time to finish: 0000:02:33:04
2020-07-04 10:53:02,243 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39549313 = 2020-07-04 01:52:05 UTC estimated time to finish: 0000:14:42:53
2020-07-04 10:53:02,609 INFO [make_surface_current_tiles] finished PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast/04jul20
2020-07-04 10:53:02,632 INFO [make_surface_current_tiles] finished rendering of tiles for 2020-07-04 forecast into /results/nowcast-sys/figures/surface_currents/forecast/04jul20
2020-07-04 10:53:02,633 INFO [make_surface_current_tiles] surface current tile figures for 2020-07-04 forecast completed
2020-07-04 10:53:03,153 INFO [manager] checklist updated with [surface current tiles] items from make_surface_current_tiles worker
2020-07-04 10:55:01,370 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 7498598 = 2020-07-04 13:44:40 UTC, 57.3% complete
2020-07-04 10:55:05,473 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07795 Point Atkinson from 2020-06-29 00:00:00 to 2020-07-06 11:55:00
2020-07-04 10:55:24,356 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 08525 Port Renfrew from 2020-06-29 00:00:00 to 2020-07-06 11:55:00
2020-07-04 10:57:35,778 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07594 Sand Heads from 2020-06-29 00:00:00 to 2020-07-06 11:55:00
2020-07-04 10:57:53,909 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15843341 = 2020-07-04 08:25:41 UTC estimated time to finish: 0000:02:27:44
2020-07-04 10:58:02,355 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39550420 = 2020-07-04 01:59:28 UTC estimated time to finish: 0000:14:40:38
2020-07-04 10:59:52,166 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07786 Sandy Cove from 2020-06-29 00:00:00 to 2020-07-06 11:55:00
2020-07-04 11:00:01,471 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 7498758 = 2020-07-04 15:31:20 UTC, 64.7% complete
2020-07-04 11:02:04,133 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07811 Squamish from 2020-06-29 00:00:00 to 2020-07-06 11:55:00
2020-07-04 11:02:23,276 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07120 Victoria from 2020-06-29 00:00:00 to 2020-07-06 11:55:00
2020-07-04 11:02:54,028 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15845074 = 2020-07-04 08:54:34 UTC estimated time to finish: 0000:02:23:16
2020-07-04 11:03:02,461 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39551374 = 2020-07-04 02:05:49 UTC estimated time to finish: 0000:14:39:14
2020-07-04 11:04:35,479 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07610 Woodwards Landing from 2020-06-29 00:00:00 to 2020-07-06 11:55:00
2020-07-04 11:05:01,572 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 7498915 = 2020-07-04 17:16:00 UTC, 72.0% complete
2020-07-04 11:06:39,248 INFO [make_plots] finished rendering nemo forecast publish 04jul20 figures
2020-07-04 11:06:39,250 INFO [make_plots] nemo publish plots for 2020-07-04 forecast completed
2020-07-04 11:06:39,263 INFO [manager] checklist updated with [plots] items from make_plots worker
2020-07-04 11:06:39,951 INFO [manager] launching NextWorker(module='nowcast.workers.make_feeds', args=['forecast', '--run-date', '2020-07-04'], host='localhost')
2020-07-04 11:06:44,431 INFO [make_feeds] running in process 21470
2020-07-04 11:06:44,432 INFO [make_feeds] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 11:06:44,433 INFO [make_feeds] publishing log messages to tcp://*:5564
2020-07-04 11:06:44,434 INFO [make_feeds] connected to skookum.eos.ubc.ca port 5555
2020-07-04 11:06:53,588 INFO [make_feeds] Note: NumExpr detected 12 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
2020-07-04 11:06:53,589 INFO [make_feeds] NumExpr defaulting to 8 threads.
2020-07-04 11:06:59,837 INFO [make_feeds] ATOM feeds for 2020-07-04 forecast run completed
2020-07-04 11:06:59,841 INFO [manager] checklist updated with [storm surge alert feeds] items from make_feeds worker
2020-07-04 11:07:54,148 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15847009 = 2020-07-04 09:26:49 UTC estimated time to finish: 0000:02:18:10
2020-07-04 11:08:02,577 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39552509 = 2020-07-04 02:13:23 UTC estimated time to finish: 0000:14:35:38
2020-07-04 11:10:01,673 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 7499071 = 2020-07-04 19:00:00 UTC, 79.2% complete
2020-07-04 11:12:54,274 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15848944 = 2020-07-04 09:59:04 UTC estimated time to finish: 0000:02:13:04
2020-07-04 11:13:02,689 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39553614 = 2020-07-04 02:20:45 UTC estimated time to finish: 0000:14:32:04
2020-07-04 11:15:01,774 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 7499230 = 2020-07-04 20:46:00 UTC, 86.6% complete
2020-07-04 11:17:54,401 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15850908 = 2020-07-04 10:31:48 UTC estimated time to finish: 0000:02:07:47
2020-07-04 11:18:02,794 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39554802 = 2020-07-04 02:28:40 UTC estimated time to finish: 0000:14:24:47
2020-07-04 11:20:01,875 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 7499387 = 2020-07-04 22:30:40 UTC, 93.8% complete
2020-07-04 11:22:54,534 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15852639 = 2020-07-04 11:00:39 UTC estimated time to finish: 0000:02:03:13
2020-07-04 11:23:02,908 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39555867 = 2020-07-04 02:35:46 UTC estimated time to finish: 0000:14:19:01
2020-07-04 11:25:01,977 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 7499520 = 2020-07-04 23:59:20 UTC, 100.0% complete
2020-07-04 11:27:54,648 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15854575 = 2020-07-04 11:32:55 UTC estimated time to finish: 0000:01:58:09
2020-07-04 11:28:03,016 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39557206 = 2020-07-04 02:44:42 UTC estimated time to finish: 0000:14:06:10
2020-07-04 11:30:02,085 INFO [watch_NEMO] nowcast-green NEMO run on arbutus.cloud-nowcast completed
2020-07-04 11:30:02,090 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2020-07-04 11:30:03,055 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_wind_file', args=['arbutus.cloud-nowcast', 'forecast'], host='arbutus.cloud-nowcast')
2020-07-04 11:30:03,063 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_current_file', args=['arbutus.cloud-nowcast', 'forecast'], host='arbutus.cloud-nowcast')
2020-07-04 11:30:03,067 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['salish-nowcast', 'nowcast+', '--shared-storage'], host='localhost')
2020-07-04 11:30:03,071 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2020-07-04'], host='localhost')
2020-07-04 11:30:04,140 INFO [download_results] running in process 21878
2020-07-04 11:30:04,141 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 11:30:04,142 INFO [download_results] publishing log messages to tcp://*:5564
2020-07-04 11:30:04,143 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2020-07-04 11:30:04,144 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/nowcast-green/04jul20 to /results2/SalishSea/nowcast-green.201905
2020-07-04 11:30:04,154 INFO [make_forcing_links] running in process 21877
2020-07-04 11:30:04,155 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 11:30:04,156 INFO [make_forcing_links] publishing log messages to tcp://*:5567
2020-07-04 11:30:04,157 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2020-07-04 11:30:05,073 INFO [make_forcing_links] nowcast+ 2020-07-04 forcing file links on salish-nowcast created
2020-07-04 11:30:05,078 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2020-07-04 11:30:05,776 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['salish-nowcast', 'nowcast-dev', '--run-date', '2020-07-04'], host='salish-nowcast')
2020-07-04 11:30:07,093 INFO [make_ww3_wind_file] running in process 6029
2020-07-04 11:30:07,095 INFO [make_ww3_wind_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 11:30:07,096 INFO [make_ww3_wind_file] publishing log messages to tcp://*:5570
2020-07-04 11:30:07,149 INFO [make_ww3_wind_file] connected to skookum.eos.ubc.ca port 5555
2020-07-04 11:30:07,151 INFO [make_ww3_wind_file] Creating wwatch3 wind forcing file for 2020-07-04 forecast run
2020-07-04 11:30:07,159 INFO [make_ww3_current_file] running in process 6028
2020-07-04 11:30:07,160 INFO [make_ww3_current_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 11:30:07,162 INFO [make_ww3_current_file] publishing log messages to tcp://*:5571
2020-07-04 11:30:07,163 INFO [make_ww3_current_file] connected to skookum.eos.ubc.ca port 5555
2020-07-04 11:30:07,164 INFO [make_ww3_current_file] Creating wwatch3 currents forcing file for 2020-07-04 forecast run
2020-07-04 11:30:07,226 INFO [run_NEMO] running in process 16753
2020-07-04 11:30:07,227 INFO [run_NEMO] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 11:30:07,228 INFO [run_NEMO] publishing log messages to tcp://*:5556
2020-07-04 11:30:07,229 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2020-07-04 11:30:08,227 WARNING [run_NEMO] There are uncommitted changes in /SalishSeaCast/SalishSeaNowcast
2020-07-04 11:30:08,796 INFO [run_NEMO] nowcast-dev: launching /SalishSeaCast/runs/04jul20nowcast-dev_2020-07-04T113007.384989-0700/SalishSeaNEMO.sh on salish-nowcast
2020-07-04 11:30:08,873 INFO [run_NEMO] nowcast-dev NEMO run for 2020-07-04 on salish-nowcast started
2020-07-04 11:30:08,878 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2020-07-04 11:30:09,548 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['salish-nowcast', 'nowcast-dev'], host='salish-nowcast')
2020-07-04 11:30:10,750 INFO [watch_NEMO] running in process 16994
2020-07-04 11:30:10,751 INFO [watch_NEMO] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 11:30:10,752 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2020-07-04 11:30:10,754 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2020-07-04 11:30:10,813 INFO [watch_NEMO] nowcast-dev on salish-nowcast: time.step not found; continuing to watch...
2020-07-04 11:30:11,205 INFO [make_ww3_wind_file] wwatch3 wind forcing file created on arbutus.cloud-nowcast for 2020-07-04 forecast run
2020-07-04 11:30:11,209 INFO [manager] checklist updated with [WW3 wind forcing] items from make_ww3_wind_file worker
2020-07-04 11:30:54,349 INFO [make_ww3_current_file] wwatch3 currents forcing file created on arbutus.cloud-nowcast for 2020-07-04 forecast run
2020-07-04 11:30:54,355 INFO [manager] checklist updated with [WW3 currents forcing] items from make_ww3_current_file worker
2020-07-04 11:30:55,044 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2020-07-04'], host='arbutus.cloud-nowcast')
2020-07-04 11:30:57,056 INFO [run_ww3] running in process 6229
2020-07-04 11:30:57,057 INFO [run_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 11:30:57,058 INFO [run_ww3] publishing log messages to tcp://*:5572
2020-07-04 11:30:57,059 INFO [run_ww3] connected to skookum.eos.ubc.ca port 5555
2020-07-04 11:30:57,107 INFO [run_ww3] Created run directory /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/nowcast_2020-07-04T113057.056541-0700
2020-07-04 11:30:57,109 INFO [run_ww3] nowcast: launching /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/nowcast_2020-07-04T113057.056541-0700/SoGWW3.sh on arbutus.cloud-nowcast
2020-07-04 11:30:57,124 INFO [run_ww3] nowcast WaveWatch3 run for 2020-07-04 on arbutus.cloud-nowcast started
2020-07-04 11:30:57,130 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker
2020-07-04 11:30:57,837 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'nowcast'], host='arbutus.cloud-nowcast')
2020-07-04 11:30:59,791 INFO [watch_ww3] running in process 6333
2020-07-04 11:30:59,793 INFO [watch_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 11:30:59,794 INFO [watch_ww3] publishing log messages to tcp://*:5573
2020-07-04 11:30:59,795 INFO [watch_ww3] connected to skookum.eos.ubc.ca port 5555
2020-07-04 11:30:59,836 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2020-07-04 11:32:54,763 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15856470 = 2020-07-04 12:04:30 UTC estimated time to finish: 0000:01:53:17
2020-07-04 11:33:03,124 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39558789 = 2020-07-04 02:55:15 UTC estimated time to finish: 0000:13:46:01
2020-07-04 11:34:07,075 INFO [download_results] nowcast-green 2020-07-04 results files from arbutus.cloud-nowcast downloaded
2020-07-04 11:34:07,086 INFO [manager] checklist updated with [results files] items from download_results worker
2020-07-04 11:34:07,772 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast-green', 'research', '--run-date', '2020-07-04'], host='localhost')
2020-07-04 11:34:07,776 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nowcast-green'], host='localhost')
2020-07-04 11:34:09,258 INFO [ping_erddap] running in process 21978
2020-07-04 11:34:09,259 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 11:34:09,259 INFO [ping_erddap] publishing log messages to tcp://*:5564
2020-07-04 11:34:09,260 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2020-07-04 11:34:09,283 INFO [ping_erddap] nowcast-green ERDDAP dataset flag file(s) created
2020-07-04 11:34:09,287 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2020-07-04 11:34:13,397 INFO [make_plots] running in process 21977
2020-07-04 11:34:13,398 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 11:34:13,399 INFO [make_plots] publishing log messages to tcp://*:5564
2020-07-04 11:34:13,400 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2020-07-04 11:34:13,816 INFO [make_plots] preparing render list for 2020-07-04 NEMO nowcast-green research figures
2020-07-04 11:34:16,915 INFO [make_plots] added temperature_thalweg_and_surface figures to 2020-07-04 NEMO nowcast-green research render list
2020-07-04 11:34:18,201 INFO [make_plots] added nitrate_thalweg_and_surface figures to 2020-07-04 NEMO nowcast-green research render list
2020-07-04 11:34:19,340 INFO [make_plots] added ammonium_thalweg_and_surface figures to 2020-07-04 NEMO nowcast-green research render list
2020-07-04 11:34:20,500 INFO [make_plots] added silicon_thalweg_and_surface figures to 2020-07-04 NEMO nowcast-green research render list
2020-07-04 11:34:21,767 INFO [make_plots] added dissolved_organic_nitrogen_thalweg_and_surface figures to 2020-07-04 NEMO nowcast-green research render list
2020-07-04 11:34:23,045 INFO [make_plots] added particulate_organic_nitrogen_thalweg_and_surface figures to 2020-07-04 NEMO nowcast-green research render list
2020-07-04 11:34:24,220 INFO [make_plots] added biogenic_silicon_thalweg_and_surface figures to 2020-07-04 NEMO nowcast-green research render list
2020-07-04 11:34:25,574 INFO [make_plots] added diatoms_thalweg_and_surface figures to 2020-07-04 NEMO nowcast-green research render list
2020-07-04 11:34:26,907 INFO [make_plots] added ciliates_thalweg_and_surface figures to 2020-07-04 NEMO nowcast-green research render list
2020-07-04 11:34:28,215 INFO [make_plots] added flagellates_thalweg_and_surface figures to 2020-07-04 NEMO nowcast-green research render list
2020-07-04 11:34:29,488 INFO [make_plots] added microzooplankton_thalweg_and_surface figures to 2020-07-04 NEMO nowcast-green research render list
2020-07-04 11:34:30,745 INFO [make_plots] added mesozooplankton_thalweg_and_surface figures to 2020-07-04 NEMO nowcast-green research render list
2020-07-04 11:34:31,944 INFO [make_plots] added Fraser_tracer_thalweg_and_surface figures to 2020-07-04 NEMO nowcast-green research render list
2020-07-04 11:34:32,031 INFO [make_plots] added temperature_salinity_timeseries figure to 2020-07-04 NEMO nowcast-green research render list
2020-07-04 11:34:32,032 INFO [make_plots] added nitrate_diatoms_timeseries figure to 2020-07-04 NEMO nowcast-green research render list
2020-07-04 11:34:32,032 INFO [make_plots] added mesodinium_flagellates_timeseries figure to 2020-07-04 NEMO nowcast-green research render list
2020-07-04 11:34:32,033 INFO [make_plots] added mesozoo_microzoo_timeseries figure to 2020-07-04 NEMO nowcast-green research render list
2020-07-04 11:34:32,033 INFO [make_plots] starting to render nemo nowcast-green research 04jul20 figures
2020-07-04 11:35:10,914 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4626748 = 2020-07-04 00:18:00 UTC, 1.3% complete
2020-07-04 11:35:59,932 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 99, 22.9% complete
2020-07-04 11:37:54,878 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15858392 = 2020-07-04 12:36:32 UTC estimated time to finish: 0000:01:48:14
2020-07-04 11:38:03,242 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39560400 = 2020-07-04 03:06:00 UTC estimated time to finish: 0000:13:26:08
2020-07-04 11:40:11,014 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4626803 = 2020-07-04 00:54:40 UTC, 3.8% complete
2020-07-04 11:41:00,032 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 210, 48.6% complete
2020-07-04 11:42:55,001 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15860136 = 2020-07-04 13:05:36 UTC estimated time to finish: 0000:01:43:43
2020-07-04 11:43:03,352 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39562025 = 2020-07-04 03:16:50 UTC estimated time to finish: 0000:13:07:32
2020-07-04 11:45:11,111 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4626857 = 2020-07-04 01:30:40 UTC, 6.3% complete
2020-07-04 11:46:00,147 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 318, 73.6% complete
2020-07-04 11:47:55,110 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15862019 = 2020-07-04 13:36:59 UTC estimated time to finish: 0000:01:38:48
2020-07-04 11:48:03,463 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39563486 = 2020-07-04 03:26:34 UTC estimated time to finish: 0000:12:52:02
2020-07-04 11:50:11,212 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4626911 = 2020-07-04 02:06:40 UTC, 8.8% complete
2020-07-04 11:51:00,235 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 426, 98.6% complete
2020-07-04 11:52:55,227 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15863954 = 2020-07-04 14:09:14 UTC estimated time to finish: 0000:01:33:44
2020-07-04 11:53:03,492 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39565070 = 2020-07-04 03:37:08 UTC estimated time to finish: 0000:12:36:55
2020-07-04 11:55:11,311 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4626964 = 2020-07-04 02:42:00 UTC, 11.3% complete
2020-07-04 11:56:00,338 INFO [watch_ww3] nowcast WWATCH3 run on arbutus.cloud-nowcast completed
2020-07-04 11:56:00,356 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2020-07-04 11:56:01,397 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2020-07-04'], host='localhost')
2020-07-04 11:56:01,400 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2020-07-04'], host='arbutus.cloud-nowcast')
2020-07-04 11:56:03,463 INFO [run_ww3] running in process 6498
2020-07-04 11:56:03,538 INFO [run_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 11:56:03,539 INFO [run_ww3] publishing log messages to tcp://*:5572
2020-07-04 11:56:03,540 INFO [run_ww3] connected to skookum.eos.ubc.ca port 5555
2020-07-04 11:56:03,543 INFO [run_ww3] Created run directory /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast_2020-07-04T115603.462558-0700
2020-07-04 11:56:03,544 INFO [run_ww3] forecast: launching /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast_2020-07-04T115603.462558-0700/SoGWW3.sh on arbutus.cloud-nowcast
2020-07-04 11:56:03,545 INFO [run_ww3] forecast WaveWatch3 run for 2020-07-04 on arbutus.cloud-nowcast started
2020-07-04 11:56:03,547 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker
2020-07-04 11:56:03,806 INFO [download_wwatch3_results] running in process 22509
2020-07-04 11:56:03,814 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 11:56:03,815 INFO [download_wwatch3_results] publishing log messages to tcp://*:5567
2020-07-04 11:56:03,816 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2020-07-04 11:56:04,182 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'forecast'], host='arbutus.cloud-nowcast')
2020-07-04 11:56:06,106 INFO [watch_ww3] running in process 6688
2020-07-04 11:56:06,107 INFO [watch_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 11:56:06,108 INFO [watch_ww3] publishing log messages to tcp://*:5573
2020-07-04 11:56:06,109 INFO [watch_ww3] connected to skookum.eos.ubc.ca port 5555
2020-07-04 11:56:06,144 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2020-07-04 11:57:16,304 INFO [download_wwatch3_results] nowcast 2020-07-04 results files from arbutus.cloud-nowcast downloaded
2020-07-04 11:57:16,307 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2020-07-04 11:57:55,344 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15865848 = 2020-07-04 14:40:48 UTC estimated time to finish: 0000:01:28:45
2020-07-04 11:58:03,603 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39566490 = 2020-07-04 03:46:36 UTC estimated time to finish: 0000:12:27:22
2020-07-04 12:00:11,412 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4627018 = 2020-07-04 03:18:00 UTC, 13.8% complete
2020-07-04 12:01:06,245 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 99, 15.3% complete
2020-07-04 12:02:55,462 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15867730 = 2020-07-04 15:12:10 UTC estimated time to finish: 0000:01:23:54
2020-07-04 12:03:03,715 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39567870 = 2020-07-04 03:55:48 UTC estimated time to finish: 0000:12:18:39
2020-07-04 12:05:11,512 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4627072 = 2020-07-04 03:54:00 UTC, 16.3% complete
2020-07-04 12:06:06,346 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 207, 31.9% complete
2020-07-04 12:07:55,561 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15869432 = 2020-07-04 15:40:32 UTC estimated time to finish: 0000:01:19:27
2020-07-04 12:08:03,827 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39569168 = 2020-07-04 04:04:27 UTC estimated time to finish: 0000:12:09:56
2020-07-04 12:10:11,612 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4627126 = 2020-07-04 04:30:00 UTC, 18.8% complete
2020-07-04 12:11:06,447 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 315, 48.6% complete
2020-07-04 12:12:55,679 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15871355 = 2020-07-04 16:12:35 UTC estimated time to finish: 0000:01:14:21
2020-07-04 12:13:03,938 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39570629 = 2020-07-04 04:14:11 UTC estimated time to finish: 0000:12:00:26
2020-07-04 12:15:11,713 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4627180 = 2020-07-04 05:06:00 UTC, 21.3% complete
2020-07-04 12:16:06,548 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 417, 64.4% complete
2020-07-04 12:17:55,799 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15873250 = 2020-07-04 16:44:10 UTC estimated time to finish: 0000:01:09:21
2020-07-04 12:18:04,051 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39572049 = 2020-07-04 04:23:39 UTC estimated time to finish: 0000:11:51:50
2020-07-04 12:20:11,812 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4627235 = 2020-07-04 05:42:40 UTC, 23.8% complete
2020-07-04 12:21:06,649 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 525, 81.0% complete
2020-07-04 12:22:55,919 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15875184 = 2020-07-04 17:16:24 UTC estimated time to finish: 0000:01:04:14
2020-07-04 12:23:04,164 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39573469 = 2020-07-04 04:33:07 UTC estimated time to finish: 0000:11:43:32
2020-07-04 12:25:11,913 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4627289 = 2020-07-04 06:18:40 UTC, 26.3% complete
2020-07-04 12:26:06,750 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 633, 97.7% complete
2020-07-04 12:27:56,038 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15876876 = 2020-07-04 17:44:36 UTC estimated time to finish: 0000:00:59:46
2020-07-04 12:28:04,277 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39574738 = 2020-07-04 04:41:35 UTC estimated time to finish: 0000:11:36:22
2020-07-04 12:30:12,014 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4627343 = 2020-07-04 06:54:40 UTC, 28.8% complete
2020-07-04 12:31:06,852 INFO [watch_ww3] forecast WWATCH3 run on arbutus.cloud-nowcast completed
2020-07-04 12:31:06,854 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2020-07-04 12:31:07,881 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2020-07-04'], host='localhost')
2020-07-04 12:31:10,048 INFO [download_wwatch3_results] running in process 23461
2020-07-04 12:31:10,049 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 12:31:10,050 INFO [download_wwatch3_results] publishing log messages to tcp://*:5567
2020-07-04 12:31:10,051 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2020-07-04 12:31:51,838 INFO [download_wwatch3_results] forecast 2020-07-04 results files from arbutus.cloud-nowcast downloaded
2020-07-04 12:31:51,842 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2020-07-04 12:31:52,514 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['wwatch3', 'forecast', '--run-date', '2020-07-04'], host='localhost')
2020-07-04 12:31:53,817 INFO [update_forecast_datasets] running in process 23514
2020-07-04 12:31:53,818 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 12:31:53,819 INFO [update_forecast_datasets] publishing log messages to tcp://*:5567
2020-07-04 12:31:53,820 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2020-07-04 12:31:53,820 INFO [update_forecast_datasets] updating wwatch3 most_recent_forecast directory from forecast/04jul20 run
2020-07-04 12:31:53,822 INFO [update_forecast_datasets] updated wwatch3 most_recent_forecast directory from forecast/04jul20 run
2020-07-04 12:31:53,823 INFO [update_forecast_datasets] updating wwatch3 forecast directory for forecast/04jul20 run
2020-07-04 12:31:54,113 INFO [update_forecast_datasets] updated wwatch3 forecast directory for forecast/04jul20 run: /results/SalishSea/rolling-forecasts/wwatch3
2020-07-04 12:31:54,114 INFO [update_forecast_datasets] wwatch3 2020-07-04 forecast rolling forecast datasets updated
2020-07-04 12:31:54,230 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2020-07-04 12:31:55,011 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['wwatch3-forecast'], host='localhost')
2020-07-04 12:31:56,144 INFO [ping_erddap] running in process 23518
2020-07-04 12:31:56,145 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 12:31:56,146 INFO [ping_erddap] publishing log messages to tcp://*:5567
2020-07-04 12:31:56,146 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2020-07-04 12:31:56,147 INFO [ping_erddap] wwatch3-forecast ERDDAP dataset flag file(s) created
2020-07-04 12:31:56,149 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2020-07-04 12:31:56,894 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['wwatch3', 'forecast2', 'publish', '--run-date', '2020-07-04'], host='localhost')
2020-07-04 12:32:07,795 INFO [make_plots] running in process 23524
2020-07-04 12:32:07,796 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 12:32:07,797 INFO [make_plots] publishing log messages to tcp://*:5567
2020-07-04 12:32:07,798 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2020-07-04 12:32:07,799 INFO [make_plots] preparing render list for 2020-07-04 WaveWatch3 forecast2 publish figures
2020-07-04 12:32:07,800 INFO [make_plots] added Halibut Bank figure to 2020-07-04 WaveWatch3 forecast2 publish render list
2020-07-04 12:32:07,801 INFO [make_plots] added Sentry Shoal figure to 2020-07-04 WaveWatch3 forecast2 publish render list
2020-07-04 12:32:07,802 INFO [make_plots] starting to render wwatch3 forecast2 publish 04jul20 figures
2020-07-04 12:32:56,158 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15878770 = 2020-07-04 18:16:10 UTC estimated time to finish: 0000:00:54:47
2020-07-04 12:33:04,390 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39576076 = 2020-07-04 04:50:30 UTC estimated time to finish: 0000:11:29:49
2020-07-04 12:33:16,601 INFO [make_plots] Note: NumExpr detected 12 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
2020-07-04 12:33:16,602 INFO [make_plots] NumExpr defaulting to 8 threads.
2020-07-04 12:33:16,611 INFO [make_plots] retrieving last 45 days of real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46146 Halibut Bank
2020-07-04 12:34:26,863 INFO [make_plots] retrieving last 45 days of real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46131 Sentry Shoal
2020-07-04 12:34:28,245 INFO [make_plots] finished rendering wwatch3 forecast2 publish 04jul20 figures
2020-07-04 12:34:28,246 INFO [make_plots] wwatch3 publish plots for 2020-07-04 forecast2 completed
2020-07-04 12:34:28,251 INFO [manager] checklist updated with [plots] items from make_plots worker
2020-07-04 12:35:12,115 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4627397 = 2020-07-04 07:30:40 UTC, 31.3% complete
2020-07-04 12:37:56,280 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15880665 = 2020-07-04 18:47:45 UTC estimated time to finish: 0000:00:49:46
2020-07-04 12:38:04,504 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39577497 = 2020-07-04 04:59:58 UTC estimated time to finish: 0000:11:22:12
2020-07-04 12:40:12,213 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4627451 = 2020-07-04 08:06:40 UTC, 33.8% complete
2020-07-04 12:42:56,400 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15882547 = 2020-07-04 19:19:07 UTC estimated time to finish: 0000:00:44:47
2020-07-04 12:43:04,618 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39578835 = 2020-07-04 05:08:54 UTC estimated time to finish: 0000:11:15:43
2020-07-04 12:45:12,313 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4627506 = 2020-07-04 08:43:20 UTC, 36.4% complete
2020-07-04 12:47:56,522 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15884280 = 2020-07-04 19:48:00 UTC estimated time to finish: 0000:00:40:11
2020-07-04 12:48:04,733 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39580255 = 2020-07-04 05:18:22 UTC estimated time to finish: 0000:11:08:21
2020-07-04 12:50:12,414 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4627560 = 2020-07-04 09:19:20 UTC, 38.9% complete
2020-07-04 12:52:56,645 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15886173 = 2020-07-04 20:19:33 UTC estimated time to finish: 0000:00:35:10
2020-07-04 12:53:04,849 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39581524 = 2020-07-04 05:26:49 UTC estimated time to finish: 0000:11:01:57
2020-07-04 12:55:12,514 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4627614 = 2020-07-04 09:55:20 UTC, 41.4% complete
2020-07-04 12:57:56,770 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15888108 = 2020-07-04 20:51:48 UTC estimated time to finish: 0000:00:30:01
2020-07-04 12:58:04,967 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39582903 = 2020-07-04 05:36:01 UTC estimated time to finish: 0000:10:55:22
2020-07-04 13:00:12,613 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4627668 = 2020-07-04 10:31:20 UTC, 43.9% complete
2020-07-04 13:02:56,894 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15889961 = 2020-07-04 21:22:41 UTC estimated time to finish: 0000:00:25:07
2020-07-04 13:03:05,083 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39584242 = 2020-07-04 05:44:56 UTC estimated time to finish: 0000:10:49:28
2020-07-04 13:03:55,620 INFO [make_plots] finished rendering nemo nowcast-green research 04jul20 figures
2020-07-04 13:03:55,635 INFO [make_plots] nemo research plots for 2020-07-04 nowcast-green completed
2020-07-04 13:03:55,792 INFO [manager] checklist updated with [plots] items from make_plots worker
2020-07-04 13:05:12,714 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4627723 = 2020-07-04 11:08:00 UTC, 46.4% complete
2020-07-04 13:07:57,019 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15891925 = 2020-07-04 21:55:25 UTC estimated time to finish: 0000:00:19:53
2020-07-04 13:08:05,200 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39585662 = 2020-07-04 05:54:24 UTC estimated time to finish: 0000:10:42:28
2020-07-04 13:10:12,801 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4627777 = 2020-07-04 11:44:00 UTC, 48.9% complete
2020-07-04 13:12:57,145 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15893657 = 2020-07-04 22:24:17 UTC estimated time to finish: 0000:00:15:16
2020-07-04 13:13:05,318 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39586931 = 2020-07-04 06:02:52 UTC estimated time to finish: 0000:10:36:02
2020-07-04 13:15:12,829 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4627831 = 2020-07-04 12:20:00 UTC, 51.4% complete
2020-07-04 13:17:57,270 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15895551 = 2020-07-04 22:55:51 UTC estimated time to finish: 0000:00:10:14
2020-07-04 13:18:05,435 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39588351 = 2020-07-04 06:12:20 UTC estimated time to finish: 0000:10:29:03
2020-07-04 13:20:12,930 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4627886 = 2020-07-04 12:56:40 UTC, 54.0% complete
2020-07-04 13:22:57,397 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15897487 = 2020-07-04 23:28:07 UTC estimated time to finish: 0000:00:05:05
2020-07-04 13:23:05,553 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39589771 = 2020-07-04 06:21:48 UTC estimated time to finish: 0000:10:22:18
2020-07-04 13:25:13,031 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4627940 = 2020-07-04 13:32:40 UTC, 56.5% complete
2020-07-04 13:27:57,503 INFO [watch_fvcom] x2 nowcast FVCOM VH-FR run on arbutus.cloud-nowcast completed
2020-07-04 13:27:57,508 INFO [manager] checklist updated with [FVCOM run] items from watch_fvcom worker
2020-07-04 13:27:58,464 INFO [manager] launching NextWorker(module='nowcast.workers.download_fvcom_results', args=['arbutus.cloud-nowcast', 'x2', 'nowcast', '--run-date', '2020-07-04'], host='localhost')
2020-07-04 13:27:58,469 INFO [manager] launching NextWorker(module='nowcast.workers.make_fvcom_boundary', args=['arbutus.cloud-nowcast', 'x2', 'forecast', '--run-date', '2020-07-04'], host='arbutus.cloud-nowcast')
2020-07-04 13:27:59,915 INFO [download_fvcom_results] running in process 24660
2020-07-04 13:27:59,916 INFO [download_fvcom_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 13:27:59,917 INFO [download_fvcom_results] publishing log messages to tcp://*:5564
2020-07-04 13:27:59,918 INFO [download_fvcom_results] connected to skookum.eos.ubc.ca port 5555
2020-07-04 13:28:02,013 INFO [make_fvcom_boundary] running in process 7059
2020-07-04 13:28:02,013 INFO [make_fvcom_boundary] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 13:28:02,014 INFO [make_fvcom_boundary] publishing log messages to tcp://*:5580
2020-07-04 13:28:02,069 INFO [make_fvcom_boundary] connected to skookum.eos.ubc.ca port 5555
2020-07-04 13:28:02,070 INFO [make_fvcom_boundary] Creating VHFR FVCOM open boundary file for x2 forecast run from 2020-07-04 NEMO run
2020-07-04 13:28:05,670 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39591191 = 2020-07-04 06:31:16 UTC estimated time to finish: 0000:10:15:30
2020-07-04 13:28:44,006 INFO [download_fvcom_results] VHFR FVCOM x2 nowcast 2020-07-04 results files from arbutus.cloud-nowcast downloaded
2020-07-04 13:28:44,012 INFO [manager] checklist updated with [VHFR FVCOM results files] items from download_fvcom_results worker
2020-07-04 13:28:44,693 INFO [manager] launching NextWorker(module='nowcast.workers.get_vfpa_hadcp', args=['--data-date', '2020-07-04'], host='localhost')
2020-07-04 13:28:44,697 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['fvcom', 'nowcast-x2', 'research', '--run-date', '2020-07-04'], host='localhost')
2020-07-04 13:28:48,115 INFO [get_vfpa_hadcp] running in process 24699
2020-07-04 13:28:48,116 INFO [get_vfpa_hadcp] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 13:28:48,117 INFO [get_vfpa_hadcp] publishing log messages to tcp://*:5564
2020-07-04 13:28:48,118 INFO [get_vfpa_hadcp] connected to skookum.eos.ubc.ca port 5555
2020-07-04 13:28:48,119 INFO [get_vfpa_hadcp] processing VFPA HADCP data from 2nd Narrows Rail Bridge for 2020-07-04
2020-07-04 13:28:49,170 INFO [get_vfpa_hadcp] added VFPA HADCP data from 2nd Narrows Rail Bridge for 2020-07-04 to /opp/observations/AISDATA/netcdf/VFPA_2ND_NARROWS_HADCP_2s_202007.nc
2020-07-04 13:28:49,171 INFO [get_vfpa_hadcp] VFPA HADCP observations added to 2020-07 netcdf file
2020-07-04 13:28:49,181 INFO [manager] checklist updated with [VFPA HADCP data] items from get_vfpa_hadcp worker
2020-07-04 13:28:49,860 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['VFPA-HADCP'], host='localhost')
2020-07-04 13:28:50,810 INFO [make_plots] running in process 24700
2020-07-04 13:28:50,811 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 13:28:50,812 INFO [make_plots] publishing log messages to tcp://*:5564
2020-07-04 13:28:50,812 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2020-07-04 13:28:50,887 INFO [make_plots] preparing render list for 2020-07-04 VHFR FVCOM nowcast-x2 research figures
2020-07-04 13:28:50,890 INFO [make_plots] added EnglishBay_surface_currents figures to 2020-07-04 VHFR FVCOM nowcast-x2 research render list
2020-07-04 13:28:50,892 INFO [make_plots] added VancouverHarbour_surface_currents figures to 2020-07-04 VHFR FVCOM nowcast-x2 research render list
2020-07-04 13:28:50,894 INFO [make_plots] added IndianArm_surface_currents figures to 2020-07-04 VHFR FVCOM nowcast-x2 research render list
2020-07-04 13:28:50,896 INFO [make_plots] added VancouverHarbour_thalweg figures to 2020-07-04 VHFR FVCOM nowcast-x2 research render list
2020-07-04 13:28:50,898 INFO [make_plots] added PortMoody_thalweg figures to 2020-07-04 VHFR FVCOM nowcast-x2 research render list
2020-07-04 13:28:50,900 INFO [make_plots] added IndianArm_thalweg figures to 2020-07-04 VHFR FVCOM nowcast-x2 research render list
2020-07-04 13:28:50,901 INFO [make_plots] starting to render fvcom nowcast-x2 research 04jul20 figures
2020-07-04 13:28:50,995 INFO [ping_erddap] running in process 24706
2020-07-04 13:28:51,001 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 13:28:51,007 INFO [ping_erddap] publishing log messages to tcp://*:5567
2020-07-04 13:28:51,025 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2020-07-04 13:28:51,026 INFO [ping_erddap] VFPA-HADCP ERDDAP dataset flag file(s) created
2020-07-04 13:28:51,028 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2020-07-04 13:28:51,556 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['fvcom', 'nowcast-x2', 'publish', '--run-date', '2020-07-04'], host='localhost')
2020-07-04 13:28:51,559 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['fvcom', 'forecast-x2', 'publish', '--run-date', '2020-07-03'], host='localhost')
2020-07-04 13:28:54,912 INFO [make_plots] running in process 24712
2020-07-04 13:28:54,913 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 13:28:54,914 INFO [make_plots] publishing log messages to tcp://*:5567
2020-07-04 13:28:54,915 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2020-07-04 13:28:54,919 INFO [make_plots] running in process 24713
2020-07-04 13:28:54,920 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 13:28:54,921 INFO [make_plots] publishing log messages to tcp://*:5568
2020-07-04 13:28:54,922 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2020-07-04 13:28:56,397 INFO [make_plots] preparing render list for 2020-07-04 VHFR FVCOM nowcast-x2 publish figures
2020-07-04 13:28:56,399 INFO [make_plots] added Calamity Point figure to 2020-07-04 VHFR FVCOM nowcast-x2 publish render list
2020-07-04 13:28:56,400 INFO [make_plots] added Indian Arm Head figure to 2020-07-04 VHFR FVCOM nowcast-x2 publish render list
2020-07-04 13:28:56,400 INFO [make_plots] added New Westminster figure to 2020-07-04 VHFR FVCOM nowcast-x2 publish render list
2020-07-04 13:28:56,401 INFO [make_plots] added Port Moody figure to 2020-07-04 VHFR FVCOM nowcast-x2 publish render list
2020-07-04 13:28:56,402 INFO [make_plots] added Sand Heads figure to 2020-07-04 VHFR FVCOM nowcast-x2 publish render list
2020-07-04 13:28:56,402 INFO [make_plots] added Sandy Cove figure to 2020-07-04 VHFR FVCOM nowcast-x2 publish render list
2020-07-04 13:28:56,403 INFO [make_plots] added Vancouver Harbour figure to 2020-07-04 VHFR FVCOM nowcast-x2 publish render list
2020-07-04 13:28:56,403 INFO [make_plots] added Woodwards Landing figure to 2020-07-04 VHFR FVCOM nowcast-x2 publish render list
2020-07-04 13:28:56,404 INFO [make_plots] added 2ndNarrows_current figure to 2020-07-04 VHFR FVCOM nowcast-x2 publish render list
2020-07-04 13:28:56,404 INFO [make_plots] starting to render fvcom nowcast-x2 publish 04jul20 figures
2020-07-04 13:28:56,408 INFO [make_plots] preparing render list for 2020-07-03 VHFR FVCOM forecast-x2 publish figures
2020-07-04 13:28:56,411 INFO [make_plots] added Calamity Point figure to 2020-07-03 VHFR FVCOM forecast-x2 publish render list
2020-07-04 13:28:56,412 INFO [make_plots] added Indian Arm Head figure to 2020-07-03 VHFR FVCOM forecast-x2 publish render list
2020-07-04 13:28:56,413 INFO [make_plots] added New Westminster figure to 2020-07-03 VHFR FVCOM forecast-x2 publish render list
2020-07-04 13:28:56,413 INFO [make_plots] added Port Moody figure to 2020-07-03 VHFR FVCOM forecast-x2 publish render list
2020-07-04 13:28:56,414 INFO [make_plots] added Sand Heads figure to 2020-07-03 VHFR FVCOM forecast-x2 publish render list
2020-07-04 13:28:56,414 INFO [make_plots] added Sandy Cove figure to 2020-07-03 VHFR FVCOM forecast-x2 publish render list
2020-07-04 13:28:56,415 INFO [make_plots] added Vancouver Harbour figure to 2020-07-03 VHFR FVCOM forecast-x2 publish render list
2020-07-04 13:28:56,415 INFO [make_plots] added Woodwards Landing figure to 2020-07-03 VHFR FVCOM forecast-x2 publish render list
2020-07-04 13:28:56,416 INFO [make_plots] added 2ndNarrows_current figure to 2020-07-03 VHFR FVCOM forecast-x2 publish render list
2020-07-04 13:28:56,417 INFO [make_plots] starting to render fvcom forecast-x2 publish 03jul20 figures
2020-07-04 13:28:56,428 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07724 Calamity Point from 2020-07-03 23:55:00 to 2020-07-05 00:00:00
2020-07-04 13:28:56,429 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07724 Calamity Point from 2020-07-02 23:55:00 to 2020-07-05 12:00:00
2020-07-04 13:29:16,859 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07724 Calamity Point from 2020-07-04 00:00:00 to 2020-07-05 00:00:00
2020-07-04 13:29:18,219 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07774 Indian Arm Head from 2020-07-03 23:55:00 to 2020-07-05 00:00:00
2020-07-04 13:29:18,784 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07774 Indian Arm Head from 2020-07-04 00:00:00 to 2020-07-05 00:00:00
2020-07-04 13:29:20,770 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07654 New Westminster from 2020-07-03 23:55:00 to 2020-07-05 00:00:00
2020-07-04 13:29:38,304 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07724 Calamity Point from 2020-07-03 00:00:00 to 2020-07-05 12:00:00
2020-07-04 13:29:39,446 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07654 New Westminster from 2020-07-04 00:00:00 to 2020-07-05 00:00:00
2020-07-04 13:29:40,026 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07774 Indian Arm Head from 2020-07-02 23:55:00 to 2020-07-05 12:00:00
2020-07-04 13:29:40,591 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07774 Indian Arm Head from 2020-07-03 00:00:00 to 2020-07-05 12:00:00
2020-07-04 13:29:42,122 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07654 New Westminster from 2020-07-02 23:55:00 to 2020-07-05 12:00:00
2020-07-04 13:29:42,368 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07755 Port Moody from 2020-07-03 23:55:00 to 2020-07-05 00:00:00
2020-07-04 13:30:01,205 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07755 Port Moody from 2020-07-04 00:00:00 to 2020-07-05 00:00:00
2020-07-04 13:30:04,428 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07594 Sand Heads from 2020-07-03 23:55:00 to 2020-07-05 00:00:00
2020-07-04 13:30:13,128 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4627995 = 2020-07-04 14:09:20 UTC, 59.0% complete
2020-07-04 13:30:22,358 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07654 New Westminster from 2020-07-03 00:00:00 to 2020-07-05 12:00:00
2020-07-04 13:30:23,148 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07594 Sand Heads from 2020-07-04 00:00:00 to 2020-07-05 00:00:00
2020-07-04 13:30:26,135 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07786 Sandy Cove from 2020-07-03 23:55:00 to 2020-07-05 00:00:00
2020-07-04 13:30:27,886 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07755 Port Moody from 2020-07-02 23:55:00 to 2020-07-05 12:00:00
2020-07-04 13:30:44,181 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07795 Point Atkinson from 2020-07-04 00:00:00 to 2020-07-05 00:00:00
2020-07-04 13:30:47,216 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07735 Vancouver Harbour from 2020-07-03 23:55:00 to 2020-07-05 00:00:00
2020-07-04 13:31:05,513 INFO [make_fvcom_boundary] Stored VHFR FVCOM open boundary file: /nemoShare/MEOPAR/nowcast-sys/fvcom-runs/input.x2/bdy_x2_forecast_brcl_20200705.nc
2020-07-04 13:31:05,519 INFO [make_fvcom_boundary] FVCOM x2 forecast run boundary condition file for 2020-07-04 created on arbutus.cloud-nowcast
2020-07-04 13:31:05,524 INFO [manager] checklist updated with [FVCOM boundary conditions] items from make_fvcom_boundary worker
2020-07-04 13:31:06,191 INFO [manager] launching NextWorker(module='nowcast.workers.make_fvcom_atmos_forcing', args=['x2', 'forecast', '--run-date', '2020-07-04'], host='localhost')
2020-07-04 13:31:06,194 INFO [manager] launching NextWorker(module='nowcast.workers.make_fvcom_rivers_forcing', args=['arbutus.cloud-nowcast', 'x2', 'forecast', '--run-date', '2020-07-04'], host='arbutus.cloud-nowcast')
2020-07-04 13:31:06,489 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07735 Vancouver Harbour from 2020-07-04 00:00:00 to 2020-07-05 00:00:00
2020-07-04 13:31:06,767 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07755 Port Moody from 2020-07-03 00:00:00 to 2020-07-05 12:00:00
2020-07-04 13:31:09,440 INFO [make_fvcom_rivers_forcing] running in process 7237
2020-07-04 13:31:09,441 INFO [make_fvcom_rivers_forcing] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 13:31:09,442 INFO [make_fvcom_rivers_forcing] publishing log messages to tcp://*:5582
2020-07-04 13:31:09,443 INFO [make_fvcom_rivers_forcing] connected to skookum.eos.ubc.ca port 5555
2020-07-04 13:31:09,444 INFO [make_fvcom_rivers_forcing] Creating VHFR FVCOM rivers forcing file for x2 forecast run from 2020-07-04 NEMO runoff forcing files
2020-07-04 13:31:09,484 INFO [make_fvcom_atmos_forcing] running in process 24781
2020-07-04 13:31:09,486 INFO [make_fvcom_atmos_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 13:31:09,486 INFO [make_fvcom_atmos_forcing] publishing log messages to tcp://*:5569
2020-07-04 13:31:09,487 INFO [make_fvcom_atmos_forcing] connected to skookum.eos.ubc.ca port 5555
2020-07-04 13:31:09,488 INFO [make_fvcom_atmos_forcing] Creating VHFR FVCOM atmospheric forcing file for x2 forecast run from 2020-07-04 HRDPS GRIB files
2020-07-04 13:31:09,632 INFO [make_fvcom_rivers_forcing] Stored VHFR FVCOM rivers forcing file: /nemoShare/MEOPAR/nowcast-sys/fvcom-runs/input.x2/rivers_x2_forecast_20200705.nc
2020-07-04 13:31:09,633 INFO [make_fvcom_rivers_forcing] FVCOM x2 forecast run rivers forcing file for 2020-07-04 created on arbutus.cloud-nowcast
2020-07-04 13:31:09,635 INFO [manager] checklist updated with [FVCOM rivers forcing] items from make_fvcom_rivers_forcing worker
2020-07-04 13:31:09,987 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07610 Woodwards Landing from 2020-07-03 23:55:00 to 2020-07-05 00:00:00
2020-07-04 13:31:11,746 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07594 Sand Heads from 2020-07-02 23:55:00 to 2020-07-05 12:00:00
2020-07-04 13:31:27,159 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07610 Woodwards Landing from 2020-07-04 00:00:00 to 2020-07-05 00:00:00
2020-07-04 13:31:31,693 INFO [make_plots] finished rendering fvcom nowcast-x2 publish 04jul20 figures
2020-07-04 13:31:32,219 INFO [make_plots] fvcom publish plots for 2020-07-04 nowcast-x2 completed
2020-07-04 13:31:32,227 INFO [manager] checklist updated with [plots] items from make_plots worker
2020-07-04 13:31:42,481 INFO [make_fvcom_atmos_forcing] Stored VHFR FVCOM atmospheric forcing hfx file: /results/forcing/atmospheric/GEM2.5/vhfr-fvcom/atmos_x2_forecast_hfx_20200705.nc
2020-07-04 13:31:50,817 INFO [make_fvcom_atmos_forcing] Stored VHFR FVCOM atmospheric forcing precip file: /results/forcing/atmospheric/GEM2.5/vhfr-fvcom/atmos_x2_forecast_precip_20200705.nc
2020-07-04 13:31:51,964 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07594 Sand Heads from 2020-07-03 00:00:00 to 2020-07-05 12:00:00
2020-07-04 13:31:56,700 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07786 Sandy Cove from 2020-07-02 23:55:00 to 2020-07-05 12:00:00
2020-07-04 13:32:04,474 INFO [make_fvcom_atmos_forcing] Stored VHFR FVCOM atmospheric forcing wnd file: /results/forcing/atmospheric/GEM2.5/vhfr-fvcom/atmos_x2_forecast_wnd_20200705.nc
2020-07-04 13:32:04,475 INFO [make_fvcom_atmos_forcing] FVCOM x2 forecast run atmospheric forcing file for 2020-07-04 created
2020-07-04 13:32:04,480 INFO [manager] checklist updated with [FVCOM atmospheric forcing] items from make_fvcom_atmos_forcing worker
2020-07-04 13:32:05,049 INFO [manager] launching NextWorker(module='nowcast.workers.upload_fvcom_atmos_forcing', args=['arbutus.cloud-nowcast', 'x2', 'forecast', '--run-date', '2020-07-04'], host='localhost')
2020-07-04 13:32:05,937 INFO [upload_fvcom_atmos_forcing] running in process 24838
2020-07-04 13:32:05,937 INFO [upload_fvcom_atmos_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 13:32:05,938 INFO [upload_fvcom_atmos_forcing] publishing log messages to tcp://*:5567
2020-07-04 13:32:05,939 INFO [upload_fvcom_atmos_forcing] connected to skookum.eos.ubc.ca port 5555
2020-07-04 13:32:05,939 INFO [upload_fvcom_atmos_forcing] Uploading VHFR FVCOM atmospheric forcing files for 2020-07-04 x2 forecast run to arbutus.cloud-nowcast
2020-07-04 13:32:10,623 INFO [upload_fvcom_atmos_forcing] FVCOM x2 forecast run atmospheric forcing files for 2020-07-04 uploaded to arbutus.cloud-nowcast
2020-07-04 13:32:10,627 INFO [manager] checklist updated with [FVCOM atmospheric forcing upload] items from upload_fvcom_atmos_forcing worker
2020-07-04 13:32:11,242 INFO [manager] launching NextWorker(module='nowcast.workers.run_fvcom', args=['arbutus.cloud-nowcast', 'x2', 'forecast', '--run-date', '2020-07-04'], host='arbutus.cloud-nowcast')
2020-07-04 13:32:13,254 INFO [run_fvcom] running in process 7675
2020-07-04 13:32:13,259 INFO [run_fvcom] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 13:32:13,260 INFO [run_fvcom] publishing log messages to tcp://*:5584
2020-07-04 13:32:13,261 INFO [run_fvcom] connected to skookum.eos.ubc.ca port 5555
2020-07-04 13:32:13,299 INFO [run_fvcom] forecast: launching /nemoShare/MEOPAR/nowcast-sys/fvcom-runs/20200704-20h32m13.290593s/VHFR_FVCOM.sh on arbutus.cloud-nowcast
2020-07-04 13:32:13,316 INFO [run_fvcom] x2 forecast FVCOM VH-FR run for 2020-07-04 on arbutus.cloud-nowcast started
2020-07-04 13:32:13,322 INFO [manager] checklist updated with [FVCOM run] items from run_fvcom worker
2020-07-04 13:32:14,255 INFO [manager] launching NextWorker(module='nowcast.workers.watch_fvcom', args=['arbutus.cloud-nowcast', 'x2', 'forecast'], host='arbutus.cloud-nowcast')
2020-07-04 13:32:16,145 INFO [watch_fvcom] running in process 7777
2020-07-04 13:32:16,146 INFO [watch_fvcom] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 13:32:16,146 INFO [watch_fvcom] publishing log messages to tcp://*:5586
2020-07-04 13:32:16,147 INFO [watch_fvcom] connected to skookum.eos.ubc.ca port 5555
2020-07-04 13:32:16,184 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: no run progress found in fvcom.log ; continuing to watch...
2020-07-04 13:32:37,146 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07795 Point Atkinson from 2020-07-03 00:00:00 to 2020-07-05 12:00:00
2020-07-04 13:32:41,982 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07735 Vancouver Harbour from 2020-07-02 23:55:00 to 2020-07-05 12:00:00
2020-07-04 13:33:05,788 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39593321 = 2020-07-04 06:45:28 UTC estimated time to finish: 0000:09:57:50
2020-07-04 13:33:22,844 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07735 Vancouver Harbour from 2020-07-03 00:00:00 to 2020-07-05 12:00:00
2020-07-04 13:33:27,788 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07610 Woodwards Landing from 2020-07-02 23:55:00 to 2020-07-05 12:00:00
2020-07-04 13:34:05,188 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07610 Woodwards Landing from 2020-07-03 00:00:00 to 2020-07-05 12:00:00
2020-07-04 13:34:10,747 INFO [make_plots] finished rendering fvcom forecast-x2 publish 03jul20 figures
2020-07-04 13:34:11,198 INFO [make_plots] fvcom publish plots for 2020-07-03 forecast-x2 completed
2020-07-04 13:34:11,206 INFO [manager] checklist updated with [plots] items from make_plots worker
2020-07-04 13:35:13,228 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4628049 = 2020-07-04 14:45:20 UTC, 61.5% complete
2020-07-04 13:37:16,285 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15901212 = 2020-07-05 00:30:12 UTC estimated time to finish: 0000:05:43:53
2020-07-04 13:38:05,906 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39594701 = 2020-07-04 06:54:40 UTC estimated time to finish: 0000:09:51:59
2020-07-04 13:40:13,329 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4628104 = 2020-07-04 15:22:00 UTC, 64.1% complete
2020-07-04 13:42:16,387 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15902957 = 2020-07-05 00:59:17 UTC estimated time to finish: 0000:05:36:54
2020-07-04 13:43:06,025 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39596080 = 2020-07-04 07:03:52 UTC estimated time to finish: 0000:09:46:05
2020-07-04 13:45:13,429 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4628158 = 2020-07-04 15:58:00 UTC, 66.6% complete
2020-07-04 13:47:16,489 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15904920 = 2020-07-05 01:32:00 UTC estimated time to finish: 0000:05:30:06
2020-07-04 13:48:06,144 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39597500 = 2020-07-04 07:13:20 UTC estimated time to finish: 0000:09:39:54
2020-07-04 13:50:13,489 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4628212 = 2020-07-04 16:34:00 UTC, 69.1% complete
2020-07-04 13:52:16,593 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15906814 = 2020-07-05 02:03:34 UTC estimated time to finish: 0000:05:25:30
2020-07-04 13:53:06,265 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39598892 = 2020-07-04 07:22:36 UTC estimated time to finish: 0000:09:33:56
2020-07-04 13:53:30,971 INFO [make_plots] finished rendering fvcom nowcast-x2 research 04jul20 figures
2020-07-04 13:53:30,971 INFO [make_plots] fvcom research plots for 2020-07-04 nowcast-x2 completed
2020-07-04 13:53:31,079 INFO [manager] checklist updated with [plots] items from make_plots worker
2020-07-04 13:55:13,588 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4628267 = 2020-07-04 17:10:40 UTC, 71.6% complete
2020-07-04 13:57:16,697 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15908778 = 2020-07-05 02:36:18 UTC estimated time to finish: 0000:05:19:36
2020-07-04 13:58:06,385 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39600190 = 2020-07-04 07:31:16 UTC estimated time to finish: 0000:09:28:18
2020-07-04 14:00:13,689 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4628321 = 2020-07-04 17:46:40 UTC, 74.1% complete
2020-07-04 14:02:16,801 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15910440 = 2020-07-05 03:04:00 UTC estimated time to finish: 0000:05:16:12
2020-07-04 14:03:06,506 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39601570 = 2020-07-04 07:40:28 UTC estimated time to finish: 0000:09:22:44
2020-07-04 14:05:13,713 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4628375 = 2020-07-04 18:22:40 UTC, 76.6% complete
2020-07-04 14:07:16,907 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15912362 = 2020-07-05 03:36:02 UTC estimated time to finish: 0000:05:11:23
2020-07-04 14:08:06,628 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39602950 = 2020-07-04 07:49:40 UTC estimated time to finish: 0000:09:17:00
2020-07-04 14:10:13,812 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4628429 = 2020-07-04 18:58:40 UTC, 79.1% complete
2020-07-04 14:12:17,014 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15914298 = 2020-07-05 04:08:18 UTC estimated time to finish: 0000:05:06:00
2020-07-04 14:13:06,750 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39604370 = 2020-07-04 07:59:08 UTC estimated time to finish: 0000:09:11:02
2020-07-04 14:15:13,912 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4628483 = 2020-07-04 19:34:40 UTC, 81.6% complete
2020-07-04 14:17:17,120 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15916192 = 2020-07-05 04:39:52 UTC estimated time to finish: 0000:05:01:16
2020-07-04 14:18:06,872 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39605598 = 2020-07-04 08:07:19 UTC estimated time to finish: 0000:09:06:05
2020-07-04 14:20:14,012 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4628538 = 2020-07-04 20:11:20 UTC, 84.2% complete
2020-07-04 14:22:17,227 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15917923 = 2020-07-05 05:08:43 UTC estimated time to finish: 0000:04:56:39
2020-07-04 14:23:06,995 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39606977 = 2020-07-04 08:16:30 UTC estimated time to finish: 0000:09:00:24
2020-07-04 14:25:14,112 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4628592 = 2020-07-04 20:47:20 UTC, 86.7% complete
2020-07-04 14:27:17,335 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15919817 = 2020-07-05 05:40:17 UTC estimated time to finish: 0000:04:51:37
2020-07-04 14:28:07,118 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39608398 = 2020-07-04 08:25:59 UTC estimated time to finish: 0000:08:54:33
2020-07-04 14:30:14,192 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4628647 = 2020-07-04 21:24:00 UTC, 89.2% complete
2020-07-04 14:32:17,443 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15921740 = 2020-07-05 06:12:20 UTC estimated time to finish: 0000:04:46:31
2020-07-04 14:33:07,241 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39609819 = 2020-07-04 08:35:27 UTC estimated time to finish: 0000:08:48:36
2020-07-04 14:35:14,293 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4628701 = 2020-07-04 22:00:00 UTC, 91.7% complete
2020-07-04 14:37:17,551 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15923676 = 2020-07-05 06:44:36 UTC estimated time to finish: 0000:04:41:11
2020-07-04 14:38:07,364 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39611280 = 2020-07-04 08:45:12 UTC estimated time to finish: 0000:08:41:15
2020-07-04 14:40:14,391 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4628756 = 2020-07-04 22:36:40 UTC, 94.3% complete
2020-07-04 14:42:17,661 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15925407 = 2020-07-05 07:13:27 UTC estimated time to finish: 0000:04:36:24
2020-07-04 14:43:07,480 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39612973 = 2020-07-04 08:56:29 UTC estimated time to finish: 0000:08:32:41
2020-07-04 14:45:14,491 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4628810 = 2020-07-04 23:12:40 UTC, 96.8% complete
2020-07-04 14:47:17,770 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15927342 = 2020-07-05 07:45:42 UTC estimated time to finish: 0000:04:31:08
2020-07-04 14:48:07,604 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39614598 = 2020-07-04 09:07:19 UTC estimated time to finish: 0000:08:24:49
2020-07-04 14:50:14,592 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4628864 = 2020-07-04 23:48:40 UTC, 99.3% complete
2020-07-04 14:52:17,881 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15929237 = 2020-07-05 08:17:17 UTC estimated time to finish: 0000:04:26:15
2020-07-04 14:53:07,730 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39616210 = 2020-07-04 09:18:04 UTC estimated time to finish: 0000:08:17:17
2020-07-04 14:55:14,702 INFO [watch_NEMO] nowcast-dev NEMO run on salish-nowcast completed
2020-07-04 14:55:14,708 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2020-07-04 14:57:17,992 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15931200 = 2020-07-05 08:50:00 UTC estimated time to finish: 0000:04:20:53
2020-07-04 14:58:07,855 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39617835 = 2020-07-04 09:28:54 UTC estimated time to finish: 0000:08:09:41
2020-07-04 15:02:18,104 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15933095 = 2020-07-05 09:21:35 UTC estimated time to finish: 0000:04:15:51
2020-07-04 15:03:07,980 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39619256 = 2020-07-04 09:38:22 UTC estimated time to finish: 0000:08:03:14
2020-07-04 15:07:01,755 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-west/18/ to /results/forcing/atmospheric/GEM2.5/GRIB/20200704/18/
2020-07-04 15:07:01,756 INFO [collect_weather] 2020-07-04 2.5km weather forecast 18 collection complete
2020-07-04 15:07:01,760 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2020-07-04 15:07:02,700 INFO [manager] launching NextWorker(module='nowcast.workers.download_weather', args=['00', '1km', '--no-verify-certs'], host='localhost')
2020-07-04 15:07:02,705 INFO [manager] launching NextWorker(module='nowcast.workers.download_weather', args=['12', '1km', '--no-verify-certs'], host='localhost')
2020-07-04 15:07:02,709 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['00', '2.5km'], host='localhost')
2020-07-04 15:07:03,895 INFO [download_weather] running in process 27091
2020-07-04 15:07:03,895 INFO [download_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 15:07:03,896 INFO [download_weather] publishing log messages to tcp://*:5562
2020-07-04 15:07:03,897 INFO [download_weather] connected to skookum.eos.ubc.ca port 5555
2020-07-04 15:07:03,898 INFO [download_weather] downloading 12 1 km forecast GRIB2 files for 20200704
2020-07-04 15:07:03,900 INFO [download_weather] running in process 27090
2020-07-04 15:07:03,901 INFO [download_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 15:07:03,902 INFO [download_weather] publishing log messages to tcp://*:5564
2020-07-04 15:07:03,903 INFO [download_weather] connected to skookum.eos.ubc.ca port 5555
2020-07-04 15:07:03,904 INFO [download_weather] downloading 00 1 km forecast GRIB2 files for 20200704
2020-07-04 15:07:04,081 INFO [collect_weather] running in process 27092
2020-07-04 15:07:04,082 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 15:07:04,083 INFO [collect_weather] publishing log messages to tcp://*:5567
2020-07-04 15:07:04,084 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2020-07-04 15:07:04,137 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-west/00/
2020-07-04 15:07:18,216 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15934826 = 2020-07-05 09:50:26 UTC estimated time to finish: 0000:04:11:15
2020-07-04 15:08:08,107 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39620909 = 2020-07-04 09:49:23 UTC estimated time to finish: 0000:07:55:28
2020-07-04 15:12:18,329 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15936720 = 2020-07-05 10:22:00 UTC estimated time to finish: 0000:04:06:13
2020-07-04 15:13:08,233 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39622573 = 2020-07-04 10:00:29 UTC estimated time to finish: 0000:07:47:44
2020-07-04 15:17:18,443 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15938685 = 2020-07-05 10:54:45 UTC estimated time to finish: 0000:04:00:55
2020-07-04 15:18:08,360 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39624034 = 2020-07-04 10:10:13 UTC estimated time to finish: 0000:07:41:52
2020-07-04 15:22:18,557 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15940579 = 2020-07-05 11:26:19 UTC estimated time to finish: 0000:03:55:59
2020-07-04 15:23:08,487 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39625332 = 2020-07-04 10:18:52 UTC estimated time to finish: 0000:07:36:54
2020-07-04 15:25:01,422 INFO [download_weather] 2020-07-04 1km weather forecast 00 downloads complete
2020-07-04 15:25:01,427 INFO [manager] checklist updated with [weather forecast] items from download_weather worker
2020-07-04 15:25:24,085 INFO [download_weather] 2020-07-04 1km weather forecast 12 downloads complete
2020-07-04 15:25:24,090 INFO [manager] checklist updated with [weather forecast] items from download_weather worker
2020-07-04 15:27:18,672 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15942310 = 2020-07-05 11:55:10 UTC estimated time to finish: 0000:03:51:20
2020-07-04 15:28:08,616 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39626753 = 2020-07-04 10:28:21 UTC estimated time to finish: 0000:07:31:20
2020-07-04 15:32:18,788 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15944205 = 2020-07-05 12:26:45 UTC estimated time to finish: 0000:03:46:23
2020-07-04 15:33:08,744 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39628174 = 2020-07-04 10:37:49 UTC estimated time to finish: 0000:07:25:50
2020-07-04 15:37:18,903 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15946100 = 2020-07-05 12:58:20 UTC estimated time to finish: 0000:03:41:22
2020-07-04 15:38:08,873 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39629635 = 2020-07-04 10:47:34 UTC estimated time to finish: 0000:07:20:12
2020-07-04 15:42:19,019 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15948023 = 2020-07-05 13:30:23 UTC estimated time to finish: 0000:03:36:16
2020-07-04 15:43:09,003 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39630904 = 2020-07-04 10:56:01 UTC estimated time to finish: 0000:07:15:18
2020-07-04 15:47:19,136 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15949768 = 2020-07-05 13:59:28 UTC estimated time to finish: 0000:03:31:33
2020-07-04 15:48:09,133 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39632324 = 2020-07-04 11:05:29 UTC estimated time to finish: 0000:07:10:00
2020-07-04 15:52:19,254 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15951649 = 2020-07-05 14:30:49 UTC estimated time to finish: 0000:03:26:33
2020-07-04 15:53:09,263 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39633745 = 2020-07-04 11:14:58 UTC estimated time to finish: 0000:07:04:28
2020-07-04 15:57:19,372 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15953584 = 2020-07-05 15:03:04 UTC estimated time to finish: 0000:03:21:23
2020-07-04 15:58:09,393 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39635165 = 2020-07-04 11:24:26 UTC estimated time to finish: 0000:06:59:05
2020-07-04 16:02:19,459 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15955479 = 2020-07-05 15:34:39 UTC estimated time to finish: 0000:03:16:20
2020-07-04 16:03:09,523 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39636546 = 2020-07-04 11:33:38 UTC estimated time to finish: 0000:06:53:59
2020-07-04 16:07:19,578 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15957400 = 2020-07-05 16:06:40 UTC estimated time to finish: 0000:03:11:11
2020-07-04 16:08:09,654 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39637843 = 2020-07-04 11:42:17 UTC estimated time to finish: 0000:06:49:06
2020-07-04 16:12:19,698 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15959144 = 2020-07-05 16:35:44 UTC estimated time to finish: 0000:03:06:33
2020-07-04 16:13:09,787 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39639235 = 2020-07-04 11:51:34 UTC estimated time to finish: 0000:06:43:50
2020-07-04 16:17:19,828 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15961067 = 2020-07-05 17:07:47 UTC estimated time to finish: 0000:03:01:21
2020-07-04 16:18:09,920 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39640614 = 2020-07-04 12:00:45 UTC estimated time to finish: 0000:06:38:42
2020-07-04 16:22:19,949 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15963002 = 2020-07-05 17:40:02 UTC estimated time to finish: 0000:02:56:09
2020-07-04 16:23:10,052 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39642185 = 2020-07-04 12:11:14 UTC estimated time to finish: 0000:06:32:22
2020-07-04 16:27:20,069 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15964966 = 2020-07-05 18:12:46 UTC estimated time to finish: 0000:02:50:51
2020-07-04 16:28:10,185 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39643646 = 2020-07-04 12:20:58 UTC estimated time to finish: 0000:06:26:12
2020-07-04 16:32:20,191 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15966670 = 2020-07-05 18:41:10 UTC estimated time to finish: 0000:02:46:19
2020-07-04 16:33:10,319 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39645231 = 2020-07-04 12:31:32 UTC estimated time to finish: 0000:06:19:40
2020-07-04 16:37:20,314 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15968592 = 2020-07-05 19:13:12 UTC estimated time to finish: 0000:02:41:07
2020-07-04 16:38:10,453 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39646884 = 2020-07-04 12:42:33 UTC estimated time to finish: 0000:06:12:43
2020-07-04 16:42:20,437 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15970447 = 2020-07-05 19:44:07 UTC estimated time to finish: 0000:02:36:17
2020-07-04 16:43:10,542 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39648427 = 2020-07-04 12:52:50 UTC estimated time to finish: 0000:06:06:39
2020-07-04 16:47:20,561 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15972381 = 2020-07-05 20:16:21 UTC estimated time to finish: 0000:02:31:09
2020-07-04 16:48:10,676 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39649888 = 2020-07-04 13:02:35 UTC estimated time to finish: 0000:06:00:41
2020-07-04 16:52:20,684 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15974153 = 2020-07-05 20:45:53 UTC estimated time to finish: 0000:02:26:22
2020-07-04 16:53:10,811 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39651500 = 2020-07-04 13:13:20 UTC estimated time to finish: 0000:05:54:06
2020-07-04 16:57:20,809 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15976080 = 2020-07-05 21:18:00 UTC estimated time to finish: 0000:02:21:13
2020-07-04 16:58:10,947 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39653124 = 2020-07-04 13:24:09 UTC estimated time to finish: 0000:05:47:31
2020-07-04 17:02:20,934 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15977971 = 2020-07-05 21:49:31 UTC estimated time to finish: 0000:02:16:11
2020-07-04 17:03:11,085 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39654587 = 2020-07-04 13:33:54 UTC estimated time to finish: 0000:05:42:06
2020-07-04 17:07:21,059 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15979906 = 2020-07-05 22:21:46 UTC estimated time to finish: 0000:02:11:01
2020-07-04 17:08:11,223 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39656007 = 2020-07-04 13:43:22 UTC estimated time to finish: 0000:05:36:50
2020-07-04 17:12:21,184 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15981840 = 2020-07-05 22:54:00 UTC estimated time to finish: 0000:02:05:50
2020-07-04 17:13:11,361 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39657304 = 2020-07-04 13:52:01 UTC estimated time to finish: 0000:05:32:04
2020-07-04 17:17:21,310 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15983573 = 2020-07-05 23:22:53 UTC estimated time to finish: 0000:02:01:11
2020-07-04 17:18:11,494 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39658724 = 2020-07-04 14:01:29 UTC estimated time to finish: 0000:05:26:51
2020-07-04 17:22:21,438 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15985537 = 2020-07-05 23:55:37 UTC estimated time to finish: 0000:01:55:53
2020-07-04 17:23:11,629 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39660144 = 2020-07-04 14:10:57 UTC estimated time to finish: 0000:05:21:37
2020-07-04 17:27:21,564 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15987471 = 2020-07-06 00:27:51 UTC estimated time to finish: 0000:01:50:42
2020-07-04 17:28:11,768 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39661564 = 2020-07-04 14:20:25 UTC estimated time to finish: 0000:05:16:24
2020-07-04 17:32:21,694 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15989367 = 2020-07-06 00:59:27 UTC estimated time to finish: 0000:01:45:40
2020-07-04 17:33:11,906 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39662835 = 2020-07-04 14:28:54 UTC estimated time to finish: 0000:05:11:46
2020-07-04 17:37:21,822 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15991098 = 2020-07-06 01:28:18 UTC estimated time to finish: 0000:01:41:03
2020-07-04 17:38:12,046 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39664256 = 2020-07-04 14:38:22 UTC estimated time to finish: 0000:05:06:32
2020-07-04 17:42:21,952 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15993020 = 2020-07-06 02:00:20 UTC estimated time to finish: 0000:01:35:57
2020-07-04 17:43:12,185 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39665676 = 2020-07-04 14:47:50 UTC estimated time to finish: 0000:05:01:21
2020-07-04 17:47:22,082 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15994915 = 2020-07-06 02:31:55 UTC estimated time to finish: 0000:01:30:55
2020-07-04 17:48:12,326 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39667096 = 2020-07-04 14:57:18 UTC estimated time to finish: 0000:04:56:13
2020-07-04 17:52:22,215 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15996850 = 2020-07-06 03:04:10 UTC estimated time to finish: 0000:01:25:45
2020-07-04 17:53:12,428 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39668516 = 2020-07-04 15:06:46 UTC estimated time to finish: 0000:04:51:00
2020-07-04 17:57:22,346 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15998581 = 2020-07-06 03:33:01 UTC estimated time to finish: 0000:01:21:07
2020-07-04 17:58:12,568 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39669826 = 2020-07-04 15:15:30 UTC estimated time to finish: 0000:04:46:10
2020-07-04 18:02:22,478 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 16000476 = 2020-07-06 04:04:36 UTC estimated time to finish: 0000:01:16:05
2020-07-04 18:03:12,710 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39671206 = 2020-07-04 15:24:42 UTC estimated time to finish: 0000:04:41:12
2020-07-04 18:07:22,609 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 16002360 = 2020-07-06 04:36:00 UTC estimated time to finish: 0000:01:11:05
2020-07-04 18:08:12,851 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39672587 = 2020-07-04 15:33:54 UTC estimated time to finish: 0000:04:36:11
2020-07-04 18:12:22,742 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 16004293 = 2020-07-06 05:08:13 UTC estimated time to finish: 0000:01:05:55
2020-07-04 18:13:12,941 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39674048 = 2020-07-04 15:43:39 UTC estimated time to finish: 0000:04:30:48
2020-07-04 18:17:22,877 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 16006148 = 2020-07-06 05:39:08 UTC estimated time to finish: 0000:01:00:59
2020-07-04 18:18:13,086 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39675304 = 2020-07-04 15:52:01 UTC estimated time to finish: 0000:04:26:14
2020-07-04 18:22:23,009 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 16007879 = 2020-07-06 06:07:59 UTC estimated time to finish: 0000:00:56:22
2020-07-04 18:23:13,228 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39676683 = 2020-07-04 16:01:13 UTC estimated time to finish: 0000:04:21:16
2020-07-04 18:27:23,143 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 16009814 = 2020-07-06 06:40:14 UTC estimated time to finish: 0000:00:51:12
2020-07-04 18:28:13,370 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39678075 = 2020-07-04 16:10:30 UTC estimated time to finish: 0000:04:16:15
2020-07-04 18:32:23,279 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 16011668 = 2020-07-06 07:11:08 UTC estimated time to finish: 0000:00:46:16
2020-07-04 18:33:13,454 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39679455 = 2020-07-04 16:19:42 UTC estimated time to finish: 0000:04:11:17
2020-07-04 18:37:23,414 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 16013590 = 2020-07-06 07:43:10 UTC estimated time to finish: 0000:00:41:09
2020-07-04 18:38:13,597 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39680712 = 2020-07-04 16:28:04 UTC estimated time to finish: 0000:04:06:43
2020-07-04 18:42:23,475 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 16015321 = 2020-07-06 08:12:01 UTC estimated time to finish: 0000:00:36:31
2020-07-04 18:43:13,741 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39682092 = 2020-07-04 16:37:16 UTC estimated time to finish: 0000:04:01:42
2020-07-04 18:47:23,611 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 16017217 = 2020-07-06 08:43:37 UTC estimated time to finish: 0000:00:31:27
2020-07-04 18:48:13,885 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39683520 = 2020-07-04 16:46:48 UTC estimated time to finish: 0000:03:56:28
2020-07-04 18:52:23,749 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 16019151 = 2020-07-06 09:15:51 UTC estimated time to finish: 0000:00:26:18
2020-07-04 18:53:14,029 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39684904 = 2020-07-04 16:56:01 UTC estimated time to finish: 0000:03:51:28
2020-07-04 18:57:23,887 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 16021080 = 2020-07-06 09:48:00 UTC estimated time to finish: 0000:00:21:09
2020-07-04 18:58:14,079 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39686324 = 2020-07-04 17:05:29 UTC estimated time to finish: 0000:03:46:19
2020-07-04 19:02:24,025 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 16022778 = 2020-07-06 10:16:18 UTC estimated time to finish: 0000:00:16:37
2020-07-04 19:03:14,224 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39687540 = 2020-07-04 17:13:36 UTC estimated time to finish: 0000:03:41:56
2020-07-04 19:07:24,164 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 16024632 = 2020-07-06 10:47:12 UTC estimated time to finish: 0000:00:11:40
2020-07-04 19:08:14,368 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39688960 = 2020-07-04 17:23:04 UTC estimated time to finish: 0000:03:36:47
2020-07-04 19:12:24,303 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 16026555 = 2020-07-06 11:19:15 UTC estimated time to finish: 0000:00:06:31
2020-07-04 19:13:14,515 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39690354 = 2020-07-04 17:32:21 UTC estimated time to finish: 0000:03:31:42
2020-07-04 19:17:24,443 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 16028490 = 2020-07-06 11:51:30 UTC estimated time to finish: 0000:00:01:21
2020-07-04 19:18:14,659 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39691774 = 2020-07-04 17:41:49 UTC estimated time to finish: 0000:03:26:32
2020-07-04 19:22:24,553 INFO [watch_fvcom] x2 forecast FVCOM VH-FR run on arbutus.cloud-nowcast completed
2020-07-04 19:22:24,559 INFO [manager] checklist updated with [FVCOM run] items from watch_fvcom worker
2020-07-04 19:22:25,504 INFO [manager] launching NextWorker(module='nowcast.workers.download_fvcom_results', args=['arbutus.cloud-nowcast', 'x2', 'forecast', '--run-date', '2020-07-04'], host='localhost')
2020-07-04 19:22:27,084 INFO [download_fvcom_results] running in process 32393
2020-07-04 19:22:27,085 INFO [download_fvcom_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 19:22:27,086 INFO [download_fvcom_results] publishing log messages to tcp://*:5562
2020-07-04 19:22:27,087 INFO [download_fvcom_results] connected to skookum.eos.ubc.ca port 5555
2020-07-04 19:23:14,806 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39693863 = 2020-07-04 17:55:45 UTC estimated time to finish: 0000:03:17:53
2020-07-04 19:23:48,185 INFO [download_fvcom_results] VHFR FVCOM x2 forecast 2020-07-04 results files from arbutus.cloud-nowcast downloaded
2020-07-04 19:23:48,193 INFO [manager] checklist updated with [VHFR FVCOM results files] items from download_fvcom_results worker
2020-07-04 19:23:48,882 INFO [manager] launching NextWorker(module='nowcast.workers.get_vfpa_hadcp', args=['--data-date', '2020-07-04'], host='localhost')
2020-07-04 19:23:48,886 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['fvcom', 'forecast-x2', 'research', '--run-date', '2020-07-04'], host='localhost')
2020-07-04 19:23:48,889 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['fvcom', 'forecast', '--run-date', '2020-07-04'], host='localhost')
2020-07-04 19:23:50,871 INFO [update_forecast_datasets] running in process 32435
2020-07-04 19:23:50,874 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 19:23:50,875 INFO [update_forecast_datasets] publishing log messages to tcp://*:5562
2020-07-04 19:23:50,876 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2020-07-04 19:23:50,877 INFO [update_forecast_datasets] updating fvcom most_recent_forecast directory from forecast/04jul20 run
2020-07-04 19:23:50,879 INFO [update_forecast_datasets] updated fvcom most_recent_forecast directory from forecast/04jul20 run
2020-07-04 19:23:50,880 INFO [update_forecast_datasets] fvcom 2020-07-04 forecast rolling forecast datasets updated
2020-07-04 19:23:50,882 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2020-07-04 19:23:51,541 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['fvcom-forecast'], host='localhost')
2020-07-04 19:23:54,424 INFO [get_vfpa_hadcp] running in process 32433
2020-07-04 19:23:54,426 INFO [get_vfpa_hadcp] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 19:23:54,427 INFO [get_vfpa_hadcp] publishing log messages to tcp://*:5562
2020-07-04 19:23:54,428 INFO [get_vfpa_hadcp] connected to skookum.eos.ubc.ca port 5555
2020-07-04 19:23:54,430 INFO [get_vfpa_hadcp] processing VFPA HADCP data from 2nd Narrows Rail Bridge for 2020-07-04
2020-07-04 19:23:56,951 INFO [get_vfpa_hadcp] added VFPA HADCP data from 2nd Narrows Rail Bridge for 2020-07-04 to /opp/observations/AISDATA/netcdf/VFPA_2ND_NARROWS_HADCP_2s_202007.nc
2020-07-04 19:23:56,952 INFO [get_vfpa_hadcp] VFPA HADCP observations added to 2020-07 netcdf file
2020-07-04 19:23:56,955 INFO [manager] checklist updated with [VFPA HADCP data] items from get_vfpa_hadcp worker
2020-07-04 19:23:57,556 INFO [make_plots] running in process 32434
2020-07-04 19:23:57,558 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 19:23:57,558 INFO [make_plots] publishing log messages to tcp://*:5564
2020-07-04 19:23:57,559 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2020-07-04 19:23:57,620 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['VFPA-HADCP'], host='localhost')
2020-07-04 19:23:58,526 INFO [ping_erddap] running in process 32452
2020-07-04 19:23:58,527 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 19:23:58,528 INFO [ping_erddap] publishing log messages to tcp://*:5562
2020-07-04 19:23:58,530 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2020-07-04 19:23:58,532 INFO [ping_erddap] VFPA-HADCP ERDDAP dataset flag file(s) created
2020-07-04 19:23:58,534 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2020-07-04 19:24:00,069 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['fvcom', 'nowcast-x2', 'publish', '--run-date', '2020-07-04'], host='localhost')
2020-07-04 19:24:00,906 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['fvcom', 'forecast-x2', 'publish', '--run-date', '2020-07-04'], host='localhost')
2020-07-04 19:24:12,076 INFO [make_plots] preparing render list for 2020-07-04 VHFR FVCOM forecast-x2 research figures
2020-07-04 19:24:12,085 INFO [make_plots] added EnglishBay_surface_currents figures to 2020-07-04 VHFR FVCOM forecast-x2 research render list
2020-07-04 19:24:12,091 INFO [make_plots] added VancouverHarbour_surface_currents figures to 2020-07-04 VHFR FVCOM forecast-x2 research render list
2020-07-04 19:24:12,097 INFO [make_plots] added IndianArm_surface_currents figures to 2020-07-04 VHFR FVCOM forecast-x2 research render list
2020-07-04 19:24:12,103 INFO [make_plots] added VancouverHarbour_thalweg figures to 2020-07-04 VHFR FVCOM forecast-x2 research render list
2020-07-04 19:24:12,110 INFO [make_plots] added PortMoody_thalweg figures to 2020-07-04 VHFR FVCOM forecast-x2 research render list
2020-07-04 19:24:12,116 INFO [make_plots] added IndianArm_thalweg figures to 2020-07-04 VHFR FVCOM forecast-x2 research render list
2020-07-04 19:24:12,117 INFO [make_plots] starting to render fvcom forecast-x2 research 04jul20 figures
2020-07-04 19:24:14,100 INFO [make_plots] running in process 32456
2020-07-04 19:24:14,101 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 19:24:14,102 INFO [make_plots] publishing log messages to tcp://*:5562
2020-07-04 19:24:14,104 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2020-07-04 19:24:14,107 INFO [make_plots] running in process 32457
2020-07-04 19:24:14,108 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 19:24:14,109 INFO [make_plots] publishing log messages to tcp://*:5568
2020-07-04 19:24:14,110 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2020-07-04 19:24:14,963 INFO [make_plots] preparing render list for 2020-07-04 VHFR FVCOM nowcast-x2 publish figures
2020-07-04 19:24:14,965 INFO [make_plots] added Calamity Point figure to 2020-07-04 VHFR FVCOM nowcast-x2 publish render list
2020-07-04 19:24:14,965 INFO [make_plots] added Indian Arm Head figure to 2020-07-04 VHFR FVCOM nowcast-x2 publish render list
2020-07-04 19:24:14,966 INFO [make_plots] added New Westminster figure to 2020-07-04 VHFR FVCOM nowcast-x2 publish render list
2020-07-04 19:24:14,967 INFO [make_plots] added Port Moody figure to 2020-07-04 VHFR FVCOM nowcast-x2 publish render list
2020-07-04 19:24:14,969 INFO [make_plots] added Sand Heads figure to 2020-07-04 VHFR FVCOM nowcast-x2 publish render list
2020-07-04 19:24:14,969 INFO [make_plots] added Sandy Cove figure to 2020-07-04 VHFR FVCOM nowcast-x2 publish render list
2020-07-04 19:24:14,970 INFO [make_plots] added Vancouver Harbour figure to 2020-07-04 VHFR FVCOM nowcast-x2 publish render list
2020-07-04 19:24:14,971 INFO [make_plots] added Woodwards Landing figure to 2020-07-04 VHFR FVCOM nowcast-x2 publish render list
2020-07-04 19:24:14,972 INFO [make_plots] added 2ndNarrows_current figure to 2020-07-04 VHFR FVCOM nowcast-x2 publish render list
2020-07-04 19:24:14,972 INFO [make_plots] starting to render fvcom nowcast-x2 publish 04jul20 figures
2020-07-04 19:24:14,988 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07724 Calamity Point from 2020-07-03 23:55:00 to 2020-07-05 00:00:00
2020-07-04 19:24:15,029 INFO [make_plots] preparing render list for 2020-07-04 VHFR FVCOM forecast-x2 publish figures
2020-07-04 19:24:15,030 INFO [make_plots] added Calamity Point figure to 2020-07-04 VHFR FVCOM forecast-x2 publish render list
2020-07-04 19:24:15,030 INFO [make_plots] added Indian Arm Head figure to 2020-07-04 VHFR FVCOM forecast-x2 publish render list
2020-07-04 19:24:15,031 INFO [make_plots] added New Westminster figure to 2020-07-04 VHFR FVCOM forecast-x2 publish render list
2020-07-04 19:24:15,032 INFO [make_plots] added Port Moody figure to 2020-07-04 VHFR FVCOM forecast-x2 publish render list
2020-07-04 19:24:15,033 INFO [make_plots] added Sand Heads figure to 2020-07-04 VHFR FVCOM forecast-x2 publish render list
2020-07-04 19:24:15,033 INFO [make_plots] added Sandy Cove figure to 2020-07-04 VHFR FVCOM forecast-x2 publish render list
2020-07-04 19:24:15,034 INFO [make_plots] added Vancouver Harbour figure to 2020-07-04 VHFR FVCOM forecast-x2 publish render list
2020-07-04 19:24:15,035 INFO [make_plots] added Woodwards Landing figure to 2020-07-04 VHFR FVCOM forecast-x2 publish render list
2020-07-04 19:24:15,036 INFO [make_plots] added 2ndNarrows_current figure to 2020-07-04 VHFR FVCOM forecast-x2 publish render list
2020-07-04 19:24:15,037 INFO [make_plots] starting to render fvcom forecast-x2 publish 04jul20 figures
2020-07-04 19:24:15,050 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07724 Calamity Point from 2020-07-03 23:55:00 to 2020-07-06 12:00:00
2020-07-04 19:24:38,312 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07724 Calamity Point from 2020-07-04 00:00:00 to 2020-07-05 00:00:00
2020-07-04 19:24:39,794 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07724 Calamity Point from 2020-07-04 00:00:00 to 2020-07-06 12:00:00
2020-07-04 19:24:40,090 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07774 Indian Arm Head from 2020-07-03 23:55:00 to 2020-07-05 00:00:00
2020-07-04 19:24:40,875 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07774 Indian Arm Head from 2020-07-04 00:00:00 to 2020-07-05 00:00:00
2020-07-04 19:24:41,475 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07774 Indian Arm Head from 2020-07-03 23:55:00 to 2020-07-06 12:00:00
2020-07-04 19:24:42,019 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07774 Indian Arm Head from 2020-07-04 00:00:00 to 2020-07-06 12:00:00
2020-07-04 19:24:42,404 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07654 New Westminster from 2020-07-03 23:55:00 to 2020-07-05 00:00:00
2020-07-04 19:24:43,410 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07654 New Westminster from 2020-07-03 23:55:00 to 2020-07-06 12:00:00
2020-07-04 19:25:03,662 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07654 New Westminster from 2020-07-04 00:00:00 to 2020-07-05 00:00:00
2020-07-04 19:25:06,666 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07755 Port Moody from 2020-07-03 23:55:00 to 2020-07-05 00:00:00
2020-07-04 19:25:07,391 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07654 New Westminster from 2020-07-04 00:00:00 to 2020-07-06 12:00:00
2020-07-04 19:25:12,163 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07755 Port Moody from 2020-07-03 23:55:00 to 2020-07-06 12:00:00
2020-07-04 19:25:28,807 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07755 Port Moody from 2020-07-04 00:00:00 to 2020-07-05 00:00:00
2020-07-04 19:25:31,927 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07594 Sand Heads from 2020-07-03 23:55:00 to 2020-07-05 00:00:00
2020-07-04 19:25:35,513 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07755 Port Moody from 2020-07-04 00:00:00 to 2020-07-06 12:00:00
2020-07-04 19:25:40,552 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07594 Sand Heads from 2020-07-03 23:55:00 to 2020-07-06 12:00:00
2020-07-04 19:25:53,781 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07594 Sand Heads from 2020-07-04 00:00:00 to 2020-07-05 00:00:00
2020-07-04 19:25:57,099 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07786 Sandy Cove from 2020-07-03 23:55:00 to 2020-07-05 00:00:00
2020-07-04 19:26:04,721 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07594 Sand Heads from 2020-07-04 00:00:00 to 2020-07-06 12:00:00
2020-07-04 19:26:09,425 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07786 Sandy Cove from 2020-07-03 23:55:00 to 2020-07-06 12:00:00
2020-07-04 19:26:18,888 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07795 Point Atkinson from 2020-07-04 00:00:00 to 2020-07-05 00:00:00
2020-07-04 19:26:22,012 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07735 Vancouver Harbour from 2020-07-03 23:55:00 to 2020-07-05 00:00:00
2020-07-04 19:26:33,270 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07795 Point Atkinson from 2020-07-04 00:00:00 to 2020-07-06 12:00:00
2020-07-04 19:26:38,262 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07735 Vancouver Harbour from 2020-07-03 23:55:00 to 2020-07-06 12:00:00
2020-07-04 19:26:43,234 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07735 Vancouver Harbour from 2020-07-04 00:00:00 to 2020-07-05 00:00:00
2020-07-04 19:26:46,208 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07610 Woodwards Landing from 2020-07-03 23:55:00 to 2020-07-05 00:00:00
2020-07-04 19:27:02,302 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07735 Vancouver Harbour from 2020-07-04 00:00:00 to 2020-07-06 12:00:00
2020-07-04 19:27:04,158 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07610 Woodwards Landing from 2020-07-04 00:00:00 to 2020-07-05 00:00:00
2020-07-04 19:27:06,888 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07610 Woodwards Landing from 2020-07-03 23:55:00 to 2020-07-06 12:00:00
2020-07-04 19:27:08,765 INFO [make_plots] finished rendering fvcom nowcast-x2 publish 04jul20 figures
2020-07-04 19:27:09,325 INFO [make_plots] fvcom publish plots for 2020-07-04 nowcast-x2 completed
2020-07-04 19:27:09,330 INFO [manager] checklist updated with [plots] items from make_plots worker
2020-07-04 19:27:26,727 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07610 Woodwards Landing from 2020-07-04 00:00:00 to 2020-07-06 12:00:00
2020-07-04 19:27:31,400 INFO [make_plots] finished rendering fvcom forecast-x2 publish 04jul20 figures
2020-07-04 19:27:31,950 INFO [make_plots] fvcom publish plots for 2020-07-04 forecast-x2 completed
2020-07-04 19:27:31,957 INFO [manager] checklist updated with [plots] items from make_plots worker
2020-07-04 19:28:14,953 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39696308 = 2020-07-04 18:12:03 UTC estimated time to finish: 0000:03:07:51
2020-07-04 19:33:15,102 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39698753 = 2020-07-04 18:28:21 UTC estimated time to finish: 0000:02:57:57
2020-07-04 19:38:15,205 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39701199 = 2020-07-04 18:44:39 UTC estimated time to finish: 0000:02:48:11
2020-07-04 19:43:15,355 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39703410 = 2020-07-04 18:59:24 UTC estimated time to finish: 0000:02:39:28
2020-07-04 19:48:15,509 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39705826 = 2020-07-04 19:15:30 UTC estimated time to finish: 0000:02:30:05
2020-07-04 19:53:15,660 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39708272 = 2020-07-04 19:31:48 UTC estimated time to finish: 0000:02:20:41
2020-07-04 19:58:15,815 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39710717 = 2020-07-04 19:48:06 UTC estimated time to finish: 0000:02:11:24
2020-07-04 20:03:15,968 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39713120 = 2020-07-04 20:04:08 UTC estimated time to finish: 0000:02:02:24
2020-07-04 20:08:16,126 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39715332 = 2020-07-04 20:18:52 UTC estimated time to finish: 0000:01:54:12
2020-07-04 20:13:16,282 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39717778 = 2020-07-04 20:35:11 UTC estimated time to finish: 0000:01:45:14
2020-07-04 20:18:16,439 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39720263 = 2020-07-04 20:51:45 UTC estimated time to finish: 0000:01:36:13
2020-07-04 20:23:16,595 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39722680 = 2020-07-04 21:07:52 UTC estimated time to finish: 0000:01:27:33
2020-07-04 20:25:31,468 INFO [make_plots] finished rendering fvcom forecast-x2 research 04jul20 figures
2020-07-04 20:25:31,469 INFO [make_plots] fvcom research plots for 2020-07-04 forecast-x2 completed
2020-07-04 20:25:31,675 INFO [manager] checklist updated with [plots] items from make_plots worker
2020-07-04 20:28:16,754 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39724891 = 2020-07-04 21:22:36 UTC estimated time to finish: 0000:01:19:42
2020-07-04 20:33:16,915 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39727338 = 2020-07-04 21:38:55 UTC estimated time to finish: 0000:01:11:06
2020-07-04 20:38:17,074 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39729781 = 2020-07-04 21:55:12 UTC estimated time to finish: 0000:01:02:36
2020-07-04 20:43:17,233 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39732185 = 2020-07-04 22:11:14 UTC estimated time to finish: 0000:00:54:19
2020-07-04 20:48:17,395 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39734398 = 2020-07-04 22:25:59 UTC estimated time to finish: 0000:00:46:45
2020-07-04 20:53:17,555 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39736843 = 2020-07-04 22:42:17 UTC estimated time to finish: 0000:00:38:28
2020-07-04 20:58:17,718 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39739300 = 2020-07-04 22:58:40 UTC estimated time to finish: 0000:00:30:14
2020-07-04 21:03:17,880 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39741745 = 2020-07-04 23:14:58 UTC estimated time to finish: 0000:00:22:06
2020-07-04 21:04:50,937 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-west/00/ to /results/forcing/atmospheric/GEM2.5/GRIB/20200705/00/
2020-07-04 21:04:50,938 INFO [collect_weather] 2020-07-05 2.5km weather forecast 00 collection complete
2020-07-04 21:04:50,942 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2020-07-04 21:04:51,896 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['06', '2.5km'], host='localhost')
2020-07-04 21:04:53,965 INFO [collect_weather] running in process 2337
2020-07-04 21:04:53,966 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 21:04:53,968 INFO [collect_weather] publishing log messages to tcp://*:5562
2020-07-04 21:04:53,970 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2020-07-04 21:04:53,988 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-west/06/
2020-07-04 21:08:18,044 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39744190 = 2020-07-04 23:31:16 UTC estimated time to finish: 0000:00:14:02
2020-07-04 21:13:18,207 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39746402 = 2020-07-04 23:46:00 UTC estimated time to finish: 0000:00:06:48
2020-07-04 21:18:18,321 INFO [watch_fvcom] r12 nowcast FVCOM VH-FR run on arbutus.cloud-nowcast completed
2020-07-04 21:18:18,327 INFO [manager] checklist updated with [FVCOM run] items from watch_fvcom worker
2020-07-04 21:18:19,300 INFO [manager] launching NextWorker(module='nowcast.workers.download_fvcom_results', args=['arbutus.cloud-nowcast', 'r12', 'nowcast', '--run-date', '2020-07-04'], host='localhost')
2020-07-04 21:18:20,628 INFO [download_fvcom_results] running in process 2625
2020-07-04 21:18:20,629 INFO [download_fvcom_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 21:18:20,630 INFO [download_fvcom_results] publishing log messages to tcp://*:5564
2020-07-04 21:18:20,631 INFO [download_fvcom_results] connected to skookum.eos.ubc.ca port 5555
2020-07-04 21:19:55,806 INFO [download_fvcom_results] VHFR FVCOM r12 nowcast 2020-07-04 results files from arbutus.cloud-nowcast downloaded
2020-07-04 21:19:55,812 INFO [manager] checklist updated with [VHFR FVCOM results files] items from download_fvcom_results worker
2020-07-04 21:19:56,500 INFO [manager] launching NextWorker(module='nowcast.workers.get_vfpa_hadcp', args=['--data-date', '2020-07-04'], host='localhost')
2020-07-04 21:19:56,503 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['fvcom', 'nowcast-r12', 'research', '--run-date', '2020-07-04'], host='localhost')
2020-07-04 21:20:01,261 INFO [get_vfpa_hadcp] running in process 2633
2020-07-04 21:20:01,262 INFO [get_vfpa_hadcp] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 21:20:01,263 INFO [get_vfpa_hadcp] publishing log messages to tcp://*:5564
2020-07-04 21:20:01,264 INFO [get_vfpa_hadcp] connected to skookum.eos.ubc.ca port 5555
2020-07-04 21:20:01,265 INFO [get_vfpa_hadcp] processing VFPA HADCP data from 2nd Narrows Rail Bridge for 2020-07-04
2020-07-04 21:20:03,880 INFO [get_vfpa_hadcp] added VFPA HADCP data from 2nd Narrows Rail Bridge for 2020-07-04 to /opp/observations/AISDATA/netcdf/VFPA_2ND_NARROWS_HADCP_2s_202007.nc
2020-07-04 21:20:03,881 INFO [get_vfpa_hadcp] VFPA HADCP observations added to 2020-07 netcdf file
2020-07-04 21:20:03,883 INFO [manager] checklist updated with [VFPA HADCP data] items from get_vfpa_hadcp worker
2020-07-04 21:20:04,341 INFO [make_plots] running in process 2634
2020-07-04 21:20:04,342 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 21:20:04,343 INFO [make_plots] publishing log messages to tcp://*:5567
2020-07-04 21:20:04,344 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2020-07-04 21:20:04,346 INFO [make_plots] preparing render list for 2020-07-04 VHFR FVCOM nowcast-r12 research figures
2020-07-04 21:20:04,350 INFO [make_plots] added EnglishBay_surface_currents figures to 2020-07-04 VHFR FVCOM nowcast-r12 research render list
2020-07-04 21:20:04,353 INFO [make_plots] added VancouverHarbour_surface_currents figures to 2020-07-04 VHFR FVCOM nowcast-r12 research render list
2020-07-04 21:20:04,356 INFO [make_plots] added IndianArm_surface_currents figures to 2020-07-04 VHFR FVCOM nowcast-r12 research render list
2020-07-04 21:20:04,360 INFO [make_plots] added VancouverHarbour_thalweg figures to 2020-07-04 VHFR FVCOM nowcast-r12 research render list
2020-07-04 21:20:04,363 INFO [make_plots] added PortMoody_thalweg figures to 2020-07-04 VHFR FVCOM nowcast-r12 research render list
2020-07-04 21:20:04,366 INFO [make_plots] added IndianArm_thalweg figures to 2020-07-04 VHFR FVCOM nowcast-r12 research render list
2020-07-04 21:20:04,367 INFO [make_plots] starting to render fvcom nowcast-r12 research 04jul20 figures
2020-07-04 21:20:04,549 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['VFPA-HADCP'], host='localhost')
2020-07-04 21:20:05,405 INFO [ping_erddap] running in process 2642
2020-07-04 21:20:05,406 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 21:20:05,407 INFO [ping_erddap] publishing log messages to tcp://*:5564
2020-07-04 21:20:05,408 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2020-07-04 21:20:05,451 INFO [ping_erddap] VFPA-HADCP ERDDAP dataset flag file(s) created
2020-07-04 21:20:05,454 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2020-07-04 21:20:06,131 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['fvcom', 'nowcast-x2', 'publish', '--run-date', '2020-07-04'], host='localhost')
2020-07-04 21:20:06,135 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['fvcom', 'nowcast-r12', 'publish', '--run-date', '2020-07-04'], host='localhost')
2020-07-04 21:20:06,137 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['fvcom', 'forecast-x2', 'publish', '--run-date', '2020-07-04'], host='localhost')
2020-07-04 21:20:08,971 INFO [make_plots] running in process 2647
2020-07-04 21:20:08,972 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 21:20:08,973 INFO [make_plots] publishing log messages to tcp://*:5564
2020-07-04 21:20:08,975 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2020-07-04 21:20:08,976 INFO [make_plots] running in process 2646
2020-07-04 21:20:08,977 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 21:20:08,978 INFO [make_plots] publishing log messages to tcp://*:5568
2020-07-04 21:20:08,979 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2020-07-04 21:20:09,000 INFO [make_plots] running in process 2648
2020-07-04 21:20:09,001 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-04 21:20:09,001 INFO [make_plots] publishing log messages to tcp://*:5569
2020-07-04 21:20:09,002 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2020-07-04 21:20:10,223 INFO [make_plots] preparing render list for 2020-07-04 VHFR FVCOM nowcast-r12 publish figures
2020-07-04 21:20:10,224 INFO [make_plots] added Calamity Point figure to 2020-07-04 VHFR FVCOM nowcast-r12 publish render list
2020-07-04 21:20:10,225 INFO [make_plots] added Indian Arm Head figure to 2020-07-04 VHFR FVCOM nowcast-r12 publish render list
2020-07-04 21:20:10,226 INFO [make_plots] added New Westminster figure to 2020-07-04 VHFR FVCOM nowcast-r12 publish render list
2020-07-04 21:20:10,226 INFO [make_plots] added Port Moody figure to 2020-07-04 VHFR FVCOM nowcast-r12 publish render list
2020-07-04 21:20:10,227 INFO [make_plots] added Sand Heads figure to 2020-07-04 VHFR FVCOM nowcast-r12 publish render list
2020-07-04 21:20:10,228 INFO [make_plots] added Sandy Cove figure to 2020-07-04 VHFR FVCOM nowcast-r12 publish render list
2020-07-04 21:20:10,229 INFO [make_plots] added Vancouver Harbour figure to 2020-07-04 VHFR FVCOM nowcast-r12 publish render list
2020-07-04 21:20:10,229 INFO [make_plots] added Woodwards Landing figure to 2020-07-04 VHFR FVCOM nowcast-r12 publish render list
2020-07-04 21:20:10,230 INFO [make_plots] added 2ndNarrows_current figure to 2020-07-04 VHFR FVCOM nowcast-r12 publish render list
2020-07-04 21:20:10,231 INFO [make_plots] starting to render fvcom nowcast-r12 publish 04jul20 figures
2020-07-04 21:20:10,246 INFO [make_plots] preparing render list for 2020-07-04 VHFR FVCOM forecast-x2 publish figures
2020-07-04 21:20:10,247 INFO [make_plots] added Calamity Point figure to 2020-07-04 VHFR FVCOM forecast-x2 publish render list
2020-07-04 21:20:10,248 INFO [make_plots] added Indian Arm Head figure to 2020-07-04 VHFR FVCOM forecast-x2 publish render list
2020-07-04 21:20:10,249 INFO [make_plots] added New Westminster figure to 2020-07-04 VHFR FVCOM forecast-x2 publish render list
2020-07-04 21:20:10,250 INFO [make_plots] added Port Moody figure to 2020-07-04 VHFR FVCOM forecast-x2 publish render list
2020-07-04 21:20:10,251 INFO [make_plots] added Sand Heads figure to 2020-07-04 VHFR FVCOM forecast-x2 publish render list
2020-07-04 21:20:10,252 INFO [make_plots] added Sandy Cove figure to 2020-07-04 VHFR FVCOM forecast-x2 publish render list
2020-07-04 21:20:10,253 INFO [make_plots] added Vancouver Harbour figure to 2020-07-04 VHFR FVCOM forecast-x2 publish render list
2020-07-04 21:20:10,253 INFO [make_plots] added Woodwards Landing figure to 2020-07-04 VHFR FVCOM forecast-x2 publish render list
2020-07-04 21:20:10,254 INFO [make_plots] added 2ndNarrows_current figure to 2020-07-04 VHFR FVCOM forecast-x2 publish render list
2020-07-04 21:20:10,254 INFO [make_plots] starting to render fvcom forecast-x2 publish 04jul20 figures
2020-07-04 21:20:10,256 INFO [make_plots] preparing render list for 2020-07-04 VHFR FVCOM nowcast-x2 publish figures
2020-07-04 21:20:10,256 INFO [make_plots] added Calamity Point figure to 2020-07-04 VHFR FVCOM nowcast-x2 publish render list
2020-07-04 21:20:10,257 INFO [make_plots] added Indian Arm Head figure to 2020-07-04 VHFR FVCOM nowcast-x2 publish render list
2020-07-04 21:20:10,258 INFO [make_plots] added New Westminster figure to 2020-07-04 VHFR FVCOM nowcast-x2 publish render list
2020-07-04 21:20:10,258 INFO [make_plots] added Port Moody figure to 2020-07-04 VHFR FVCOM nowcast-x2 publish render list
2020-07-04 21:20:10,259 INFO [make_plots] added Sand Heads figure to 2020-07-04 VHFR FVCOM nowcast-x2 publish render list
2020-07-04 21:20:10,260 INFO [make_plots] added Sandy Cove figure to 2020-07-04 VHFR FVCOM nowcast-x2 publish render list
2020-07-04 21:20:10,260 INFO [make_plots] added Vancouver Harbour figure to 2020-07-04 VHFR FVCOM nowcast-x2 publish render list
2020-07-04 21:20:10,261 INFO [make_plots] added Woodwards Landing figure to 2020-07-04 VHFR FVCOM nowcast-x2 publish render list
2020-07-04 21:20:10,261 INFO [make_plots] added 2ndNarrows_current figure to 2020-07-04 VHFR FVCOM nowcast-x2 publish render list
2020-07-04 21:20:10,262 INFO [make_plots] starting to render fvcom nowcast-x2 publish 04jul20 figures
2020-07-04 21:20:10,264 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07724 Calamity Point from 1858-11-16 23:55:00 to 2020-07-04 00:20:00
2020-07-04 21:20:10,282 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07724 Calamity Point from 1858-11-16 23:55:00 to 2020-07-04 00:20:00
2020-07-04 21:20:10,285 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07724 Calamity Point from 2020-07-03 23:55:00 to 2020-07-06 12:00:00
2020-07-04 21:20:37,328 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07724 Calamity Point from 2020-07-04 00:00:00 to 2020-07-06 12:00:00
2020-07-04 21:20:39,203 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07774 Indian Arm Head from 2020-07-03 23:55:00 to 2020-07-06 12:00:00
2020-07-04 21:20:39,805 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07774 Indian Arm Head from 2020-07-04 00:00:00 to 2020-07-06 12:00:00
2020-07-04 21:20:41,427 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07654 New Westminster from 2020-07-03 23:55:00 to 2020-07-06 12:00:00
2020-07-04 21:21:07,644 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07654 New Westminster from 2020-07-04 00:00:00 to 2020-07-06 12:00:00
2020-07-04 21:21:12,337 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07755 Port Moody from 2020-07-03 23:55:00 to 2020-07-06 12:00:00
2020-07-04 21:21:38,442 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07755 Port Moody from 2020-07-04 00:00:00 to 2020-07-06 12:00:00
2020-07-04 21:21:43,317 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07594 Sand Heads from 2020-07-03 23:55:00 to 2020-07-06 12:00:00
2020-07-04 21:22:09,724 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07594 Sand Heads from 2020-07-04 00:00:00 to 2020-07-06 12:00:00
2020-07-04 21:22:14,758 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07786 Sandy Cove from 2020-07-03 23:55:00 to 2020-07-06 12:00:00
2020-07-04 21:22:40,877 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07795 Point Atkinson from 2020-07-04 00:00:00 to 2020-07-06 12:00:00
2020-07-04 21:22:46,242 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07735 Vancouver Harbour from 2020-07-03 23:55:00 to 2020-07-06 12:00:00
2020-07-04 21:23:12,540 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07735 Vancouver Harbour from 2020-07-04 00:00:00 to 2020-07-06 12:00:00
2020-07-04 21:23:17,972 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07610 Woodwards Landing from 2020-07-03 23:55:00 to 2020-07-06 12:00:00
2020-07-04 21:23:38,293 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07610 Woodwards Landing from 2020-07-04 00:00:00 to 2020-07-06 12:00:00
2020-07-04 21:23:43,397 INFO [make_plots] finished rendering fvcom forecast-x2 publish 04jul20 figures
2020-07-04 21:23:43,840 INFO [make_plots] fvcom publish plots for 2020-07-04 forecast-x2 completed
2020-07-04 21:23:43,846 INFO [manager] checklist updated with [plots] items from make_plots worker
2020-07-04 22:39:11,607 INFO [make_plots] finished rendering fvcom nowcast-r12 research 04jul20 figures
2020-07-04 22:39:11,608 INFO [make_plots] fvcom research plots for 2020-07-04 nowcast-r12 completed
2020-07-04 22:39:11,720 INFO [manager] checklist updated with [plots] items from make_plots worker
2020-07-05 00:45:45,413 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07724 Calamity Point from 1858-11-17 00:00:00 to 2020-07-04 00:20:00
2020-07-05 00:45:49,213 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07774 Indian Arm Head from 1858-11-16 23:55:00 to 2020-07-04 00:20:00
2020-07-05 00:45:50,016 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07774 Indian Arm Head from 1858-11-17 00:00:00 to 2020-07-04 00:20:00
2020-07-05 00:45:52,030 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07654 New Westminster from 1858-11-16 23:55:00 to 2020-07-04 00:20:00
2020-07-05 00:46:23,825 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07724 Calamity Point from 1858-11-17 00:00:00 to 2020-07-04 00:20:00
2020-07-05 00:46:26,164 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07774 Indian Arm Head from 1858-11-16 23:55:00 to 2020-07-04 00:20:00
2020-07-05 00:46:26,980 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07774 Indian Arm Head from 1858-11-17 00:00:00 to 2020-07-04 00:20:00
2020-07-05 00:46:28,515 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07654 New Westminster from 1858-11-16 23:55:00 to 2020-07-04 00:20:00
2020-07-05 03:08:31,104 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-west/06/ to /results/forcing/atmospheric/GEM2.5/GRIB/20200705/06/
2020-07-05 03:08:31,105 INFO [collect_weather] 2020-07-05 2.5km weather forecast 06 collection complete
2020-07-05 03:08:31,107 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2020-07-05 03:08:32,196 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['Capilano', '--data-date', '2020-07-04'], host='localhost')
2020-07-05 03:08:32,203 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ChilliwackVedder', '--data-date', '2020-07-04'], host='localhost')
2020-07-05 03:08:32,204 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ClowhomClowhomLake', '--data-date', '2020-07-04'], host='localhost')
2020-07-05 03:08:32,207 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['Englishman', '--data-date', '2020-07-04'], host='localhost')
2020-07-05 03:08:32,215 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['Fraser', '--data-date', '2020-07-04'], host='localhost')
2020-07-05 03:08:32,221 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['HomathkoMouth', '--data-date', '2020-07-04'], host='localhost')
2020-07-05 03:08:32,235 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['SalmonSayward', '--data-date', '2020-07-04'], host='localhost')
2020-07-05 03:08:32,244 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['SanJuanPortRenfrew', '--data-date', '2020-07-04'], host='localhost')
2020-07-05 03:08:32,261 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['SquamishBrackendale', '--data-date', '2020-07-04'], host='localhost')
2020-07-05 03:08:32,278 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['TheodosiaScotty', '--data-date', '2020-07-04'], host='localhost')
2020-07-05 03:08:32,284 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['TheodosiaBypass', '--data-date', '2020-07-04'], host='localhost')
2020-07-05 03:08:32,298 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['TheodosiaDiversion', '--data-date', '2020-07-04'], host='localhost')
2020-07-05 03:08:32,313 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ctd', args=['SCVIP'], host='localhost')
2020-07-05 03:08:32,324 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ctd', args=['SEVIP'], host='localhost')
2020-07-05 03:08:32,338 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ferry', args=['TWDP'], host='localhost')
2020-07-05 03:08:32,357 INFO [manager] launching NextWorker(module='nowcast.workers.get_NeahBay_ssh', args=['forecast2'], host='localhost')
2020-07-05 03:08:32,367 INFO [manager] launching NextWorker(module='nowcast.workers.grib_to_netcdf', args=['forecast2'], host='localhost')
2020-07-05 03:08:32,373 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['12', '2.5km'], host='localhost')
2020-07-05 03:08:35,123 INFO [collect_weather] running in process 10778
2020-07-05 03:08:35,124 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-05 03:08:35,131 INFO [collect_weather] publishing log messages to tcp://*:5562
2020-07-05 03:08:35,133 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2020-07-05 03:08:35,843 INFO [grib_to_netcdf] running in process 10777
2020-07-05 03:08:35,853 INFO [grib_to_netcdf] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-05 03:08:35,854 INFO [grib_to_netcdf] publishing log messages to tcp://*:5567
2020-07-05 03:08:35,855 INFO [grib_to_netcdf] connected to skookum.eos.ubc.ca port 5555
2020-07-05 03:08:35,856 INFO [grib_to_netcdf] forecast section: 20200705/06
2020-07-05 03:08:35,856 INFO [grib_to_netcdf] next day forecast section: 20200705/06
2020-07-05 03:08:36,329 INFO [collect_river_data] running in process 10760
2020-07-05 03:08:36,330 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-05 03:08:36,330 INFO [collect_river_data] publishing log messages to tcp://*:5569
2020-07-05 03:08:36,331 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2020-07-05 03:08:36,332 INFO [collect_river_data] Collecting SalmonSayward river data for 2020-07-04
2020-07-05 03:08:36,401 INFO [collect_river_data] running in process 10763
2020-07-05 03:08:36,401 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-05 03:08:36,402 INFO [collect_river_data] publishing log messages to tcp://*:5570
2020-07-05 03:08:36,402 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2020-07-05 03:08:36,403 INFO [collect_river_data] Collecting SquamishBrackendale river data for 2020-07-04
2020-07-05 03:08:36,424 INFO [collect_river_data] running in process 10755
2020-07-05 03:08:36,425 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-05 03:08:36,426 INFO [collect_river_data] publishing log messages to tcp://*:5571
2020-07-05 03:08:36,426 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2020-07-05 03:08:36,427 INFO [collect_river_data] Collecting ChilliwackVedder river data for 2020-07-04
2020-07-05 03:08:36,474 INFO [collect_river_data] running in process 10762
2020-07-05 03:08:36,505 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-05 03:08:36,506 INFO [collect_river_data] publishing log messages to tcp://*:5572
2020-07-05 03:08:36,507 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2020-07-05 03:08:36,574 INFO [collect_river_data] Collecting SanJuanPortRenfrew river data for 2020-07-04
2020-07-05 03:08:36,575 INFO [collect_river_data] running in process 10769
2020-07-05 03:08:36,577 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-05 03:08:36,599 INFO [collect_river_data] publishing log messages to tcp://*:5573
2020-07-05 03:08:36,600 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2020-07-05 03:08:36,601 INFO [collect_river_data] Collecting TheodosiaDiversion river data for 2020-07-04
2020-07-05 03:08:36,607 INFO [collect_river_data] running in process 10758
2020-07-05 03:08:36,608 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-05 03:08:36,609 INFO [collect_river_data] publishing log messages to tcp://*:5574
2020-07-05 03:08:36,612 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2020-07-05 03:08:36,613 INFO [collect_river_data] Collecting Fraser river data for 2020-07-04
2020-07-05 03:08:36,614 INFO [collect_river_data] running in process 10764
2020-07-05 03:08:36,615 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-05 03:08:36,615 INFO [collect_river_data] publishing log messages to tcp://*:5575
2020-07-05 03:08:36,616 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2020-07-05 03:08:36,621 INFO [collect_river_data] Collecting TheodosiaScotty river data for 2020-07-04
2020-07-05 03:08:36,637 INFO [collect_river_data] running in process 10757
2020-07-05 03:08:36,638 INFO [collect_river_data] running in process 10759
2020-07-05 03:08:36,639 INFO [collect_river_data] running in process 10756
2020-07-05 03:08:36,640 INFO [collect_river_data] running in process 10754
2020-07-05 03:08:36,641 INFO [collect_river_data] running in process 10766
2020-07-05 03:08:36,642 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-05 03:08:36,643 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-05 03:08:36,644 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-05 03:08:36,645 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-05 03:08:36,646 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-05 03:08:36,647 INFO [collect_river_data] publishing log messages to tcp://*:5576
2020-07-05 03:08:36,647 INFO [collect_river_data] publishing log messages to tcp://*:5577
2020-07-05 03:08:36,648 INFO [collect_river_data] publishing log messages to tcp://*:5578
2020-07-05 03:08:36,649 INFO [collect_river_data] publishing log messages to tcp://*:5579
2020-07-05 03:08:36,649 INFO [collect_river_data] publishing log messages to tcp://*:5590
2020-07-05 03:08:36,650 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2020-07-05 03:08:36,651 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2020-07-05 03:08:36,652 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2020-07-05 03:08:36,653 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2020-07-05 03:08:36,653 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2020-07-05 03:08:36,654 INFO [collect_river_data] Collecting Englishman river data for 2020-07-04
2020-07-05 03:08:36,655 INFO [collect_river_data] Collecting HomathkoMouth river data for 2020-07-04
2020-07-05 03:08:36,655 INFO [collect_river_data] Collecting ClowhomClowhomLake river data for 2020-07-04
2020-07-05 03:08:36,656 INFO [collect_river_data] Collecting Capilano river data for 2020-07-04
2020-07-05 03:08:36,657 INFO [collect_river_data] Collecting TheodosiaBypass river data for 2020-07-04
2020-07-05 03:08:37,084 INFO [collect_river_data] Appended Englishman river average discharge for 2020-07-04 to: /data/dlatorne/SOG-projects/SOG-forcing/ECget/Englishman_flow
2020-07-05 03:08:37,085 INFO [collect_river_data] Englishman river average discharge for 2020-07-04 calculated and appended to Englishman_flow file
2020-07-05 03:08:37,087 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2020-07-05 03:08:37,088 INFO [collect_river_data] Appended Capilano river average discharge for 2020-07-04 to: /opp/observations/rivers/Capilano/Caplilano_08GA010_day_avg_flow
2020-07-05 03:08:37,088 INFO [collect_river_data] Capilano river average discharge for 2020-07-04 calculated and appended to Capilano_flow file
2020-07-05 03:08:37,278 INFO [collect_river_data] Appended Fraser river average discharge for 2020-07-04 to: /data/dlatorne/SOG-projects/SOG-forcing/ECget/Fraser_flow
2020-07-05 03:08:37,279 INFO [collect_river_data] Fraser river average discharge for 2020-07-04 calculated and appended to Fraser_flow file
2020-07-05 03:08:37,725 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2020-07-05 03:08:38,213 INFO [collect_river_data] Appended TheodosiaDiversion river average discharge for 2020-07-04 to: /results/forcing/rivers/observations/Theodosia_Diversion_flow
2020-07-05 03:08:38,214 INFO [collect_river_data] TheodosiaDiversion river average discharge for 2020-07-04 calculated and appended to TheodosiaDiversion_flow file
2020-07-05 03:08:38,216 INFO [collect_river_data] Appended TheodosiaBypass river average discharge for 2020-07-04 to: /results/forcing/rivers/observations/Theodosia_Bypass_flow
2020-07-05 03:08:38,216 INFO [collect_river_data] TheodosiaBypass river average discharge for 2020-07-04 calculated and appended to TheodosiaBypass_flow file
2020-07-05 03:08:38,267 INFO [collect_river_data] Appended SalmonSayward river average discharge for 2020-07-04 to: /results/forcing/rivers/observations/Salmon_Sayward_flow
2020-07-05 03:08:38,268 INFO [collect_river_data] SalmonSayward river average discharge for 2020-07-04 calculated and appended to SalmonSayward_flow file
2020-07-05 03:08:38,363 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2020-07-05 03:08:38,716 INFO [get_NeahBay_ssh] running in process 10776
2020-07-05 03:08:38,717 INFO [get_NeahBay_ssh] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-05 03:08:38,718 INFO [get_NeahBay_ssh] publishing log messages to tcp://*:5576
2020-07-05 03:08:38,719 INFO [get_NeahBay_ssh] connected to skookum.eos.ubc.ca port 5555
2020-07-05 03:08:39,045 INFO [manager] launching NextWorker(module='nowcast.workers.make_runoff_file', args=[], host='localhost')
2020-07-05 03:08:39,050 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2020-07-05 03:08:39,186 INFO [get_onc_ferry] running in process 10775
2020-07-05 03:08:39,187 INFO [get_onc_ferry] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-05 03:08:39,188 INFO [get_onc_ferry] publishing log messages to tcp://*:5579
2020-07-05 03:08:39,188 INFO [get_onc_ferry] connected to skookum.eos.ubc.ca port 5555
2020-07-05 03:08:39,189 INFO [get_onc_ferry] requesting ONC TWDP.N1 NAV data for 2020-07-04
2020-07-05 03:08:39,193 INFO [get_onc_ctd] running in process 10772
2020-07-05 03:08:39,194 INFO [get_onc_ctd] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-05 03:08:39,194 INFO [get_onc_ctd] publishing log messages to tcp://*:5591
2020-07-05 03:08:39,195 INFO [get_onc_ctd] connected to skookum.eos.ubc.ca port 5555
2020-07-05 03:08:39,195 INFO [get_onc_ctd] requesting ONC SCVIP CTD T&S data for 2020-07-04
2020-07-05 03:08:39,273 INFO [get_onc_ctd] running in process 10773
2020-07-05 03:08:39,274 INFO [get_onc_ctd] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-05 03:08:39,275 INFO [get_onc_ctd] publishing log messages to tcp://*:5592
2020-07-05 03:08:39,276 INFO [get_onc_ctd] connected to skookum.eos.ubc.ca port 5555
2020-07-05 03:08:39,276 INFO [get_onc_ctd] requesting ONC SEVIP CTD T&S data for 2020-07-04
2020-07-05 03:08:39,664 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2020-07-05 03:08:40,152 INFO [make_runoff_file] running in process 10866
2020-07-05 03:08:40,153 INFO [make_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-05 03:08:40,154 INFO [make_runoff_file] publishing log messages to tcp://*:5573
2020-07-05 03:08:40,155 INFO [make_runoff_file] connected to skookum.eos.ubc.ca port 5555
2020-07-05 03:08:40,252 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2020-07-05 03:08:40,676 INFO [make_runoff_file] calculating runoff forcing for bathymetry type: b201702
2020-07-05 03:08:40,735 INFO [collect_river_data] Appended SanJuanPortRenfrew river average discharge for 2020-07-04 to: /results/forcing/rivers/observations/SanJuan_PortRenfrew_flow
2020-07-05 03:08:40,736 INFO [collect_river_data] SanJuanPortRenfrew river average discharge for 2020-07-04 calculated and appended to SanJuanPortRenfrew_flow file
2020-07-05 03:08:40,741 INFO [collect_river_data] Appended HomathkoMouth river average discharge for 2020-07-04 to: /results/forcing/rivers/observations/Homathko_Mouth_flow
2020-07-05 03:08:40,742 INFO [collect_river_data] HomathkoMouth river average discharge for 2020-07-04 calculated and appended to HomathkoMouth_flow file
2020-07-05 03:08:40,749 INFO [collect_river_data] Appended SquamishBrackendale river average discharge for 2020-07-04 to: /results/forcing/rivers/observations/Squamish_Brackendale_flow
2020-07-05 03:08:40,750 INFO [collect_river_data] SquamishBrackendale river average discharge for 2020-07-04 calculated and appended to SquamishBrackendale_flow file
2020-07-05 03:08:40,752 INFO [collect_river_data] Appended ChilliwackVedder river average discharge for 2020-07-04 to: /results/forcing/rivers/observations/Chilliwack_Vedder_flow
2020-07-05 03:08:40,753 INFO [collect_river_data] ChilliwackVedder river average discharge for 2020-07-04 calculated and appended to ChilliwackVedder_flow file
2020-07-05 03:08:40,773 INFO [collect_river_data] Appended TheodosiaScotty river average discharge for 2020-07-04 to: /results/forcing/rivers/observations/Theodosia_Scotty_flow
2020-07-05 03:08:40,774 INFO [collect_river_data] TheodosiaScotty river average discharge for 2020-07-04 calculated and appended to TheodosiaScotty_flow file
2020-07-05 03:08:40,786 INFO [collect_river_data] Appended ClowhomClowhomLake river average discharge for 2020-07-04 to: /results/forcing/rivers/observations/Clowhom_ClowhomLake_flow
2020-07-05 03:08:40,787 INFO [collect_river_data] ClowhomClowhomLake river average discharge for 2020-07-04 calculated and appended to ClowhomClowhomLake_flow file
2020-07-05 03:08:40,831 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2020-07-05 03:08:40,988 INFO [make_runoff_file] 2020-07-05 runoff file creation from Fraser at Hope and climatology elsewhere complete
2020-07-05 03:08:41,111 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-west/12/
2020-07-05 03:08:41,378 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2020-07-05 03:08:41,940 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2020-07-05 03:08:42,540 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2020-07-05 03:08:43,123 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2020-07-05 03:08:43,698 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2020-07-05 03:08:44,260 INFO [manager] checklist updated with [rivers forcing] items from make_runoff_file worker
2020-07-05 03:08:45,662 INFO [get_NeahBay_ssh] Neah Bay sea surface height web scraping and open boundary file creation for forecast2 complete
2020-07-05 03:08:45,667 INFO [manager] checklist updated with [Neah Bay ssh] items from get_NeahBay_ssh worker
2020-07-05 03:09:03,959 INFO [get_onc_ctd] 2020-07-04 ONC SEVIP CTD T&S file created
2020-07-05 03:09:03,962 INFO [manager] checklist updated with [ONC CTD data] items from get_onc_ctd worker
2020-07-05 03:09:04,655 INFO [get_onc_ctd] 2020-07-04 ONC SCVIP CTD T&S file created
2020-07-05 03:09:04,722 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['SEVIP-CTD'], host='localhost')
2020-07-05 03:09:04,729 INFO [manager] checklist updated with [ONC CTD data] items from get_onc_ctd worker
2020-07-05 03:09:05,414 INFO [get_onc_ferry] requesting ONC TWDP TSG data for 2020-07-04
2020-07-05 03:09:05,419 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['SCVIP-CTD'], host='localhost')
2020-07-05 03:09:05,735 INFO [ping_erddap] running in process 11148
2020-07-05 03:09:05,736 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-05 03:09:05,737 INFO [ping_erddap] publishing log messages to tcp://*:5569
2020-07-05 03:09:05,738 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2020-07-05 03:09:05,739 INFO [ping_erddap] SEVIP-CTD ERDDAP dataset flag file(s) created
2020-07-05 03:09:05,741 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2020-07-05 03:09:06,218 INFO [ping_erddap] running in process 11152
2020-07-05 03:09:06,219 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-05 03:09:06,219 INFO [ping_erddap] publishing log messages to tcp://*:5570
2020-07-05 03:09:06,220 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2020-07-05 03:09:06,221 INFO [ping_erddap] SCVIP-CTD ERDDAP dataset flag file(s) created
2020-07-05 03:09:06,347 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2020-07-05 03:09:38,409 INFO [get_onc_ferry] requesting ONC TWDP OXYSENSOR data for 2020-07-04
2020-07-05 03:10:02,766 INFO [grib_to_netcdf] 2020-07-05 NEMO-atmos forcing file for forecast2 created
2020-07-05 03:10:02,769 INFO [manager] checklist updated with [weather forcing] items from grib_to_netcdf worker
2020-07-05 03:10:03,387 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'forecast2'], host='localhost')
2020-07-05 03:10:03,391 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'forecast2'], host='localhost')
2020-07-05 03:10:03,394 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['graham-hindcast', 'forecast2'], host='localhost')
2020-07-05 03:10:03,397 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'forecast2'], host='localhost')
2020-07-05 03:10:04,403 INFO [upload_forcing] running in process 11593
2020-07-05 03:10:04,403 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-05 03:10:04,404 INFO [upload_forcing] publishing log messages to tcp://*:5567
2020-07-05 03:10:04,405 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2020-07-05 03:10:04,411 INFO [upload_forcing] running in process 11595
2020-07-05 03:10:04,412 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-05 03:10:04,413 INFO [upload_forcing] publishing log messages to tcp://*:5569
2020-07-05 03:10:04,414 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2020-07-05 03:10:04,475 INFO [upload_forcing] running in process 11594
2020-07-05 03:10:04,476 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-05 03:10:04,477 INFO [upload_forcing] publishing log messages to tcp://*:5570
2020-07-05 03:10:04,477 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2020-07-05 03:10:04,527 INFO [upload_forcing] running in process 11596
2020-07-05 03:10:04,528 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-05 03:10:04,528 INFO [upload_forcing] publishing log messages to tcp://*:5571
2020-07-05 03:10:04,529 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2020-07-05 03:10:09,805 INFO [upload_forcing] LiveOcean boundary conditions file not found; created symlink to /results/forcing/LiveOcean/boundary_conditions/LiveOcean_v201905_y2020m07d04.nc
2020-07-05 03:10:09,806 INFO [upload_forcing] LiveOcean boundary conditions file not found; created symlink to /results/forcing/LiveOcean/boundary_conditions/LiveOcean_v201905_y2020m07d04.nc
2020-07-05 03:10:09,807 INFO [upload_forcing] LiveOcean boundary conditions file not found; created symlink to /results/forcing/LiveOcean/boundary_conditions/LiveOcean_v201905_y2020m07d04.nc
2020-07-05 03:10:09,925 INFO [upload_forcing] forecast2 2020-07-05 forcing files upload to optimum-hindcast completed
2020-07-05 03:10:09,929 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2020-07-05 03:10:09,936 INFO [upload_forcing] forecast2 2020-07-05 forcing files upload to arbutus.cloud-nowcast completed
2020-07-05 03:10:09,938 INFO [upload_forcing] forecast2 2020-07-05 forcing files upload to orcinus-nowcast-agrif completed
2020-07-05 03:10:10,597 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2020-07-05 03:10:11,190 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2020-07-05'], host='localhost')
2020-07-05 03:10:11,196 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2020-07-05 03:10:11,761 INFO [upload_forcing] forecast2 2020-07-05 forcing files upload to graham-hindcast completed
2020-07-05 03:10:11,843 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2020-07-05 03:10:12,165 INFO [make_forcing_links] running in process 11617
2020-07-05 03:10:12,166 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-05 03:10:12,167 INFO [make_forcing_links] publishing log messages to tcp://*:5567
2020-07-05 03:10:12,168 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2020-07-05 03:10:13,392 INFO [make_forcing_links] forecast2 2020-07-05 forcing file links on arbutus.cloud-nowcast created
2020-07-05 03:10:13,396 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2020-07-05 03:10:14,084 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2020-07-05'], host='arbutus.cloud-nowcast')
2020-07-05 03:10:16,739 INFO [run_NEMO] running in process 9897
2020-07-05 03:10:16,740 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2020-07-05 03:10:16,741 INFO [run_NEMO] publishing log messages to tcp://*:5556
2020-07-05 03:10:16,750 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2020-07-05 03:10:17,293 INFO [get_onc_ferry] requesting ONC TWDP TURBCHLFL data for 2020-07-04
2020-07-05 03:10:17,791 INFO [run_NEMO] forecast2: launching /nemoShare/MEOPAR/nowcast-sys/runs/04jul20forecast2_2020-07-05T031016.835926-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2020-07-05 03:10:17,807 INFO [run_NEMO] forecast2 NEMO run for 2020-07-05 on arbutus.cloud-nowcast started
2020-07-05 03:10:17,811 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2020-07-05 03:10:18,414 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'forecast2'], host='arbutus.cloud-nowcast')
2020-07-05 03:10:20,436 INFO [watch_NEMO] running in process 10162
2020-07-05 03:10:20,437 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2020-07-05 03:10:20,438 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2020-07-05 03:10:20,439 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2020-07-05 03:10:20,555 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2020-07-05 03:10:50,196 INFO [get_onc_ferry] requesting ONC TWDP CO2SENSOR data for 2020-07-04
2020-07-05 03:11:04,875 INFO [get_onc_ferry] requesting ONC TWDP TEMPHUMID data for 2020-07-04
2020-07-05 03:11:31,392 INFO [get_onc_ferry] requesting ONC TWDP BARPRESS data for 2020-07-04
2020-07-05 03:11:44,177 INFO [get_onc_ferry] requesting ONC TWDP PYRANOMETER data for 2020-07-04
2020-07-05 03:11:56,156 INFO [get_onc_ferry] requesting ONC TWDP PYRGEOMETER data for 2020-07-04
2020-07-05 03:12:11,591 INFO [get_onc_ferry] 2020-07-04 ONC TWDP ferry data file created
2020-07-05 03:12:11,595 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['TWDP-ferry'], host='localhost')
2020-07-05 03:12:12,661 INFO [ping_erddap] running in process 11679
2020-07-05 03:12:12,662 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-05 03:12:12,663 INFO [ping_erddap] publishing log messages to tcp://*:5567
2020-07-05 03:12:12,664 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2020-07-05 03:12:15,534 INFO [ping_erddap] TWDP-ferry ERDDAP dataset flag file(s) created
2020-07-05 03:12:15,537 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2020-07-05 03:15:20,657 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 7502127 = 2020-07-06 04:57:20 UTC, 16.5% complete
2020-07-05 03:20:20,759 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 7502634 = 2020-07-06 10:35:20 UTC, 35.3% complete
2020-07-05 03:25:20,861 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 7503140 = 2020-07-06 16:12:40 UTC, 54.1% complete
2020-07-05 03:30:20,962 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 7503641 = 2020-07-06 21:46:40 UTC, 72.6% complete
2020-07-05 03:35:21,062 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 7504131 = 2020-07-07 03:13:20 UTC, 90.8% complete
2020-07-05 03:36:00,591 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07654 New Westminster from 1858-11-17 00:00:00 to 2020-07-04 00:20:00
2020-07-05 03:37:22,327 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07654 New Westminster from 1858-11-17 00:00:00 to 2020-07-04 00:20:00
2020-07-05 03:40:21,166 INFO [watch_NEMO] forecast2 NEMO run on arbutus.cloud-nowcast completed
2020-07-05 03:40:21,170 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2020-07-05 03:40:22,046 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_wind_file', args=['arbutus.cloud-nowcast', 'forecast2'], host='arbutus.cloud-nowcast')
2020-07-05 03:40:22,200 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_current_file', args=['arbutus.cloud-nowcast', 'forecast2'], host='arbutus.cloud-nowcast')
2020-07-05 03:40:22,203 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2020-07-04'], host='localhost')
2020-07-05 03:40:23,312 INFO [download_results] running in process 12388
2020-07-05 03:40:23,313 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-05 03:40:23,314 INFO [download_results] publishing log messages to tcp://*:5567
2020-07-05 03:40:23,315 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2020-07-05 03:40:23,316 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/forecast2/04jul20 to /results/SalishSea/forecast2.201905
2020-07-05 03:40:26,069 INFO [make_ww3_wind_file] running in process 10420
2020-07-05 03:40:26,069 INFO [make_ww3_wind_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2020-07-05 03:40:26,070 INFO [make_ww3_wind_file] publishing log messages to tcp://*:5570
2020-07-05 03:40:26,071 INFO [make_ww3_wind_file] connected to skookum.eos.ubc.ca port 5555
2020-07-05 03:40:26,072 INFO [make_ww3_wind_file] Creating wwatch3 wind forcing file for 2020-07-05 forecast2 run
2020-07-05 03:40:26,188 INFO [make_ww3_current_file] running in process 10419
2020-07-05 03:40:26,197 INFO [make_ww3_current_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2020-07-05 03:40:26,198 INFO [make_ww3_current_file] publishing log messages to tcp://*:5571
2020-07-05 03:40:26,200 INFO [make_ww3_current_file] connected to skookum.eos.ubc.ca port 5555
2020-07-05 03:40:26,201 INFO [make_ww3_current_file] Creating wwatch3 currents forcing file for 2020-07-05 forecast2 run
2020-07-05 03:40:29,590 INFO [make_ww3_wind_file] wwatch3 wind forcing file created on arbutus.cloud-nowcast for 2020-07-05 forecast2 run
2020-07-05 03:40:29,595 INFO [manager] checklist updated with [WW3 wind forcing] items from make_ww3_wind_file worker
2020-07-05 03:42:17,810 INFO [make_ww3_current_file] wwatch3 currents forcing file created on arbutus.cloud-nowcast for 2020-07-05 forecast2 run
2020-07-05 03:42:17,815 INFO [manager] checklist updated with [WW3 currents forcing] items from make_ww3_current_file worker
2020-07-05 03:42:18,519 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2020-07-05'], host='arbutus.cloud-nowcast')
2020-07-05 03:42:20,683 INFO [run_ww3] running in process 10617
2020-07-05 03:42:20,684 INFO [run_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2020-07-05 03:42:20,685 INFO [run_ww3] publishing log messages to tcp://*:5572
2020-07-05 03:42:20,686 INFO [run_ww3] connected to skookum.eos.ubc.ca port 5555
2020-07-05 03:42:20,689 INFO [run_ww3] Created run directory /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast2_2020-07-05T034220.681518-0700
2020-07-05 03:42:20,690 INFO [run_ww3] forecast2: launching /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast2_2020-07-05T034220.681518-0700/SoGWW3.sh on arbutus.cloud-nowcast
2020-07-05 03:42:20,704 INFO [run_ww3] forecast2 WaveWatch3 run for 2020-07-05 on arbutus.cloud-nowcast started
2020-07-05 03:42:20,711 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker
2020-07-05 03:42:21,416 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'forecast2'], host='arbutus.cloud-nowcast')
2020-07-05 03:42:23,332 INFO [watch_ww3] running in process 10721
2020-07-05 03:42:23,334 INFO [watch_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2020-07-05 03:42:23,335 INFO [watch_ww3] publishing log messages to tcp://*:5573
2020-07-05 03:42:23,336 INFO [watch_ww3] connected to skookum.eos.ubc.ca port 5555
2020-07-05 03:42:23,370 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2020-07-05 03:44:20,688 INFO [download_results] forecast2 2020-07-04 results files from arbutus.cloud-nowcast downloaded
2020-07-05 03:44:20,694 INFO [manager] checklist updated with [results files] items from download_results worker
2020-07-05 03:44:21,314 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['forecast2', '--run-date', '2020-07-04'], host='localhost')
2020-07-05 03:44:24,469 INFO [make_CHS_currents_file] running in process 12481
2020-07-05 03:44:24,470 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-05 03:44:24,471 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5567
2020-07-05 03:44:24,471 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555
2020-07-05 03:47:23,470 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 90, 16.7% complete
2020-07-05 03:47:54,434 INFO [make_CHS_currents_file] Made CHS currents file for 2020-07-04 for forecast2
2020-07-05 03:47:54,440 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker
2020-07-05 03:47:55,130 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['nemo', 'forecast2', '--run-date', '2020-07-04'], host='localhost')
2020-07-05 03:47:56,197 INFO [update_forecast_datasets] running in process 12599
2020-07-05 03:47:56,198 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-05 03:47:56,198 INFO [update_forecast_datasets] publishing log messages to tcp://*:5567
2020-07-05 03:47:56,200 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2020-07-05 03:47:56,201 INFO [update_forecast_datasets] updating nemo forecast directory for forecast2/04jul20 run
2020-07-05 03:52:23,507 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 192, 35.6% complete
2020-07-05 03:57:23,608 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 300, 55.6% complete
2020-07-05 04:00:18,548 INFO [update_forecast_datasets] updated nemo forecast directory for forecast2/04jul20 run: /results/SalishSea/rolling-forecasts/nemo
2020-07-05 04:00:18,549 INFO [update_forecast_datasets] nemo 2020-07-04 forecast2 rolling forecast datasets updated
2020-07-05 04:00:18,554 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2020-07-05 04:00:19,234 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nemo-forecast'], host='localhost')
2020-07-05 04:00:19,259 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'forecast2', 'publish', '--run-date', '2020-07-04'], host='localhost')
2020-07-05 04:00:19,425 INFO [manager] launching NextWorker(module='nowcast.workers.make_surface_current_tiles', args=['forecast2', '--run-date', '2020-07-04'], host='localhost')
2020-07-05 04:00:20,964 INFO [ping_erddap] running in process 12880
2020-07-05 04:00:20,964 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-05 04:00:20,965 INFO [ping_erddap] publishing log messages to tcp://*:5567
2020-07-05 04:00:20,966 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2020-07-05 04:00:21,346 INFO [make_surface_current_tiles] running in process 12882
2020-07-05 04:00:21,365 INFO [make_surface_current_tiles] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-05 04:00:21,365 INFO [make_surface_current_tiles] publishing log messages to tcp://*:5569
2020-07-05 04:00:21,366 INFO [make_surface_current_tiles] connected to skookum.eos.ubc.ca port 5555
2020-07-05 04:00:21,583 INFO [ping_erddap] nemo-forecast ERDDAP dataset flag file(s) created
2020-07-05 04:00:21,585 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2020-07-05 04:00:21,713 INFO [make_surface_current_tiles] starting rendering of tiles for 2020-07-04 forecast2 in 6 processes into /results/nowcast-sys/figures/surface_currents/forecast2/04jul20
2020-07-05 04:00:29,112 INFO [make_plots] running in process 12881
2020-07-05 04:00:29,113 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-05 04:00:29,114 INFO [make_plots] publishing log messages to tcp://*:5567
2020-07-05 04:00:29,115 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2020-07-05 04:00:30,452 INFO [make_plots] preparing render list for 2020-07-04 NEMO forecast2 publish figures
2020-07-05 04:00:31,093 INFO [make_plots] added Website_thumbnail figure to 2020-07-04 NEMO forecast2 publish render list
2020-07-05 04:00:31,094 INFO [make_plots] added Threshold_website figure to 2020-07-04 NEMO forecast2 publish render list
2020-07-05 04:00:31,095 INFO [make_plots] added PA_tidal_predictions figure to 2020-07-04 NEMO forecast2 publish render list
2020-07-05 04:00:31,095 INFO [make_plots] added Boundary Bay figure to 2020-07-04 NEMO forecast2 publish render list
2020-07-05 04:00:31,096 INFO [make_plots] added Campbell River figure to 2020-07-04 NEMO forecast2 publish render list
2020-07-05 04:00:31,097 INFO [make_plots] added Cherry Point figure to 2020-07-04 NEMO forecast2 publish render list
2020-07-05 04:00:31,097 INFO [make_plots] added Friday Harbor figure to 2020-07-04 NEMO forecast2 publish render list
2020-07-05 04:00:31,098 INFO [make_plots] added Halfmoon Bay figure to 2020-07-04 NEMO forecast2 publish render list
2020-07-05 04:00:31,099 INFO [make_plots] added Nanaimo figure to 2020-07-04 NEMO forecast2 publish render list
2020-07-05 04:00:31,100 INFO [make_plots] added Neah Bay figure to 2020-07-04 NEMO forecast2 publish render list
2020-07-05 04:00:31,100 INFO [make_plots] added New Westminster figure to 2020-07-04 NEMO forecast2 publish render list
2020-07-05 04:00:31,101 INFO [make_plots] added Patricia Bay figure to 2020-07-04 NEMO forecast2 publish render list
2020-07-05 04:00:31,102 INFO [make_plots] added Point Atkinson figure to 2020-07-04 NEMO forecast2 publish render list
2020-07-05 04:00:31,102 INFO [make_plots] added Port Renfrew figure to 2020-07-04 NEMO forecast2 publish render list
2020-07-05 04:00:31,103 INFO [make_plots] added Sand Heads figure to 2020-07-04 NEMO forecast2 publish render list
2020-07-05 04:00:31,104 INFO [make_plots] added Sandy Cove figure to 2020-07-04 NEMO forecast2 publish render list
2020-07-05 04:00:31,104 INFO [make_plots] added Squamish figure to 2020-07-04 NEMO forecast2 publish render list
2020-07-05 04:00:31,105 INFO [make_plots] added Victoria figure to 2020-07-04 NEMO forecast2 publish render list
2020-07-05 04:00:31,106 INFO [make_plots] added Woodwards Landing figure to 2020-07-04 NEMO forecast2 publish render list
2020-07-05 04:00:31,107 INFO [make_plots] starting to render nemo forecast2 publish 04jul20 figures
2020-07-05 04:00:55,066 INFO [make_plots] Note: NumExpr detected 12 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
2020-07-05 04:00:55,067 INFO [make_plots] NumExpr defaulting to 8 threads.
2020-07-05 04:02:23,709 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 408, 75.6% complete
2020-07-05 04:07:23,811 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 513, 95.0% complete
2020-07-05 04:12:23,913 INFO [watch_ww3] forecast2 WWATCH3 run on arbutus.cloud-nowcast completed
2020-07-05 04:12:23,922 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2020-07-05 04:12:26,189 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2020-07-05'], host='localhost')
2020-07-05 04:12:28,044 INFO [download_wwatch3_results] running in process 13960
2020-07-05 04:12:28,045 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-05 04:12:28,046 INFO [download_wwatch3_results] publishing log messages to tcp://*:5570
2020-07-05 04:12:28,047 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2020-07-05 04:13:01,908 INFO [download_wwatch3_results] forecast2 2020-07-05 results files from arbutus.cloud-nowcast downloaded
2020-07-05 04:13:01,917 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2020-07-05 04:13:03,849 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['wwatch3', 'forecast2', '--run-date', '2020-07-05'], host='localhost')
2020-07-05 04:13:05,527 INFO [update_forecast_datasets] running in process 13986
2020-07-05 04:13:05,528 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-05 04:13:05,529 INFO [update_forecast_datasets] publishing log messages to tcp://*:5570
2020-07-05 04:13:05,530 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2020-07-05 04:13:05,530 INFO [update_forecast_datasets] updating wwatch3 most_recent_forecast directory from forecast2/05jul20 run
2020-07-05 04:13:05,532 INFO [update_forecast_datasets] updated wwatch3 most_recent_forecast directory from forecast2/05jul20 run
2020-07-05 04:13:05,532 INFO [update_forecast_datasets] updating wwatch3 forecast directory for forecast2/05jul20 run
2020-07-05 04:13:46,951 INFO [update_forecast_datasets] updated wwatch3 forecast directory for forecast2/05jul20 run: /results/SalishSea/rolling-forecasts/wwatch3
2020-07-05 04:13:46,954 INFO [update_forecast_datasets] wwatch3 2020-07-05 forecast2 rolling forecast datasets updated
2020-07-05 04:13:46,961 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2020-07-05 04:13:48,612 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['wwatch3-forecast'], host='localhost')
2020-07-05 04:13:49,778 INFO [ping_erddap] running in process 14013
2020-07-05 04:13:49,791 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-05 04:13:49,791 INFO [ping_erddap] publishing log messages to tcp://*:5570
2020-07-05 04:13:49,792 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2020-07-05 04:13:49,897 INFO [ping_erddap] wwatch3-forecast ERDDAP dataset flag file(s) created
2020-07-05 04:13:49,904 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2020-07-05 04:13:52,273 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['wwatch3', 'forecast2', 'publish', '--run-date', '2020-07-05'], host='localhost')
2020-07-05 04:14:05,508 INFO [make_plots] running in process 14017
2020-07-05 04:14:05,509 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-05 04:14:05,510 INFO [make_plots] publishing log messages to tcp://*:5570
2020-07-05 04:14:05,512 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2020-07-05 04:14:05,512 INFO [make_plots] preparing render list for 2020-07-05 WaveWatch3 forecast2 publish figures
2020-07-05 04:14:05,513 INFO [make_plots] added Halibut Bank figure to 2020-07-05 WaveWatch3 forecast2 publish render list
2020-07-05 04:14:05,514 INFO [make_plots] added Sentry Shoal figure to 2020-07-05 WaveWatch3 forecast2 publish render list
2020-07-05 04:14:05,515 INFO [make_plots] starting to render wwatch3 forecast2 publish 05jul20 figures
2020-07-05 04:14:39,789 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 08074 Campbell River from 2020-06-30 00:00:00 to 2020-07-07 05:55:00
2020-07-05 04:16:24,718 INFO [make_plots] Note: NumExpr detected 12 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
2020-07-05 04:16:24,720 INFO [make_plots] NumExpr defaulting to 8 threads.
2020-07-05 04:16:24,752 INFO [make_plots] retrieving last 45 days of real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46146 Halibut Bank
2020-07-05 04:18:12,530 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 9449424 Cherry Point from 2020-06-30 00:00:00 to 2020-07-07 05:55:00
2020-07-05 04:19:08,962 INFO [make_plots] retrieving last 45 days of real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46131 Sentry Shoal
2020-07-05 04:19:17,416 INFO [make_plots] finished rendering wwatch3 forecast2 publish 05jul20 figures
2020-07-05 04:19:17,422 INFO [make_plots] wwatch3 publish plots for 2020-07-05 forecast2 completed
2020-07-05 04:19:17,425 INFO [manager] checklist updated with [plots] items from make_plots worker
2020-07-05 04:19:18,179 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 9449880 Friday Harbor from 2020-06-30 00:00:00 to 2020-07-07 05:55:00
2020-07-05 04:20:13,564 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07830 Halfmoon Bay from 2020-06-30 00:00:00 to 2020-07-07 05:55:00
2020-07-05 04:21:20,818 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07917 Nanaimo from 2020-06-30 00:00:00 to 2020-07-07 05:55:00
2020-07-05 04:25:04,622 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 9443090 Neah Bay from 2020-06-30 00:00:00 to 2020-07-07 05:55:00
2020-07-05 04:25:56,527 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07654 New Westminster from 2020-06-30 00:00:00 to 2020-07-07 05:55:00
2020-07-05 04:29:08,549 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07277 Patricia Bay from 2020-06-30 00:00:00 to 2020-07-07 05:55:00
2020-07-05 04:32:38,190 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07795 Point Atkinson from 2020-06-30 00:00:00 to 2020-07-07 05:55:00
2020-07-05 04:33:42,719 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 08525 Port Renfrew from 2020-06-30 00:00:00 to 2020-07-07 05:55:00
2020-07-05 04:37:44,335 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07594 Sand Heads from 2020-06-30 00:00:00 to 2020-07-07 05:55:00
2020-07-05 04:39:11,868 INFO [make_surface_current_tiles] starting PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast2/04jul20
2020-07-05 04:40:32,755 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07786 Sandy Cove from 2020-06-30 00:00:00 to 2020-07-07 05:55:00
2020-07-05 04:43:01,505 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07811 Squamish from 2020-06-30 00:00:00 to 2020-07-07 05:55:00
2020-07-05 04:43:28,496 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07120 Victoria from 2020-06-30 00:00:00 to 2020-07-07 05:55:00
2020-07-05 04:45:53,251 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07610 Woodwards Landing from 2020-06-30 00:00:00 to 2020-07-07 05:55:00
2020-07-05 04:47:57,244 INFO [make_plots] finished rendering nemo forecast2 publish 04jul20 figures
2020-07-05 04:47:57,246 INFO [make_plots] nemo publish plots for 2020-07-04 forecast2 completed
2020-07-05 04:47:57,258 INFO [manager] checklist updated with [plots] items from make_plots worker
2020-07-05 04:47:57,877 INFO [manager] launching NextWorker(module='nowcast.workers.make_feeds', args=['forecast2', '--run-date', '2020-07-04'], host='localhost')
2020-07-05 04:48:04,559 INFO [make_feeds] running in process 16094
2020-07-05 04:48:04,560 INFO [make_feeds] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-05 04:48:04,561 INFO [make_feeds] publishing log messages to tcp://*:5567
2020-07-05 04:48:04,562 INFO [make_feeds] connected to skookum.eos.ubc.ca port 5555
2020-07-05 04:48:14,058 INFO [make_feeds] Note: NumExpr detected 12 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
2020-07-05 04:48:14,058 INFO [make_feeds] NumExpr defaulting to 8 threads.
2020-07-05 04:48:24,093 INFO [make_feeds] ATOM feeds for 2020-07-04 forecast2 run completed
2020-07-05 04:48:24,096 INFO [manager] checklist updated with [storm surge alert feeds] items from make_feeds worker
2020-07-05 04:48:24,660 INFO [manager] launching NextWorker(module='nemo_nowcast.workers.clear_checklist', args=[], host='localhost')
2020-07-05 04:48:25,591 INFO [clear_checklist] running in process 16139
2020-07-05 04:48:25,592 INFO [clear_checklist] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-05 04:48:25,593 INFO [clear_checklist] publishing log messages to tcp://*:5567
2020-07-05 04:48:25,594 INFO [clear_checklist] connected to skookum.eos.ubc.ca port 5555
2020-07-05 04:48:25,595 INFO [clear_checklist] requesting that manager clear system state checklist
2020-07-05 04:48:25,597 INFO [manager] writing checklist to log file
2020-07-05 04:48:25,670 INFO [manager] checklist cleared
2020-07-05 04:48:25,673 INFO [clear_checklist] nowcast system checklist cleared
2020-07-05 04:48:25,675 INFO [manager] launching NextWorker(module='nemo_nowcast.workers.rotate_logs', args=[], host='localhost')
2020-07-05 04:48:26,552 INFO [rotate_logs] running in process 16143
2020-07-05 04:48:26,553 INFO [rotate_logs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-05 04:48:26,553 INFO [rotate_logs] publishing log messages to tcp://*:5567
2020-07-05 04:48:26,554 INFO [rotate_logs] connected to skookum.eos.ubc.ca port 5555
2020-07-05 04:48:26,555 INFO [rotate_logs] rotating log files