2020-07-02 04:25:50,137 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/nowcast.log
2020-07-02 04:25:50,152 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/nowcast.debug.log
2020-07-02 04:25:50,158 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/checklist.log
2020-07-02 04:25:50,159 INFO [rotate_logs] log files rotated
2020-07-02 04:25:50,165 INFO [manager] checklist updated with [log rotation] items from rotate_logs worker
2020-07-02 09:58:40,123 INFO [collect_weather] termination signal (SIGTERM) received; shutting down
2020-07-02 09:58:52,686 INFO [collect_weather] running in process 27674
2020-07-02 09:58:52,687 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-02 09:58:52,688 INFO [collect_weather] publishing log messages to tcp://*:5562
2020-07-02 09:58:52,689 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2020-07-02 09:58:52,731 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-west/18/
2020-07-02 09:59:16,927 INFO [download_weather] running in process 27688
2020-07-02 09:59:16,928 INFO [download_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-02 09:59:16,929 INFO [download_weather] publishing log messages to tcp://*:5563
2020-07-02 09:59:16,930 INFO [download_weather] connected to skookum.eos.ubc.ca port 5555
2020-07-02 09:59:16,931 INFO [download_weather] downloading 12 2.5 km forecast GRIB2 files for 20200702
2020-07-02 10:04:06,211 INFO [download_weather] 2020-07-02 2.5km weather forecast 12 downloads complete
2020-07-02 10:04:06,215 INFO [manager] checklist updated with [weather forecast] items from download_weather worker
2020-07-02 10:04:06,222 INFO [manager] launching NextWorker(module='nowcast.workers.get_NeahBay_ssh', args=['nowcast'], host='localhost')
2020-07-02 10:04:06,228 INFO [manager] launching NextWorker(module='nowcast.workers.grib_to_netcdf', args=['nowcast+'], host='localhost')
2020-07-02 10:04:06,232 INFO [manager] launching NextWorker(module='nowcast.workers.download_live_ocean', args=[], host='localhost')
2020-07-02 10:04:07,558 INFO [grib_to_netcdf] running in process 27895
2020-07-02 10:04:07,559 INFO [grib_to_netcdf] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-02 10:04:07,560 INFO [grib_to_netcdf] publishing log messages to tcp://*:5563
2020-07-02 10:04:07,561 INFO [grib_to_netcdf] connected to skookum.eos.ubc.ca port 5555
2020-07-02 10:04:07,664 INFO [download_live_ocean] running in process 27896
2020-07-02 10:04:07,665 INFO [download_live_ocean] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-02 10:04:07,666 INFO [download_live_ocean] publishing log messages to tcp://*:5564
2020-07-02 10:04:07,667 INFO [download_live_ocean] connected to skookum.eos.ubc.ca port 5555
2020-07-02 10:04:07,667 INFO [download_live_ocean] downloading Salish Sea western boundary daily averaged Live Ocean file for 2020-07-02
2020-07-02 10:04:08,275 INFO [get_NeahBay_ssh] running in process 27894
2020-07-02 10:04:08,276 INFO [get_NeahBay_ssh] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-02 10:04:08,282 INFO [get_NeahBay_ssh] publishing log messages to tcp://*:5565
2020-07-02 10:04:08,283 INFO [get_NeahBay_ssh] connected to skookum.eos.ubc.ca port 5555
2020-07-02 10:04:11,370 INFO [download_live_ocean] Deflating in up to 6 concurrent sub-processes
2020-07-02 10:04:12,085 INFO [get_NeahBay_ssh] Neah Bay sea surface height web scraping and open boundary file creation for nowcast complete
2020-07-02 10:04:12,089 INFO [manager] checklist updated with [Neah Bay ssh] items from get_NeahBay_ssh worker
2020-07-02 10:04:12,433 INFO [download_live_ocean] netCDF4 deflated /results/forcing/LiveOcean/downloaded/20200702/low_passed_UBC.nc
2020-07-02 10:04:12,434 INFO [download_live_ocean] 2020-07-02 Live Ocean file for Salish Sea western boundary downloaded
2020-07-02 10:04:12,436 INFO [manager] checklist updated with [Live Ocean products] items from download_live_ocean worker
2020-07-02 10:04:12,723 INFO [manager] launching NextWorker(module='nowcast.workers.make_live_ocean_files', args=['--run-date', '2020-07-02'], host='localhost')
2020-07-02 10:04:15,312 INFO [make_live_ocean_files] running in process 27960
2020-07-02 10:04:15,313 INFO [make_live_ocean_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-02 10:04:15,314 INFO [make_live_ocean_files] publishing log messages to tcp://*:5564
2020-07-02 10:04:15,315 INFO [make_live_ocean_files] connected to skookum.eos.ubc.ca port 5555
2020-07-02 10:04:15,316 INFO [make_live_ocean_files] Creating T&S western boundary conditions file from 2020-07-02 Live Ocean run
2020-07-02 10:05:21,212 INFO [make_live_ocean_files] Stored T&S western boundary conditions file: /
2020-07-02 10:05:21,213 INFO [make_live_ocean_files] 2020-07-02 Live Ocean western boundary conditions files created
2020-07-02 10:05:21,215 INFO [manager] checklist updated with [Live Ocean boundary conditions] items from make_live_ocean_files worker
2020-07-02 10:05:53,275 INFO [grib_to_netcdf] 2020-07-02 NEMO-atmos forcing file for nowcast+ created
2020-07-02 10:05:53,279 INFO [manager] checklist updated with [weather forcing] items from grib_to_netcdf worker
2020-07-02 10:05:53,286 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'nowcast+'], host='localhost')
2020-07-02 10:05:53,291 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'nowcast+'], host='localhost')
2020-07-02 10:05:53,295 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['graham-hindcast', 'nowcast+'], host='localhost')
2020-07-02 10:05:53,298 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'nowcast+'], host='localhost')
2020-07-02 10:05:53,302 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['weather'], host='localhost')
2020-07-02 10:05:54,234 INFO [ping_erddap] running in process 29313
2020-07-02 10:05:54,235 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-02 10:05:54,236 INFO [ping_erddap] publishing log messages to tcp://*:5563
2020-07-02 10:05:54,237 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2020-07-02 10:05:54,238 INFO [ping_erddap] weather ERDDAP dataset flag file(s) created
2020-07-02 10:05:54,240 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2020-07-02 10:05:54,290 INFO [upload_forcing] running in process 29312
2020-07-02 10:05:54,291 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-02 10:05:54,292 INFO [upload_forcing] publishing log messages to tcp://*:5564
2020-07-02 10:05:54,293 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2020-07-02 10:05:54,297 INFO [upload_forcing] running in process 29311
2020-07-02 10:05:54,298 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-02 10:05:54,299 INFO [upload_forcing] publishing log messages to tcp://*:5565
2020-07-02 10:05:54,300 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2020-07-02 10:05:54,312 INFO [upload_forcing] running in process 29309
2020-07-02 10:05:54,312 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-02 10:05:54,313 INFO [upload_forcing] publishing log messages to tcp://*:5566
2020-07-02 10:05:54,314 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2020-07-02 10:05:54,315 INFO [upload_forcing] running in process 29310
2020-07-02 10:05:54,316 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-02 10:05:54,316 INFO [upload_forcing] publishing log messages to tcp://*:5567
2020-07-02 10:05:54,317 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2020-07-02 10:05:58,526 INFO [upload_forcing] nowcast+ 2020-07-02 forcing files upload to optimum-hindcast completed
2020-07-02 10:05:58,530 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2020-07-02 10:05:58,786 INFO [upload_forcing] nowcast+ 2020-07-02 forcing files upload to orcinus-nowcast-agrif completed
2020-07-02 10:05:58,789 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2020-07-02 10:05:59,424 INFO [upload_forcing] nowcast+ 2020-07-02 forcing files upload to arbutus.cloud-nowcast completed
2020-07-02 10:05:59,429 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2020-07-02 10:05:59,437 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'nowcast+', '--run-date', '2020-07-02'], host='localhost')
2020-07-02 10:06:00,373 INFO [make_forcing_links] running in process 29345
2020-07-02 10:06:00,374 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-02 10:06:00,375 INFO [make_forcing_links] publishing log messages to tcp://*:5563
2020-07-02 10:06:00,376 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2020-07-02 10:06:01,591 INFO [make_forcing_links] nowcast+ 2020-07-02 forcing file links on arbutus.cloud-nowcast created
2020-07-02 10:06:01,595 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2020-07-02 10:06:01,602 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2020-07-02'], host='arbutus.cloud-nowcast')
2020-07-02 10:06:03,894 INFO [run_NEMO] running in process 8368
2020-07-02 10:06:03,895 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2020-07-02 10:06:03,896 INFO [run_NEMO] publishing log messages to tcp://*:5556
2020-07-02 10:06:03,950 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2020-07-02 10:06:04,862 INFO [run_NEMO] nowcast: launching /nemoShare/MEOPAR/nowcast-sys/runs/02jul20nowcast_2020-07-02T100603.959617-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2020-07-02 10:06:04,877 INFO [run_NEMO] nowcast NEMO run for 2020-07-02 on arbutus.cloud-nowcast started
2020-07-02 10:06:04,884 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2020-07-02 10:06:04,894 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'nowcast'], host='arbutus.cloud-nowcast')
2020-07-02 10:06:05,713 INFO [upload_forcing] nowcast+ 2020-07-02 forcing files upload to graham-hindcast completed
2020-07-02 10:06:05,720 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2020-07-02 10:06:06,858 INFO [watch_NEMO] running in process 8632
2020-07-02 10:06:06,859 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2020-07-02 10:06:06,860 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2020-07-02 10:06:06,861 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2020-07-02 10:06:06,961 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2020-07-02 10:11:07,064 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 7493469 = 2020-07-02 04:45:20 UTC, 19.8% complete
2020-07-02 10:16:07,165 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 7493969 = 2020-07-02 10:18:40 UTC, 43.0% complete
2020-07-02 10:21:07,267 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 7494466 = 2020-07-02 15:50:00 UTC, 66.0% complete
2020-07-02 10:26:07,369 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 7494957 = 2020-07-02 21:17:20 UTC, 88.7% complete
2020-07-02 10:31:07,461 INFO [watch_NEMO] nowcast NEMO run on arbutus.cloud-nowcast completed
2020-07-02 10:31:07,466 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2020-07-02 10:31:07,748 INFO [manager] launching NextWorker(module='nowcast.workers.get_NeahBay_ssh', args=['forecast'], host='localhost')
2020-07-02 10:31:07,754 INFO [manager] launching NextWorker(module='nowcast.workers.make_fvcom_boundary', args=['arbutus.cloud-nowcast', 'x2', 'nowcast'], host='arbutus.cloud-nowcast')
2020-07-02 10:31:07,761 INFO [manager] launching NextWorker(module='nowcast.workers.make_fvcom_boundary', args=['arbutus.cloud-nowcast', 'r12', 'nowcast'], host='arbutus.cloud-nowcast')
2020-07-02 10:31:07,765 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2020-07-02'], host='localhost')
2020-07-02 10:31:08,793 INFO [download_results] running in process 29803
2020-07-02 10:31:08,793 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-02 10:31:08,794 INFO [download_results] publishing log messages to tcp://*:5563
2020-07-02 10:31:08,795 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2020-07-02 10:31:08,796 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/nowcast/02jul20 to /results/SalishSea/nowcast-blue.201905
2020-07-02 10:31:10,192 INFO [get_NeahBay_ssh] running in process 29800
2020-07-02 10:31:10,194 INFO [get_NeahBay_ssh] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-02 10:31:10,195 INFO [get_NeahBay_ssh] publishing log messages to tcp://*:5564
2020-07-02 10:31:10,196 INFO [get_NeahBay_ssh] connected to skookum.eos.ubc.ca port 5555
2020-07-02 10:31:12,098 INFO [make_fvcom_boundary] running in process 8927
2020-07-02 10:31:12,099 INFO [make_fvcom_boundary] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2020-07-02 10:31:12,100 INFO [make_fvcom_boundary] publishing log messages to tcp://*:5580
2020-07-02 10:31:12,101 INFO [make_fvcom_boundary] connected to skookum.eos.ubc.ca port 5555
2020-07-02 10:31:12,103 INFO [make_fvcom_boundary] Creating VHFR FVCOM open boundary file for r12 nowcast run from 2020-07-02 NEMO run
2020-07-02 10:31:12,104 INFO [make_fvcom_boundary] running in process 8926
2020-07-02 10:31:12,105 INFO [make_fvcom_boundary] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2020-07-02 10:31:12,106 INFO [make_fvcom_boundary] publishing log messages to tcp://*:5581
2020-07-02 10:31:12,107 INFO [make_fvcom_boundary] connected to skookum.eos.ubc.ca port 5555
2020-07-02 10:31:12,108 INFO [make_fvcom_boundary] Creating VHFR FVCOM open boundary file for x2 nowcast run from 2020-07-02 NEMO run
2020-07-02 10:31:14,241 INFO [get_NeahBay_ssh] Neah Bay sea surface height web scraping and open boundary file creation for forecast complete
2020-07-02 10:31:14,246 INFO [manager] checklist updated with [Neah Bay ssh] items from get_NeahBay_ssh worker
2020-07-02 10:31:14,257 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'ssh'], host='localhost')
2020-07-02 10:31:15,356 INFO [upload_forcing] running in process 29846
2020-07-02 10:31:15,357 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-02 10:31:15,359 INFO [upload_forcing] publishing log messages to tcp://*:5564
2020-07-02 10:31:15,360 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2020-07-02 10:31:16,559 INFO [upload_forcing] ssh 2020-07-02 forcing files upload to arbutus.cloud-nowcast completed
2020-07-02 10:31:16,565 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2020-07-02 10:31:16,576 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'ssh', '--run-date', '2020-07-02'], host='localhost')
2020-07-02 10:31:17,623 INFO [make_forcing_links] running in process 29852
2020-07-02 10:31:17,624 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-02 10:31:17,626 INFO [make_forcing_links] publishing log messages to tcp://*:5564
2020-07-02 10:31:17,627 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2020-07-02 10:31:18,747 INFO [make_forcing_links] ssh 2020-07-02 forcing file links on arbutus.cloud-nowcast created
2020-07-02 10:31:18,752 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2020-07-02 10:31:18,764 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2020-07-02'], host='arbutus.cloud-nowcast')
2020-07-02 10:31:21,028 INFO [run_NEMO] running in process 9286
2020-07-02 10:31:21,029 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2020-07-02 10:31:21,030 INFO [run_NEMO] publishing log messages to tcp://*:5556
2020-07-02 10:31:21,031 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2020-07-02 10:31:22,081 INFO [run_NEMO] forecast: launching /nemoShare/MEOPAR/nowcast-sys/runs/02jul20forecast_2020-07-02T103121.114465-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2020-07-02 10:31:22,102 INFO [run_NEMO] forecast NEMO run for 2020-07-02 on arbutus.cloud-nowcast started
2020-07-02 10:31:22,107 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2020-07-02 10:31:22,121 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'forecast'], host='arbutus.cloud-nowcast')
2020-07-02 10:31:24,164 INFO [watch_NEMO] running in process 9551
2020-07-02 10:31:24,165 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2020-07-02 10:31:24,167 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2020-07-02 10:31:24,168 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2020-07-02 10:31:24,259 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2020-07-02 10:33:19,850 INFO [download_results] nowcast 2020-07-02 results files from arbutus.cloud-nowcast downloaded
2020-07-02 10:33:19,857 INFO [manager] checklist updated with [results files] items from download_results worker
2020-07-02 10:33:19,872 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast', 'research', '--run-date', '2020-07-02'], host='localhost')
2020-07-02 10:33:19,878 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast', 'comparison', '--run-date', '2020-07-01'], host='localhost')
2020-07-02 10:33:19,883 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['nowcast', '--run-date', '2020-07-02'], host='localhost')
2020-07-02 10:33:22,524 INFO [make_CHS_currents_file] running in process 29915
2020-07-02 10:33:22,525 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-02 10:33:22,526 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5563
2020-07-02 10:33:22,527 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555
2020-07-02 10:33:23,762 INFO [make_plots] running in process 29914
2020-07-02 10:33:23,763 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-02 10:33:23,764 INFO [make_plots] publishing log messages to tcp://*:5564
2020-07-02 10:33:23,770 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2020-07-02 10:33:23,796 INFO [make_plots] running in process 29913
2020-07-02 10:33:23,838 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-02 10:33:23,839 INFO [make_plots] publishing log messages to tcp://*:5565
2020-07-02 10:33:23,844 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2020-07-02 10:33:24,253 INFO [make_plots] preparing render list for 2020-07-01 NEMO nowcast-blue comparison figures
2020-07-02 10:33:24,282 INFO [make_plots] preparing render list for 2020-07-02 NEMO nowcast-blue research figures
2020-07-02 10:33:25,783 INFO [make_plots] added salinity_thalweg_and_surface figures to 2020-07-02 NEMO nowcast-blue research render list
2020-07-02 10:33:27,268 INFO [make_plots] added temperature_thalweg_and_surface figures to 2020-07-02 NEMO nowcast-blue research render list
2020-07-02 10:33:27,269 INFO [make_plots] added Currents_sections_and_surface figure to 2020-07-02 NEMO nowcast-blue research render list
2020-07-02 10:33:27,271 INFO [make_plots] added Currents_at_VENUS_Central figure to 2020-07-02 NEMO nowcast-blue research render list
2020-07-02 10:33:27,272 INFO [make_plots] added Currents_at_VENUS_East figure to 2020-07-02 NEMO nowcast-blue research render list
2020-07-02 10:33:27,273 INFO [make_plots] starting to render nemo nowcast research 02jul20 figures
2020-07-02 10:33:35,319 INFO [make_plots] added SH_wind figure to 2020-07-01 NEMO nowcast-blue comparison render list
2020-07-02 10:33:35,320 INFO [make_plots] added Compare_VENUS_East figure to 2020-07-01 NEMO nowcast-blue comparison render list
2020-07-02 10:33:35,321 INFO [make_plots] added Compare_VENUS_Central figure to 2020-07-01 NEMO nowcast-blue comparison render list
2020-07-02 10:33:35,321 INFO [make_plots] added Compare_VENUS_Delta_DDL figure to 2020-07-01 NEMO nowcast-blue comparison render list
2020-07-02 10:33:35,322 INFO [make_plots] added Central_ADCP figure to 2020-07-01 NEMO nowcast-blue comparison render list
2020-07-02 10:33:35,323 INFO [make_plots] added Central_depavADCP figure to 2020-07-01 NEMO nowcast-blue comparison render list
2020-07-02 10:33:35,324 INFO [make_plots] added Central_timeavADCP figure to 2020-07-01 NEMO nowcast-blue comparison render list
2020-07-02 10:33:35,324 INFO [make_plots] added East_ADCP figure to 2020-07-01 NEMO nowcast-blue comparison render list
2020-07-02 10:33:35,325 INFO [make_plots] added East_depavADCP figure to 2020-07-01 NEMO nowcast-blue comparison render list
2020-07-02 10:33:35,326 INFO [make_plots] added East_timeavADCP figure to 2020-07-01 NEMO nowcast-blue comparison render list
2020-07-02 10:33:35,326 INFO [make_plots] added ddl_ADCP figure to 2020-07-01 NEMO nowcast-blue comparison render list
2020-07-02 10:33:35,327 INFO [make_plots] added ddl_depavADCP figure to 2020-07-01 NEMO nowcast-blue comparison render list
2020-07-02 10:33:35,328 INFO [make_plots] added ddl_timeavADCP figure to 2020-07-01 NEMO nowcast-blue comparison render list
2020-07-02 10:33:35,329 INFO [make_plots] starting to render nemo nowcast comparison 01jul20 figures
2020-07-02 10:33:36,522 INFO [make_fvcom_boundary] Stored VHFR FVCOM open boundary file: /nemoShare/MEOPAR/nowcast-sys/fvcom-runs/input.r12/bdy_r12_nowcast_brcl_20200702.nc
2020-07-02 10:33:36,526 INFO [make_fvcom_boundary] FVCOM r12 nowcast run boundary condition file for 2020-07-02 created on arbutus.cloud-nowcast
2020-07-02 10:33:36,533 INFO [manager] checklist updated with [FVCOM boundary conditions] items from make_fvcom_boundary worker
2020-07-02 10:33:36,541 INFO [manager] launching NextWorker(module='nowcast.workers.make_fvcom_atmos_forcing', args=['r12', 'nowcast', '--run-date', '2020-07-02'], host='localhost')
2020-07-02 10:33:36,599 INFO [manager] launching NextWorker(module='nowcast.workers.make_fvcom_rivers_forcing', args=['arbutus.cloud-nowcast', 'r12', 'nowcast', '--run-date', '2020-07-02'], host='arbutus.cloud-nowcast')
2020-07-02 10:33:39,761 INFO [make_fvcom_atmos_forcing] running in process 29944
2020-07-02 10:33:39,762 INFO [make_fvcom_atmos_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-02 10:33:39,763 INFO [make_fvcom_atmos_forcing] publishing log messages to tcp://*:5566
2020-07-02 10:33:39,764 INFO [make_fvcom_atmos_forcing] connected to skookum.eos.ubc.ca port 5555
2020-07-02 10:33:39,765 INFO [make_fvcom_atmos_forcing] Creating VHFR FVCOM atmospheric forcing file for r12 nowcast run from 2020-07-02 HRDPS GRIB files
2020-07-02 10:33:40,642 INFO [make_fvcom_rivers_forcing] running in process 9646
2020-07-02 10:33:40,643 INFO [make_fvcom_rivers_forcing] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2020-07-02 10:33:40,644 INFO [make_fvcom_rivers_forcing] publishing log messages to tcp://*:5582
2020-07-02 10:33:40,646 INFO [make_fvcom_rivers_forcing] connected to skookum.eos.ubc.ca port 5555
2020-07-02 10:33:40,647 INFO [make_fvcom_rivers_forcing] Creating VHFR FVCOM rivers forcing file for r12 nowcast run from 2020-07-02 NEMO runoff forcing files
2020-07-02 10:33:40,867 INFO [make_fvcom_rivers_forcing] Stored VHFR FVCOM rivers forcing file: /nemoShare/MEOPAR/nowcast-sys/fvcom-runs/input.r12/rivers_r12_nowcast_20200702.nc
2020-07-02 10:33:40,868 INFO [make_fvcom_rivers_forcing] FVCOM r12 nowcast run rivers forcing file for 2020-07-02 created on arbutus.cloud-nowcast
2020-07-02 10:33:40,873 INFO [manager] checklist updated with [FVCOM rivers forcing] items from make_fvcom_rivers_forcing worker
2020-07-02 10:33:43,157 INFO [make_fvcom_boundary] Stored VHFR FVCOM open boundary file: /nemoShare/MEOPAR/nowcast-sys/fvcom-runs/input.x2/bdy_x2_nowcast_brcl_20200702.nc
2020-07-02 10:33:43,164 INFO [make_fvcom_boundary] FVCOM x2 nowcast run boundary condition file for 2020-07-02 created on arbutus.cloud-nowcast
2020-07-02 10:33:43,168 INFO [manager] checklist updated with [FVCOM boundary conditions] items from make_fvcom_boundary worker
2020-07-02 10:33:43,181 INFO [manager] launching NextWorker(module='nowcast.workers.make_fvcom_atmos_forcing', args=['x2', 'nowcast', '--run-date', '2020-07-02'], host='localhost')
2020-07-02 10:33:43,184 INFO [manager] launching NextWorker(module='nowcast.workers.make_fvcom_rivers_forcing', args=['arbutus.cloud-nowcast', 'x2', 'nowcast', '--run-date', '2020-07-02'], host='arbutus.cloud-nowcast')
2020-07-02 10:33:45,656 INFO [make_fvcom_atmos_forcing] running in process 29964
2020-07-02 10:33:45,657 INFO [make_fvcom_atmos_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-02 10:33:45,658 INFO [make_fvcom_atmos_forcing] publishing log messages to tcp://*:5567
2020-07-02 10:33:45,659 INFO [make_fvcom_atmos_forcing] connected to skookum.eos.ubc.ca port 5555
2020-07-02 10:33:45,660 INFO [make_fvcom_atmos_forcing] Creating VHFR FVCOM atmospheric forcing file for x2 nowcast run from 2020-07-02 HRDPS GRIB files
2020-07-02 10:33:47,364 INFO [make_fvcom_rivers_forcing] running in process 9739
2020-07-02 10:33:47,366 INFO [make_fvcom_rivers_forcing] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2020-07-02 10:33:47,367 INFO [make_fvcom_rivers_forcing] publishing log messages to tcp://*:5582
2020-07-02 10:33:47,368 INFO [make_fvcom_rivers_forcing] connected to skookum.eos.ubc.ca port 5555
2020-07-02 10:33:47,369 INFO [make_fvcom_rivers_forcing] Creating VHFR FVCOM rivers forcing file for x2 nowcast run from 2020-07-02 NEMO runoff forcing files
2020-07-02 10:33:47,617 INFO [make_fvcom_rivers_forcing] Stored VHFR FVCOM rivers forcing file: /nemoShare/MEOPAR/nowcast-sys/fvcom-runs/input.x2/rivers_x2_nowcast_20200702.nc
2020-07-02 10:33:47,618 INFO [make_fvcom_rivers_forcing] FVCOM x2 nowcast run rivers forcing file for 2020-07-02 created on arbutus.cloud-nowcast
2020-07-02 10:33:47,619 INFO [manager] checklist updated with [FVCOM rivers forcing] items from make_fvcom_rivers_forcing worker
2020-07-02 10:34:05,353 INFO [make_fvcom_atmos_forcing] Stored VHFR FVCOM atmospheric forcing hfx file: /results/forcing/atmospheric/GEM2.5/vhfr-fvcom/atmos_r12_nowcast_hfx_20200702.nc
2020-07-02 10:34:06,811 INFO [make_CHS_currents_file] Made CHS currents file for 2020-07-02 for nowcast
2020-07-02 10:34:06,813 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker
2020-07-02 10:34:07,562 INFO [make_fvcom_atmos_forcing] Stored VHFR FVCOM atmospheric forcing hfx file: /results/forcing/atmospheric/GEM2.5/vhfr-fvcom/atmos_x2_nowcast_hfx_20200702.nc
2020-07-02 10:34:13,278 INFO [make_fvcom_atmos_forcing] Stored VHFR FVCOM atmospheric forcing precip file: /results/forcing/atmospheric/GEM2.5/vhfr-fvcom/atmos_r12_nowcast_precip_20200702.nc
2020-07-02 10:34:14,307 INFO [make_fvcom_atmos_forcing] Stored VHFR FVCOM atmospheric forcing precip file: /results/forcing/atmospheric/GEM2.5/vhfr-fvcom/atmos_x2_nowcast_precip_20200702.nc
2020-07-02 10:34:23,636 INFO [make_fvcom_atmos_forcing] Stored VHFR FVCOM atmospheric forcing wnd file: /results/forcing/atmospheric/GEM2.5/vhfr-fvcom/atmos_x2_nowcast_wnd_20200702.nc
2020-07-02 10:34:23,637 INFO [make_fvcom_atmos_forcing] FVCOM x2 nowcast run atmospheric forcing file for 2020-07-02 created
2020-07-02 10:34:23,640 INFO [manager] checklist updated with [FVCOM atmospheric forcing] items from make_fvcom_atmos_forcing worker
2020-07-02 10:34:23,649 INFO [manager] launching NextWorker(module='nowcast.workers.upload_fvcom_atmos_forcing', args=['arbutus.cloud-nowcast', 'x2', 'nowcast', '--run-date', '2020-07-02'], host='localhost')
2020-07-02 10:34:24,113 INFO [make_fvcom_atmos_forcing] Stored VHFR FVCOM atmospheric forcing wnd file: /results/forcing/atmospheric/GEM2.5/vhfr-fvcom/atmos_r12_nowcast_wnd_20200702.nc
2020-07-02 10:34:24,114 INFO [make_fvcom_atmos_forcing] FVCOM r12 nowcast run atmospheric forcing file for 2020-07-02 created
2020-07-02 10:34:24,119 INFO [manager] checklist updated with [FVCOM atmospheric forcing] items from make_fvcom_atmos_forcing worker
2020-07-02 10:34:24,134 INFO [manager] launching NextWorker(module='nowcast.workers.upload_fvcom_atmos_forcing', args=['arbutus.cloud-nowcast', 'r12', 'nowcast', '--run-date', '2020-07-02'], host='localhost')
2020-07-02 10:34:24,702 INFO [upload_fvcom_atmos_forcing] running in process 29972
2020-07-02 10:34:24,703 INFO [upload_fvcom_atmos_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-02 10:34:24,704 INFO [upload_fvcom_atmos_forcing] publishing log messages to tcp://*:5563
2020-07-02 10:34:24,705 INFO [upload_fvcom_atmos_forcing] connected to skookum.eos.ubc.ca port 5555
2020-07-02 10:34:24,706 INFO [upload_fvcom_atmos_forcing] Uploading VHFR FVCOM atmospheric forcing files for 2020-07-02 x2 nowcast run to arbutus.cloud-nowcast
2020-07-02 10:34:25,043 INFO [upload_fvcom_atmos_forcing] running in process 29974
2020-07-02 10:34:25,044 INFO [upload_fvcom_atmos_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-02 10:34:25,045 INFO [upload_fvcom_atmos_forcing] publishing log messages to tcp://*:5566
2020-07-02 10:34:25,046 INFO [upload_fvcom_atmos_forcing] connected to skookum.eos.ubc.ca port 5555
2020-07-02 10:34:25,047 INFO [upload_fvcom_atmos_forcing] Uploading VHFR FVCOM atmospheric forcing files for 2020-07-02 r12 nowcast run to arbutus.cloud-nowcast
2020-07-02 10:34:29,775 INFO [upload_fvcom_atmos_forcing] FVCOM x2 nowcast run atmospheric forcing files for 2020-07-02 uploaded to arbutus.cloud-nowcast
2020-07-02 10:34:29,778 INFO [manager] checklist updated with [FVCOM atmospheric forcing upload] items from upload_fvcom_atmos_forcing worker
2020-07-02 10:34:29,790 INFO [manager] launching NextWorker(module='nowcast.workers.run_fvcom', args=['arbutus.cloud-nowcast', 'x2', 'nowcast', '--run-date', '2020-07-02'], host='arbutus.cloud-nowcast')
2020-07-02 10:34:30,976 INFO [upload_fvcom_atmos_forcing] FVCOM r12 nowcast run atmospheric forcing files for 2020-07-02 uploaded to arbutus.cloud-nowcast
2020-07-02 10:34:30,980 INFO [manager] checklist updated with [FVCOM atmospheric forcing upload] items from upload_fvcom_atmos_forcing worker
2020-07-02 10:34:30,990 INFO [manager] launching NextWorker(module='nowcast.workers.run_fvcom', args=['arbutus.cloud-nowcast', 'r12', 'nowcast', '--run-date', '2020-07-02'], host='arbutus.cloud-nowcast')
2020-07-02 10:34:32,523 INFO [run_fvcom] running in process 10361
2020-07-02 10:34:32,524 INFO [run_fvcom] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2020-07-02 10:34:32,525 INFO [run_fvcom] publishing log messages to tcp://*:5584
2020-07-02 10:34:32,526 INFO [run_fvcom] connected to skookum.eos.ubc.ca port 5555
2020-07-02 10:34:32,582 INFO [run_fvcom] nowcast: launching /nemoShare/MEOPAR/nowcast-sys/fvcom-runs/20200702-17h34m32.570812s/VHFR_FVCOM.sh on arbutus.cloud-nowcast
2020-07-02 10:34:32,615 INFO [run_fvcom] x2 nowcast FVCOM VH-FR run for 2020-07-02 on arbutus.cloud-nowcast started
2020-07-02 10:34:32,620 INFO [manager] checklist updated with [FVCOM run] items from run_fvcom worker
2020-07-02 10:34:32,630 INFO [manager] launching NextWorker(module='nowcast.workers.watch_fvcom', args=['arbutus.cloud-nowcast', 'x2', 'nowcast'], host='arbutus.cloud-nowcast')
2020-07-02 10:34:33,489 INFO [run_fvcom] running in process 10432
2020-07-02 10:34:33,490 INFO [run_fvcom] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2020-07-02 10:34:33,491 INFO [run_fvcom] publishing log messages to tcp://*:5584
2020-07-02 10:34:33,492 INFO [run_fvcom] connected to skookum.eos.ubc.ca port 5555
2020-07-02 10:34:33,545 INFO [run_fvcom] nowcast: launching /nemoShare/MEOPAR/nowcast-sys/fvcom-runs/20200702-17h34m33.533541s/VHFR_FVCOM.sh on arbutus.cloud-nowcast
2020-07-02 10:34:33,571 INFO [run_fvcom] r12 nowcast FVCOM VH-FR run for 2020-07-02 on arbutus.cloud-nowcast started
2020-07-02 10:34:33,578 INFO [manager] checklist updated with [FVCOM run] items from run_fvcom worker
2020-07-02 10:34:33,596 INFO [manager] launching NextWorker(module='nowcast.workers.watch_fvcom', args=['arbutus.cloud-nowcast', 'r12', 'nowcast'], host='arbutus.cloud-nowcast')
2020-07-02 10:34:34,932 INFO [watch_fvcom] running in process 10568
2020-07-02 10:34:34,933 INFO [watch_fvcom] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2020-07-02 10:34:34,934 INFO [watch_fvcom] publishing log messages to tcp://*:5586
2020-07-02 10:34:34,935 INFO [watch_fvcom] connected to skookum.eos.ubc.ca port 5555
2020-07-02 10:34:34,981 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: no run progress found in fvcom.log ; continuing to watch...
2020-07-02 10:34:35,852 INFO [watch_fvcom] running in process 10640
2020-07-02 10:34:35,853 INFO [watch_fvcom] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2020-07-02 10:34:35,854 INFO [watch_fvcom] publishing log messages to tcp://*:5587
2020-07-02 10:34:35,855 INFO [watch_fvcom] connected to skookum.eos.ubc.ca port 5555
2020-07-02 10:34:35,901 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: no run progress found in fvcom.log ; continuing to watch...
2020-07-02 10:35:26,747 INFO [make_plots] finished rendering nemo nowcast research 02jul20 figures
2020-07-02 10:35:26,750 INFO [make_plots] nemo research plots for 2020-07-02 nowcast completed
2020-07-02 10:35:26,779 INFO [manager] checklist updated with [plots] items from make_plots worker
2020-07-02 10:35:44,402 WARNING [make_plots] VENUS Delta DDL node CTD comparison figure failed: No observations available
2020-07-02 10:35:45,218 WARNING [make_plots] VENUS Central ADCP comparison figure failed: No observations available
2020-07-02 10:35:45,772 WARNING [make_plots] VENUS Central ADCP comparison figure failed: No observations available
2020-07-02 10:35:46,323 WARNING [make_plots] VENUS Central ADCP comparison figure failed: No observations available
2020-07-02 10:35:46,933 WARNING [make_plots] VENUS East ADCP comparison figure failed: No observations available
2020-07-02 10:35:47,484 WARNING [make_plots] VENUS East ADCP comparison figure failed: No observations available
2020-07-02 10:35:48,034 WARNING [make_plots] VENUS East ADCP comparison figure failed: No observations available
2020-07-02 10:35:48,613 WARNING [make_plots] VENUS ddl ADCP comparison figure failed: No observations available
2020-07-02 10:35:49,166 WARNING [make_plots] VENUS ddl ADCP comparison figure failed: No observations available
2020-07-02 10:35:49,719 WARNING [make_plots] VENUS ddl ADCP comparison figure failed: No observations available
2020-07-02 10:35:49,720 INFO [make_plots] finished rendering nemo nowcast comparison 01jul20 figures
2020-07-02 10:35:49,801 INFO [make_plots] nemo comparison plots for 2020-07-01 nowcast completed
2020-07-02 10:35:49,806 INFO [manager] checklist updated with [plots] items from make_plots worker
2020-07-02 10:36:24,360 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 7495624 = 2020-07-03 04:42:00 UTC, 13.1% complete
2020-07-02 10:39:35,086 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15641984 = 2020-07-02 00:29:44 UTC estimated time to finish: 0000:03:28:01
2020-07-02 10:39:36,003 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39101610 = 2020-07-02 00:07:24 UTC estimated time to finish: 0000:13:51:03
2020-07-02 10:41:24,461 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 7496106 = 2020-07-03 10:03:20 UTC, 27.9% complete
2020-07-02 10:44:35,189 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15644030 = 2020-07-02 01:03:50 UTC estimated time to finish: 0000:03:23:38
2020-07-02 10:44:36,105 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39102989 = 2020-07-02 00:16:35 UTC estimated time to finish: 0000:13:38:13
2020-07-02 10:46:24,562 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 7496570 = 2020-07-03 15:12:40 UTC, 42.3% complete
2020-07-02 10:49:35,294 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15646115 = 2020-07-02 01:38:35 UTC estimated time to finish: 0000:03:17:34
2020-07-02 10:49:36,208 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39104327 = 2020-07-02 00:25:30 UTC estimated time to finish: 0000:13:33:13
2020-07-02 10:51:24,663 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 7497047 = 2020-07-03 20:30:40 UTC, 57.0% complete
2020-07-02 10:54:35,401 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15648241 = 2020-07-02 02:14:01 UTC estimated time to finish: 0000:03:12:00
2020-07-02 10:54:36,309 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39105516 = 2020-07-02 00:33:26 UTC estimated time to finish: 0000:13:29:58
2020-07-02 10:56:24,764 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 7497485 = 2020-07-04 01:22:40 UTC, 70.5% complete
2020-07-02 10:59:35,424 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15650369 = 2020-07-02 02:49:29 UTC estimated time to finish: 0000:03:06:15
2020-07-02 10:59:36,411 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39106895 = 2020-07-02 00:42:38 UTC estimated time to finish: 0000:13:16:59
2020-07-02 11:01:24,866 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 7497975 = 2020-07-04 06:49:20 UTC, 85.6% complete
2020-07-02 11:04:35,532 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15652221 = 2020-07-02 03:20:21 UTC estimated time to finish: 0000:03:02:01
2020-07-02 11:04:36,510 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39108315 = 2020-07-02 00:52:06 UTC estimated time to finish: 0000:13:03:51
2020-07-02 11:06:24,967 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 7498440 = 2020-07-04 11:59:20 UTC, 100.0% complete
2020-07-02 11:09:35,637 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15654267 = 2020-07-02 03:54:27 UTC estimated time to finish: 0000:02:57:19
2020-07-02 11:09:36,614 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39109899 = 2020-07-02 01:02:39 UTC estimated time to finish: 0000:12:40:43
2020-07-02 11:11:25,071 INFO [watch_NEMO] forecast NEMO run on arbutus.cloud-nowcast completed
2020-07-02 11:11:25,076 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2020-07-02 11:11:25,375 INFO [manager] launching NextWorker(module='nowcast.workers.make_turbidity_file', args=['--run-date', '2020-07-02'], host='localhost')
2020-07-02 11:11:25,380 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2020-07-02'], host='localhost')
2020-07-02 11:11:26,461 INFO [download_results] running in process 30878
2020-07-02 11:11:26,462 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-02 11:11:26,463 INFO [download_results] publishing log messages to tcp://*:5563
2020-07-02 11:11:26,464 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2020-07-02 11:11:26,465 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/forecast/02jul20 to /results/SalishSea/forecast.201905
2020-07-02 11:11:26,862 INFO [make_turbidity_file] running in process 30877
2020-07-02 11:11:26,863 INFO [make_turbidity_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-02 11:11:26,864 INFO [make_turbidity_file] publishing log messages to tcp://*:5564
2020-07-02 11:11:26,866 INFO [make_turbidity_file] connected to skookum.eos.ubc.ca port 5555
2020-07-02 11:11:26,867 INFO [make_turbidity_file] Creating Fraser River turbidity forcing file for 2020-07-02
2020-07-02 11:11:27,151 INFO [make_turbidity_file] Note: NumExpr detected 12 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
2020-07-02 11:11:27,152 INFO [make_turbidity_file] NumExpr defaulting to 8 threads.
2020-07-02 11:11:28,639 INFO [make_turbidity_file] 2020-07-02 Fraser River turbidity file creation complete
2020-07-02 11:11:28,642 INFO [manager] checklist updated with [Fraser River turbidity] items from make_turbidity_file worker
2020-07-02 11:11:28,673 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'turbidity'], host='localhost')
2020-07-02 11:11:28,677 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'turbidity'], host='localhost')
2020-07-02 11:11:28,680 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['graham-hindcast', 'turbidity'], host='localhost')
2020-07-02 11:11:28,685 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'turbidity'], host='localhost')
2020-07-02 11:11:29,541 INFO [upload_forcing] running in process 30902
2020-07-02 11:11:29,542 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-02 11:11:29,543 INFO [upload_forcing] publishing log messages to tcp://*:5564
2020-07-02 11:11:29,544 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2020-07-02 11:11:29,552 INFO [upload_forcing] running in process 30903
2020-07-02 11:11:29,553 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-02 11:11:29,554 INFO [upload_forcing] publishing log messages to tcp://*:5565
2020-07-02 11:11:29,554 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2020-07-02 11:11:29,558 INFO [upload_forcing] running in process 30900
2020-07-02 11:11:29,559 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-02 11:11:29,560 INFO [upload_forcing] publishing log messages to tcp://*:5566
2020-07-02 11:11:29,561 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2020-07-02 11:11:29,571 INFO [upload_forcing] running in process 30901
2020-07-02 11:11:29,572 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-02 11:11:29,573 INFO [upload_forcing] publishing log messages to tcp://*:5567
2020-07-02 11:11:29,574 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2020-07-02 11:11:30,033 INFO [upload_forcing] turbidity 2020-07-02 forcing files upload to optimum-hindcast completed
2020-07-02 11:11:30,039 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2020-07-02 11:11:30,131 INFO [upload_forcing] turbidity 2020-07-02 forcing files upload to orcinus-nowcast-agrif completed
2020-07-02 11:11:30,135 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2020-07-02 11:11:30,167 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['orcinus-nowcast-agrif', 'nowcast-agrif', '--run-date', '2020-07-02'], host='localhost')
2020-07-02 11:11:30,753 INFO [upload_forcing] turbidity 2020-07-02 forcing files upload to arbutus.cloud-nowcast completed
2020-07-02 11:11:30,758 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2020-07-02 11:11:30,788 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2020-07-02'], host='localhost')
2020-07-02 11:11:31,227 INFO [make_forcing_links] running in process 30925
2020-07-02 11:11:31,228 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-02 11:11:31,229 INFO [make_forcing_links] publishing log messages to tcp://*:5565
2020-07-02 11:11:31,230 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2020-07-02 11:11:31,754 INFO [make_forcing_links] running in process 30927
2020-07-02 11:11:31,756 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-02 11:11:31,757 INFO [make_forcing_links] publishing log messages to tcp://*:5566
2020-07-02 11:11:31,758 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2020-07-02 11:11:31,789 INFO [make_forcing_links] nowcast-agrif 2020-07-02 forcing file links on orcinus-nowcast-agrif created
2020-07-02 11:11:31,793 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2020-07-02 11:11:31,823 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_agrif', args=['orcinus-nowcast-agrif', 'nowcast-agrif', '--run-date', '2020-07-02'], host='localhost')
2020-07-02 11:11:32,817 INFO [upload_forcing] turbidity 2020-07-02 forcing files upload to graham-hindcast completed
2020-07-02 11:11:32,823 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2020-07-02 11:11:32,843 INFO [run_NEMO_agrif] running in process 30935
2020-07-02 11:11:32,844 INFO [run_NEMO_agrif] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-02 11:11:32,845 INFO [run_NEMO_agrif] publishing log messages to tcp://*:5565
2020-07-02 11:11:32,846 INFO [run_NEMO_agrif] connected to skookum.eos.ubc.ca port 5555
2020-07-02 11:11:32,997 INFO [make_forcing_links] nowcast-green 2020-07-02 forcing file links on arbutus.cloud-nowcast created
2020-07-02 11:11:33,001 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2020-07-02 11:11:33,028 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2020-07-02'], host='arbutus.cloud-nowcast')
2020-07-02 11:11:35,334 INFO [run_NEMO] running in process 11114
2020-07-02 11:11:35,335 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2020-07-02 11:11:35,336 INFO [run_NEMO] publishing log messages to tcp://*:5556
2020-07-02 11:11:35,392 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2020-07-02 11:11:36,376 INFO [run_NEMO] nowcast-green: launching /nemoShare/MEOPAR/nowcast-sys/runs/02jul20nowcast-green_2020-07-02T111135.407149-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2020-07-02 11:11:36,397 INFO [run_NEMO] nowcast-green NEMO run for 2020-07-02 on arbutus.cloud-nowcast started
2020-07-02 11:11:36,404 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2020-07-02 11:11:36,436 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'nowcast-green'], host='arbutus.cloud-nowcast')
2020-07-02 11:11:38,429 INFO [watch_NEMO] running in process 11379
2020-07-02 11:11:38,430 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2020-07-02 11:11:38,431 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2020-07-02 11:11:38,432 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2020-07-02 11:11:38,544 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2020-07-02 11:11:54,116 INFO [run_NEMO_agrif] job id for 02jul20nowcast-agrif: 10846109.orca2.ibb
2020-07-02 11:11:54,117 INFO [run_NEMO_agrif] NEMO nowcast-agrif run for 2020-07-02 queued on orcinus-nowcast-agrif
2020-07-02 11:11:54,123 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_agrif worker
2020-07-02 11:11:54,157 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_agrif', args=['orcinus-nowcast-agrif', '10846109.orca2.ibb'], host='localhost')
2020-07-02 11:11:55,209 INFO [watch_NEMO_agrif] running in process 30962
2020-07-02 11:11:55,210 INFO [watch_NEMO_agrif] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-02 11:11:55,211 INFO [watch_NEMO_agrif] publishing log messages to tcp://*:5564
2020-07-02 11:11:55,212 INFO [watch_NEMO_agrif] connected to skookum.eos.ubc.ca port 5555
2020-07-02 11:11:55,879 INFO [watch_NEMO_agrif] watching 02jul20nowcast-agrif job 10846109 on orcinus-nowcast-agrif
2020-07-02 11:11:56,116 INFO [watch_NEMO_agrif] 02jul20nowcast-agrif job 10846109 is queued on orcinus-nowcast-agrif
2020-07-02 11:12:56,385 INFO [watch_NEMO_agrif] 02jul20nowcast-agrif job 10846109 is queued on orcinus-nowcast-agrif
2020-07-02 11:13:57,349 INFO [watch_NEMO_agrif] 02jul20nowcast-agrif on orcinus-nowcast-agrif: time.step not found; continuing to watch...
2020-07-02 11:14:35,745 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15656365 = 2020-07-02 04:29:25 UTC estimated time to finish: 0000:02:52:20
2020-07-02 11:14:36,718 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39111360 = 2020-07-02 01:12:24 UTC estimated time to finish: 0000:12:29:11
2020-07-02 11:14:41,912 INFO [download_results] forecast 2020-07-02 results files from arbutus.cloud-nowcast downloaded
2020-07-02 11:14:41,920 INFO [manager] checklist updated with [results files] items from download_results worker
2020-07-02 11:14:41,957 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['forecast', '--run-date', '2020-07-02'], host='localhost')
2020-07-02 11:14:44,969 INFO [make_CHS_currents_file] running in process 31044
2020-07-02 11:14:44,970 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-02 11:14:44,971 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5563
2020-07-02 11:14:44,972 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555
2020-07-02 11:15:55,478 INFO [make_CHS_currents_file] Made CHS currents file for 2020-07-02 for forecast
2020-07-02 11:15:55,482 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker
2020-07-02 11:15:55,519 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['nemo', 'forecast', '--run-date', '2020-07-02'], host='localhost')
2020-07-02 11:15:56,391 INFO [update_forecast_datasets] running in process 31050
2020-07-02 11:15:56,391 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-02 11:15:56,393 INFO [update_forecast_datasets] publishing log messages to tcp://*:5563
2020-07-02 11:15:56,394 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2020-07-02 11:15:56,395 INFO [update_forecast_datasets] updating nemo forecast directory for forecast/02jul20 run
2020-07-02 11:15:56,714 INFO [update_forecast_datasets] updated nemo forecast directory for forecast/02jul20 run: /results/SalishSea/rolling-forecasts/nemo
2020-07-02 11:15:56,715 INFO [update_forecast_datasets] nemo 2020-07-02 forecast rolling forecast datasets updated
2020-07-02 11:15:56,718 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2020-07-02 11:15:56,738 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nemo-forecast'], host='localhost')
2020-07-02 11:15:56,741 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'forecast', 'publish', '--run-date', '2020-07-02'], host='localhost')
2020-07-02 11:15:56,743 INFO [manager] launching NextWorker(module='nowcast.workers.make_surface_current_tiles', args=['forecast', '--run-date', '2020-07-02'], host='localhost')
2020-07-02 11:15:57,583 INFO [ping_erddap] running in process 31054
2020-07-02 11:15:57,584 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-02 11:15:57,585 INFO [ping_erddap] publishing log messages to tcp://*:5563
2020-07-02 11:15:57,585 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2020-07-02 11:15:57,593 INFO [ping_erddap] nemo-forecast ERDDAP dataset flag file(s) created
2020-07-02 11:15:57,597 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2020-07-02 11:15:57,984 INFO [make_surface_current_tiles] running in process 31056
2020-07-02 11:15:57,985 INFO [make_surface_current_tiles] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-02 11:15:57,986 INFO [make_surface_current_tiles] publishing log messages to tcp://*:5563
2020-07-02 11:15:57,986 INFO [make_surface_current_tiles] connected to skookum.eos.ubc.ca port 5555
2020-07-02 11:15:58,043 INFO [make_surface_current_tiles] starting rendering of tiles for 2020-07-02 forecast in 6 processes into /results/nowcast-sys/figures/surface_currents/forecast/02jul20
2020-07-02 11:16:02,846 INFO [make_plots] running in process 31055
2020-07-02 11:16:02,847 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-02 11:16:02,848 INFO [make_plots] publishing log messages to tcp://*:5565
2020-07-02 11:16:02,849 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2020-07-02 11:16:03,877 INFO [make_plots] preparing render list for 2020-07-02 NEMO forecast publish figures
2020-07-02 11:16:04,015 INFO [make_plots] added Website_thumbnail figure to 2020-07-02 NEMO forecast publish render list
2020-07-02 11:16:04,016 INFO [make_plots] added Threshold_website figure to 2020-07-02 NEMO forecast publish render list
2020-07-02 11:16:04,017 INFO [make_plots] added PA_tidal_predictions figure to 2020-07-02 NEMO forecast publish render list
2020-07-02 11:16:04,018 INFO [make_plots] added Boundary Bay figure to 2020-07-02 NEMO forecast publish render list
2020-07-02 11:16:04,018 INFO [make_plots] added Campbell River figure to 2020-07-02 NEMO forecast publish render list
2020-07-02 11:16:04,019 INFO [make_plots] added Cherry Point figure to 2020-07-02 NEMO forecast publish render list
2020-07-02 11:16:04,020 INFO [make_plots] added Friday Harbor figure to 2020-07-02 NEMO forecast publish render list
2020-07-02 11:16:04,020 INFO [make_plots] added Halfmoon Bay figure to 2020-07-02 NEMO forecast publish render list
2020-07-02 11:16:04,021 INFO [make_plots] added Nanaimo figure to 2020-07-02 NEMO forecast publish render list
2020-07-02 11:16:04,022 INFO [make_plots] added Neah Bay figure to 2020-07-02 NEMO forecast publish render list
2020-07-02 11:16:04,022 INFO [make_plots] added New Westminster figure to 2020-07-02 NEMO forecast publish render list
2020-07-02 11:16:04,023 INFO [make_plots] added Patricia Bay figure to 2020-07-02 NEMO forecast publish render list
2020-07-02 11:16:04,024 INFO [make_plots] added Point Atkinson figure to 2020-07-02 NEMO forecast publish render list
2020-07-02 11:16:04,024 INFO [make_plots] added Port Renfrew figure to 2020-07-02 NEMO forecast publish render list
2020-07-02 11:16:04,025 INFO [make_plots] added Sand Heads figure to 2020-07-02 NEMO forecast publish render list
2020-07-02 11:16:04,025 INFO [make_plots] added Sandy Cove figure to 2020-07-02 NEMO forecast publish render list
2020-07-02 11:16:04,026 INFO [make_plots] added Squamish figure to 2020-07-02 NEMO forecast publish render list
2020-07-02 11:16:04,027 INFO [make_plots] added Victoria figure to 2020-07-02 NEMO forecast publish render list
2020-07-02 11:16:04,027 INFO [make_plots] added Woodwards Landing figure to 2020-07-02 NEMO forecast publish render list
2020-07-02 11:16:04,028 INFO [make_plots] starting to render nemo forecast publish 02jul20 figures
2020-07-02 11:16:36,225 INFO [make_plots] Note: NumExpr detected 12 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
2020-07-02 11:16:36,226 INFO [make_plots] NumExpr defaulting to 8 threads.
2020-07-02 11:16:38,645 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 7493178 = 2020-07-02 01:31:20 UTC, 6.3% complete
2020-07-02 11:18:57,949 INFO [watch_NEMO_agrif] 02jul20nowcast-agrif on orcinus-nowcast-agrif: timestep: 4622651 = 2020-07-02 02:46:40 UTC, 11.6% complete
2020-07-02 11:19:35,856 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15658450 = 2020-07-02 05:04:10 UTC estimated time to finish: 0000:02:47:08
2020-07-02 11:19:36,827 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39112575 = 2020-07-02 01:20:30 UTC estimated time to finish: 0000:12:27:00
2020-07-02 11:21:38,747 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 7493338 = 2020-07-02 03:18:00 UTC, 13.8% complete
2020-07-02 11:23:58,532 INFO [watch_NEMO_agrif] 02jul20nowcast-agrif on orcinus-nowcast-agrif: timestep: 4622914 = 2020-07-02 05:42:00 UTC, 23.8% complete
2020-07-02 11:24:35,976 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15660536 = 2020-07-02 05:38:56 UTC estimated time to finish: 0000:02:41:58
2020-07-02 11:24:36,935 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39114000 = 2020-07-02 01:30:00 UTC estimated time to finish: 0000:12:20:54
2020-07-02 11:26:38,849 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 7493499 = 2020-07-02 05:05:20 UTC, 21.2% complete
2020-07-02 11:28:59,137 INFO [watch_NEMO_agrif] 02jul20nowcast-agrif on orcinus-nowcast-agrif: timestep: 4623178 = 2020-07-02 08:38:00 UTC, 36.0% complete
2020-07-02 11:29:36,096 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15662470 = 2020-07-02 06:11:10 UTC estimated time to finish: 0000:02:37:04
2020-07-02 11:29:37,050 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39115457 = 2020-07-02 01:39:42 UTC estimated time to finish: 0000:12:15:52
2020-07-02 11:31:38,952 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 7493661 = 2020-07-02 06:53:20 UTC, 28.7% complete
2020-07-02 11:33:59,722 INFO [watch_NEMO_agrif] 02jul20nowcast-agrif on orcinus-nowcast-agrif: timestep: 4623439 = 2020-07-02 11:32:00 UTC, 48.1% complete
2020-07-02 11:34:36,213 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15664557 = 2020-07-02 06:45:57 UTC estimated time to finish: 0000:02:31:53
2020-07-02 11:34:37,167 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39116767 = 2020-07-02 01:48:26 UTC estimated time to finish: 0000:12:10:28
2020-07-02 11:34:39,070 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 08074 Campbell River from 2020-06-27 00:00:00 to 2020-07-04 11:55:00
2020-07-02 11:36:39,054 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 7493818 = 2020-07-02 08:38:00 UTC, 36.0% complete
2020-07-02 11:38:38,285 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 9449424 Cherry Point from 2020-06-27 00:00:00 to 2020-07-04 11:55:00
2020-07-02 11:39:00,269 INFO [watch_NEMO_agrif] 02jul20nowcast-agrif on orcinus-nowcast-agrif: timestep: 4623699 = 2020-07-02 14:25:20 UTC, 60.1% complete
2020-07-02 11:39:36,330 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15666683 = 2020-07-02 07:21:23 UTC estimated time to finish: 0000:02:26:34
2020-07-02 11:39:36,661 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 9449880 Friday Harbor from 2020-06-27 00:00:00 to 2020-07-04 11:55:00
2020-07-02 11:39:37,273 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39118146 = 2020-07-02 01:57:38 UTC estimated time to finish: 0000:12:05:31
2020-07-02 11:40:38,400 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07830 Halfmoon Bay from 2020-06-27 00:00:00 to 2020-07-04 11:55:00
2020-07-02 11:41:36,423 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07917 Nanaimo from 2020-06-27 00:00:00 to 2020-07-04 11:55:00
2020-07-02 11:41:39,155 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 7493979 = 2020-07-02 10:25:20 UTC, 43.4% complete
2020-07-02 11:43:52,458 INFO [make_surface_current_tiles] starting PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast/02jul20
2020-07-02 11:44:00,853 INFO [watch_NEMO_agrif] 02jul20nowcast-agrif on orcinus-nowcast-agrif: timestep: 4623960 = 2020-07-02 17:19:20 UTC, 72.2% complete
2020-07-02 11:44:33,298 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 9443090 Neah Bay from 2020-06-27 00:00:00 to 2020-07-04 11:55:00
2020-07-02 11:44:36,450 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15668770 = 2020-07-02 07:56:10 UTC estimated time to finish: 0000:02:21:22
2020-07-02 11:44:37,386 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39119484 = 2020-07-02 02:06:33 UTC estimated time to finish: 0000:12:02:59
2020-07-02 11:44:50,306 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07654 New Westminster from 2020-06-27 00:00:00 to 2020-07-04 11:55:00
2020-07-02 11:46:39,257 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 7494138 = 2020-07-02 12:11:20 UTC, 50.8% complete
2020-07-02 11:47:03,400 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07277 Patricia Bay from 2020-06-27 00:00:00 to 2020-07-04 11:55:00
2020-07-02 11:48:10,657 INFO [make_surface_current_tiles] finished PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast/02jul20
2020-07-02 11:48:10,680 INFO [make_surface_current_tiles] finished rendering of tiles for 2020-07-02 forecast into /results/nowcast-sys/figures/surface_currents/forecast/02jul20
2020-07-02 11:48:10,681 INFO [make_surface_current_tiles] surface current tile figures for 2020-07-02 forecast completed
2020-07-02 11:48:11,229 INFO [manager] checklist updated with [surface current tiles] items from make_surface_current_tiles worker
2020-07-02 11:49:01,438 INFO [watch_NEMO_agrif] 02jul20nowcast-agrif on orcinus-nowcast-agrif: timestep: 4624220 = 2020-07-02 20:12:40 UTC, 84.3% complete
2020-07-02 11:49:16,804 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07795 Point Atkinson from 2020-06-27 00:00:00 to 2020-07-04 11:55:00
2020-07-02 11:49:35,300 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 08525 Port Renfrew from 2020-06-27 00:00:00 to 2020-07-04 11:55:00
2020-07-02 11:49:36,569 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15670664 = 2020-07-02 08:27:44 UTC estimated time to finish: 0000:02:16:39
2020-07-02 11:49:37,497 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39120672 = 2020-07-02 02:14:28 UTC estimated time to finish: 0000:12:00:53
2020-07-02 11:51:39,358 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 7494297 = 2020-07-02 13:57:20 UTC, 58.2% complete
2020-07-02 11:51:49,956 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07594 Sand Heads from 2020-06-27 00:00:00 to 2020-07-04 11:55:00
2020-07-02 11:54:02,020 INFO [watch_NEMO_agrif] 02jul20nowcast-agrif on orcinus-nowcast-agrif: timestep: 4624481 = 2020-07-02 23:06:40 UTC, 96.3% complete
2020-07-02 11:54:02,791 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07786 Sandy Cove from 2020-06-27 00:00:00 to 2020-07-04 11:55:00
2020-07-02 11:54:36,697 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15672791 = 2020-07-02 09:03:11 UTC estimated time to finish: 0000:02:11:24
2020-07-02 11:54:37,609 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39121970 = 2020-07-02 02:23:08 UTC estimated time to finish: 0000:11:58:34
2020-07-02 11:56:19,438 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07811 Squamish from 2020-06-27 00:00:00 to 2020-07-04 11:55:00
2020-07-02 11:56:37,576 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07120 Victoria from 2020-06-27 00:00:00 to 2020-07-04 11:55:00
2020-07-02 11:56:39,388 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 7494458 = 2020-07-02 15:44:40 UTC, 65.6% complete
2020-07-02 11:58:52,005 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07610 Woodwards Landing from 2020-06-27 00:00:00 to 2020-07-04 11:55:00
2020-07-02 11:59:02,511 INFO [watch_NEMO_agrif] 02jul20nowcast-agrif on orcinus-nowcast-agrif: timestep: 4624560 = 2020-07-02 23:59:20 UTC, 100.0% complete
2020-07-02 11:59:36,825 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15674877 = 2020-07-02 09:37:57 UTC estimated time to finish: 0000:02:06:23
2020-07-02 11:59:37,722 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39123390 = 2020-07-02 02:32:36 UTC estimated time to finish: 0000:11:52:47
2020-07-02 12:00:58,050 INFO [make_plots] finished rendering nemo forecast publish 02jul20 figures
2020-07-02 12:00:58,052 INFO [make_plots] nemo publish plots for 2020-07-02 forecast completed
2020-07-02 12:00:58,067 INFO [manager] checklist updated with [plots] items from make_plots worker
2020-07-02 12:00:58,352 INFO [manager] launching NextWorker(module='nowcast.workers.make_feeds', args=['forecast', '--run-date', '2020-07-02'], host='localhost')
2020-07-02 12:01:03,353 INFO [make_feeds] running in process 1647
2020-07-02 12:01:03,354 INFO [make_feeds] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-02 12:01:03,355 INFO [make_feeds] publishing log messages to tcp://*:5563
2020-07-02 12:01:03,357 INFO [make_feeds] connected to skookum.eos.ubc.ca port 5555
2020-07-02 12:01:11,975 INFO [make_feeds] Note: NumExpr detected 12 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
2020-07-02 12:01:11,976 INFO [make_feeds] NumExpr defaulting to 8 threads.
2020-07-02 12:01:18,155 INFO [make_feeds] ATOM feeds for 2020-07-02 forecast run completed
2020-07-02 12:01:18,159 INFO [manager] checklist updated with [storm surge alert feeds] items from make_feeds worker
2020-07-02 12:01:39,491 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 7494617 = 2020-07-02 17:30:40 UTC, 73.0% complete
2020-07-02 12:04:02,783 INFO [watch_NEMO_agrif] NEMO AGRIF run 10846109.orca2.ibb on orcinus-nowcast-agrif completed
2020-07-02 12:04:02,788 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_agrif worker
2020-07-02 12:04:03,329 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['orcinus-nowcast-agrif', 'nowcast-agrif', '--run-date', '2020-07-02'], host='localhost')
2020-07-02 12:04:04,773 INFO [download_results] running in process 1742
2020-07-02 12:04:04,774 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-02 12:04:04,775 INFO [download_results] publishing log messages to tcp://*:5563
2020-07-02 12:04:04,777 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2020-07-02 12:04:04,778 INFO [download_results] downloading results from orcinus-nowcast-agrif:/global/scratch/dlatorne/nowcast-agrif/02jul20 to /results/SalishSea/nowcast-agrif.201702
2020-07-02 12:04:36,945 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15676921 = 2020-07-02 10:12:01 UTC estimated time to finish: 0000:02:01:29
2020-07-02 12:04:37,842 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39124781 = 2020-07-02 02:41:52 UTC estimated time to finish: 0000:11:47:50
2020-07-02 12:04:46,619 INFO [download_results] nowcast-agrif 2020-07-02 results files from orcinus-nowcast-agrif downloaded
2020-07-02 12:04:46,632 INFO [manager] checklist updated with [results files] items from download_results worker
2020-07-02 12:04:47,111 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast-agrif', 'research', '--run-date', '2020-07-02'], host='localhost')
2020-07-02 12:04:52,045 INFO [make_plots] running in process 1749
2020-07-02 12:04:52,046 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-02 12:04:52,047 INFO [make_plots] publishing log messages to tcp://*:5563
2020-07-02 12:04:52,047 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2020-07-02 12:04:52,654 INFO [make_plots] preparing render list for 2020-07-02 NEMO nowcast-agrif research figures
2020-07-02 12:04:52,655 INFO [make_plots] added baynes_sound_surface figure to 2020-07-02 NEMO nowcast-agrif research render list
2020-07-02 12:04:52,657 INFO [make_plots] starting to render nemo nowcast-agrif research 02jul20 figures
2020-07-02 12:05:00,708 INFO [make_plots] finished rendering nemo nowcast-agrif research 02jul20 figures
2020-07-02 12:05:00,709 INFO [make_plots] nemo research plots for 2020-07-02 nowcast-agrif completed
2020-07-02 12:05:00,714 INFO [manager] checklist updated with [plots] items from make_plots worker
2020-07-02 12:06:39,593 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 7494768 = 2020-07-02 19:11:20 UTC, 80.0% complete
2020-07-02 12:09:37,068 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15678817 = 2020-07-02 10:43:37 UTC estimated time to finish: 0000:01:56:52
2020-07-02 12:09:37,958 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39126038 = 2020-07-02 02:50:15 UTC estimated time to finish: 0000:11:42:58
2020-07-02 12:11:39,694 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 7494925 = 2020-07-02 20:56:00 UTC, 87.3% complete
2020-07-02 12:14:37,195 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15680902 = 2020-07-02 11:18:22 UTC estimated time to finish: 0000:01:51:49
2020-07-02 12:14:38,076 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39127459 = 2020-07-02 02:59:43 UTC estimated time to finish: 0000:11:36:38
2020-07-02 12:16:39,795 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 7495079 = 2020-07-02 22:38:40 UTC, 94.4% complete
2020-07-02 12:19:37,327 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15682960 = 2020-07-02 11:52:40 UTC estimated time to finish: 0000:01:46:48
2020-07-02 12:19:38,199 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39128797 = 2020-07-02 03:08:38 UTC estimated time to finish: 0000:11:33:11
2020-07-02 12:21:39,897 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 7495200 = 2020-07-02 23:59:20 UTC, 100.0% complete
2020-07-02 12:24:37,442 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15685005 = 2020-07-02 12:26:45 UTC estimated time to finish: 0000:01:41:55
2020-07-02 12:24:38,309 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39130380 = 2020-07-02 03:19:12 UTC estimated time to finish: 0000:11:23:06
2020-07-02 12:26:40,006 INFO [watch_NEMO] nowcast-green NEMO run on arbutus.cloud-nowcast completed
2020-07-02 12:26:40,010 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2020-07-02 12:26:40,581 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_wind_file', args=['arbutus.cloud-nowcast', 'forecast'], host='arbutus.cloud-nowcast')
2020-07-02 12:26:40,588 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_current_file', args=['arbutus.cloud-nowcast', 'forecast'], host='arbutus.cloud-nowcast')
2020-07-02 12:26:40,592 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['salish-nowcast', 'nowcast+', '--shared-storage'], host='localhost')
2020-07-02 12:26:40,597 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2020-07-02'], host='localhost')
2020-07-02 12:26:41,534 INFO [download_results] running in process 2168
2020-07-02 12:26:41,535 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-02 12:26:41,536 INFO [download_results] publishing log messages to tcp://*:5563
2020-07-02 12:26:41,537 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2020-07-02 12:26:41,538 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/nowcast-green/02jul20 to /results2/SalishSea/nowcast-green.201905
2020-07-02 12:26:41,539 INFO [make_forcing_links] running in process 2167
2020-07-02 12:26:41,539 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-02 12:26:41,540 INFO [make_forcing_links] publishing log messages to tcp://*:5564
2020-07-02 12:26:41,541 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2020-07-02 12:26:42,381 INFO [make_forcing_links] nowcast+ 2020-07-02 forcing file links on salish-nowcast created
2020-07-02 12:26:42,387 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2020-07-02 12:26:42,666 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['salish-nowcast', 'nowcast-dev', '--run-date', '2020-07-02'], host='salish-nowcast')
2020-07-02 12:26:44,215 INFO [make_ww3_wind_file] running in process 11813
2020-07-02 12:26:44,270 INFO [make_ww3_wind_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2020-07-02 12:26:44,324 INFO [make_ww3_wind_file] publishing log messages to tcp://*:5570
2020-07-02 12:26:44,395 INFO [make_ww3_wind_file] connected to skookum.eos.ubc.ca port 5555
2020-07-02 12:26:44,484 INFO [make_ww3_wind_file] Creating wwatch3 wind forcing file for 2020-07-02 forecast run
2020-07-02 12:26:44,604 INFO [run_NEMO] running in process 23750
2020-07-02 12:26:44,605 INFO [make_ww3_current_file] running in process 11814
2020-07-02 12:26:44,606 INFO [run_NEMO] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-02 12:26:44,607 INFO [make_ww3_current_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2020-07-02 12:26:44,608 INFO [run_NEMO] publishing log messages to tcp://*:5556
2020-07-02 12:26:44,609 INFO [make_ww3_current_file] publishing log messages to tcp://*:5571
2020-07-02 12:26:44,610 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2020-07-02 12:26:44,611 INFO [make_ww3_current_file] connected to skookum.eos.ubc.ca port 5555
2020-07-02 12:26:44,613 INFO [make_ww3_current_file] Creating wwatch3 currents forcing file for 2020-07-02 forecast run
2020-07-02 12:26:45,159 WARNING [run_NEMO] There are uncommitted changes in /SalishSeaCast/SalishSeaNowcast
2020-07-02 12:26:45,868 INFO [run_NEMO] nowcast-dev: launching /SalishSeaCast/runs/02jul20nowcast-dev_2020-07-02T122644.303772-0700/SalishSeaNEMO.sh on salish-nowcast
2020-07-02 12:26:45,958 INFO [run_NEMO] nowcast-dev NEMO run for 2020-07-02 on salish-nowcast started
2020-07-02 12:26:45,966 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2020-07-02 12:26:46,239 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['salish-nowcast', 'nowcast-dev'], host='salish-nowcast')
2020-07-02 12:26:48,034 INFO [make_ww3_wind_file] wwatch3 wind forcing file created on arbutus.cloud-nowcast for 2020-07-02 forecast run
2020-07-02 12:26:48,040 INFO [manager] checklist updated with [WW3 wind forcing] items from make_ww3_wind_file worker
2020-07-02 12:27:31,351 INFO [make_ww3_current_file] wwatch3 currents forcing file created on arbutus.cloud-nowcast for 2020-07-02 forecast run
2020-07-02 12:27:31,521 INFO [manager] checklist updated with [WW3 currents forcing] items from make_ww3_current_file worker
2020-07-02 12:27:31,941 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2020-07-02'], host='arbutus.cloud-nowcast')
2020-07-02 12:27:33,805 INFO [run_ww3] running in process 12017
2020-07-02 12:27:33,807 INFO [run_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2020-07-02 12:27:33,808 INFO [run_ww3] publishing log messages to tcp://*:5572
2020-07-02 12:27:33,809 INFO [run_ww3] connected to skookum.eos.ubc.ca port 5555
2020-07-02 12:27:33,813 INFO [run_ww3] Created run directory /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/nowcast_2020-07-02T122733.803350-0700
2020-07-02 12:27:33,814 INFO [run_ww3] nowcast: launching /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/nowcast_2020-07-02T122733.803350-0700/SoGWW3.sh on arbutus.cloud-nowcast
2020-07-02 12:27:33,830 INFO [run_ww3] nowcast WaveWatch3 run for 2020-07-02 on arbutus.cloud-nowcast started
2020-07-02 12:27:33,836 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker
2020-07-02 12:27:34,106 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'nowcast'], host='arbutus.cloud-nowcast')
2020-07-02 12:27:36,074 INFO [watch_ww3] running in process 12120
2020-07-02 12:27:36,075 INFO [watch_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2020-07-02 12:27:36,076 INFO [watch_ww3] publishing log messages to tcp://*:5573
2020-07-02 12:27:36,077 INFO [watch_ww3] connected to skookum.eos.ubc.ca port 5555
2020-07-02 12:27:36,111 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2020-07-02 12:29:37,558 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15687010 = 2020-07-02 13:00:10 UTC estimated time to finish: 0000:01:37:09
2020-07-02 12:29:38,423 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39132074 = 2020-07-02 03:30:29 UTC estimated time to finish: 0000:11:10:12
2020-07-02 12:30:29,561 INFO [download_results] nowcast-green 2020-07-02 results files from arbutus.cloud-nowcast downloaded
2020-07-02 12:30:29,572 INFO [manager] checklist updated with [results files] items from download_results worker
2020-07-02 12:30:29,849 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast-green', 'research', '--run-date', '2020-07-02'], host='localhost')
2020-07-02 12:30:29,854 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nowcast-green'], host='localhost')
2020-07-02 12:30:31,362 INFO [ping_erddap] running in process 2238
2020-07-02 12:30:31,364 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-02 12:30:31,364 INFO [ping_erddap] publishing log messages to tcp://*:5563
2020-07-02 12:30:31,365 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2020-07-02 12:30:31,368 INFO [ping_erddap] nowcast-green ERDDAP dataset flag file(s) created
2020-07-02 12:30:31,371 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2020-07-02 12:30:34,760 INFO [make_plots] running in process 2237
2020-07-02 12:30:34,762 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-02 12:30:34,763 INFO [make_plots] publishing log messages to tcp://*:5563
2020-07-02 12:30:34,764 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2020-07-02 12:30:35,201 INFO [make_plots] preparing render list for 2020-07-02 NEMO nowcast-green research figures
2020-07-02 12:30:41,228 INFO [make_plots] added temperature_thalweg_and_surface figures to 2020-07-02 NEMO nowcast-green research render list
2020-07-02 12:30:43,599 INFO [make_plots] added nitrate_thalweg_and_surface figures to 2020-07-02 NEMO nowcast-green research render list
2020-07-02 12:30:45,743 INFO [make_plots] added ammonium_thalweg_and_surface figures to 2020-07-02 NEMO nowcast-green research render list
2020-07-02 12:30:48,026 INFO [make_plots] added silicon_thalweg_and_surface figures to 2020-07-02 NEMO nowcast-green research render list
2020-07-02 12:30:50,446 INFO [make_plots] added dissolved_organic_nitrogen_thalweg_and_surface figures to 2020-07-02 NEMO nowcast-green research render list
2020-07-02 12:30:52,843 INFO [make_plots] added particulate_organic_nitrogen_thalweg_and_surface figures to 2020-07-02 NEMO nowcast-green research render list
2020-07-02 12:30:55,155 INFO [make_plots] added biogenic_silicon_thalweg_and_surface figures to 2020-07-02 NEMO nowcast-green research render list
2020-07-02 12:30:57,830 INFO [make_plots] added diatoms_thalweg_and_surface figures to 2020-07-02 NEMO nowcast-green research render list
2020-07-02 12:31:00,382 INFO [make_plots] added ciliates_thalweg_and_surface figures to 2020-07-02 NEMO nowcast-green research render list
2020-07-02 12:31:02,919 INFO [make_plots] added flagellates_thalweg_and_surface figures to 2020-07-02 NEMO nowcast-green research render list
2020-07-02 12:31:05,486 INFO [make_plots] added microzooplankton_thalweg_and_surface figures to 2020-07-02 NEMO nowcast-green research render list
2020-07-02 12:31:08,025 INFO [make_plots] added mesozooplankton_thalweg_and_surface figures to 2020-07-02 NEMO nowcast-green research render list
2020-07-02 12:31:10,352 INFO [make_plots] added Fraser_tracer_thalweg_and_surface figures to 2020-07-02 NEMO nowcast-green research render list
2020-07-02 12:31:10,441 INFO [make_plots] added temperature_salinity_timeseries figure to 2020-07-02 NEMO nowcast-green research render list
2020-07-02 12:31:10,442 INFO [make_plots] added nitrate_diatoms_timeseries figure to 2020-07-02 NEMO nowcast-green research render list
2020-07-02 12:31:10,443 INFO [make_plots] added mesodinium_flagellates_timeseries figure to 2020-07-02 NEMO nowcast-green research render list
2020-07-02 12:31:10,444 INFO [make_plots] added mesozoo_microzoo_timeseries figure to 2020-07-02 NEMO nowcast-green research render list
2020-07-02 12:31:10,444 INFO [make_plots] starting to render nemo nowcast-green research 02jul20 figures
2020-07-02 12:32:36,212 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 102, 23.6% complete
2020-07-02 12:34:37,674 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15688864 = 2020-07-02 13:31:04 UTC estimated time to finish: 0000:01:32:43
2020-07-02 12:34:38,533 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39133576 = 2020-07-02 03:40:30 UTC estimated time to finish: 0000:10:59:42
2020-07-02 12:37:36,313 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 210, 48.6% complete
2020-07-02 12:39:37,792 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15690881 = 2020-07-02 14:04:41 UTC estimated time to finish: 0000:01:27:49
2020-07-02 12:39:38,645 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39135310 = 2020-07-02 03:52:04 UTC estimated time to finish: 0000:10:47:46
2020-07-02 12:42:36,414 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 324, 75.0% complete
2020-07-02 12:44:37,908 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15692967 = 2020-07-02 14:39:27 UTC estimated time to finish: 0000:01:22:44
2020-07-02 12:44:38,768 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39137017 = 2020-07-02 04:03:26 UTC estimated time to finish: 0000:10:36:54
2020-07-02 12:47:36,515 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2020-07-02 12:49:38,027 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15695011 = 2020-07-02 15:13:31 UTC estimated time to finish: 0000:01:17:43
2020-07-02 12:49:38,882 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39138710 = 2020-07-02 04:14:44 UTC estimated time to finish: 0000:10:26:27
2020-07-02 12:52:36,617 INFO [watch_ww3] nowcast WWATCH3 run on arbutus.cloud-nowcast completed
2020-07-02 12:52:36,620 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2020-07-02 12:52:37,115 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2020-07-02'], host='localhost')
2020-07-02 12:52:37,123 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2020-07-02'], host='arbutus.cloud-nowcast')
2020-07-02 12:52:38,492 INFO [download_wwatch3_results] running in process 2730
2020-07-02 12:52:38,493 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-02 12:52:38,494 INFO [download_wwatch3_results] publishing log messages to tcp://*:5564
2020-07-02 12:52:38,495 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2020-07-02 12:52:39,034 INFO [run_ww3] running in process 12286
2020-07-02 12:52:39,036 INFO [run_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2020-07-02 12:52:39,038 INFO [run_ww3] publishing log messages to tcp://*:5572
2020-07-02 12:52:39,039 INFO [run_ww3] connected to skookum.eos.ubc.ca port 5555
2020-07-02 12:52:39,041 INFO [run_ww3] Created run directory /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast_2020-07-02T125239.030915-0700
2020-07-02 12:52:39,042 INFO [run_ww3] forecast: launching /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast_2020-07-02T125239.030915-0700/SoGWW3.sh on arbutus.cloud-nowcast
2020-07-02 12:52:39,055 INFO [run_ww3] forecast WaveWatch3 run for 2020-07-02 on arbutus.cloud-nowcast started
2020-07-02 12:52:39,062 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker
2020-07-02 12:52:39,318 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'forecast'], host='arbutus.cloud-nowcast')
2020-07-02 12:52:41,252 INFO [watch_ww3] running in process 12476
2020-07-02 12:52:41,253 INFO [watch_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2020-07-02 12:52:41,255 INFO [watch_ww3] publishing log messages to tcp://*:5573
2020-07-02 12:52:41,255 INFO [watch_ww3] connected to skookum.eos.ubc.ca port 5555
2020-07-02 12:52:41,289 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2020-07-02 12:53:14,252 INFO [download_wwatch3_results] nowcast 2020-07-02 results files from arbutus.cloud-nowcast downloaded
2020-07-02 12:53:14,255 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2020-07-02 12:54:38,145 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15696865 = 2020-07-02 15:44:25 UTC estimated time to finish: 0000:01:13:10
2020-07-02 12:54:39,000 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39140280 = 2020-07-02 04:25:12 UTC estimated time to finish: 0000:10:17:06
2020-07-02 12:57:41,390 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 99, 15.3% complete
2020-07-02 12:59:38,264 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15698950 = 2020-07-02 16:19:10 UTC estimated time to finish: 0000:01:08:04
2020-07-02 12:59:39,107 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39141987 = 2020-07-02 04:36:34 UTC estimated time to finish: 0000:10:07:42
2020-07-02 13:02:41,490 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 210, 32.4% complete
2020-07-02 13:04:38,384 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15701008 = 2020-07-02 16:53:28 UTC estimated time to finish: 0000:01:03:02
2020-07-02 13:04:39,221 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39143640 = 2020-07-02 04:47:36 UTC estimated time to finish: 0000:09:59:06
2020-07-02 13:07:41,591 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 318, 49.1% complete
2020-07-02 13:09:38,505 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15703093 = 2020-07-02 17:28:13 UTC estimated time to finish: 0000:00:57:53
2020-07-02 13:09:39,334 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39145374 = 2020-07-02 04:59:09 UTC estimated time to finish: 0000:09:49:44
2020-07-02 13:12:41,692 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 432, 66.7% complete
2020-07-02 13:14:38,625 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15704948 = 2020-07-02 17:59:08 UTC estimated time to finish: 0000:00:53:19
2020-07-02 13:14:39,449 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39146917 = 2020-07-02 05:09:26 UTC estimated time to finish: 0000:09:41:48
2020-07-02 13:17:41,793 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 543, 83.8% complete
2020-07-02 13:19:38,747 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15706993 = 2020-07-02 18:33:13 UTC estimated time to finish: 0000:00:48:19
2020-07-02 13:19:39,547 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39148650 = 2020-07-02 05:21:00 UTC estimated time to finish: 0000:09:32:52
2020-07-02 13:22:41,894 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2020-07-02 13:24:38,869 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15709079 = 2020-07-02 19:07:59 UTC estimated time to finish: 0000:00:43:09
2020-07-02 13:24:39,663 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39150385 = 2020-07-02 05:32:34 UTC estimated time to finish: 0000:09:24:37
2020-07-02 13:27:42,159 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2020-07-02 13:27:42,160 INFO [watch_ww3] forecast WWATCH3 run on arbutus.cloud-nowcast completed
2020-07-02 13:27:42,723 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2020-07-02'], host='localhost')
2020-07-02 13:27:44,488 INFO [download_wwatch3_results] running in process 3592
2020-07-02 13:27:44,489 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-02 13:27:44,489 INFO [download_wwatch3_results] publishing log messages to tcp://*:5564
2020-07-02 13:27:44,490 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2020-07-02 13:28:31,454 INFO [download_wwatch3_results] forecast 2020-07-02 results files from arbutus.cloud-nowcast downloaded
2020-07-02 13:28:31,459 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2020-07-02 13:28:31,863 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['wwatch3', 'forecast', '--run-date', '2020-07-02'], host='localhost')
2020-07-02 13:28:33,153 INFO [update_forecast_datasets] running in process 3632
2020-07-02 13:28:33,154 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-02 13:28:33,155 INFO [update_forecast_datasets] publishing log messages to tcp://*:5564
2020-07-02 13:28:33,155 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2020-07-02 13:28:33,156 INFO [update_forecast_datasets] updating wwatch3 most_recent_forecast directory from forecast/02jul20 run
2020-07-02 13:28:33,181 INFO [update_forecast_datasets] updated wwatch3 most_recent_forecast directory from forecast/02jul20 run
2020-07-02 13:28:33,182 INFO [update_forecast_datasets] updating wwatch3 forecast directory for forecast/02jul20 run
2020-07-02 13:28:33,533 INFO [update_forecast_datasets] updated wwatch3 forecast directory for forecast/02jul20 run: /results/SalishSea/rolling-forecasts/wwatch3
2020-07-02 13:28:33,534 INFO [update_forecast_datasets] wwatch3 2020-07-02 forecast rolling forecast datasets updated
2020-07-02 13:28:33,538 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2020-07-02 13:28:33,900 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['wwatch3-forecast'], host='localhost')
2020-07-02 13:28:34,920 INFO [ping_erddap] running in process 3637
2020-07-02 13:28:34,922 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-02 13:28:34,923 INFO [ping_erddap] publishing log messages to tcp://*:5564
2020-07-02 13:28:34,924 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2020-07-02 13:28:34,925 INFO [ping_erddap] wwatch3-forecast ERDDAP dataset flag file(s) created
2020-07-02 13:28:34,927 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2020-07-02 13:28:35,182 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['wwatch3', 'forecast', 'publish', '--run-date', '2020-07-02'], host='localhost')
2020-07-02 13:28:45,168 INFO [make_plots] running in process 3641
2020-07-02 13:28:45,169 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-02 13:28:45,170 INFO [make_plots] publishing log messages to tcp://*:5564
2020-07-02 13:28:45,171 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2020-07-02 13:28:45,171 INFO [make_plots] preparing render list for 2020-07-02 WaveWatch3 forecast publish figures
2020-07-02 13:28:45,172 INFO [make_plots] added Halibut Bank figure to 2020-07-02 WaveWatch3 forecast publish render list
2020-07-02 13:28:45,172 INFO [make_plots] added Sentry Shoal figure to 2020-07-02 WaveWatch3 forecast publish render list
2020-07-02 13:28:45,173 INFO [make_plots] starting to render wwatch3 forecast publish 02jul20 figures
2020-07-02 13:29:38,992 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15711206 = 2020-07-02 19:43:26 UTC estimated time to finish: 0000:00:37:54
2020-07-02 13:29:39,779 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39152132 = 2020-07-02 05:44:12 UTC estimated time to finish: 0000:09:16:06
2020-07-02 13:29:53,104 INFO [make_plots] Note: NumExpr detected 12 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
2020-07-02 13:29:53,104 INFO [make_plots] NumExpr defaulting to 8 threads.
2020-07-02 13:29:53,122 INFO [make_plots] retrieving last 45 days of real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46146 Halibut Bank
2020-07-02 13:31:00,799 INFO [make_plots] retrieving last 45 days of real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46131 Sentry Shoal
2020-07-02 13:31:02,042 INFO [make_plots] finished rendering wwatch3 forecast publish 02jul20 figures
2020-07-02 13:31:02,043 INFO [make_plots] wwatch3 publish plots for 2020-07-02 forecast completed
2020-07-02 13:31:02,044 INFO [manager] checklist updated with [plots] items from make_plots worker
2020-07-02 13:34:39,116 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15713263 = 2020-07-02 20:17:43 UTC estimated time to finish: 0000:00:32:50
2020-07-02 13:34:39,895 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39153784 = 2020-07-02 05:55:13 UTC estimated time to finish: 0000:09:08:48
2020-07-02 13:39:39,239 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15715104 = 2020-07-02 20:48:24 UTC estimated time to finish: 0000:00:28:19
2020-07-02 13:39:40,013 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39155327 = 2020-07-02 06:05:30 UTC estimated time to finish: 0000:09:01:51
2020-07-02 13:44:39,364 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15717161 = 2020-07-02 21:22:41 UTC estimated time to finish: 0000:00:23:15
2020-07-02 13:44:40,131 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39157061 = 2020-07-02 06:17:04 UTC estimated time to finish: 0000:08:54:06
2020-07-02 13:49:39,443 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15719207 = 2020-07-02 21:56:47 UTC estimated time to finish: 0000:00:18:13
2020-07-02 13:49:40,249 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39158796 = 2020-07-02 06:28:38 UTC estimated time to finish: 0000:08:46:24
2020-07-02 13:54:39,570 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15721252 = 2020-07-02 22:30:52 UTC estimated time to finish: 0000:00:13:11
2020-07-02 13:54:40,368 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39160490 = 2020-07-02 06:39:56 UTC estimated time to finish: 0000:08:39:24
2020-07-02 13:55:58,316 INFO [watch_NEMO] termination signal (SIGTERM) received; shutting down
2020-07-02 13:56:53,017 INFO [watch_NEMO] running in process 26773
2020-07-02 13:56:53,018 INFO [watch_NEMO] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-02 13:56:53,018 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2020-07-02 13:56:53,019 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2020-07-02 13:56:53,086 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4623335 = 2020-07-02 10:22:40 UTC, 43.3% complete
2020-07-02 13:59:39,697 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15723106 = 2020-07-02 23:01:46 UTC estimated time to finish: 0000:00:08:37
2020-07-02 13:59:40,487 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39162031 = 2020-07-02 06:50:12 UTC estimated time to finish: 0000:08:33:06
2020-07-02 14:01:54,142 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4623389 = 2020-07-02 10:58:40 UTC, 45.8% complete
2020-07-02 14:04:39,824 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15725160 = 2020-07-02 23:36:00 UTC estimated time to finish: 0000:00:03:33
2020-07-02 14:04:40,608 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39163725 = 2020-07-02 07:01:30 UTC estimated time to finish: 0000:08:26:12
2020-07-02 14:06:54,242 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4623444 = 2020-07-02 11:35:20 UTC, 48.3% complete
2020-07-02 14:09:39,932 INFO [watch_fvcom] x2 nowcast FVCOM VH-FR run on arbutus.cloud-nowcast completed
2020-07-02 14:09:39,935 INFO [manager] checklist updated with [FVCOM run] items from watch_fvcom worker
2020-07-02 14:09:40,524 INFO [manager] launching NextWorker(module='nowcast.workers.download_fvcom_results', args=['arbutus.cloud-nowcast', 'x2', 'nowcast', '--run-date', '2020-07-02'], host='localhost')
2020-07-02 14:09:40,527 INFO [manager] launching NextWorker(module='nowcast.workers.make_fvcom_boundary', args=['arbutus.cloud-nowcast', 'x2', 'forecast', '--run-date', '2020-07-02'], host='arbutus.cloud-nowcast')
2020-07-02 14:09:40,728 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39165664 = 2020-07-02 07:14:25 UTC estimated time to finish: 0000:08:16:48
2020-07-02 14:09:41,821 INFO [download_fvcom_results] running in process 4879
2020-07-02 14:09:41,822 INFO [download_fvcom_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-02 14:09:41,823 INFO [download_fvcom_results] publishing log messages to tcp://*:5565
2020-07-02 14:09:41,824 INFO [download_fvcom_results] connected to skookum.eos.ubc.ca port 5555
2020-07-02 14:09:44,201 INFO [make_fvcom_boundary] running in process 13135
2020-07-02 14:09:44,239 INFO [make_fvcom_boundary] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2020-07-02 14:09:44,239 INFO [make_fvcom_boundary] publishing log messages to tcp://*:5580
2020-07-02 14:09:44,257 INFO [make_fvcom_boundary] connected to skookum.eos.ubc.ca port 5555
2020-07-02 14:09:44,257 INFO [make_fvcom_boundary] Creating VHFR FVCOM open boundary file for x2 forecast run from 2020-07-02 NEMO run
2020-07-02 14:10:34,438 INFO [download_fvcom_results] VHFR FVCOM x2 nowcast 2020-07-02 results files from arbutus.cloud-nowcast downloaded
2020-07-02 14:10:34,444 INFO [manager] checklist updated with [VHFR FVCOM results files] items from download_fvcom_results worker
2020-07-02 14:10:34,767 INFO [manager] launching NextWorker(module='nowcast.workers.get_vfpa_hadcp', args=['--data-date', '2020-07-02'], host='localhost')
2020-07-02 14:10:34,775 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['fvcom', 'nowcast-x2', 'research', '--run-date', '2020-07-02'], host='localhost')
2020-07-02 14:10:38,811 INFO [get_vfpa_hadcp] running in process 4887
2020-07-02 14:10:38,820 INFO [get_vfpa_hadcp] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-02 14:10:38,821 INFO [get_vfpa_hadcp] publishing log messages to tcp://*:5565
2020-07-02 14:10:38,822 INFO [get_vfpa_hadcp] connected to skookum.eos.ubc.ca port 5555
2020-07-02 14:10:38,823 INFO [get_vfpa_hadcp] processing VFPA HADCP data from 2nd Narrows Rail Bridge for 2020-07-02
2020-07-02 14:10:40,696 INFO [get_vfpa_hadcp] added VFPA HADCP data from 2nd Narrows Rail Bridge for 2020-07-02 to /opp/observations/AISDATA/netcdf/VFPA_2ND_NARROWS_HADCP_2s_202007.nc
2020-07-02 14:10:40,697 INFO [get_vfpa_hadcp] VFPA HADCP observations added to 2020-07 netcdf file
2020-07-02 14:10:40,699 INFO [manager] checklist updated with [VFPA HADCP data] items from get_vfpa_hadcp worker
2020-07-02 14:10:40,980 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['VFPA-HADCP'], host='localhost')
2020-07-02 14:10:41,225 INFO [make_plots] running in process 4888
2020-07-02 14:10:41,226 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-02 14:10:41,227 INFO [make_plots] publishing log messages to tcp://*:5566
2020-07-02 14:10:41,227 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2020-07-02 14:10:41,230 INFO [make_plots] preparing render list for 2020-07-02 VHFR FVCOM nowcast-x2 research figures
2020-07-02 14:10:41,233 INFO [make_plots] added EnglishBay_surface_currents figures to 2020-07-02 VHFR FVCOM nowcast-x2 research render list
2020-07-02 14:10:41,236 INFO [make_plots] added VancouverHarbour_surface_currents figures to 2020-07-02 VHFR FVCOM nowcast-x2 research render list
2020-07-02 14:10:41,239 INFO [make_plots] added IndianArm_surface_currents figures to 2020-07-02 VHFR FVCOM nowcast-x2 research render list
2020-07-02 14:10:41,242 INFO [make_plots] added VancouverHarbour_thalweg figures to 2020-07-02 VHFR FVCOM nowcast-x2 research render list
2020-07-02 14:10:41,244 INFO [make_plots] added PortMoody_thalweg figures to 2020-07-02 VHFR FVCOM nowcast-x2 research render list
2020-07-02 14:10:41,246 INFO [make_plots] added IndianArm_thalweg figures to 2020-07-02 VHFR FVCOM nowcast-x2 research render list
2020-07-02 14:10:41,247 INFO [make_plots] starting to render fvcom nowcast-x2 research 02jul20 figures
2020-07-02 14:10:42,157 INFO [ping_erddap] running in process 4897
2020-07-02 14:10:42,158 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-02 14:10:42,158 INFO [ping_erddap] publishing log messages to tcp://*:5565
2020-07-02 14:10:42,159 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2020-07-02 14:10:42,160 INFO [ping_erddap] VFPA-HADCP ERDDAP dataset flag file(s) created
2020-07-02 14:10:42,163 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2020-07-02 14:10:42,441 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['fvcom', 'nowcast-x2', 'publish', '--run-date', '2020-07-02'], host='localhost')
2020-07-02 14:10:46,968 INFO [make_plots] running in process 4902
2020-07-02 14:10:46,969 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-02 14:10:46,969 INFO [make_plots] publishing log messages to tcp://*:5565
2020-07-02 14:10:46,970 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2020-07-02 14:10:48,087 INFO [make_plots] preparing render list for 2020-07-02 VHFR FVCOM nowcast-x2 publish figures
2020-07-02 14:10:48,089 INFO [make_plots] added Calamity Point figure to 2020-07-02 VHFR FVCOM nowcast-x2 publish render list
2020-07-02 14:10:48,090 INFO [make_plots] added Indian Arm Head figure to 2020-07-02 VHFR FVCOM nowcast-x2 publish render list
2020-07-02 14:10:48,091 INFO [make_plots] added New Westminster figure to 2020-07-02 VHFR FVCOM nowcast-x2 publish render list
2020-07-02 14:10:48,092 INFO [make_plots] added Port Moody figure to 2020-07-02 VHFR FVCOM nowcast-x2 publish render list
2020-07-02 14:10:48,093 INFO [make_plots] added Sand Heads figure to 2020-07-02 VHFR FVCOM nowcast-x2 publish render list
2020-07-02 14:10:48,094 INFO [make_plots] added Sandy Cove figure to 2020-07-02 VHFR FVCOM nowcast-x2 publish render list
2020-07-02 14:10:48,095 INFO [make_plots] added Vancouver Harbour figure to 2020-07-02 VHFR FVCOM nowcast-x2 publish render list
2020-07-02 14:10:48,095 INFO [make_plots] added Woodwards Landing figure to 2020-07-02 VHFR FVCOM nowcast-x2 publish render list
2020-07-02 14:10:48,096 INFO [make_plots] added 2ndNarrows_current figure to 2020-07-02 VHFR FVCOM nowcast-x2 publish render list
2020-07-02 14:10:48,097 INFO [make_plots] starting to render fvcom nowcast-x2 publish 02jul20 figures
2020-07-02 14:10:48,111 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07724 Calamity Point from 2020-07-01 23:55:00 to 2020-07-03 00:00:00
2020-07-02 14:11:08,340 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07724 Calamity Point from 2020-07-02 00:00:00 to 2020-07-03 00:00:00
2020-07-02 14:11:10,191 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07774 Indian Arm Head from 2020-07-01 23:55:00 to 2020-07-03 00:00:00
2020-07-02 14:11:11,056 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07774 Indian Arm Head from 2020-07-02 00:00:00 to 2020-07-03 00:00:00
2020-07-02 14:11:12,699 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07654 New Westminster from 2020-07-01 23:55:00 to 2020-07-03 00:00:00
2020-07-02 14:11:32,568 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07654 New Westminster from 2020-07-02 00:00:00 to 2020-07-03 00:00:00
2020-07-02 14:11:35,597 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07755 Port Moody from 2020-07-01 23:55:00 to 2020-07-03 00:00:00
2020-07-02 14:11:54,343 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4623498 = 2020-07-02 12:11:20 UTC, 50.8% complete
2020-07-02 14:11:55,877 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07755 Port Moody from 2020-07-02 00:00:00 to 2020-07-03 00:00:00
2020-07-02 14:11:58,790 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07594 Sand Heads from 2020-07-01 23:55:00 to 2020-07-03 00:00:00
2020-07-02 14:12:19,096 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07594 Sand Heads from 2020-07-02 00:00:00 to 2020-07-03 00:00:00
2020-07-02 14:12:22,116 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07786 Sandy Cove from 2020-07-01 23:55:00 to 2020-07-03 00:00:00
2020-07-02 14:12:41,304 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07795 Point Atkinson from 2020-07-02 00:00:00 to 2020-07-03 00:00:00
2020-07-02 14:12:44,446 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07735 Vancouver Harbour from 2020-07-01 23:55:00 to 2020-07-03 00:00:00
2020-07-02 14:13:04,534 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07735 Vancouver Harbour from 2020-07-02 00:00:00 to 2020-07-03 00:00:00
2020-07-02 14:13:07,687 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07610 Woodwards Landing from 2020-07-01 23:55:00 to 2020-07-03 00:00:00
2020-07-02 14:13:19,021 INFO [make_fvcom_boundary] Stored VHFR FVCOM open boundary file: /nemoShare/MEOPAR/nowcast-sys/fvcom-runs/input.x2/bdy_x2_forecast_brcl_20200703.nc
2020-07-02 14:13:19,066 INFO [make_fvcom_boundary] FVCOM x2 forecast run boundary condition file for 2020-07-02 created on arbutus.cloud-nowcast
2020-07-02 14:13:19,067 INFO [manager] checklist updated with [FVCOM boundary conditions] items from make_fvcom_boundary worker
2020-07-02 14:13:19,573 INFO [manager] launching NextWorker(module='nowcast.workers.make_fvcom_atmos_forcing', args=['x2', 'forecast', '--run-date', '2020-07-02'], host='localhost')
2020-07-02 14:13:19,581 INFO [manager] launching NextWorker(module='nowcast.workers.make_fvcom_rivers_forcing', args=['arbutus.cloud-nowcast', 'x2', 'forecast', '--run-date', '2020-07-02'], host='arbutus.cloud-nowcast')
2020-07-02 14:13:22,995 INFO [make_fvcom_rivers_forcing] running in process 13316
2020-07-02 14:13:23,083 INFO [make_fvcom_rivers_forcing] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2020-07-02 14:13:23,084 INFO [make_fvcom_rivers_forcing] publishing log messages to tcp://*:5582
2020-07-02 14:13:23,085 INFO [make_fvcom_rivers_forcing] connected to skookum.eos.ubc.ca port 5555
2020-07-02 14:13:23,086 INFO [make_fvcom_rivers_forcing] Creating VHFR FVCOM rivers forcing file for x2 forecast run from 2020-07-02 NEMO runoff forcing files
2020-07-02 14:13:23,187 INFO [make_fvcom_rivers_forcing] Stored VHFR FVCOM rivers forcing file: /nemoShare/MEOPAR/nowcast-sys/fvcom-runs/input.x2/rivers_x2_forecast_20200703.nc
2020-07-02 14:13:23,189 INFO [make_fvcom_rivers_forcing] FVCOM x2 forecast run rivers forcing file for 2020-07-02 created on arbutus.cloud-nowcast
2020-07-02 14:13:23,192 INFO [manager] checklist updated with [FVCOM rivers forcing] items from make_fvcom_rivers_forcing worker
2020-07-02 14:13:25,067 INFO [make_fvcom_atmos_forcing] running in process 5051
2020-07-02 14:13:25,068 INFO [make_fvcom_atmos_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-02 14:13:25,069 INFO [make_fvcom_atmos_forcing] publishing log messages to tcp://*:5567
2020-07-02 14:13:25,069 INFO [make_fvcom_atmos_forcing] connected to skookum.eos.ubc.ca port 5555
2020-07-02 14:13:25,070 INFO [make_fvcom_atmos_forcing] Creating VHFR FVCOM atmospheric forcing file for x2 forecast run from 2020-07-02 HRDPS GRIB files
2020-07-02 14:13:27,603 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07610 Woodwards Landing from 2020-07-02 00:00:00 to 2020-07-03 00:00:00
2020-07-02 14:13:33,214 INFO [make_plots] finished rendering fvcom nowcast-x2 publish 02jul20 figures
2020-07-02 14:13:33,751 INFO [make_plots] fvcom publish plots for 2020-07-02 nowcast-x2 completed
2020-07-02 14:13:33,758 INFO [manager] checklist updated with [plots] items from make_plots worker
2020-07-02 14:14:01,376 INFO [make_fvcom_atmos_forcing] Stored VHFR FVCOM atmospheric forcing hfx file: /results/forcing/atmospheric/GEM2.5/vhfr-fvcom/atmos_x2_forecast_hfx_20200703.nc
2020-07-02 14:14:11,129 INFO [make_fvcom_atmos_forcing] Stored VHFR FVCOM atmospheric forcing precip file: /results/forcing/atmospheric/GEM2.5/vhfr-fvcom/atmos_x2_forecast_precip_20200703.nc
2020-07-02 14:14:26,037 INFO [make_fvcom_atmos_forcing] Stored VHFR FVCOM atmospheric forcing wnd file: /results/forcing/atmospheric/GEM2.5/vhfr-fvcom/atmos_x2_forecast_wnd_20200703.nc
2020-07-02 14:14:26,107 INFO [make_fvcom_atmos_forcing] FVCOM x2 forecast run atmospheric forcing file for 2020-07-02 created
2020-07-02 14:14:26,109 INFO [manager] checklist updated with [FVCOM atmospheric forcing] items from make_fvcom_atmos_forcing worker
2020-07-02 14:14:26,523 INFO [manager] launching NextWorker(module='nowcast.workers.upload_fvcom_atmos_forcing', args=['arbutus.cloud-nowcast', 'x2', 'forecast', '--run-date', '2020-07-02'], host='localhost')
2020-07-02 14:14:27,595 INFO [upload_fvcom_atmos_forcing] running in process 5084
2020-07-02 14:14:27,596 INFO [upload_fvcom_atmos_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-02 14:14:27,597 INFO [upload_fvcom_atmos_forcing] publishing log messages to tcp://*:5564
2020-07-02 14:14:27,598 INFO [upload_fvcom_atmos_forcing] connected to skookum.eos.ubc.ca port 5555
2020-07-02 14:14:27,599 INFO [upload_fvcom_atmos_forcing] Uploading VHFR FVCOM atmospheric forcing files for 2020-07-02 x2 forecast run to arbutus.cloud-nowcast
2020-07-02 14:14:32,449 INFO [upload_fvcom_atmos_forcing] FVCOM x2 forecast run atmospheric forcing files for 2020-07-02 uploaded to arbutus.cloud-nowcast
2020-07-02 14:14:32,454 INFO [manager] checklist updated with [FVCOM atmospheric forcing upload] items from upload_fvcom_atmos_forcing worker
2020-07-02 14:14:32,732 INFO [manager] launching NextWorker(module='nowcast.workers.run_fvcom', args=['arbutus.cloud-nowcast', 'x2', 'forecast', '--run-date', '2020-07-02'], host='arbutus.cloud-nowcast')
2020-07-02 14:14:34,756 INFO [run_fvcom] running in process 13666
2020-07-02 14:14:34,757 INFO [run_fvcom] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2020-07-02 14:14:34,758 INFO [run_fvcom] publishing log messages to tcp://*:5584
2020-07-02 14:14:34,759 INFO [run_fvcom] connected to skookum.eos.ubc.ca port 5555
2020-07-02 14:14:34,801 INFO [run_fvcom] forecast: launching /nemoShare/MEOPAR/nowcast-sys/fvcom-runs/20200702-21h14m34.792429s/VHFR_FVCOM.sh on arbutus.cloud-nowcast
2020-07-02 14:14:34,818 INFO [run_fvcom] x2 forecast FVCOM VH-FR run for 2020-07-02 on arbutus.cloud-nowcast started
2020-07-02 14:14:34,822 INFO [manager] checklist updated with [FVCOM run] items from run_fvcom worker
2020-07-02 14:14:35,064 INFO [manager] launching NextWorker(module='nowcast.workers.watch_fvcom', args=['arbutus.cloud-nowcast', 'x2', 'forecast'], host='arbutus.cloud-nowcast')
2020-07-02 14:14:36,965 INFO [watch_fvcom] running in process 13769
2020-07-02 14:14:37,013 INFO [watch_fvcom] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2020-07-02 14:14:37,013 INFO [watch_fvcom] publishing log messages to tcp://*:5586
2020-07-02 14:14:37,014 INFO [watch_fvcom] connected to skookum.eos.ubc.ca port 5555
2020-07-02 14:14:37,018 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: no run progress found in fvcom.log ; continuing to watch...
2020-07-02 14:14:40,806 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39168109 = 2020-07-02 07:30:43 UTC estimated time to finish: 0000:08:02:08
2020-07-02 14:16:54,443 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4623552 = 2020-07-02 12:47:20 UTC, 53.3% complete
2020-07-02 14:17:29,318 CRITICAL [make_plots] unhandled exception:
Traceback (most recent call last):
  File "/SalishSeaCast/NEMO_Nowcast/nemo_nowcast/worker.py", line 390, in _do_work
    checklist = self.worker_func(
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 325, in make_plots
    checklist = _render_figures(
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1038, in _render_figures
    fig = _calc_figure(fig_func, args, kwargs)
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure
    fig = fig_func(*args, **kwargs)
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure
    plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname)
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 80, in _prep_plot_data
    date_bytes = fvcom_results_dataset.variables["Times"][time_index, :]
  File "netCDF4/_netCDF4.pyx", line 4384, in netCDF4._netCDF4.Variable.__getitem__
  File "netCDF4/_netCDF4.pyx", line 4072, in netCDF4._netCDF4.Variable.shape.__get__
  File "netCDF4/_netCDF4.pyx", line 4020, in netCDF4._netCDF4.Variable._getdims
  File "netCDF4/_netCDF4.pyx", line 1887, in netCDF4._netCDF4._ensure_nc_success
RuntimeError: NetCDF: Not a valid ID
2020-07-02 14:19:37,107 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15728494 = 2020-07-03 00:31:34 UTC estimated time to finish: 0000:05:28:02
2020-07-02 14:19:40,929 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39169692 = 2020-07-02 07:41:16 UTC estimated time to finish: 0000:07:55:52
2020-07-02 14:21:54,541 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4623606 = 2020-07-02 13:23:20 UTC, 55.8% complete
2020-07-02 14:22:29,725 INFO [make_plots] finished rendering nemo nowcast-green research 02jul20 figures
2020-07-02 14:22:29,900 INFO [make_plots] nemo research plots for 2020-07-02 nowcast-green completed
2020-07-02 14:22:30,056 INFO [manager] checklist updated with [plots] items from make_plots worker
2020-07-02 14:24:37,209 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15730266 = 2020-07-03 01:01:06 UTC estimated time to finish: 0000:05:23:24
2020-07-02 14:24:41,050 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39171427 = 2020-07-02 07:52:50 UTC estimated time to finish: 0000:07:49:11
2020-07-02 14:26:54,621 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4623661 = 2020-07-02 14:00:00 UTC, 58.4% complete
2020-07-02 14:29:37,313 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15732283 = 2020-07-03 01:34:43 UTC estimated time to finish: 0000:05:18:24
2020-07-02 14:29:41,173 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39173161 = 2020-07-02 08:04:24 UTC estimated time to finish: 0000:07:42:46
2020-07-02 14:31:54,722 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4623715 = 2020-07-02 14:36:00 UTC, 60.9% complete
2020-07-02 14:34:37,416 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15734246 = 2020-07-03 02:07:26 UTC estimated time to finish: 0000:05:14:15
2020-07-02 14:34:41,297 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39174895 = 2020-07-02 08:15:58 UTC estimated time to finish: 0000:07:36:13
2020-07-02 14:36:54,823 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4623770 = 2020-07-02 15:12:40 UTC, 63.4% complete
2020-07-02 14:39:37,521 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15736263 = 2020-07-03 02:41:03 UTC estimated time to finish: 0000:05:07:55
2020-07-02 14:39:41,420 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39176640 = 2020-07-02 08:27:36 UTC estimated time to finish: 0000:07:29:38
2020-07-02 14:41:54,848 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4623825 = 2020-07-02 15:49:20 UTC, 66.0% complete
2020-07-02 14:44:37,625 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15738076 = 2020-07-03 03:11:16 UTC estimated time to finish: 0000:05:02:51
2020-07-02 14:44:41,543 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39178215 = 2020-07-02 08:38:06 UTC estimated time to finish: 0000:07:23:49
2020-07-02 14:46:54,949 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4623879 = 2020-07-02 16:25:20 UTC, 68.5% complete
2020-07-02 14:49:37,730 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15740080 = 2020-07-03 03:44:40 UTC estimated time to finish: 0000:04:57:29
2020-07-02 14:49:41,668 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39179950 = 2020-07-02 08:49:40 UTC estimated time to finish: 0000:07:17:21
2020-07-02 14:51:55,050 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4623934 = 2020-07-02 17:02:00 UTC, 71.0% complete
2020-07-02 14:54:37,837 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15742056 = 2020-07-03 04:17:36 UTC estimated time to finish: 0000:04:53:04
2020-07-02 14:54:41,794 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39181683 = 2020-07-02 09:01:13 UTC estimated time to finish: 0000:07:11:08
2020-07-02 14:56:55,151 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4623989 = 2020-07-02 17:38:40 UTC, 73.6% complete
2020-07-02 14:59:37,943 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15744060 = 2020-07-03 04:51:00 UTC estimated time to finish: 0000:04:48:03
2020-07-02 14:59:41,920 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39183430 = 2020-07-02 09:12:52 UTC estimated time to finish: 0000:07:04:49
2020-07-02 15:01:52,127 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-west/18/ to /results/forcing/atmospheric/GEM2.5/GRIB/20200702/18/
2020-07-02 15:01:52,128 INFO [collect_weather] 2020-07-02 2.5km weather forecast 18 collection complete
2020-07-02 15:01:52,131 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2020-07-02 15:01:52,757 INFO [manager] launching NextWorker(module='nowcast.workers.download_weather', args=['00', '1km', '--no-verify-certs'], host='localhost')
2020-07-02 15:01:52,762 INFO [manager] launching NextWorker(module='nowcast.workers.download_weather', args=['12', '1km', '--no-verify-certs'], host='localhost')
2020-07-02 15:01:52,766 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['00', '2.5km'], host='localhost')
2020-07-02 15:01:53,862 INFO [download_weather] running in process 6093
2020-07-02 15:01:53,863 INFO [download_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-02 15:01:53,864 INFO [download_weather] publishing log messages to tcp://*:5562
2020-07-02 15:01:53,864 INFO [download_weather] connected to skookum.eos.ubc.ca port 5555
2020-07-02 15:01:53,865 INFO [download_weather] downloading 12 1 km forecast GRIB2 files for 20200702
2020-07-02 15:01:53,866 INFO [download_weather] running in process 6092
2020-07-02 15:01:53,867 INFO [download_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-02 15:01:53,869 INFO [download_weather] publishing log messages to tcp://*:5563
2020-07-02 15:01:53,870 INFO [download_weather] connected to skookum.eos.ubc.ca port 5555
2020-07-02 15:01:53,870 INFO [download_weather] downloading 00 1 km forecast GRIB2 files for 20200702
2020-07-02 15:01:53,889 INFO [collect_weather] running in process 6094
2020-07-02 15:01:53,890 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-02 15:01:53,891 INFO [collect_weather] publishing log messages to tcp://*:5564
2020-07-02 15:01:53,892 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2020-07-02 15:01:53,941 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-west/00/
2020-07-02 15:01:55,252 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4624043 = 2020-07-02 18:14:40 UTC, 76.1% complete
2020-07-02 15:04:38,051 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15745791 = 2020-07-03 05:19:51 UTC estimated time to finish: 0000:04:44:28
2020-07-02 15:04:42,046 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39185001 = 2020-07-02 09:23:20 UTC estimated time to finish: 0000:06:59:13
2020-07-02 15:06:55,352 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4624098 = 2020-07-02 18:51:20 UTC, 78.6% complete
2020-07-02 15:09:38,159 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15747808 = 2020-07-03 05:53:28 UTC estimated time to finish: 0000:04:38:56
2020-07-02 15:09:42,174 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39186738 = 2020-07-02 09:34:55 UTC estimated time to finish: 0000:06:53:04
2020-07-02 15:11:55,453 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4624153 = 2020-07-02 19:28:00 UTC, 81.1% complete
2020-07-02 15:14:38,267 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15749893 = 2020-07-03 06:28:13 UTC estimated time to finish: 0000:04:33:18
2020-07-02 15:14:42,301 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39188443 = 2020-07-02 09:46:17 UTC estimated time to finish: 0000:06:47:04
2020-07-02 15:16:55,554 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4624207 = 2020-07-02 20:04:00 UTC, 83.6% complete
2020-07-02 15:19:38,376 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15751789 = 2020-07-03 06:59:49 UTC estimated time to finish: 0000:04:28:12
2020-07-02 15:19:42,428 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39190219 = 2020-07-02 09:58:07 UTC estimated time to finish: 0000:06:40:49
2020-07-02 15:19:53,603 INFO [download_weather] 2020-07-02 1km weather forecast 00 downloads complete
2020-07-02 15:19:53,608 INFO [manager] checklist updated with [weather forecast] items from download_weather worker
2020-07-02 15:20:24,558 INFO [download_weather] 2020-07-02 1km weather forecast 12 downloads complete
2020-07-02 15:20:24,562 INFO [manager] checklist updated with [weather forecast] items from download_weather worker
2020-07-02 15:21:55,655 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4624262 = 2020-07-02 20:40:40 UTC, 86.2% complete
2020-07-02 15:24:38,486 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15753793 = 2020-07-03 07:33:13 UTC estimated time to finish: 0000:04:22:53
2020-07-02 15:24:42,556 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39191790 = 2020-07-02 10:08:36 UTC estimated time to finish: 0000:06:35:19
2020-07-02 15:26:55,755 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4624316 = 2020-07-02 21:16:40 UTC, 88.7% complete
2020-07-02 15:29:38,596 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15755606 = 2020-07-03 08:03:26 UTC estimated time to finish: 0000:04:18:30
2020-07-02 15:29:42,684 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39193536 = 2020-07-02 10:20:14 UTC estimated time to finish: 0000:06:29:16
2020-07-02 15:31:55,856 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4624371 = 2020-07-02 21:53:20 UTC, 91.2% complete
2020-07-02 15:34:38,707 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15757570 = 2020-07-03 08:36:10 UTC estimated time to finish: 0000:04:13:40
2020-07-02 15:34:42,814 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39195272 = 2020-07-02 10:31:48 UTC estimated time to finish: 0000:06:23:18
2020-07-02 15:36:55,957 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4624426 = 2020-07-02 22:30:00 UTC, 93.8% complete
2020-07-02 15:39:38,819 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15759546 = 2020-07-03 09:09:06 UTC estimated time to finish: 0000:04:08:31
2020-07-02 15:39:42,944 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39197007 = 2020-07-02 10:43:22 UTC estimated time to finish: 0000:06:17:21
2020-07-02 15:41:56,058 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4624480 = 2020-07-02 23:06:00 UTC, 96.3% complete
2020-07-02 15:44:38,932 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15761550 = 2020-07-03 09:42:30 UTC estimated time to finish: 0000:04:03:25
2020-07-02 15:44:43,075 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39198550 = 2020-07-02 10:53:40 UTC estimated time to finish: 0000:06:12:06
2020-07-02 15:46:56,159 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4624535 = 2020-07-02 23:42:40 UTC, 98.8% complete
2020-07-02 15:49:39,045 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15763362 = 2020-07-03 10:12:42 UTC estimated time to finish: 0000:03:58:35
2020-07-02 15:49:43,206 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39200324 = 2020-07-02 11:05:29 UTC estimated time to finish: 0000:06:06:04
2020-07-02 15:51:56,269 INFO [watch_NEMO] nowcast-dev NEMO run on salish-nowcast completed
2020-07-02 15:51:56,276 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2020-07-02 15:54:39,160 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15765380 = 2020-07-03 10:46:20 UTC estimated time to finish: 0000:03:53:19
2020-07-02 15:54:43,337 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39202059 = 2020-07-02 11:17:03 UTC estimated time to finish: 0000:06:00:10
2020-07-02 15:59:39,274 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15767343 = 2020-07-03 11:19:03 UTC estimated time to finish: 0000:03:48:17
2020-07-02 15:59:43,468 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39203794 = 2020-07-02 11:28:37 UTC estimated time to finish: 0000:05:54:21
2020-07-02 16:04:39,389 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15769320 = 2020-07-03 11:52:00 UTC estimated time to finish: 0000:03:43:16
2020-07-02 16:04:43,564 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39205297 = 2020-07-02 11:38:38 UTC estimated time to finish: 0000:05:49:28
2020-07-02 16:09:39,504 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15771050 = 2020-07-03 12:20:50 UTC estimated time to finish: 0000:03:39:10
2020-07-02 16:09:43,698 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39207030 = 2020-07-02 11:50:12 UTC estimated time to finish: 0000:05:43:42
2020-07-02 16:14:39,549 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15773056 = 2020-07-03 12:54:16 UTC estimated time to finish: 0000:03:34:00
2020-07-02 16:14:43,833 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39208765 = 2020-07-02 12:01:46 UTC estimated time to finish: 0000:05:37:54
2020-07-02 16:19:39,665 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15775031 = 2020-07-03 13:27:11 UTC estimated time to finish: 0000:03:28:55
2020-07-02 16:19:43,965 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39210500 = 2020-07-02 12:13:20 UTC estimated time to finish: 0000:05:32:12
2020-07-02 16:24:39,781 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15777036 = 2020-07-03 14:00:36 UTC estimated time to finish: 0000:03:23:48
2020-07-02 16:24:44,098 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39212247 = 2020-07-02 12:24:58 UTC estimated time to finish: 0000:05:26:25
2020-07-02 16:29:39,876 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15779012 = 2020-07-03 14:33:32 UTC estimated time to finish: 0000:03:18:42
2020-07-02 16:29:44,232 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39213819 = 2020-07-02 12:35:27 UTC estimated time to finish: 0000:05:21:13
2020-07-02 16:34:39,999 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15780825 = 2020-07-03 15:03:45 UTC estimated time to finish: 0000:03:13:57
2020-07-02 16:34:44,366 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39215520 = 2020-07-02 12:46:48 UTC estimated time to finish: 0000:05:15:42
2020-07-02 16:39:40,112 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15782829 = 2020-07-03 15:37:09 UTC estimated time to finish: 0000:03:08:41
2020-07-02 16:39:44,501 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39217219 = 2020-07-02 12:58:07 UTC estimated time to finish: 0000:05:10:09
2020-07-02 16:44:40,232 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15784845 = 2020-07-03 16:10:45 UTC estimated time to finish: 0000:03:03:30
2020-07-02 16:44:44,636 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39218994 = 2020-07-02 13:09:57 UTC estimated time to finish: 0000:05:04:20
2020-07-02 16:49:40,361 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15786809 = 2020-07-03 16:43:29 UTC estimated time to finish: 0000:02:58:29
2020-07-02 16:49:44,773 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39220536 = 2020-07-02 13:20:14 UTC estimated time to finish: 0000:04:59:19
2020-07-02 16:54:40,484 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15788580 = 2020-07-03 17:13:00 UTC estimated time to finish: 0000:02:54:04
2020-07-02 16:54:44,911 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39222231 = 2020-07-02 13:31:32 UTC estimated time to finish: 0000:04:53:56
2020-07-02 16:59:40,604 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15790557 = 2020-07-03 17:45:57 UTC estimated time to finish: 0000:02:48:56
2020-07-02 16:59:45,048 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39223925 = 2020-07-02 13:42:50 UTC estimated time to finish: 0000:04:48:28
2020-07-02 17:04:40,727 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15792520 = 2020-07-03 18:18:40 UTC estimated time to finish: 0000:02:43:58
2020-07-02 17:04:45,185 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39225660 = 2020-07-02 13:54:24 UTC estimated time to finish: 0000:04:42:54
2020-07-02 17:09:40,847 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15794497 = 2020-07-03 18:51:37 UTC estimated time to finish: 0000:02:38:52
2020-07-02 17:09:45,323 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39227202 = 2020-07-02 14:04:40 UTC estimated time to finish: 0000:04:37:56
2020-07-02 17:14:40,971 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15796391 = 2020-07-03 19:23:11 UTC estimated time to finish: 0000:02:33:59
2020-07-02 17:14:45,461 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39228936 = 2020-07-02 14:16:14 UTC estimated time to finish: 0000:04:32:22
2020-07-02 17:19:41,094 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15798355 = 2020-07-03 19:55:55 UTC estimated time to finish: 0000:02:28:52
2020-07-02 17:19:45,600 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39230671 = 2020-07-02 14:27:48 UTC estimated time to finish: 0000:04:26:48
2020-07-02 17:24:41,219 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15800331 = 2020-07-03 20:28:51 UTC estimated time to finish: 0000:02:23:47
2020-07-02 17:24:45,739 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39232420 = 2020-07-02 14:39:28 UTC estimated time to finish: 0000:04:21:10
2020-07-02 17:29:41,343 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15802294 = 2020-07-03 21:01:34 UTC estimated time to finish: 0000:02:18:46
2020-07-02 17:29:45,878 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39234153 = 2020-07-02 14:51:01 UTC estimated time to finish: 0000:04:15:38
2020-07-02 17:34:41,469 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15803998 = 2020-07-03 21:29:58 UTC estimated time to finish: 0000:02:14:33
2020-07-02 17:34:46,019 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39235655 = 2020-07-02 15:01:02 UTC estimated time to finish: 0000:04:10:55
2020-07-02 17:39:41,592 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15805920 = 2020-07-03 22:02:00 UTC estimated time to finish: 0000:02:09:39
2020-07-02 17:39:46,160 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39237390 = 2020-07-02 15:12:36 UTC estimated time to finish: 0000:04:05:23
2020-07-02 17:44:41,719 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15807898 = 2020-07-03 22:34:58 UTC estimated time to finish: 0000:02:04:35
2020-07-02 17:44:46,301 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39239083 = 2020-07-02 15:23:53 UTC estimated time to finish: 0000:04:00:01
2020-07-02 17:49:41,844 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15809820 = 2020-07-03 23:07:00 UTC estimated time to finish: 0000:01:59:43
2020-07-02 17:49:46,440 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39240819 = 2020-07-02 15:35:27 UTC estimated time to finish: 0000:03:54:31
2020-07-02 17:54:41,971 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15811797 = 2020-07-03 23:39:57 UTC estimated time to finish: 0000:01:54:39
2020-07-02 17:54:46,580 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39242361 = 2020-07-02 15:45:44 UTC estimated time to finish: 0000:03:49:38
2020-07-02 17:59:42,098 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15813568 = 2020-07-04 00:09:28 UTC estimated time to finish: 0000:01:50:05
2020-07-02 17:59:46,723 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39244096 = 2020-07-02 15:57:18 UTC estimated time to finish: 0000:03:44:09
2020-07-02 18:04:42,228 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15815531 = 2020-07-04 00:42:11 UTC estimated time to finish: 0000:01:45:03
2020-07-02 18:04:46,865 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39245760 = 2020-07-02 16:08:24 UTC estimated time to finish: 0000:03:38:59
2020-07-02 18:09:42,356 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15817507 = 2020-07-04 01:15:07 UTC estimated time to finish: 0000:01:39:56
2020-07-02 18:09:47,010 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39247455 = 2020-07-02 16:19:42 UTC estimated time to finish: 0000:03:33:41
2020-07-02 18:14:42,485 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15819512 = 2020-07-04 01:48:32 UTC estimated time to finish: 0000:01:34:45
2020-07-02 18:14:47,154 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39248999 = 2020-07-02 16:29:59 UTC estimated time to finish: 0000:03:28:49
2020-07-02 18:19:42,614 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15821325 = 2020-07-04 02:18:45 UTC estimated time to finish: 0000:01:30:02
2020-07-02 18:19:47,295 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39250733 = 2020-07-02 16:41:33 UTC estimated time to finish: 0000:03:23:20
2020-07-02 18:24:42,746 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15823341 = 2020-07-04 02:52:21 UTC estimated time to finish: 0000:01:24:48
2020-07-02 18:24:47,440 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39252468 = 2020-07-02 16:53:07 UTC estimated time to finish: 0000:03:17:52
2020-07-02 18:29:42,876 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15825304 = 2020-07-04 03:25:04 UTC estimated time to finish: 0000:01:19:45
2020-07-02 18:29:47,596 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39254202 = 2020-07-02 17:04:40 UTC estimated time to finish: 0000:03:12:25
2020-07-02 18:34:43,007 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15827240 = 2020-07-04 03:57:20 UTC estimated time to finish: 0000:01:14:46
2020-07-02 18:34:47,745 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39255936 = 2020-07-02 17:16:14 UTC estimated time to finish: 0000:03:07:00
2020-07-02 18:39:43,139 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15829052 = 2020-07-04 04:27:32 UTC estimated time to finish: 0000:01:10:06
2020-07-02 18:39:47,892 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39257480 = 2020-07-02 17:26:32 UTC estimated time to finish: 0000:03:02:09
2020-07-02 18:44:43,273 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15830989 = 2020-07-04 04:59:49 UTC estimated time to finish: 0000:01:05:07
2020-07-02 18:44:48,038 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39259215 = 2020-07-02 17:38:06 UTC estimated time to finish: 0000:02:56:43
2020-07-02 18:49:43,406 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15832870 = 2020-07-04 05:31:10 UTC estimated time to finish: 0000:01:00:18
2020-07-02 18:49:48,186 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39260950 = 2020-07-02 17:49:40 UTC estimated time to finish: 0000:02:51:17
2020-07-02 18:54:43,543 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15834847 = 2020-07-04 06:04:07 UTC estimated time to finish: 0000:00:55:12
2020-07-02 18:54:48,334 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39262683 = 2020-07-02 18:01:13 UTC estimated time to finish: 0000:02:45:52
2020-07-02 18:59:43,677 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15836770 = 2020-07-04 06:36:10 UTC estimated time to finish: 0000:00:50:15
2020-07-02 18:59:48,482 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39264226 = 2020-07-02 18:11:30 UTC estimated time to finish: 0000:02:41:04
2020-07-02 19:04:43,812 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15838513 = 2020-07-04 07:05:13 UTC estimated time to finish: 0000:00:45:46
2020-07-02 19:04:48,632 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39265920 = 2020-07-02 18:22:48 UTC estimated time to finish: 0000:02:35:47
2020-07-02 19:09:43,947 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15840436 = 2020-07-04 07:37:16 UTC estimated time to finish: 0000:00:40:48
2020-07-02 19:09:48,781 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39267669 = 2020-07-02 18:34:27 UTC estimated time to finish: 0000:02:30:20
2020-07-02 19:14:44,084 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15842411 = 2020-07-04 08:10:11 UTC estimated time to finish: 0000:00:35:42
2020-07-02 19:14:48,931 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39269402 = 2020-07-02 18:46:00 UTC estimated time to finish: 0000:02:24:56
2020-07-02 19:19:44,219 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15844335 = 2020-07-04 08:42:15 UTC estimated time to finish: 0000:00:30:43
2020-07-02 19:19:49,080 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39270973 = 2020-07-02 18:56:29 UTC estimated time to finish: 0000:02:20:02
2020-07-02 19:24:44,357 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15846120 = 2020-07-04 09:12:00 UTC estimated time to finish: 0000:00:26:06
2020-07-02 19:24:49,232 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39272760 = 2020-07-02 19:08:24 UTC estimated time to finish: 0000:02:14:28
2020-07-02 19:29:44,496 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15848084 = 2020-07-04 09:44:44 UTC estimated time to finish: 0000:00:21:01
2020-07-02 19:29:49,383 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39274496 = 2020-07-02 19:19:58 UTC estimated time to finish: 0000:02:09:04
2020-07-02 19:34:44,635 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15850019 = 2020-07-04 10:16:59 UTC estimated time to finish: 0000:00:16:00
2020-07-02 19:34:49,536 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39276231 = 2020-07-02 19:31:32 UTC estimated time to finish: 0000:02:03:41
2020-07-02 19:39:44,774 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15851983 = 2020-07-04 10:49:43 UTC estimated time to finish: 0000:00:10:55
2020-07-02 19:39:49,688 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39277966 = 2020-07-02 19:43:06 UTC estimated time to finish: 0000:01:58:19
2020-07-02 19:44:44,914 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15853755 = 2020-07-04 11:19:15 UTC estimated time to finish: 0000:00:06:20
2020-07-02 19:44:49,840 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39279550 = 2020-07-02 19:53:40 UTC estimated time to finish: 0000:01:53:24
2020-07-02 19:49:45,053 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15855772 = 2020-07-04 11:52:52 UTC estimated time to finish: 0000:00:01:06
2020-07-02 19:49:50,000 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39281283 = 2020-07-02 20:05:13 UTC estimated time to finish: 0000:01:48:03
2020-07-02 19:54:45,162 INFO [watch_fvcom] x2 forecast FVCOM VH-FR run on arbutus.cloud-nowcast completed
2020-07-02 19:54:45,168 INFO [manager] checklist updated with [FVCOM run] items from watch_fvcom worker
2020-07-02 19:54:45,766 INFO [manager] launching NextWorker(module='nowcast.workers.download_fvcom_results', args=['arbutus.cloud-nowcast', 'x2', 'forecast', '--run-date', '2020-07-02'], host='localhost')
2020-07-02 19:54:47,061 INFO [download_fvcom_results] running in process 12579
2020-07-02 19:54:47,062 INFO [download_fvcom_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-02 19:54:47,063 INFO [download_fvcom_results] publishing log messages to tcp://*:5562
2020-07-02 19:54:47,064 INFO [download_fvcom_results] connected to skookum.eos.ubc.ca port 5555
2020-07-02 19:54:50,149 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39283646 = 2020-07-02 20:20:58 UTC estimated time to finish: 0000:01:40:25
2020-07-02 19:55:50,274 INFO [download_fvcom_results] VHFR FVCOM x2 forecast 2020-07-02 results files from arbutus.cloud-nowcast downloaded
2020-07-02 19:55:50,282 INFO [manager] checklist updated with [VHFR FVCOM results files] items from download_fvcom_results worker
2020-07-02 19:55:50,604 INFO [manager] launching NextWorker(module='nowcast.workers.get_vfpa_hadcp', args=['--data-date', '2020-07-02'], host='localhost')
2020-07-02 19:55:50,608 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['fvcom', 'forecast-x2', 'research', '--run-date', '2020-07-02'], host='localhost')
2020-07-02 19:55:50,611 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['fvcom', 'forecast', '--run-date', '2020-07-02'], host='localhost')
2020-07-02 19:55:51,569 INFO [update_forecast_datasets] running in process 12624
2020-07-02 19:55:51,570 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-02 19:55:51,571 INFO [update_forecast_datasets] publishing log messages to tcp://*:5562
2020-07-02 19:55:51,571 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2020-07-02 19:55:51,572 INFO [update_forecast_datasets] updating fvcom most_recent_forecast directory from forecast/02jul20 run
2020-07-02 19:55:51,573 INFO [update_forecast_datasets] updated fvcom most_recent_forecast directory from forecast/02jul20 run
2020-07-02 19:55:51,574 INFO [update_forecast_datasets] fvcom 2020-07-02 forecast rolling forecast datasets updated
2020-07-02 19:55:51,575 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2020-07-02 19:55:51,899 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['fvcom-forecast'], host='localhost')
2020-07-02 19:55:52,587 INFO [get_vfpa_hadcp] running in process 12622
2020-07-02 19:55:52,588 INFO [get_vfpa_hadcp] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-02 19:55:52,589 INFO [get_vfpa_hadcp] publishing log messages to tcp://*:5562
2020-07-02 19:55:52,590 INFO [get_vfpa_hadcp] connected to skookum.eos.ubc.ca port 5555
2020-07-02 19:55:52,591 INFO [get_vfpa_hadcp] processing VFPA HADCP data from 2nd Narrows Rail Bridge for 2020-07-02
2020-07-02 19:55:53,508 INFO [get_vfpa_hadcp] added VFPA HADCP data from 2nd Narrows Rail Bridge for 2020-07-02 to /opp/observations/AISDATA/netcdf/VFPA_2ND_NARROWS_HADCP_2s_202007.nc
2020-07-02 19:55:53,509 INFO [get_vfpa_hadcp] VFPA HADCP observations added to 2020-07 netcdf file
2020-07-02 19:55:53,511 INFO [manager] checklist updated with [VFPA HADCP data] items from get_vfpa_hadcp worker
2020-07-02 19:55:53,784 INFO [make_plots] running in process 12623
2020-07-02 19:55:53,785 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-02 19:55:53,786 INFO [make_plots] publishing log messages to tcp://*:5563
2020-07-02 19:55:53,787 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2020-07-02 19:55:53,798 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['VFPA-HADCP'], host='localhost')
2020-07-02 19:55:58,470 INFO [ping_erddap] running in process 12641
2020-07-02 19:55:58,471 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-02 19:55:58,471 INFO [ping_erddap] publishing log messages to tcp://*:5562
2020-07-02 19:55:58,472 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2020-07-02 19:55:58,473 INFO [ping_erddap] VFPA-HADCP ERDDAP dataset flag file(s) created
2020-07-02 19:55:58,475 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2020-07-02 19:55:58,780 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['fvcom', 'nowcast-x2', 'publish', '--run-date', '2020-07-02'], host='localhost')
2020-07-02 19:55:58,783 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['fvcom', 'forecast-x2', 'publish', '--run-date', '2020-07-02'], host='localhost')
2020-07-02 19:56:01,746 INFO [make_plots] running in process 12647
2020-07-02 19:56:01,747 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-02 19:56:01,748 INFO [make_plots] publishing log messages to tcp://*:5562
2020-07-02 19:56:01,749 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2020-07-02 19:56:01,750 INFO [make_plots] running in process 12648
2020-07-02 19:56:01,751 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-02 19:56:01,752 INFO [make_plots] publishing log messages to tcp://*:5565
2020-07-02 19:56:01,753 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2020-07-02 19:56:05,848 INFO [make_plots] preparing render list for 2020-07-02 VHFR FVCOM forecast-x2 research figures
2020-07-02 19:56:05,855 INFO [make_plots] added EnglishBay_surface_currents figures to 2020-07-02 VHFR FVCOM forecast-x2 research render list
2020-07-02 19:56:05,862 INFO [make_plots] added VancouverHarbour_surface_currents figures to 2020-07-02 VHFR FVCOM forecast-x2 research render list
2020-07-02 19:56:05,868 INFO [make_plots] added IndianArm_surface_currents figures to 2020-07-02 VHFR FVCOM forecast-x2 research render list
2020-07-02 19:56:05,874 INFO [make_plots] added VancouverHarbour_thalweg figures to 2020-07-02 VHFR FVCOM forecast-x2 research render list
2020-07-02 19:56:05,881 INFO [make_plots] added PortMoody_thalweg figures to 2020-07-02 VHFR FVCOM forecast-x2 research render list
2020-07-02 19:56:05,888 INFO [make_plots] added IndianArm_thalweg figures to 2020-07-02 VHFR FVCOM forecast-x2 research render list
2020-07-02 19:56:05,889 INFO [make_plots] starting to render fvcom forecast-x2 research 02jul20 figures
2020-07-02 19:56:06,901 INFO [make_plots] preparing render list for 2020-07-02 VHFR FVCOM forecast-x2 publish figures
2020-07-02 19:56:06,902 INFO [make_plots] added Calamity Point figure to 2020-07-02 VHFR FVCOM forecast-x2 publish render list
2020-07-02 19:56:06,902 INFO [make_plots] added Indian Arm Head figure to 2020-07-02 VHFR FVCOM forecast-x2 publish render list
2020-07-02 19:56:06,903 INFO [make_plots] added New Westminster figure to 2020-07-02 VHFR FVCOM forecast-x2 publish render list
2020-07-02 19:56:06,904 INFO [make_plots] added Port Moody figure to 2020-07-02 VHFR FVCOM forecast-x2 publish render list
2020-07-02 19:56:06,904 INFO [make_plots] added Sand Heads figure to 2020-07-02 VHFR FVCOM forecast-x2 publish render list
2020-07-02 19:56:06,905 INFO [make_plots] added Sandy Cove figure to 2020-07-02 VHFR FVCOM forecast-x2 publish render list
2020-07-02 19:56:06,905 INFO [make_plots] added Vancouver Harbour figure to 2020-07-02 VHFR FVCOM forecast-x2 publish render list
2020-07-02 19:56:06,906 INFO [make_plots] added Woodwards Landing figure to 2020-07-02 VHFR FVCOM forecast-x2 publish render list
2020-07-02 19:56:06,907 INFO [make_plots] added 2ndNarrows_current figure to 2020-07-02 VHFR FVCOM forecast-x2 publish render list
2020-07-02 19:56:06,907 INFO [make_plots] starting to render fvcom forecast-x2 publish 02jul20 figures
2020-07-02 19:56:06,908 INFO [make_plots] preparing render list for 2020-07-02 VHFR FVCOM nowcast-x2 publish figures
2020-07-02 19:56:06,909 INFO [make_plots] added Calamity Point figure to 2020-07-02 VHFR FVCOM nowcast-x2 publish render list
2020-07-02 19:56:06,910 INFO [make_plots] added Indian Arm Head figure to 2020-07-02 VHFR FVCOM nowcast-x2 publish render list
2020-07-02 19:56:06,910 INFO [make_plots] added New Westminster figure to 2020-07-02 VHFR FVCOM nowcast-x2 publish render list
2020-07-02 19:56:06,911 INFO [make_plots] added Port Moody figure to 2020-07-02 VHFR FVCOM nowcast-x2 publish render list
2020-07-02 19:56:06,912 INFO [make_plots] added Sand Heads figure to 2020-07-02 VHFR FVCOM nowcast-x2 publish render list
2020-07-02 19:56:06,912 INFO [make_plots] added Sandy Cove figure to 2020-07-02 VHFR FVCOM nowcast-x2 publish render list
2020-07-02 19:56:06,913 INFO [make_plots] added Vancouver Harbour figure to 2020-07-02 VHFR FVCOM nowcast-x2 publish render list
2020-07-02 19:56:06,914 INFO [make_plots] added Woodwards Landing figure to 2020-07-02 VHFR FVCOM nowcast-x2 publish render list
2020-07-02 19:56:06,915 INFO [make_plots] added 2ndNarrows_current figure to 2020-07-02 VHFR FVCOM nowcast-x2 publish render list
2020-07-02 19:56:06,915 INFO [make_plots] starting to render fvcom nowcast-x2 publish 02jul20 figures
2020-07-02 19:56:06,929 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07724 Calamity Point from 2020-07-01 23:55:00 to 2020-07-03 00:00:00
2020-07-02 19:56:06,930 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07724 Calamity Point from 2020-07-01 23:55:00 to 2020-07-04 12:00:00
2020-07-02 19:56:30,280 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07724 Calamity Point from 2020-07-02 00:00:00 to 2020-07-03 00:00:00
2020-07-02 19:56:31,765 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07774 Indian Arm Head from 2020-07-01 23:55:00 to 2020-07-03 00:00:00
2020-07-02 19:56:32,347 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07724 Calamity Point from 2020-07-02 00:00:00 to 2020-07-04 12:00:00
2020-07-02 19:56:32,466 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07774 Indian Arm Head from 2020-07-02 00:00:00 to 2020-07-03 00:00:00
2020-07-02 19:56:34,099 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07774 Indian Arm Head from 2020-07-01 23:55:00 to 2020-07-04 12:00:00
2020-07-02 19:56:34,265 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07654 New Westminster from 2020-07-01 23:55:00 to 2020-07-03 00:00:00
2020-07-02 19:56:34,660 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07774 Indian Arm Head from 2020-07-02 00:00:00 to 2020-07-04 12:00:00
2020-07-02 19:56:36,009 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07654 New Westminster from 2020-07-01 23:55:00 to 2020-07-04 12:00:00
2020-07-02 19:56:56,239 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07654 New Westminster from 2020-07-02 00:00:00 to 2020-07-03 00:00:00
2020-07-02 19:56:59,218 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07755 Port Moody from 2020-07-01 23:55:00 to 2020-07-03 00:00:00
2020-07-02 19:57:00,466 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07654 New Westminster from 2020-07-02 00:00:00 to 2020-07-04 12:00:00
2020-07-02 19:57:05,334 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07755 Port Moody from 2020-07-01 23:55:00 to 2020-07-04 12:00:00
2020-07-02 19:57:20,574 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07755 Port Moody from 2020-07-02 00:00:00 to 2020-07-03 00:00:00
2020-07-02 19:57:23,480 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07594 Sand Heads from 2020-07-01 23:55:00 to 2020-07-03 00:00:00
2020-07-02 19:57:29,246 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07755 Port Moody from 2020-07-02 00:00:00 to 2020-07-04 12:00:00
2020-07-02 19:57:34,195 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07594 Sand Heads from 2020-07-01 23:55:00 to 2020-07-04 12:00:00
2020-07-02 19:57:45,354 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07594 Sand Heads from 2020-07-02 00:00:00 to 2020-07-03 00:00:00
2020-07-02 19:57:48,544 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07786 Sandy Cove from 2020-07-01 23:55:00 to 2020-07-03 00:00:00
2020-07-02 19:57:57,636 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07594 Sand Heads from 2020-07-02 00:00:00 to 2020-07-04 12:00:00
2020-07-02 19:58:02,519 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07786 Sandy Cove from 2020-07-01 23:55:00 to 2020-07-04 12:00:00
2020-07-02 19:58:09,619 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07795 Point Atkinson from 2020-07-02 00:00:00 to 2020-07-03 00:00:00
2020-07-02 19:58:12,323 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07735 Vancouver Harbour from 2020-07-01 23:55:00 to 2020-07-03 00:00:00
2020-07-02 19:58:26,036 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07795 Point Atkinson from 2020-07-02 00:00:00 to 2020-07-04 12:00:00
2020-07-02 19:58:30,807 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07735 Vancouver Harbour from 2020-07-01 23:55:00 to 2020-07-04 12:00:00
2020-07-02 19:58:33,759 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07735 Vancouver Harbour from 2020-07-02 00:00:00 to 2020-07-03 00:00:00
2020-07-02 19:58:36,569 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07610 Woodwards Landing from 2020-07-01 23:55:00 to 2020-07-03 00:00:00
2020-07-02 19:58:54,212 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07735 Vancouver Harbour from 2020-07-02 00:00:00 to 2020-07-04 12:00:00
2020-07-02 19:58:58,227 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07610 Woodwards Landing from 2020-07-02 00:00:00 to 2020-07-03 00:00:00
2020-07-02 19:58:59,214 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07610 Woodwards Landing from 2020-07-01 23:55:00 to 2020-07-04 12:00:00
2020-07-02 19:59:02,847 INFO [make_plots] finished rendering fvcom nowcast-x2 publish 02jul20 figures
2020-07-02 19:59:03,428 INFO [make_plots] fvcom publish plots for 2020-07-02 nowcast-x2 completed
2020-07-02 19:59:03,436 INFO [manager] checklist updated with [plots] items from make_plots worker
2020-07-02 19:59:23,279 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07610 Woodwards Landing from 2020-07-02 00:00:00 to 2020-07-04 12:00:00
2020-07-02 19:59:28,203 INFO [make_plots] finished rendering fvcom forecast-x2 publish 02jul20 figures
2020-07-02 19:59:28,731 INFO [make_plots] fvcom publish plots for 2020-07-02 forecast-x2 completed
2020-07-02 19:59:28,738 INFO [manager] checklist updated with [plots] items from make_plots worker
2020-07-02 19:59:50,305 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39286133 = 2020-07-02 20:37:33 UTC estimated time to finish: 0000:01:32:24
2020-07-02 20:04:50,460 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39288427 = 2020-07-02 20:52:50 UTC estimated time to finish: 0000:01:25:04
2020-07-02 20:09:50,617 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39290953 = 2020-07-02 21:09:41 UTC estimated time to finish: 0000:01:17:05
2020-07-02 20:14:50,777 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39293439 = 2020-07-02 21:26:15 UTC estimated time to finish: 0000:01:09:17
2020-07-02 20:19:50,940 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39295966 = 2020-07-02 21:43:06 UTC estimated time to finish: 0000:01:01:26
2020-07-02 20:24:51,098 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39298219 = 2020-07-02 21:58:07 UTC estimated time to finish: 0000:00:54:30
2020-07-02 20:29:51,258 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39300745 = 2020-07-02 22:14:58 UTC estimated time to finish: 0000:00:46:46
2020-07-02 20:34:51,418 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39303272 = 2020-07-02 22:31:48 UTC estimated time to finish: 0000:00:39:07
2020-07-02 20:39:51,579 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39305758 = 2020-07-02 22:48:23 UTC estimated time to finish: 0000:00:31:39
2020-07-02 20:44:51,741 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39308283 = 2020-07-02 23:05:13 UTC estimated time to finish: 0000:00:24:07
2020-07-02 20:49:51,904 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39310536 = 2020-07-02 23:20:14 UTC estimated time to finish: 0000:00:17:27
2020-07-02 20:54:52,065 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39313064 = 2020-07-02 23:37:05 UTC estimated time to finish: 0000:00:10:01
2020-07-02 20:59:52,232 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39315590 = 2020-07-02 23:53:56 UTC estimated time to finish: 0000:00:02:38
2020-07-02 21:00:04,370 INFO [make_plots] finished rendering fvcom forecast-x2 research 02jul20 figures
2020-07-02 21:00:04,371 INFO [make_plots] fvcom research plots for 2020-07-02 forecast-x2 completed
2020-07-02 21:00:04,794 INFO [manager] checklist updated with [plots] items from make_plots worker
2020-07-02 21:04:16,051 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-west/00/ to /results/forcing/atmospheric/GEM2.5/GRIB/20200703/00/
2020-07-02 21:04:16,052 INFO [collect_weather] 2020-07-03 2.5km weather forecast 00 collection complete
2020-07-02 21:04:16,055 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2020-07-02 21:04:16,750 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['06', '2.5km'], host='localhost')
2020-07-02 21:04:17,806 INFO [collect_weather] running in process 14177
2020-07-02 21:04:17,807 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-02 21:04:17,808 INFO [collect_weather] publishing log messages to tcp://*:5562
2020-07-02 21:04:17,809 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2020-07-02 21:04:17,827 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-west/06/
2020-07-02 21:04:52,347 INFO [watch_fvcom] r12 nowcast FVCOM VH-FR run on arbutus.cloud-nowcast completed
2020-07-02 21:04:52,353 INFO [manager] checklist updated with [FVCOM run] items from watch_fvcom worker
2020-07-02 21:04:53,004 INFO [manager] launching NextWorker(module='nowcast.workers.download_fvcom_results', args=['arbutus.cloud-nowcast', 'r12', 'nowcast', '--run-date', '2020-07-02'], host='localhost')
2020-07-02 21:04:53,957 INFO [download_fvcom_results] running in process 14186
2020-07-02 21:04:53,958 INFO [download_fvcom_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-02 21:04:53,959 INFO [download_fvcom_results] publishing log messages to tcp://*:5563
2020-07-02 21:04:53,960 INFO [download_fvcom_results] connected to skookum.eos.ubc.ca port 5555
2020-07-02 21:06:27,324 INFO [download_fvcom_results] VHFR FVCOM r12 nowcast 2020-07-02 results files from arbutus.cloud-nowcast downloaded
2020-07-02 21:06:27,330 INFO [manager] checklist updated with [VHFR FVCOM results files] items from download_fvcom_results worker
2020-07-02 21:06:27,720 INFO [manager] launching NextWorker(module='nowcast.workers.get_vfpa_hadcp', args=['--data-date', '2020-07-02'], host='localhost')
2020-07-02 21:06:27,723 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['fvcom', 'nowcast-r12', 'research', '--run-date', '2020-07-02'], host='localhost')
2020-07-02 21:06:30,567 INFO [get_vfpa_hadcp] running in process 14221
2020-07-02 21:06:30,579 INFO [get_vfpa_hadcp] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-02 21:06:30,580 INFO [get_vfpa_hadcp] publishing log messages to tcp://*:5563
2020-07-02 21:06:30,581 INFO [get_vfpa_hadcp] connected to skookum.eos.ubc.ca port 5555
2020-07-02 21:06:30,581 INFO [get_vfpa_hadcp] processing VFPA HADCP data from 2nd Narrows Rail Bridge for 2020-07-02
2020-07-02 21:06:31,720 INFO [get_vfpa_hadcp] added VFPA HADCP data from 2nd Narrows Rail Bridge for 2020-07-02 to /opp/observations/AISDATA/netcdf/VFPA_2ND_NARROWS_HADCP_2s_202007.nc
2020-07-02 21:06:31,720 INFO [get_vfpa_hadcp] VFPA HADCP observations added to 2020-07 netcdf file
2020-07-02 21:06:31,722 INFO [manager] checklist updated with [VFPA HADCP data] items from get_vfpa_hadcp worker
2020-07-02 21:06:31,987 INFO [make_plots] running in process 14222
2020-07-02 21:06:31,988 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-02 21:06:31,989 INFO [make_plots] publishing log messages to tcp://*:5564
2020-07-02 21:06:31,989 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2020-07-02 21:06:31,991 INFO [make_plots] preparing render list for 2020-07-02 VHFR FVCOM nowcast-r12 research figures
2020-07-02 21:06:31,992 INFO [make_plots] added EnglishBay_surface_currents figures to 2020-07-02 VHFR FVCOM nowcast-r12 research render list
2020-07-02 21:06:31,994 INFO [make_plots] added VancouverHarbour_surface_currents figures to 2020-07-02 VHFR FVCOM nowcast-r12 research render list
2020-07-02 21:06:31,996 INFO [make_plots] added IndianArm_surface_currents figures to 2020-07-02 VHFR FVCOM nowcast-r12 research render list
2020-07-02 21:06:31,998 INFO [make_plots] added VancouverHarbour_thalweg figures to 2020-07-02 VHFR FVCOM nowcast-r12 research render list
2020-07-02 21:06:32,000 INFO [make_plots] added PortMoody_thalweg figures to 2020-07-02 VHFR FVCOM nowcast-r12 research render list
2020-07-02 21:06:32,001 INFO [make_plots] added IndianArm_thalweg figures to 2020-07-02 VHFR FVCOM nowcast-r12 research render list
2020-07-02 21:06:32,002 INFO [make_plots] starting to render fvcom nowcast-r12 research 02jul20 figures
2020-07-02 21:06:32,159 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['VFPA-HADCP'], host='localhost')
2020-07-02 21:06:32,947 INFO [ping_erddap] running in process 14239
2020-07-02 21:06:32,947 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-02 21:06:32,948 INFO [ping_erddap] publishing log messages to tcp://*:5563
2020-07-02 21:06:32,949 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2020-07-02 21:06:32,962 INFO [ping_erddap] VFPA-HADCP ERDDAP dataset flag file(s) created
2020-07-02 21:06:32,964 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2020-07-02 21:06:33,331 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['fvcom', 'nowcast-x2', 'publish', '--run-date', '2020-07-02'], host='localhost')
2020-07-02 21:06:33,334 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['fvcom', 'nowcast-r12', 'publish', '--run-date', '2020-07-02'], host='localhost')
2020-07-02 21:06:33,336 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['fvcom', 'forecast-x2', 'publish', '--run-date', '2020-07-02'], host='localhost')
2020-07-02 21:06:35,974 INFO [make_plots] running in process 14244
2020-07-02 21:06:35,975 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-02 21:06:35,976 INFO [make_plots] publishing log messages to tcp://*:5563
2020-07-02 21:06:35,977 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2020-07-02 21:06:36,077 INFO [make_plots] running in process 14246
2020-07-02 21:06:36,078 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-02 21:06:36,079 INFO [make_plots] publishing log messages to tcp://*:5565
2020-07-02 21:06:36,080 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2020-07-02 21:06:36,095 INFO [make_plots] running in process 14245
2020-07-02 21:06:36,096 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-02 21:06:36,096 INFO [make_plots] publishing log messages to tcp://*:5566
2020-07-02 21:06:36,097 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2020-07-02 21:06:37,131 INFO [make_plots] preparing render list for 2020-07-02 VHFR FVCOM nowcast-x2 publish figures
2020-07-02 21:06:37,132 INFO [make_plots] added Calamity Point figure to 2020-07-02 VHFR FVCOM nowcast-x2 publish render list
2020-07-02 21:06:37,132 INFO [make_plots] added Indian Arm Head figure to 2020-07-02 VHFR FVCOM nowcast-x2 publish render list
2020-07-02 21:06:37,133 INFO [make_plots] added New Westminster figure to 2020-07-02 VHFR FVCOM nowcast-x2 publish render list
2020-07-02 21:06:37,134 INFO [make_plots] added Port Moody figure to 2020-07-02 VHFR FVCOM nowcast-x2 publish render list
2020-07-02 21:06:37,134 INFO [make_plots] added Sand Heads figure to 2020-07-02 VHFR FVCOM nowcast-x2 publish render list
2020-07-02 21:06:37,135 INFO [make_plots] added Sandy Cove figure to 2020-07-02 VHFR FVCOM nowcast-x2 publish render list
2020-07-02 21:06:37,135 INFO [make_plots] added Vancouver Harbour figure to 2020-07-02 VHFR FVCOM nowcast-x2 publish render list
2020-07-02 21:06:37,136 INFO [make_plots] added Woodwards Landing figure to 2020-07-02 VHFR FVCOM nowcast-x2 publish render list
2020-07-02 21:06:37,136 INFO [make_plots] added 2ndNarrows_current figure to 2020-07-02 VHFR FVCOM nowcast-x2 publish render list
2020-07-02 21:06:37,137 INFO [make_plots] starting to render fvcom nowcast-x2 publish 02jul20 figures
2020-07-02 21:06:37,149 INFO [make_plots] preparing render list for 2020-07-02 VHFR FVCOM nowcast-r12 publish figures
2020-07-02 21:06:37,150 INFO [make_plots] added Calamity Point figure to 2020-07-02 VHFR FVCOM nowcast-r12 publish render list
2020-07-02 21:06:37,151 INFO [make_plots] added Indian Arm Head figure to 2020-07-02 VHFR FVCOM nowcast-r12 publish render list
2020-07-02 21:06:37,151 INFO [make_plots] added New Westminster figure to 2020-07-02 VHFR FVCOM nowcast-r12 publish render list
2020-07-02 21:06:37,152 INFO [make_plots] added Port Moody figure to 2020-07-02 VHFR FVCOM nowcast-r12 publish render list
2020-07-02 21:06:37,153 INFO [make_plots] added Sand Heads figure to 2020-07-02 VHFR FVCOM nowcast-r12 publish render list
2020-07-02 21:06:37,154 INFO [make_plots] added Sandy Cove figure to 2020-07-02 VHFR FVCOM nowcast-r12 publish render list
2020-07-02 21:06:37,154 INFO [make_plots] added Vancouver Harbour figure to 2020-07-02 VHFR FVCOM nowcast-r12 publish render list
2020-07-02 21:06:37,155 INFO [make_plots] added Woodwards Landing figure to 2020-07-02 VHFR FVCOM nowcast-r12 publish render list
2020-07-02 21:06:37,156 INFO [make_plots] added 2ndNarrows_current figure to 2020-07-02 VHFR FVCOM nowcast-r12 publish render list
2020-07-02 21:06:37,156 INFO [make_plots] starting to render fvcom nowcast-r12 publish 02jul20 figures
2020-07-02 21:06:37,159 INFO [make_plots] preparing render list for 2020-07-02 VHFR FVCOM forecast-x2 publish figures
2020-07-02 21:06:37,159 INFO [make_plots] added Calamity Point figure to 2020-07-02 VHFR FVCOM forecast-x2 publish render list
2020-07-02 21:06:37,160 INFO [make_plots] added Indian Arm Head figure to 2020-07-02 VHFR FVCOM forecast-x2 publish render list
2020-07-02 21:06:37,161 INFO [make_plots] added New Westminster figure to 2020-07-02 VHFR FVCOM forecast-x2 publish render list
2020-07-02 21:06:37,162 INFO [make_plots] added Port Moody figure to 2020-07-02 VHFR FVCOM forecast-x2 publish render list
2020-07-02 21:06:37,163 INFO [make_plots] added Sand Heads figure to 2020-07-02 VHFR FVCOM forecast-x2 publish render list
2020-07-02 21:06:37,163 INFO [make_plots] added Sandy Cove figure to 2020-07-02 VHFR FVCOM forecast-x2 publish render list
2020-07-02 21:06:37,164 INFO [make_plots] added Vancouver Harbour figure to 2020-07-02 VHFR FVCOM forecast-x2 publish render list
2020-07-02 21:06:37,165 INFO [make_plots] added Woodwards Landing figure to 2020-07-02 VHFR FVCOM forecast-x2 publish render list
2020-07-02 21:06:37,166 INFO [make_plots] added 2ndNarrows_current figure to 2020-07-02 VHFR FVCOM forecast-x2 publish render list
2020-07-02 21:06:37,166 INFO [make_plots] starting to render fvcom forecast-x2 publish 02jul20 figures
2020-07-02 21:06:37,168 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07724 Calamity Point from 2020-07-01 23:55:00 to 2020-07-03 00:00:00
2020-07-02 21:06:37,174 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07724 Calamity Point from 2020-07-01 23:55:00 to 2020-07-03 00:00:00
2020-07-02 21:06:37,181 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07724 Calamity Point from 2020-07-01 23:55:00 to 2020-07-04 12:00:00
2020-07-02 21:07:00,884 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07724 Calamity Point from 2020-07-02 00:00:00 to 2020-07-03 00:00:00
2020-07-02 21:07:01,138 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07724 Calamity Point from 2020-07-02 00:00:00 to 2020-07-03 00:00:00
2020-07-02 21:07:02,435 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07774 Indian Arm Head from 2020-07-01 23:55:00 to 2020-07-03 00:00:00
2020-07-02 21:07:02,611 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07774 Indian Arm Head from 2020-07-01 23:55:00 to 2020-07-03 00:00:00
2020-07-02 21:07:03,172 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07774 Indian Arm Head from 2020-07-02 00:00:00 to 2020-07-03 00:00:00
2020-07-02 21:07:03,348 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07774 Indian Arm Head from 2020-07-02 00:00:00 to 2020-07-03 00:00:00
2020-07-02 21:07:04,597 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07654 New Westminster from 2020-07-01 23:55:00 to 2020-07-03 00:00:00
2020-07-02 21:07:04,654 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07654 New Westminster from 2020-07-01 23:55:00 to 2020-07-03 00:00:00
2020-07-02 21:07:04,665 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07724 Calamity Point from 2020-07-02 00:00:00 to 2020-07-04 12:00:00
2020-07-02 21:07:06,267 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07774 Indian Arm Head from 2020-07-01 23:55:00 to 2020-07-04 12:00:00
2020-07-02 21:07:06,828 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07774 Indian Arm Head from 2020-07-02 00:00:00 to 2020-07-04 12:00:00
2020-07-02 21:07:08,214 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07654 New Westminster from 2020-07-01 23:55:00 to 2020-07-04 12:00:00
2020-07-02 21:07:27,007 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07654 New Westminster from 2020-07-02 00:00:00 to 2020-07-03 00:00:00
2020-07-02 21:07:27,102 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07654 New Westminster from 2020-07-02 00:00:00 to 2020-07-03 00:00:00
2020-07-02 21:08:51,509 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07755 Port Moody from 2020-07-01 23:55:00 to 2020-07-03 00:00:00
2020-07-02 21:09:14,536 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07755 Port Moody from 2020-07-02 00:00:00 to 2020-07-03 00:00:00
2020-07-02 21:09:17,558 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07594 Sand Heads from 2020-07-01 23:55:00 to 2020-07-03 00:00:00
2020-07-02 21:09:40,126 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07594 Sand Heads from 2020-07-02 00:00:00 to 2020-07-03 00:00:00
2020-07-02 21:09:43,415 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07786 Sandy Cove from 2020-07-01 23:55:00 to 2020-07-03 00:00:00
2020-07-02 21:10:05,667 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07795 Point Atkinson from 2020-07-02 00:00:00 to 2020-07-03 00:00:00
2020-07-02 21:10:08,845 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07735 Vancouver Harbour from 2020-07-01 23:55:00 to 2020-07-03 00:00:00
2020-07-02 21:10:31,471 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07735 Vancouver Harbour from 2020-07-02 00:00:00 to 2020-07-03 00:00:00
2020-07-02 21:10:34,713 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07610 Woodwards Landing from 2020-07-01 23:55:00 to 2020-07-03 00:00:00
2020-07-02 21:10:56,981 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07610 Woodwards Landing from 2020-07-02 00:00:00 to 2020-07-03 00:00:00
2020-07-02 21:11:02,135 INFO [make_plots] finished rendering fvcom nowcast-x2 publish 02jul20 figures
2020-07-02 21:11:02,739 INFO [make_plots] fvcom publish plots for 2020-07-02 nowcast-x2 completed
2020-07-02 21:11:02,744 INFO [manager] checklist updated with [plots] items from make_plots worker
2020-07-02 22:26:31,730 INFO [make_plots] finished rendering fvcom nowcast-r12 research 02jul20 figures
2020-07-02 22:26:31,731 INFO [make_plots] fvcom research plots for 2020-07-02 nowcast-r12 completed
2020-07-02 22:26:31,881 INFO [manager] checklist updated with [plots] items from make_plots worker
2020-07-03 03:02:00,597 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-west/06/ to /results/forcing/atmospheric/GEM2.5/GRIB/20200703/06/
2020-07-03 03:02:00,598 INFO [collect_weather] 2020-07-03 2.5km weather forecast 06 collection complete
2020-07-03 03:02:00,601 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2020-07-03 03:02:01,360 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['Capilano', '--data-date', '2020-07-02'], host='localhost')
2020-07-03 03:02:01,367 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ChilliwackVedder', '--data-date', '2020-07-02'], host='localhost')
2020-07-03 03:02:01,372 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ClowhomClowhomLake', '--data-date', '2020-07-02'], host='localhost')
2020-07-03 03:02:01,377 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['Englishman', '--data-date', '2020-07-02'], host='localhost')
2020-07-03 03:02:01,382 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['Fraser', '--data-date', '2020-07-02'], host='localhost')
2020-07-03 03:02:01,386 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['HomathkoMouth', '--data-date', '2020-07-02'], host='localhost')
2020-07-03 03:02:01,391 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['SalmonSayward', '--data-date', '2020-07-02'], host='localhost')
2020-07-03 03:02:01,396 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['SanJuanPortRenfrew', '--data-date', '2020-07-02'], host='localhost')
2020-07-03 03:02:01,401 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['SquamishBrackendale', '--data-date', '2020-07-02'], host='localhost')
2020-07-03 03:02:01,406 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['TheodosiaScotty', '--data-date', '2020-07-02'], host='localhost')
2020-07-03 03:02:01,413 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['TheodosiaBypass', '--data-date', '2020-07-02'], host='localhost')
2020-07-03 03:02:01,418 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['TheodosiaDiversion', '--data-date', '2020-07-02'], host='localhost')
2020-07-03 03:02:01,422 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ctd', args=['SCVIP'], host='localhost')
2020-07-03 03:02:01,426 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ctd', args=['SEVIP'], host='localhost')
2020-07-03 03:02:01,429 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ferry', args=['TWDP'], host='localhost')
2020-07-03 03:02:01,433 INFO [manager] launching NextWorker(module='nowcast.workers.get_NeahBay_ssh', args=['forecast2'], host='localhost')
2020-07-03 03:02:01,438 INFO [manager] launching NextWorker(module='nowcast.workers.grib_to_netcdf', args=['forecast2'], host='localhost')
2020-07-03 03:02:01,448 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['12', '2.5km'], host='localhost')
2020-07-03 03:02:03,210 INFO [collect_weather] running in process 21798
2020-07-03 03:02:03,217 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 03:02:03,219 INFO [collect_weather] publishing log messages to tcp://*:5562
2020-07-03 03:02:03,221 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2020-07-03 03:02:03,260 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-west/12/
2020-07-03 03:02:04,183 INFO [collect_river_data] running in process 21778
2020-07-03 03:02:04,210 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 03:02:04,211 INFO [collect_river_data] publishing log messages to tcp://*:5563
2020-07-03 03:02:04,220 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2020-07-03 03:02:04,250 INFO [collect_river_data] Collecting Capilano river data for 2020-07-02
2020-07-03 03:02:04,251 INFO [collect_river_data] running in process 21788
2020-07-03 03:02:04,252 INFO [collect_river_data] running in process 21782
2020-07-03 03:02:04,253 INFO [collect_river_data] running in process 21781
2020-07-03 03:02:04,254 INFO [collect_river_data] running in process 21789
2020-07-03 03:02:04,266 INFO [collect_river_data] running in process 21785
2020-07-03 03:02:04,267 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 03:02:04,270 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 03:02:04,271 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 03:02:04,272 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 03:02:04,273 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 03:02:04,274 INFO [collect_river_data] publishing log messages to tcp://*:5564
2020-07-03 03:02:04,275 INFO [collect_river_data] publishing log messages to tcp://*:5567
2020-07-03 03:02:04,276 INFO [collect_river_data] publishing log messages to tcp://*:5568
2020-07-03 03:02:04,277 INFO [collect_river_data] publishing log messages to tcp://*:5569
2020-07-03 03:02:04,278 INFO [collect_river_data] publishing log messages to tcp://*:5570
2020-07-03 03:02:04,279 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2020-07-03 03:02:04,279 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2020-07-03 03:02:04,280 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2020-07-03 03:02:04,281 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2020-07-03 03:02:04,281 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2020-07-03 03:02:04,282 INFO [collect_river_data] Collecting TheodosiaScotty river data for 2020-07-02
2020-07-03 03:02:04,283 INFO [collect_river_data] Collecting Fraser river data for 2020-07-02
2020-07-03 03:02:04,284 INFO [collect_river_data] Collecting Englishman river data for 2020-07-02
2020-07-03 03:02:04,284 INFO [collect_river_data] Collecting TheodosiaBypass river data for 2020-07-02
2020-07-03 03:02:04,285 INFO [collect_river_data] Collecting SalmonSayward river data for 2020-07-02
2020-07-03 03:02:04,286 INFO [grib_to_netcdf] running in process 21796
2020-07-03 03:02:04,287 INFO [collect_river_data] running in process 21787
2020-07-03 03:02:04,287 INFO [grib_to_netcdf] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 03:02:04,288 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 03:02:04,288 INFO [grib_to_netcdf] publishing log messages to tcp://*:5571
2020-07-03 03:02:04,289 INFO [collect_river_data] publishing log messages to tcp://*:5572
2020-07-03 03:02:04,290 INFO [grib_to_netcdf] connected to skookum.eos.ubc.ca port 5555
2020-07-03 03:02:04,290 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2020-07-03 03:02:04,291 INFO [grib_to_netcdf] forecast section: 20200703/06
2020-07-03 03:02:04,291 INFO [collect_river_data] Collecting SquamishBrackendale river data for 2020-07-02
2020-07-03 03:02:04,292 INFO [grib_to_netcdf] next day forecast section: 20200703/06
2020-07-03 03:02:04,313 INFO [collect_river_data] running in process 21780
2020-07-03 03:02:04,314 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 03:02:04,315 INFO [collect_river_data] publishing log messages to tcp://*:5573
2020-07-03 03:02:04,315 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2020-07-03 03:02:04,316 INFO [collect_river_data] Collecting ClowhomClowhomLake river data for 2020-07-02
2020-07-03 03:02:04,319 INFO [collect_river_data] running in process 21790
2020-07-03 03:02:04,321 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 03:02:04,322 INFO [collect_river_data] publishing log messages to tcp://*:5574
2020-07-03 03:02:04,322 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2020-07-03 03:02:04,323 INFO [collect_river_data] Collecting TheodosiaDiversion river data for 2020-07-02
2020-07-03 03:02:04,324 INFO [collect_river_data] running in process 21786
2020-07-03 03:02:04,325 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 03:02:04,326 INFO [collect_river_data] publishing log messages to tcp://*:5575
2020-07-03 03:02:04,327 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2020-07-03 03:02:04,328 INFO [collect_river_data] Collecting SanJuanPortRenfrew river data for 2020-07-02
2020-07-03 03:02:04,366 INFO [collect_river_data] running in process 21779
2020-07-03 03:02:04,498 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 03:02:04,499 INFO [collect_river_data] publishing log messages to tcp://*:5576
2020-07-03 03:02:04,535 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2020-07-03 03:02:04,536 INFO [collect_river_data] Collecting ChilliwackVedder river data for 2020-07-02
2020-07-03 03:02:04,537 INFO [collect_river_data] running in process 21784
2020-07-03 03:02:04,621 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 03:02:04,625 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2020-07-03 03:02:04,629 INFO [collect_river_data] publishing log messages to tcp://*:5577
2020-07-03 03:02:04,629 INFO [collect_river_data] Appended TheodosiaScotty river average discharge for 2020-07-02 to: /results/forcing/rivers/observations/Theodosia_Scotty_flow
2020-07-03 03:02:04,630 INFO [collect_river_data] Appended Capilano river average discharge for 2020-07-02 to: /opp/observations/rivers/Capilano/Caplilano_08GA010_day_avg_flow
2020-07-03 03:02:04,631 INFO [collect_river_data] Appended SquamishBrackendale river average discharge for 2020-07-02 to: /results/forcing/rivers/observations/Squamish_Brackendale_flow
2020-07-03 03:02:04,632 INFO [collect_river_data] Appended ClowhomClowhomLake river average discharge for 2020-07-02 to: /results/forcing/rivers/observations/Clowhom_ClowhomLake_flow
2020-07-03 03:02:04,633 INFO [collect_river_data] Appended TheodosiaBypass river average discharge for 2020-07-02 to: /results/forcing/rivers/observations/Theodosia_Bypass_flow
2020-07-03 03:02:04,634 INFO [collect_river_data] Appended TheodosiaDiversion river average discharge for 2020-07-02 to: /results/forcing/rivers/observations/Theodosia_Diversion_flow
2020-07-03 03:02:04,635 INFO [collect_river_data] Appended SalmonSayward river average discharge for 2020-07-02 to: /results/forcing/rivers/observations/Salmon_Sayward_flow
2020-07-03 03:02:04,636 INFO [collect_river_data] Appended SanJuanPortRenfrew river average discharge for 2020-07-02 to: /results/forcing/rivers/observations/SanJuan_PortRenfrew_flow
2020-07-03 03:02:04,636 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2020-07-03 03:02:04,637 INFO [collect_river_data] TheodosiaScotty river average discharge for 2020-07-02 calculated and appended to TheodosiaScotty_flow file
2020-07-03 03:02:04,638 INFO [collect_river_data] Capilano river average discharge for 2020-07-02 calculated and appended to Capilano_flow file
2020-07-03 03:02:04,639 INFO [collect_river_data] SquamishBrackendale river average discharge for 2020-07-02 calculated and appended to SquamishBrackendale_flow file
2020-07-03 03:02:04,639 INFO [collect_river_data] ClowhomClowhomLake river average discharge for 2020-07-02 calculated and appended to ClowhomClowhomLake_flow file
2020-07-03 03:02:04,652 INFO [collect_river_data] TheodosiaBypass river average discharge for 2020-07-02 calculated and appended to TheodosiaBypass_flow file
2020-07-03 03:02:04,653 INFO [collect_river_data] TheodosiaDiversion river average discharge for 2020-07-02 calculated and appended to TheodosiaDiversion_flow file
2020-07-03 03:02:04,656 INFO [collect_river_data] SalmonSayward river average discharge for 2020-07-02 calculated and appended to SalmonSayward_flow file
2020-07-03 03:02:04,656 INFO [collect_river_data] SanJuanPortRenfrew river average discharge for 2020-07-02 calculated and appended to SanJuanPortRenfrew_flow file
2020-07-03 03:02:04,657 INFO [collect_river_data] Collecting HomathkoMouth river data for 2020-07-02
2020-07-03 03:02:04,665 INFO [collect_river_data] Appended HomathkoMouth river average discharge for 2020-07-02 to: /results/forcing/rivers/observations/Homathko_Mouth_flow
2020-07-03 03:02:04,665 INFO [collect_river_data] Appended ChilliwackVedder river average discharge for 2020-07-02 to: /results/forcing/rivers/observations/Chilliwack_Vedder_flow
2020-07-03 03:02:04,666 INFO [collect_river_data] HomathkoMouth river average discharge for 2020-07-02 calculated and appended to HomathkoMouth_flow file
2020-07-03 03:02:04,667 INFO [collect_river_data] ChilliwackVedder river average discharge for 2020-07-02 calculated and appended to ChilliwackVedder_flow file
2020-07-03 03:02:04,794 INFO [collect_river_data] Appended Fraser river average discharge for 2020-07-02 to: /data/dlatorne/SOG-projects/SOG-forcing/ECget/Fraser_flow
2020-07-03 03:02:04,795 INFO [collect_river_data] Fraser river average discharge for 2020-07-02 calculated and appended to Fraser_flow file
2020-07-03 03:02:04,797 INFO [collect_river_data] Appended Englishman river average discharge for 2020-07-02 to: /data/dlatorne/SOG-projects/SOG-forcing/ECget/Englishman_flow
2020-07-03 03:02:04,799 INFO [collect_river_data] Englishman river average discharge for 2020-07-02 calculated and appended to Englishman_flow file
2020-07-03 03:02:04,979 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2020-07-03 03:02:05,378 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2020-07-03 03:02:05,732 INFO [get_NeahBay_ssh] running in process 21794
2020-07-03 03:02:05,733 INFO [get_NeahBay_ssh] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 03:02:05,733 INFO [get_NeahBay_ssh] publishing log messages to tcp://*:5569
2020-07-03 03:02:05,734 INFO [get_NeahBay_ssh] connected to skookum.eos.ubc.ca port 5555
2020-07-03 03:02:05,786 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2020-07-03 03:02:06,103 INFO [get_onc_ferry] running in process 21793
2020-07-03 03:02:06,104 INFO [get_onc_ferry] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 03:02:06,105 INFO [get_onc_ferry] publishing log messages to tcp://*:5574
2020-07-03 03:02:06,106 INFO [get_onc_ferry] connected to skookum.eos.ubc.ca port 5555
2020-07-03 03:02:06,106 INFO [get_onc_ferry] requesting ONC TWDP.N1 NAV data for 2020-07-02
2020-07-03 03:02:06,107 INFO [get_onc_ctd] running in process 21791
2020-07-03 03:02:06,108 INFO [get_onc_ctd] running in process 21792
2020-07-03 03:02:06,109 INFO [get_onc_ctd] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 03:02:06,109 INFO [get_onc_ctd] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 03:02:06,110 INFO [get_onc_ctd] publishing log messages to tcp://*:5575
2020-07-03 03:02:06,111 INFO [get_onc_ctd] publishing log messages to tcp://*:5578
2020-07-03 03:02:06,112 INFO [get_onc_ctd] connected to skookum.eos.ubc.ca port 5555
2020-07-03 03:02:06,113 INFO [get_onc_ctd] connected to skookum.eos.ubc.ca port 5555
2020-07-03 03:02:06,113 INFO [get_onc_ctd] requesting ONC SCVIP CTD T&S data for 2020-07-02
2020-07-03 03:02:06,114 INFO [get_onc_ctd] requesting ONC SEVIP CTD T&S data for 2020-07-02
2020-07-03 03:02:06,179 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2020-07-03 03:02:06,596 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2020-07-03 03:02:06,965 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2020-07-03 03:02:07,321 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2020-07-03 03:02:07,678 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2020-07-03 03:02:08,038 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2020-07-03 03:02:08,388 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2020-07-03 03:02:08,773 INFO [manager] launching NextWorker(module='nowcast.workers.make_runoff_file', args=[], host='localhost')
2020-07-03 03:02:08,778 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2020-07-03 03:02:09,870 INFO [make_runoff_file] running in process 21912
2020-07-03 03:02:09,879 INFO [make_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 03:02:09,880 INFO [make_runoff_file] publishing log messages to tcp://*:5563
2020-07-03 03:02:09,881 INFO [make_runoff_file] connected to skookum.eos.ubc.ca port 5555
2020-07-03 03:02:09,998 INFO [get_NeahBay_ssh] Neah Bay sea surface height web scraping and open boundary file creation for forecast2 complete
2020-07-03 03:02:10,002 INFO [manager] checklist updated with [Neah Bay ssh] items from get_NeahBay_ssh worker
2020-07-03 03:02:10,453 INFO [make_runoff_file] calculating runoff forcing for bathymetry type: b201702
2020-07-03 03:02:10,533 INFO [make_runoff_file] 2020-07-03 runoff file creation from Fraser at Hope and climatology elsewhere complete
2020-07-03 03:02:10,535 INFO [manager] checklist updated with [rivers forcing] items from make_runoff_file worker
2020-07-03 03:02:29,397 INFO [get_onc_ferry] requesting ONC TWDP TSG data for 2020-07-02
2020-07-03 03:02:30,222 INFO [get_onc_ctd] 2020-07-02 ONC SCVIP CTD T&S file created
2020-07-03 03:02:30,223 INFO [get_onc_ctd] 2020-07-02 ONC SEVIP CTD T&S file created
2020-07-03 03:02:30,225 INFO [manager] checklist updated with [ONC CTD data] items from get_onc_ctd worker
2020-07-03 03:02:30,681 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['SCVIP-CTD'], host='localhost')
2020-07-03 03:02:30,686 INFO [manager] checklist updated with [ONC CTD data] items from get_onc_ctd worker
2020-07-03 03:02:31,105 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['SEVIP-CTD'], host='localhost')
2020-07-03 03:02:31,506 INFO [ping_erddap] running in process 22218
2020-07-03 03:02:31,507 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 03:02:31,508 INFO [ping_erddap] publishing log messages to tcp://*:5563
2020-07-03 03:02:31,509 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2020-07-03 03:02:31,510 INFO [ping_erddap] SCVIP-CTD ERDDAP dataset flag file(s) created
2020-07-03 03:02:31,512 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2020-07-03 03:02:31,979 INFO [ping_erddap] running in process 22224
2020-07-03 03:02:31,980 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 03:02:31,981 INFO [ping_erddap] publishing log messages to tcp://*:5564
2020-07-03 03:02:31,982 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2020-07-03 03:02:31,983 INFO [ping_erddap] SEVIP-CTD ERDDAP dataset flag file(s) created
2020-07-03 03:02:31,984 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2020-07-03 03:02:58,978 INFO [grib_to_netcdf] 2020-07-03 NEMO-atmos forcing file for forecast2 created
2020-07-03 03:02:58,982 INFO [manager] checklist updated with [weather forcing] items from grib_to_netcdf worker
2020-07-03 03:02:59,413 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'forecast2'], host='localhost')
2020-07-03 03:02:59,417 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'forecast2'], host='localhost')
2020-07-03 03:02:59,420 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['graham-hindcast', 'forecast2'], host='localhost')
2020-07-03 03:02:59,424 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'forecast2'], host='localhost')
2020-07-03 03:03:00,275 INFO [upload_forcing] running in process 22672
2020-07-03 03:03:00,276 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 03:03:00,277 INFO [upload_forcing] publishing log messages to tcp://*:5563
2020-07-03 03:03:00,278 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2020-07-03 03:03:00,279 INFO [upload_forcing] running in process 22670
2020-07-03 03:03:00,280 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 03:03:00,281 INFO [upload_forcing] publishing log messages to tcp://*:5564
2020-07-03 03:03:00,281 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2020-07-03 03:03:00,282 INFO [upload_forcing] running in process 22671
2020-07-03 03:03:00,283 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 03:03:00,284 INFO [upload_forcing] publishing log messages to tcp://*:5567
2020-07-03 03:03:00,285 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2020-07-03 03:03:00,311 INFO [upload_forcing] running in process 22669
2020-07-03 03:03:00,312 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 03:03:00,313 INFO [upload_forcing] publishing log messages to tcp://*:5568
2020-07-03 03:03:00,313 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2020-07-03 03:03:02,116 INFO [get_onc_ferry] requesting ONC TWDP OXYSENSOR data for 2020-07-02
2020-07-03 03:03:02,578 INFO [upload_forcing] LiveOcean boundary conditions file not found; created symlink to /results/forcing/LiveOcean/boundary_conditions/LiveOcean_v201905_y2020m07d02.nc
2020-07-03 03:03:02,679 INFO [upload_forcing] forecast2 2020-07-03 forcing files upload to optimum-hindcast completed
2020-07-03 03:03:02,684 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2020-07-03 03:03:02,940 INFO [upload_forcing] forecast2 2020-07-03 forcing files upload to orcinus-nowcast-agrif completed
2020-07-03 03:03:03,112 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2020-07-03 03:03:03,648 INFO [upload_forcing] forecast2 2020-07-03 forcing files upload to arbutus.cloud-nowcast completed
2020-07-03 03:03:03,652 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2020-07-03 03:03:04,030 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2020-07-03'], host='localhost')
2020-07-03 03:03:05,100 INFO [make_forcing_links] running in process 22714
2020-07-03 03:03:05,101 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 03:03:05,102 INFO [make_forcing_links] publishing log messages to tcp://*:5563
2020-07-03 03:03:05,103 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2020-07-03 03:03:06,356 INFO [make_forcing_links] forecast2 2020-07-03 forcing file links on arbutus.cloud-nowcast created
2020-07-03 03:03:06,362 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2020-07-03 03:03:06,816 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2020-07-03'], host='arbutus.cloud-nowcast')
2020-07-03 03:03:07,520 INFO [upload_forcing] forecast2 2020-07-03 forcing files upload to graham-hindcast completed
2020-07-03 03:03:07,527 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2020-07-03 03:03:09,106 INFO [run_NEMO] running in process 15313
2020-07-03 03:03:09,107 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 03:03:09,108 INFO [run_NEMO] publishing log messages to tcp://*:5556
2020-07-03 03:03:09,117 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2020-07-03 03:03:10,130 INFO [run_NEMO] forecast2: launching /nemoShare/MEOPAR/nowcast-sys/runs/02jul20forecast2_2020-07-03T030309.211391-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2020-07-03 03:03:10,146 INFO [run_NEMO] forecast2 NEMO run for 2020-07-03 on arbutus.cloud-nowcast started
2020-07-03 03:03:10,152 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2020-07-03 03:03:10,607 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'forecast2'], host='arbutus.cloud-nowcast')
2020-07-03 03:03:12,561 INFO [watch_NEMO] running in process 15581
2020-07-03 03:03:12,562 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 03:03:12,563 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2020-07-03 03:03:12,565 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2020-07-03 03:03:12,676 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2020-07-03 03:03:37,031 INFO [get_onc_ferry] requesting ONC TWDP TURBCHLFL data for 2020-07-02
2020-07-03 03:04:07,194 INFO [get_onc_ferry] requesting ONC TWDP CO2SENSOR data for 2020-07-02
2020-07-03 03:04:21,071 INFO [get_onc_ferry] requesting ONC TWDP TEMPHUMID data for 2020-07-02
2020-07-03 03:04:44,918 INFO [get_onc_ferry] requesting ONC TWDP BARPRESS data for 2020-07-02
2020-07-03 03:04:56,832 INFO [get_onc_ferry] requesting ONC TWDP PYRANOMETER data for 2020-07-02
2020-07-03 03:05:07,541 INFO [get_onc_ferry] requesting ONC TWDP PYRGEOMETER data for 2020-07-02
2020-07-03 03:05:21,877 INFO [get_onc_ferry] 2020-07-02 ONC TWDP ferry data file created
2020-07-03 03:05:21,883 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['TWDP-ferry'], host='localhost')
2020-07-03 03:05:22,985 INFO [ping_erddap] running in process 22738
2020-07-03 03:05:22,986 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 03:05:22,987 INFO [ping_erddap] publishing log messages to tcp://*:5563
2020-07-03 03:05:22,988 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2020-07-03 03:05:22,989 INFO [ping_erddap] TWDP-ferry ERDDAP dataset flag file(s) created
2020-07-03 03:05:22,991 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2020-07-03 03:08:12,777 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 7497824 = 2020-07-04 05:08:40 UTC, 17.2% complete
2020-07-03 03:13:12,879 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 7498349 = 2020-07-04 10:58:40 UTC, 36.6% complete
2020-07-03 03:18:12,980 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 7498843 = 2020-07-04 16:28:00 UTC, 54.9% complete
2020-07-03 03:23:13,082 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 7499363 = 2020-07-04 22:14:40 UTC, 74.2% complete
2020-07-03 03:28:13,183 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 7499878 = 2020-07-05 03:58:00 UTC, 93.3% complete
2020-07-03 03:33:13,287 INFO [watch_NEMO] forecast2 NEMO run on arbutus.cloud-nowcast completed
2020-07-03 03:33:13,292 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2020-07-03 03:33:14,020 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_wind_file', args=['arbutus.cloud-nowcast', 'forecast2'], host='arbutus.cloud-nowcast')
2020-07-03 03:33:14,030 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_current_file', args=['arbutus.cloud-nowcast', 'forecast2'], host='arbutus.cloud-nowcast')
2020-07-03 03:33:14,033 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2020-07-02'], host='localhost')
2020-07-03 03:33:15,186 INFO [download_results] running in process 23327
2020-07-03 03:33:15,186 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 03:33:15,188 INFO [download_results] publishing log messages to tcp://*:5563
2020-07-03 03:33:15,189 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2020-07-03 03:33:15,190 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/forecast2/02jul20 to /results/SalishSea/forecast2.201905
2020-07-03 03:33:17,245 INFO [make_ww3_wind_file] running in process 15808
2020-07-03 03:33:17,247 INFO [make_ww3_wind_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 03:33:17,248 INFO [make_ww3_wind_file] publishing log messages to tcp://*:5570
2020-07-03 03:33:17,249 INFO [make_ww3_wind_file] connected to skookum.eos.ubc.ca port 5555
2020-07-03 03:33:17,250 INFO [make_ww3_wind_file] Creating wwatch3 wind forcing file for 2020-07-03 forecast2 run
2020-07-03 03:33:17,343 INFO [make_ww3_current_file] running in process 15823
2020-07-03 03:33:17,345 INFO [make_ww3_current_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 03:33:17,346 INFO [make_ww3_current_file] publishing log messages to tcp://*:5571
2020-07-03 03:33:17,347 INFO [make_ww3_current_file] connected to skookum.eos.ubc.ca port 5555
2020-07-03 03:33:17,348 INFO [make_ww3_current_file] Creating wwatch3 currents forcing file for 2020-07-03 forecast2 run
2020-07-03 03:33:20,678 INFO [make_ww3_wind_file] wwatch3 wind forcing file created on arbutus.cloud-nowcast for 2020-07-03 forecast2 run
2020-07-03 03:33:20,682 INFO [manager] checklist updated with [WW3 wind forcing] items from make_ww3_wind_file worker
2020-07-03 03:35:08,811 INFO [make_ww3_current_file] wwatch3 currents forcing file created on arbutus.cloud-nowcast for 2020-07-03 forecast2 run
2020-07-03 03:35:08,817 INFO [manager] checklist updated with [WW3 currents forcing] items from make_ww3_current_file worker
2020-07-03 03:35:09,279 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2020-07-03'], host='arbutus.cloud-nowcast')
2020-07-03 03:35:11,313 INFO [run_ww3] running in process 16034
2020-07-03 03:35:11,315 INFO [run_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 03:35:11,316 INFO [run_ww3] publishing log messages to tcp://*:5572
2020-07-03 03:35:11,316 INFO [run_ww3] connected to skookum.eos.ubc.ca port 5555
2020-07-03 03:35:11,320 INFO [run_ww3] Created run directory /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast2_2020-07-03T033511.310931-0700
2020-07-03 03:35:11,321 INFO [run_ww3] forecast2: launching /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast2_2020-07-03T033511.310931-0700/SoGWW3.sh on arbutus.cloud-nowcast
2020-07-03 03:35:11,334 INFO [run_ww3] forecast2 WaveWatch3 run for 2020-07-03 on arbutus.cloud-nowcast started
2020-07-03 03:35:11,340 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker
2020-07-03 03:35:11,801 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'forecast2'], host='arbutus.cloud-nowcast')
2020-07-03 03:35:13,730 INFO [watch_ww3] running in process 16138
2020-07-03 03:35:13,732 INFO [watch_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 03:35:13,733 INFO [watch_ww3] publishing log messages to tcp://*:5573
2020-07-03 03:35:13,734 INFO [watch_ww3] connected to skookum.eos.ubc.ca port 5555
2020-07-03 03:35:13,769 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2020-07-03 03:35:48,413 INFO [download_results] forecast2 2020-07-02 results files from arbutus.cloud-nowcast downloaded
2020-07-03 03:35:48,421 INFO [manager] checklist updated with [results files] items from download_results worker
2020-07-03 03:35:48,876 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['forecast2', '--run-date', '2020-07-02'], host='localhost')
2020-07-03 03:35:52,545 INFO [make_CHS_currents_file] running in process 23338
2020-07-03 03:35:52,547 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 03:35:52,548 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5563
2020-07-03 03:35:52,549 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555
2020-07-03 03:36:35,768 INFO [make_CHS_currents_file] Made CHS currents file for 2020-07-02 for forecast2
2020-07-03 03:36:35,773 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker
2020-07-03 03:36:36,235 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['nemo', 'forecast2', '--run-date', '2020-07-02'], host='localhost')
2020-07-03 03:36:37,170 INFO [update_forecast_datasets] running in process 23377
2020-07-03 03:36:37,171 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 03:36:37,172 INFO [update_forecast_datasets] publishing log messages to tcp://*:5563
2020-07-03 03:36:37,173 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2020-07-03 03:36:37,174 INFO [update_forecast_datasets] updating nemo forecast directory for forecast2/02jul20 run
2020-07-03 03:40:13,869 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 93, 17.2% complete
2020-07-03 03:40:41,153 INFO [update_forecast_datasets] updated nemo forecast directory for forecast2/02jul20 run: /results/SalishSea/rolling-forecasts/nemo
2020-07-03 03:40:41,154 INFO [update_forecast_datasets] nemo 2020-07-02 forecast2 rolling forecast datasets updated
2020-07-03 03:40:41,158 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2020-07-03 03:40:41,605 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nemo-forecast'], host='localhost')
2020-07-03 03:40:41,608 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'forecast2', 'publish', '--run-date', '2020-07-02'], host='localhost')
2020-07-03 03:40:41,611 INFO [manager] launching NextWorker(module='nowcast.workers.make_surface_current_tiles', args=['forecast2', '--run-date', '2020-07-02'], host='localhost')
2020-07-03 03:40:43,056 INFO [ping_erddap] running in process 23499
2020-07-03 03:40:43,057 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 03:40:43,058 INFO [ping_erddap] publishing log messages to tcp://*:5563
2020-07-03 03:40:43,058 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2020-07-03 03:40:43,068 INFO [ping_erddap] nemo-forecast ERDDAP dataset flag file(s) created
2020-07-03 03:40:43,069 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2020-07-03 03:40:43,343 INFO [make_surface_current_tiles] running in process 23501
2020-07-03 03:40:43,348 INFO [make_surface_current_tiles] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 03:40:43,349 INFO [make_surface_current_tiles] publishing log messages to tcp://*:5564
2020-07-03 03:40:43,349 INFO [make_surface_current_tiles] connected to skookum.eos.ubc.ca port 5555
2020-07-03 03:40:43,405 INFO [make_surface_current_tiles] starting rendering of tiles for 2020-07-02 forecast2 in 6 processes into /results/nowcast-sys/figures/surface_currents/forecast2/02jul20
2020-07-03 03:40:48,563 INFO [make_plots] running in process 23500
2020-07-03 03:40:48,564 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 03:40:48,565 INFO [make_plots] publishing log messages to tcp://*:5563
2020-07-03 03:40:48,566 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2020-07-03 03:40:49,636 INFO [make_plots] preparing render list for 2020-07-02 NEMO forecast2 publish figures
2020-07-03 03:40:49,857 INFO [make_plots] added Website_thumbnail figure to 2020-07-02 NEMO forecast2 publish render list
2020-07-03 03:40:49,858 INFO [make_plots] added Threshold_website figure to 2020-07-02 NEMO forecast2 publish render list
2020-07-03 03:40:49,859 INFO [make_plots] added PA_tidal_predictions figure to 2020-07-02 NEMO forecast2 publish render list
2020-07-03 03:40:49,859 INFO [make_plots] added Boundary Bay figure to 2020-07-02 NEMO forecast2 publish render list
2020-07-03 03:40:49,860 INFO [make_plots] added Campbell River figure to 2020-07-02 NEMO forecast2 publish render list
2020-07-03 03:40:49,861 INFO [make_plots] added Cherry Point figure to 2020-07-02 NEMO forecast2 publish render list
2020-07-03 03:40:49,861 INFO [make_plots] added Friday Harbor figure to 2020-07-02 NEMO forecast2 publish render list
2020-07-03 03:40:49,862 INFO [make_plots] added Halfmoon Bay figure to 2020-07-02 NEMO forecast2 publish render list
2020-07-03 03:40:49,863 INFO [make_plots] added Nanaimo figure to 2020-07-02 NEMO forecast2 publish render list
2020-07-03 03:40:49,863 INFO [make_plots] added Neah Bay figure to 2020-07-02 NEMO forecast2 publish render list
2020-07-03 03:40:49,864 INFO [make_plots] added New Westminster figure to 2020-07-02 NEMO forecast2 publish render list
2020-07-03 03:40:49,865 INFO [make_plots] added Patricia Bay figure to 2020-07-02 NEMO forecast2 publish render list
2020-07-03 03:40:49,865 INFO [make_plots] added Point Atkinson figure to 2020-07-02 NEMO forecast2 publish render list
2020-07-03 03:40:49,866 INFO [make_plots] added Port Renfrew figure to 2020-07-02 NEMO forecast2 publish render list
2020-07-03 03:40:49,867 INFO [make_plots] added Sand Heads figure to 2020-07-02 NEMO forecast2 publish render list
2020-07-03 03:40:49,867 INFO [make_plots] added Sandy Cove figure to 2020-07-02 NEMO forecast2 publish render list
2020-07-03 03:40:49,868 INFO [make_plots] added Squamish figure to 2020-07-02 NEMO forecast2 publish render list
2020-07-03 03:40:49,869 INFO [make_plots] added Victoria figure to 2020-07-02 NEMO forecast2 publish render list
2020-07-03 03:40:49,869 INFO [make_plots] added Woodwards Landing figure to 2020-07-02 NEMO forecast2 publish render list
2020-07-03 03:40:49,870 INFO [make_plots] starting to render nemo forecast2 publish 02jul20 figures
2020-07-03 03:41:16,577 INFO [make_plots] Note: NumExpr detected 12 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
2020-07-03 03:41:16,578 INFO [make_plots] NumExpr defaulting to 8 threads.
2020-07-03 03:45:13,969 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 201, 37.2% complete
2020-07-03 03:50:14,070 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 306, 56.7% complete
2020-07-03 03:55:14,171 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 408, 75.6% complete
2020-07-03 03:57:35,950 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 08074 Campbell River from 2020-06-28 00:00:00 to 2020-07-05 05:55:00
2020-07-03 04:00:14,272 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 516, 95.6% complete
2020-07-03 04:01:04,404 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 9449424 Cherry Point from 2020-06-28 00:00:00 to 2020-07-05 05:55:00
2020-07-03 04:02:04,813 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 9449880 Friday Harbor from 2020-06-28 00:00:00 to 2020-07-05 05:55:00
2020-07-03 04:03:04,248 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07830 Halfmoon Bay from 2020-06-28 00:00:00 to 2020-07-05 05:55:00
2020-07-03 04:04:12,656 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07917 Nanaimo from 2020-06-28 00:00:00 to 2020-07-05 05:55:00
2020-07-03 04:05:14,374 INFO [watch_ww3] forecast2 WWATCH3 run on arbutus.cloud-nowcast completed
2020-07-03 04:05:14,377 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2020-07-03 04:05:16,163 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2020-07-03'], host='localhost')
2020-07-03 04:05:18,264 INFO [download_wwatch3_results] running in process 25835
2020-07-03 04:05:18,265 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 04:05:18,266 INFO [download_wwatch3_results] publishing log messages to tcp://*:5567
2020-07-03 04:05:18,267 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2020-07-03 04:05:53,002 INFO [download_wwatch3_results] forecast2 2020-07-03 results files from arbutus.cloud-nowcast downloaded
2020-07-03 04:05:53,006 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2020-07-03 04:05:54,212 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['wwatch3', 'forecast2', '--run-date', '2020-07-03'], host='localhost')
2020-07-03 04:05:56,348 INFO [update_forecast_datasets] running in process 25931
2020-07-03 04:05:56,349 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 04:05:56,350 INFO [update_forecast_datasets] publishing log messages to tcp://*:5567
2020-07-03 04:05:56,351 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2020-07-03 04:05:56,352 INFO [update_forecast_datasets] updating wwatch3 most_recent_forecast directory from forecast2/03jul20 run
2020-07-03 04:05:56,355 INFO [update_forecast_datasets] updated wwatch3 most_recent_forecast directory from forecast2/03jul20 run
2020-07-03 04:05:56,355 INFO [update_forecast_datasets] updating wwatch3 forecast directory for forecast2/03jul20 run
2020-07-03 04:06:53,997 INFO [update_forecast_datasets] updated wwatch3 forecast directory for forecast2/03jul20 run: /results/SalishSea/rolling-forecasts/wwatch3
2020-07-03 04:06:53,997 INFO [update_forecast_datasets] wwatch3 2020-07-03 forecast2 rolling forecast datasets updated
2020-07-03 04:06:54,000 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2020-07-03 04:06:55,379 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['wwatch3-forecast'], host='localhost')
2020-07-03 04:06:57,850 INFO [ping_erddap] running in process 25987
2020-07-03 04:06:57,851 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 04:06:57,852 INFO [ping_erddap] publishing log messages to tcp://*:5567
2020-07-03 04:06:57,853 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2020-07-03 04:06:57,882 INFO [ping_erddap] wwatch3-forecast ERDDAP dataset flag file(s) created
2020-07-03 04:06:57,886 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2020-07-03 04:06:59,035 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['wwatch3', 'forecast2', 'publish', '--run-date', '2020-07-03'], host='localhost')
2020-07-03 04:07:09,487 INFO [make_plots] running in process 25991
2020-07-03 04:07:09,488 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 04:07:09,489 INFO [make_plots] publishing log messages to tcp://*:5567
2020-07-03 04:07:09,490 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2020-07-03 04:07:09,490 INFO [make_plots] preparing render list for 2020-07-03 WaveWatch3 forecast2 publish figures
2020-07-03 04:07:09,491 INFO [make_plots] added Halibut Bank figure to 2020-07-03 WaveWatch3 forecast2 publish render list
2020-07-03 04:07:09,492 INFO [make_plots] added Sentry Shoal figure to 2020-07-03 WaveWatch3 forecast2 publish render list
2020-07-03 04:07:09,493 INFO [make_plots] starting to render wwatch3 forecast2 publish 03jul20 figures
2020-07-03 04:08:07,935 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 9443090 Neah Bay from 2020-06-28 00:00:00 to 2020-07-05 05:55:00
2020-07-03 04:08:19,495 INFO [make_plots] Note: NumExpr detected 12 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
2020-07-03 04:08:19,496 INFO [make_plots] NumExpr defaulting to 8 threads.
2020-07-03 04:08:19,507 INFO [make_plots] retrieving last 45 days of real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46146 Halibut Bank
2020-07-03 04:08:59,384 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07654 New Westminster from 2020-06-28 00:00:00 to 2020-07-05 05:55:00
2020-07-03 04:09:31,102 INFO [make_plots] retrieving last 45 days of real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46131 Sentry Shoal
2020-07-03 04:09:33,359 INFO [make_plots] finished rendering wwatch3 forecast2 publish 03jul20 figures
2020-07-03 04:09:33,363 INFO [make_plots] wwatch3 publish plots for 2020-07-03 forecast2 completed
2020-07-03 04:09:33,367 INFO [manager] checklist updated with [plots] items from make_plots worker
2020-07-03 04:12:10,400 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07277 Patricia Bay from 2020-06-28 00:00:00 to 2020-07-05 05:55:00
2020-07-03 04:12:12,025 INFO [make_surface_current_tiles] starting PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast2/02jul20
2020-07-03 04:14:18,999 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07795 Point Atkinson from 2020-06-28 00:00:00 to 2020-07-05 05:55:00
2020-07-03 04:14:37,846 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 08525 Port Renfrew from 2020-06-28 00:00:00 to 2020-07-05 05:55:00
2020-07-03 04:16:39,412 INFO [make_surface_current_tiles] finished PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast2/02jul20
2020-07-03 04:16:39,422 INFO [make_surface_current_tiles] finished rendering of tiles for 2020-07-02 forecast2 into /results/nowcast-sys/figures/surface_currents/forecast2/02jul20
2020-07-03 04:16:39,423 INFO [make_surface_current_tiles] surface current tile figures for 2020-07-02 forecast2 completed
2020-07-03 04:16:39,818 INFO [manager] checklist updated with [surface current tiles] items from make_surface_current_tiles worker
2020-07-03 04:16:44,512 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07594 Sand Heads from 2020-06-28 00:00:00 to 2020-07-05 05:55:00
2020-07-03 04:18:50,123 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07786 Sandy Cove from 2020-06-28 00:00:00 to 2020-07-05 05:55:00
2020-07-03 04:20:59,689 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07811 Squamish from 2020-06-28 00:00:00 to 2020-07-05 05:55:00
2020-07-03 04:21:18,351 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07120 Victoria from 2020-06-28 00:00:00 to 2020-07-05 05:55:00
2020-07-03 04:23:24,411 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07610 Woodwards Landing from 2020-06-28 00:00:00 to 2020-07-05 05:55:00
2020-07-03 04:25:24,461 INFO [make_plots] finished rendering nemo forecast2 publish 02jul20 figures
2020-07-03 04:25:24,462 INFO [make_plots] nemo publish plots for 2020-07-02 forecast2 completed
2020-07-03 04:25:24,473 INFO [manager] checklist updated with [plots] items from make_plots worker
2020-07-03 04:25:25,072 INFO [manager] launching NextWorker(module='nowcast.workers.make_feeds', args=['forecast2', '--run-date', '2020-07-02'], host='localhost')
2020-07-03 04:25:28,890 INFO [make_feeds] running in process 26998
2020-07-03 04:25:28,891 INFO [make_feeds] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 04:25:28,892 INFO [make_feeds] publishing log messages to tcp://*:5563
2020-07-03 04:25:28,893 INFO [make_feeds] connected to skookum.eos.ubc.ca port 5555
2020-07-03 04:25:37,316 INFO [make_feeds] Note: NumExpr detected 12 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
2020-07-03 04:25:37,317 INFO [make_feeds] NumExpr defaulting to 8 threads.
2020-07-03 04:25:43,204 INFO [make_feeds] ATOM feeds for 2020-07-02 forecast2 run completed
2020-07-03 04:25:43,207 INFO [manager] checklist updated with [storm surge alert feeds] items from make_feeds worker
2020-07-03 04:25:43,771 INFO [manager] launching NextWorker(module='nemo_nowcast.workers.clear_checklist', args=[], host='localhost')
2020-07-03 04:25:44,717 INFO [clear_checklist] running in process 27010
2020-07-03 04:25:44,718 INFO [clear_checklist] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 04:25:44,719 INFO [clear_checklist] publishing log messages to tcp://*:5563
2020-07-03 04:25:44,720 INFO [clear_checklist] connected to skookum.eos.ubc.ca port 5555
2020-07-03 04:25:44,721 INFO [clear_checklist] requesting that manager clear system state checklist
2020-07-03 04:25:44,723 INFO [manager] writing checklist to log file
2020-07-03 04:25:44,817 INFO [manager] checklist cleared
2020-07-03 04:25:44,820 INFO [clear_checklist] nowcast system checklist cleared
2020-07-03 04:25:44,823 INFO [manager] launching NextWorker(module='nemo_nowcast.workers.rotate_logs', args=[], host='localhost')
2020-07-03 04:25:45,907 INFO [rotate_logs] running in process 27014
2020-07-03 04:25:45,908 INFO [rotate_logs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-03 04:25:45,909 INFO [rotate_logs] publishing log messages to tcp://*:5563
2020-07-03 04:25:45,910 INFO [rotate_logs] connected to skookum.eos.ubc.ca port 5555
2020-07-03 04:25:45,910 INFO [rotate_logs] rotating log files