2020-06-30 04:33:04,802 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/nowcast.log
2020-06-30 04:33:04,810 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/nowcast.debug.log
2020-06-30 04:33:04,815 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/checklist.log
2020-06-30 04:33:04,816 INFO [rotate_logs] log files rotated
2020-06-30 04:33:04,822 INFO [manager] checklist updated with [log rotation] items from rotate_logs worker
2020-06-30 09:06:08,936 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-west/12/ to /results/forcing/atmospheric/GEM2.5/GRIB/20200630/12/
2020-06-30 09:06:08,937 INFO [collect_weather] 2020-06-30 2.5km weather forecast 12 collection complete
2020-06-30 09:06:08,941 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2020-06-30 09:06:09,206 INFO [manager] launching NextWorker(module='nowcast.workers.get_NeahBay_ssh', args=['nowcast'], host='localhost')
2020-06-30 09:06:09,211 INFO [manager] launching NextWorker(module='nowcast.workers.grib_to_netcdf', args=['nowcast+'], host='localhost')
2020-06-30 09:06:09,215 INFO [manager] launching NextWorker(module='nowcast.workers.download_live_ocean', args=[], host='localhost')
2020-06-30 09:06:09,219 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['18', '2.5km'], host='localhost')
2020-06-30 09:06:10,349 INFO [collect_weather] running in process 28345
2020-06-30 09:06:10,349 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-06-30 09:06:10,350 INFO [collect_weather] publishing log messages to tcp://*:5562
2020-06-30 09:06:10,351 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2020-06-30 09:06:10,400 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-west/18/
2020-06-30 09:06:10,515 INFO [grib_to_netcdf] running in process 28343
2020-06-30 09:06:10,516 INFO [grib_to_netcdf] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-06-30 09:06:10,517 INFO [grib_to_netcdf] publishing log messages to tcp://*:5563
2020-06-30 09:06:10,518 INFO [grib_to_netcdf] connected to skookum.eos.ubc.ca port 5555
2020-06-30 09:06:10,654 INFO [download_live_ocean] running in process 28344
2020-06-30 09:06:10,655 INFO [download_live_ocean] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-06-30 09:06:10,656 INFO [download_live_ocean] publishing log messages to tcp://*:5564
2020-06-30 09:06:10,656 INFO [download_live_ocean] connected to skookum.eos.ubc.ca port 5555
2020-06-30 09:06:10,657 INFO [download_live_ocean] downloading Salish Sea western boundary daily averaged Live Ocean file for 2020-06-30
2020-06-30 09:06:11,257 INFO [get_NeahBay_ssh] running in process 28342
2020-06-30 09:06:11,258 INFO [get_NeahBay_ssh] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-06-30 09:06:11,259 INFO [get_NeahBay_ssh] publishing log messages to tcp://*:5565
2020-06-30 09:06:11,260 INFO [get_NeahBay_ssh] connected to skookum.eos.ubc.ca port 5555
2020-06-30 09:06:14,717 INFO [download_live_ocean] Deflating in up to 6 concurrent sub-processes
2020-06-30 09:06:14,892 INFO [get_NeahBay_ssh] Neah Bay sea surface height web scraping and open boundary file creation for nowcast complete
2020-06-30 09:06:14,896 INFO [manager] checklist updated with [Neah Bay ssh] items from get_NeahBay_ssh worker
2020-06-30 09:06:15,764 INFO [download_live_ocean] netCDF4 deflated /results/forcing/LiveOcean/downloaded/20200630/low_passed_UBC.nc
2020-06-30 09:06:15,764 INFO [download_live_ocean] 2020-06-30 Live Ocean file for Salish Sea western boundary downloaded
2020-06-30 09:06:15,769 INFO [manager] checklist updated with [Live Ocean products] items from download_live_ocean worker
2020-06-30 09:06:16,004 INFO [manager] launching NextWorker(module='nowcast.workers.make_live_ocean_files', args=['--run-date', '2020-06-30'], host='localhost')
2020-06-30 09:06:18,635 INFO [make_live_ocean_files] running in process 28452
2020-06-30 09:06:18,636 INFO [make_live_ocean_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-06-30 09:06:18,637 INFO [make_live_ocean_files] publishing log messages to tcp://*:5564
2020-06-30 09:06:18,638 INFO [make_live_ocean_files] connected to skookum.eos.ubc.ca port 5555
2020-06-30 09:06:18,639 INFO [make_live_ocean_files] Creating T&S western boundary conditions file from 2020-06-30 Live Ocean run
2020-06-30 09:07:18,746 INFO [make_live_ocean_files] Stored T&S western boundary conditions file: /
2020-06-30 09:07:18,746 INFO [make_live_ocean_files] 2020-06-30 Live Ocean western boundary conditions files created
2020-06-30 09:07:18,749 INFO [manager] checklist updated with [Live Ocean boundary conditions] items from make_live_ocean_files worker
2020-06-30 09:07:53,033 INFO [grib_to_netcdf] 2020-06-30 NEMO-atmos forcing file for nowcast+ created
2020-06-30 09:07:53,038 INFO [manager] checklist updated with [weather forcing] items from grib_to_netcdf worker
2020-06-30 09:07:53,044 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'nowcast+'], host='localhost')
2020-06-30 09:07:53,048 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'nowcast+'], host='localhost')
2020-06-30 09:07:53,086 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['graham-hindcast', 'nowcast+'], host='localhost')
2020-06-30 09:07:53,093 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'nowcast+'], host='localhost')
2020-06-30 09:07:53,096 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['weather'], host='localhost')
2020-06-30 09:07:54,030 INFO [ping_erddap] running in process 29835
2020-06-30 09:07:54,031 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-06-30 09:07:54,032 INFO [ping_erddap] publishing log messages to tcp://*:5563
2020-06-30 09:07:54,033 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2020-06-30 09:07:54,034 INFO [ping_erddap] weather ERDDAP dataset flag file(s) created
2020-06-30 09:07:54,036 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2020-06-30 09:07:54,079 INFO [upload_forcing] running in process 29833
2020-06-30 09:07:54,080 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-06-30 09:07:54,081 INFO [upload_forcing] publishing log messages to tcp://*:5564
2020-06-30 09:07:54,082 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2020-06-30 09:07:54,083 INFO [upload_forcing] running in process 29834
2020-06-30 09:07:54,084 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-06-30 09:07:54,085 INFO [upload_forcing] publishing log messages to tcp://*:5565
2020-06-30 09:07:54,085 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2020-06-30 09:07:54,086 INFO [upload_forcing] running in process 29831
2020-06-30 09:07:54,087 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-06-30 09:07:54,088 INFO [upload_forcing] publishing log messages to tcp://*:5566
2020-06-30 09:07:54,088 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2020-06-30 09:07:54,109 INFO [upload_forcing] running in process 29830
2020-06-30 09:07:54,110 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-06-30 09:07:54,111 INFO [upload_forcing] publishing log messages to tcp://*:5567
2020-06-30 09:07:54,112 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2020-06-30 09:07:58,362 INFO [upload_forcing] nowcast+ 2020-06-30 forcing files upload to optimum-hindcast completed
2020-06-30 09:07:58,366 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2020-06-30 09:07:58,716 INFO [upload_forcing] nowcast+ 2020-06-30 forcing files upload to orcinus-nowcast-agrif completed
2020-06-30 09:07:58,721 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2020-06-30 09:07:59,132 INFO [upload_forcing] nowcast+ 2020-06-30 forcing files upload to arbutus.cloud-nowcast completed
2020-06-30 09:07:59,137 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2020-06-30 09:07:59,146 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'nowcast+', '--run-date', '2020-06-30'], host='localhost')
2020-06-30 09:08:00,085 INFO [make_forcing_links] running in process 29864
2020-06-30 09:08:00,086 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-06-30 09:08:00,087 INFO [make_forcing_links] publishing log messages to tcp://*:5563
2020-06-30 09:08:00,088 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2020-06-30 09:08:01,349 INFO [make_forcing_links] nowcast+ 2020-06-30 forcing file links on arbutus.cloud-nowcast created
2020-06-30 09:08:01,354 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2020-06-30 09:08:01,365 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2020-06-30'], host='arbutus.cloud-nowcast')
2020-06-30 09:08:03,708 INFO [run_NEMO] running in process 22152
2020-06-30 09:08:03,709 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2020-06-30 09:08:03,710 INFO [run_NEMO] publishing log messages to tcp://*:5556
2020-06-30 09:08:03,720 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2020-06-30 09:08:04,697 INFO [run_NEMO] nowcast: launching /nemoShare/MEOPAR/nowcast-sys/runs/30jun20nowcast_2020-06-30T090803.774314-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2020-06-30 09:08:04,715 INFO [run_NEMO] nowcast NEMO run for 2020-06-30 on arbutus.cloud-nowcast started
2020-06-30 09:08:04,721 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2020-06-30 09:08:04,733 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'nowcast'], host='arbutus.cloud-nowcast')
2020-06-30 09:08:05,116 INFO [upload_forcing] nowcast+ 2020-06-30 forcing files upload to graham-hindcast completed
2020-06-30 09:08:05,291 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2020-06-30 09:08:06,733 INFO [watch_NEMO] running in process 22417
2020-06-30 09:08:06,734 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2020-06-30 09:08:06,735 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2020-06-30 09:08:06,736 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2020-06-30 09:08:06,818 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2020-06-30 09:13:06,919 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 7489155 = 2020-06-30 04:49:20 UTC, 20.1% complete
2020-06-30 09:18:07,021 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 7489647 = 2020-06-30 10:17:20 UTC, 42.9% complete
2020-06-30 09:23:07,128 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 7490139 = 2020-06-30 15:45:20 UTC, 65.7% complete
2020-06-30 09:28:07,224 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 7490611 = 2020-06-30 21:00:00 UTC, 87.5% complete
2020-06-30 09:33:07,329 INFO [watch_NEMO] nowcast NEMO run on arbutus.cloud-nowcast completed
2020-06-30 09:33:07,333 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2020-06-30 09:33:07,628 INFO [manager] launching NextWorker(module='nowcast.workers.get_NeahBay_ssh', args=['forecast'], host='localhost')
2020-06-30 09:33:07,633 INFO [manager] launching NextWorker(module='nowcast.workers.make_fvcom_boundary', args=['arbutus.cloud-nowcast', 'x2', 'nowcast'], host='arbutus.cloud-nowcast')
2020-06-30 09:33:07,640 INFO [manager] launching NextWorker(module='nowcast.workers.make_fvcom_boundary', args=['arbutus.cloud-nowcast', 'r12', 'nowcast'], host='arbutus.cloud-nowcast')
2020-06-30 09:33:07,644 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2020-06-30'], host='localhost')
2020-06-30 09:33:08,660 INFO [download_results] running in process 30486
2020-06-30 09:33:08,661 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-06-30 09:33:08,662 INFO [download_results] publishing log messages to tcp://*:5563
2020-06-30 09:33:08,662 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2020-06-30 09:33:08,663 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/nowcast/30jun20 to /results/SalishSea/nowcast-blue.201905
2020-06-30 09:33:09,830 INFO [get_NeahBay_ssh] running in process 30483
2020-06-30 09:33:09,831 INFO [get_NeahBay_ssh] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-06-30 09:33:09,832 INFO [get_NeahBay_ssh] publishing log messages to tcp://*:5564
2020-06-30 09:33:09,833 INFO [get_NeahBay_ssh] connected to skookum.eos.ubc.ca port 5555
2020-06-30 09:33:11,427 INFO [make_fvcom_boundary] running in process 22652
2020-06-30 09:33:11,428 INFO [make_fvcom_boundary] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2020-06-30 09:33:11,429 INFO [make_fvcom_boundary] publishing log messages to tcp://*:5580
2020-06-30 09:33:11,430 INFO [make_fvcom_boundary] connected to skookum.eos.ubc.ca port 5555
2020-06-30 09:33:11,432 INFO [make_fvcom_boundary] Creating VHFR FVCOM open boundary file for r12 nowcast run from 2020-06-30 NEMO run
2020-06-30 09:33:11,439 INFO [make_fvcom_boundary] running in process 22651
2020-06-30 09:33:11,440 INFO [make_fvcom_boundary] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2020-06-30 09:33:11,441 INFO [make_fvcom_boundary] publishing log messages to tcp://*:5581
2020-06-30 09:33:11,442 INFO [make_fvcom_boundary] connected to skookum.eos.ubc.ca port 5555
2020-06-30 09:33:11,444 INFO [make_fvcom_boundary] Creating VHFR FVCOM open boundary file for x2 nowcast run from 2020-06-30 NEMO run
2020-06-30 09:33:14,305 INFO [get_NeahBay_ssh] Neah Bay sea surface height web scraping and open boundary file creation for forecast complete
2020-06-30 09:33:14,311 INFO [manager] checklist updated with [Neah Bay ssh] items from get_NeahBay_ssh worker
2020-06-30 09:33:14,323 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'ssh'], host='localhost')
2020-06-30 09:33:15,269 INFO [upload_forcing] running in process 30514
2020-06-30 09:33:15,270 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-06-30 09:33:15,271 INFO [upload_forcing] publishing log messages to tcp://*:5564
2020-06-30 09:33:15,272 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2020-06-30 09:33:16,409 INFO [upload_forcing] ssh 2020-06-30 forcing files upload to arbutus.cloud-nowcast completed
2020-06-30 09:33:16,415 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2020-06-30 09:33:16,426 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'ssh', '--run-date', '2020-06-30'], host='localhost')
2020-06-30 09:33:17,489 INFO [make_forcing_links] running in process 30520
2020-06-30 09:33:17,491 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-06-30 09:33:17,492 INFO [make_forcing_links] publishing log messages to tcp://*:5564
2020-06-30 09:33:17,493 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2020-06-30 09:33:18,600 INFO [make_forcing_links] ssh 2020-06-30 forcing file links on arbutus.cloud-nowcast created
2020-06-30 09:33:18,606 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2020-06-30 09:33:18,618 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2020-06-30'], host='arbutus.cloud-nowcast')
2020-06-30 09:33:20,876 INFO [run_NEMO] running in process 23005
2020-06-30 09:33:20,878 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2020-06-30 09:33:20,878 INFO [run_NEMO] publishing log messages to tcp://*:5556
2020-06-30 09:33:20,880 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2020-06-30 09:33:21,905 INFO [run_NEMO] forecast: launching /nemoShare/MEOPAR/nowcast-sys/runs/30jun20forecast_2020-06-30T093320.963084-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2020-06-30 09:33:21,924 INFO [run_NEMO] forecast NEMO run for 2020-06-30 on arbutus.cloud-nowcast started
2020-06-30 09:33:21,930 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2020-06-30 09:33:21,944 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'forecast'], host='arbutus.cloud-nowcast')
2020-06-30 09:33:24,039 INFO [watch_NEMO] running in process 23270
2020-06-30 09:33:24,040 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2020-06-30 09:33:24,042 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2020-06-30 09:33:24,043 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2020-06-30 09:33:24,133 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2020-06-30 09:35:14,072 INFO [download_results] nowcast 2020-06-30 results files from arbutus.cloud-nowcast downloaded
2020-06-30 09:35:14,079 INFO [manager] checklist updated with [results files] items from download_results worker
2020-06-30 09:35:14,099 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast', 'research', '--run-date', '2020-06-30'], host='localhost')
2020-06-30 09:35:14,107 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast', 'comparison', '--run-date', '2020-06-29'], host='localhost')
2020-06-30 09:35:14,115 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['nowcast', '--run-date', '2020-06-30'], host='localhost')
2020-06-30 09:35:16,990 INFO [make_CHS_currents_file] running in process 30533
2020-06-30 09:35:17,058 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-06-30 09:35:17,059 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5563
2020-06-30 09:35:17,060 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555
2020-06-30 09:35:18,258 INFO [make_plots] running in process 30531
2020-06-30 09:35:18,259 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-06-30 09:35:18,260 INFO [make_plots] publishing log messages to tcp://*:5564
2020-06-30 09:35:18,266 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2020-06-30 09:35:18,267 INFO [make_plots] running in process 30532
2020-06-30 09:35:18,267 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-06-30 09:35:18,269 INFO [make_plots] publishing log messages to tcp://*:5565
2020-06-30 09:35:18,270 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2020-06-30 09:35:18,990 INFO [make_plots] preparing render list for 2020-06-30 NEMO nowcast-blue research figures
2020-06-30 09:35:19,012 INFO [make_plots] preparing render list for 2020-06-29 NEMO nowcast-blue comparison figures
2020-06-30 09:35:24,699 INFO [make_plots] added salinity_thalweg_and_surface figures to 2020-06-30 NEMO nowcast-blue research render list
2020-06-30 09:35:28,543 INFO [make_plots] added temperature_thalweg_and_surface figures to 2020-06-30 NEMO nowcast-blue research render list
2020-06-30 09:35:28,544 INFO [make_plots] added Currents_sections_and_surface figure to 2020-06-30 NEMO nowcast-blue research render list
2020-06-30 09:35:28,545 INFO [make_plots] added Currents_at_VENUS_Central figure to 2020-06-30 NEMO nowcast-blue research render list
2020-06-30 09:35:28,546 INFO [make_plots] added Currents_at_VENUS_East figure to 2020-06-30 NEMO nowcast-blue research render list
2020-06-30 09:35:28,547 INFO [make_plots] starting to render nemo nowcast research 30jun20 figures
2020-06-30 09:35:31,966 INFO [make_plots] added SH_wind figure to 2020-06-29 NEMO nowcast-blue comparison render list
2020-06-30 09:35:31,967 INFO [make_plots] added Compare_VENUS_East figure to 2020-06-29 NEMO nowcast-blue comparison render list
2020-06-30 09:35:31,968 INFO [make_plots] added Compare_VENUS_Central figure to 2020-06-29 NEMO nowcast-blue comparison render list
2020-06-30 09:35:31,968 INFO [make_plots] added Compare_VENUS_Delta_DDL figure to 2020-06-29 NEMO nowcast-blue comparison render list
2020-06-30 09:35:31,969 INFO [make_plots] added Central_ADCP figure to 2020-06-29 NEMO nowcast-blue comparison render list
2020-06-30 09:35:31,970 INFO [make_plots] added Central_depavADCP figure to 2020-06-29 NEMO nowcast-blue comparison render list
2020-06-30 09:35:31,970 INFO [make_plots] added Central_timeavADCP figure to 2020-06-29 NEMO nowcast-blue comparison render list
2020-06-30 09:35:31,971 INFO [make_plots] added East_ADCP figure to 2020-06-29 NEMO nowcast-blue comparison render list
2020-06-30 09:35:31,971 INFO [make_plots] added East_depavADCP figure to 2020-06-29 NEMO nowcast-blue comparison render list
2020-06-30 09:35:31,972 INFO [make_plots] added East_timeavADCP figure to 2020-06-29 NEMO nowcast-blue comparison render list
2020-06-30 09:35:31,973 INFO [make_plots] added ddl_ADCP figure to 2020-06-29 NEMO nowcast-blue comparison render list
2020-06-30 09:35:31,973 INFO [make_plots] added ddl_depavADCP figure to 2020-06-29 NEMO nowcast-blue comparison render list
2020-06-30 09:35:31,974 INFO [make_plots] added ddl_timeavADCP figure to 2020-06-29 NEMO nowcast-blue comparison render list
2020-06-30 09:35:31,974 INFO [make_plots] starting to render nemo nowcast comparison 29jun20 figures
2020-06-30 09:35:33,406 INFO [make_fvcom_boundary] Stored VHFR FVCOM open boundary file: /nemoShare/MEOPAR/nowcast-sys/fvcom-runs/input.r12/bdy_r12_nowcast_brcl_20200630.nc
2020-06-30 09:35:33,412 INFO [make_fvcom_boundary] FVCOM r12 nowcast run boundary condition file for 2020-06-30 created on arbutus.cloud-nowcast
2020-06-30 09:35:33,419 INFO [manager] checklist updated with [FVCOM boundary conditions] items from make_fvcom_boundary worker
2020-06-30 09:35:33,434 INFO [manager] launching NextWorker(module='nowcast.workers.make_fvcom_atmos_forcing', args=['r12', 'nowcast', '--run-date', '2020-06-30'], host='localhost')
2020-06-30 09:35:33,438 INFO [manager] launching NextWorker(module='nowcast.workers.make_fvcom_rivers_forcing', args=['arbutus.cloud-nowcast', 'r12', 'nowcast', '--run-date', '2020-06-30'], host='arbutus.cloud-nowcast')
2020-06-30 09:35:36,710 INFO [make_fvcom_atmos_forcing] running in process 30565
2020-06-30 09:35:36,711 INFO [make_fvcom_atmos_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-06-30 09:35:36,712 INFO [make_fvcom_atmos_forcing] publishing log messages to tcp://*:5566
2020-06-30 09:35:36,713 INFO [make_fvcom_atmos_forcing] connected to skookum.eos.ubc.ca port 5555
2020-06-30 09:35:36,714 INFO [make_fvcom_atmos_forcing] Creating VHFR FVCOM atmospheric forcing file for r12 nowcast run from 2020-06-30 HRDPS GRIB files
2020-06-30 09:35:37,479 INFO [make_fvcom_rivers_forcing] running in process 23364
2020-06-30 09:35:37,481 INFO [make_fvcom_rivers_forcing] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2020-06-30 09:35:37,482 INFO [make_fvcom_rivers_forcing] publishing log messages to tcp://*:5582
2020-06-30 09:35:37,483 INFO [make_fvcom_rivers_forcing] connected to skookum.eos.ubc.ca port 5555
2020-06-30 09:35:37,484 INFO [make_fvcom_rivers_forcing] Creating VHFR FVCOM rivers forcing file for r12 nowcast run from 2020-06-30 NEMO runoff forcing files
2020-06-30 09:35:37,701 INFO [make_fvcom_rivers_forcing] Stored VHFR FVCOM rivers forcing file: /nemoShare/MEOPAR/nowcast-sys/fvcom-runs/input.r12/rivers_r12_nowcast_20200630.nc
2020-06-30 09:35:37,702 INFO [make_fvcom_rivers_forcing] FVCOM r12 nowcast run rivers forcing file for 2020-06-30 created on arbutus.cloud-nowcast
2020-06-30 09:35:37,707 INFO [manager] checklist updated with [FVCOM rivers forcing] items from make_fvcom_rivers_forcing worker
2020-06-30 09:35:44,118 INFO [make_fvcom_boundary] Stored VHFR FVCOM open boundary file: /nemoShare/MEOPAR/nowcast-sys/fvcom-runs/input.x2/bdy_x2_nowcast_brcl_20200630.nc
2020-06-30 09:35:44,123 INFO [make_fvcom_boundary] FVCOM x2 nowcast run boundary condition file for 2020-06-30 created on arbutus.cloud-nowcast
2020-06-30 09:35:44,130 INFO [manager] checklist updated with [FVCOM boundary conditions] items from make_fvcom_boundary worker
2020-06-30 09:35:44,148 INFO [manager] launching NextWorker(module='nowcast.workers.make_fvcom_atmos_forcing', args=['x2', 'nowcast', '--run-date', '2020-06-30'], host='localhost')
2020-06-30 09:35:44,155 INFO [manager] launching NextWorker(module='nowcast.workers.make_fvcom_rivers_forcing', args=['arbutus.cloud-nowcast', 'x2', 'nowcast', '--run-date', '2020-06-30'], host='arbutus.cloud-nowcast')
2020-06-30 09:35:47,073 INFO [make_fvcom_atmos_forcing] running in process 30584
2020-06-30 09:35:47,075 INFO [make_fvcom_atmos_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-06-30 09:35:47,076 INFO [make_fvcom_atmos_forcing] publishing log messages to tcp://*:5567
2020-06-30 09:35:47,077 INFO [make_fvcom_atmos_forcing] connected to skookum.eos.ubc.ca port 5555
2020-06-30 09:35:47,078 INFO [make_fvcom_atmos_forcing] Creating VHFR FVCOM atmospheric forcing file for x2 nowcast run from 2020-06-30 HRDPS GRIB files
2020-06-30 09:35:48,434 INFO [make_fvcom_rivers_forcing] running in process 23456
2020-06-30 09:35:48,436 INFO [make_fvcom_rivers_forcing] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2020-06-30 09:35:48,437 INFO [make_fvcom_rivers_forcing] publishing log messages to tcp://*:5582
2020-06-30 09:35:48,438 INFO [make_fvcom_rivers_forcing] connected to skookum.eos.ubc.ca port 5555
2020-06-30 09:35:48,438 INFO [make_fvcom_rivers_forcing] Creating VHFR FVCOM rivers forcing file for x2 nowcast run from 2020-06-30 NEMO runoff forcing files
2020-06-30 09:35:48,656 INFO [make_fvcom_rivers_forcing] Stored VHFR FVCOM rivers forcing file: /nemoShare/MEOPAR/nowcast-sys/fvcom-runs/input.x2/rivers_x2_nowcast_20200630.nc
2020-06-30 09:35:48,657 INFO [make_fvcom_rivers_forcing] FVCOM x2 nowcast run rivers forcing file for 2020-06-30 created on arbutus.cloud-nowcast
2020-06-30 09:35:48,661 INFO [manager] checklist updated with [FVCOM rivers forcing] items from make_fvcom_rivers_forcing worker
2020-06-30 09:36:03,344 INFO [make_fvcom_atmos_forcing] Stored VHFR FVCOM atmospheric forcing hfx file: /results/forcing/atmospheric/GEM2.5/vhfr-fvcom/atmos_r12_nowcast_hfx_20200630.nc
2020-06-30 09:36:10,389 INFO [make_fvcom_atmos_forcing] Stored VHFR FVCOM atmospheric forcing hfx file: /results/forcing/atmospheric/GEM2.5/vhfr-fvcom/atmos_x2_nowcast_hfx_20200630.nc
2020-06-30 09:36:11,951 INFO [make_fvcom_atmos_forcing] Stored VHFR FVCOM atmospheric forcing precip file: /results/forcing/atmospheric/GEM2.5/vhfr-fvcom/atmos_r12_nowcast_precip_20200630.nc
2020-06-30 09:36:17,417 INFO [make_fvcom_atmos_forcing] Stored VHFR FVCOM atmospheric forcing precip file: /results/forcing/atmospheric/GEM2.5/vhfr-fvcom/atmos_x2_nowcast_precip_20200630.nc
2020-06-30 09:36:24,120 INFO [make_fvcom_atmos_forcing] Stored VHFR FVCOM atmospheric forcing wnd file: /results/forcing/atmospheric/GEM2.5/vhfr-fvcom/atmos_r12_nowcast_wnd_20200630.nc
2020-06-30 09:36:24,121 INFO [make_fvcom_atmos_forcing] FVCOM r12 nowcast run atmospheric forcing file for 2020-06-30 created
2020-06-30 09:36:24,125 INFO [manager] checklist updated with [FVCOM atmospheric forcing] items from make_fvcom_atmos_forcing worker
2020-06-30 09:36:24,139 INFO [manager] launching NextWorker(module='nowcast.workers.upload_fvcom_atmos_forcing', args=['arbutus.cloud-nowcast', 'r12', 'nowcast', '--run-date', '2020-06-30'], host='localhost')
2020-06-30 09:36:25,155 INFO [upload_fvcom_atmos_forcing] running in process 30618
2020-06-30 09:36:25,156 INFO [upload_fvcom_atmos_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-06-30 09:36:25,156 INFO [upload_fvcom_atmos_forcing] publishing log messages to tcp://*:5566
2020-06-30 09:36:25,157 INFO [upload_fvcom_atmos_forcing] connected to skookum.eos.ubc.ca port 5555
2020-06-30 09:36:25,158 INFO [upload_fvcom_atmos_forcing] Uploading VHFR FVCOM atmospheric forcing files for 2020-06-30 r12 nowcast run to arbutus.cloud-nowcast
2020-06-30 09:36:27,648 INFO [make_fvcom_atmos_forcing] Stored VHFR FVCOM atmospheric forcing wnd file: /results/forcing/atmospheric/GEM2.5/vhfr-fvcom/atmos_x2_nowcast_wnd_20200630.nc
2020-06-30 09:36:27,649 INFO [make_fvcom_atmos_forcing] FVCOM x2 nowcast run atmospheric forcing file for 2020-06-30 created
2020-06-30 09:36:27,652 INFO [manager] checklist updated with [FVCOM atmospheric forcing] items from make_fvcom_atmos_forcing worker
2020-06-30 09:36:27,661 INFO [manager] launching NextWorker(module='nowcast.workers.upload_fvcom_atmos_forcing', args=['arbutus.cloud-nowcast', 'x2', 'nowcast', '--run-date', '2020-06-30'], host='localhost')
2020-06-30 09:36:28,668 INFO [upload_fvcom_atmos_forcing] running in process 30626
2020-06-30 09:36:28,669 INFO [upload_fvcom_atmos_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-06-30 09:36:28,670 INFO [upload_fvcom_atmos_forcing] publishing log messages to tcp://*:5567
2020-06-30 09:36:28,671 INFO [upload_fvcom_atmos_forcing] connected to skookum.eos.ubc.ca port 5555
2020-06-30 09:36:28,672 INFO [upload_fvcom_atmos_forcing] Uploading VHFR FVCOM atmospheric forcing files for 2020-06-30 x2 nowcast run to arbutus.cloud-nowcast
2020-06-30 09:36:31,139 INFO [upload_fvcom_atmos_forcing] FVCOM r12 nowcast run atmospheric forcing files for 2020-06-30 uploaded to arbutus.cloud-nowcast
2020-06-30 09:36:31,144 INFO [manager] checklist updated with [FVCOM atmospheric forcing upload] items from upload_fvcom_atmos_forcing worker
2020-06-30 09:36:31,155 INFO [manager] launching NextWorker(module='nowcast.workers.run_fvcom', args=['arbutus.cloud-nowcast', 'r12', 'nowcast', '--run-date', '2020-06-30'], host='arbutus.cloud-nowcast')
2020-06-30 09:36:33,509 INFO [run_fvcom] running in process 23997
2020-06-30 09:36:33,510 INFO [run_fvcom] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2020-06-30 09:36:33,511 INFO [run_fvcom] publishing log messages to tcp://*:5584
2020-06-30 09:36:33,512 INFO [run_fvcom] connected to skookum.eos.ubc.ca port 5555
2020-06-30 09:36:33,591 INFO [run_fvcom] nowcast: launching /nemoShare/MEOPAR/nowcast-sys/fvcom-runs/20200630-16h36m33.579622s/VHFR_FVCOM.sh on arbutus.cloud-nowcast
2020-06-30 09:36:33,622 INFO [upload_fvcom_atmos_forcing] FVCOM x2 nowcast run atmospheric forcing files for 2020-06-30 uploaded to arbutus.cloud-nowcast
2020-06-30 09:36:33,623 INFO [run_fvcom] r12 nowcast FVCOM VH-FR run for 2020-06-30 on arbutus.cloud-nowcast started
2020-06-30 09:36:33,628 INFO [manager] checklist updated with [FVCOM atmospheric forcing upload] items from upload_fvcom_atmos_forcing worker
2020-06-30 09:36:33,646 INFO [manager] launching NextWorker(module='nowcast.workers.run_fvcom', args=['arbutus.cloud-nowcast', 'x2', 'nowcast', '--run-date', '2020-06-30'], host='arbutus.cloud-nowcast')
2020-06-30 09:36:33,655 INFO [manager] checklist updated with [FVCOM run] items from run_fvcom worker
2020-06-30 09:36:33,672 INFO [manager] launching NextWorker(module='nowcast.workers.watch_fvcom', args=['arbutus.cloud-nowcast', 'r12', 'nowcast'], host='arbutus.cloud-nowcast')
2020-06-30 09:36:35,943 INFO [watch_fvcom] running in process 24253
2020-06-30 09:36:35,945 INFO [watch_fvcom] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2020-06-30 09:36:35,946 INFO [watch_fvcom] publishing log messages to tcp://*:5586
2020-06-30 09:36:35,947 INFO [watch_fvcom] connected to skookum.eos.ubc.ca port 5555
2020-06-30 09:36:35,988 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: no run progress found in fvcom.log ; continuing to watch...
2020-06-30 09:36:36,105 INFO [run_fvcom] running in process 24255
2020-06-30 09:36:36,106 INFO [run_fvcom] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2020-06-30 09:36:36,107 INFO [run_fvcom] publishing log messages to tcp://*:5584
2020-06-30 09:36:36,108 INFO [run_fvcom] connected to skookum.eos.ubc.ca port 5555
2020-06-30 09:36:36,155 INFO [run_fvcom] nowcast: launching /nemoShare/MEOPAR/nowcast-sys/fvcom-runs/20200630-16h36m36.144818s/VHFR_FVCOM.sh on arbutus.cloud-nowcast
2020-06-30 09:36:36,179 INFO [run_fvcom] x2 nowcast FVCOM VH-FR run for 2020-06-30 on arbutus.cloud-nowcast started
2020-06-30 09:36:36,188 INFO [manager] checklist updated with [FVCOM run] items from run_fvcom worker
2020-06-30 09:36:36,209 INFO [manager] launching NextWorker(module='nowcast.workers.watch_fvcom', args=['arbutus.cloud-nowcast', 'x2', 'nowcast'], host='arbutus.cloud-nowcast')
2020-06-30 09:36:38,350 INFO [watch_fvcom] running in process 24361
2020-06-30 09:36:38,351 INFO [watch_fvcom] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2020-06-30 09:36:38,352 INFO [watch_fvcom] publishing log messages to tcp://*:5587
2020-06-30 09:36:38,353 INFO [watch_fvcom] connected to skookum.eos.ubc.ca port 5555
2020-06-30 09:36:38,390 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: no run progress found in fvcom.log ; continuing to watch...
2020-06-30 09:37:14,958 INFO [make_CHS_currents_file] Made CHS currents file for 2020-06-30 for nowcast
2020-06-30 09:37:14,962 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker
2020-06-30 09:37:53,302 WARNING [make_plots] VENUS Delta DDL node CTD comparison figure failed: No observations available
2020-06-30 09:37:54,138 WARNING [make_plots] VENUS Central ADCP comparison figure failed: No observations available
2020-06-30 09:37:54,699 WARNING [make_plots] VENUS Central ADCP comparison figure failed: No observations available
2020-06-30 09:37:55,269 WARNING [make_plots] VENUS Central ADCP comparison figure failed: No observations available
2020-06-30 09:37:55,887 WARNING [make_plots] VENUS East ADCP comparison figure failed: No observations available
2020-06-30 09:37:56,469 WARNING [make_plots] VENUS East ADCP comparison figure failed: No observations available
2020-06-30 09:37:57,052 WARNING [make_plots] VENUS East ADCP comparison figure failed: No observations available
2020-06-30 09:37:57,695 WARNING [make_plots] VENUS ddl ADCP comparison figure failed: No observations available
2020-06-30 09:37:58,266 WARNING [make_plots] VENUS ddl ADCP comparison figure failed: No observations available
2020-06-30 09:37:58,820 WARNING [make_plots] VENUS ddl ADCP comparison figure failed: No observations available
2020-06-30 09:37:58,821 INFO [make_plots] finished rendering nemo nowcast comparison 29jun20 figures
2020-06-30 09:37:58,910 INFO [make_plots] nemo comparison plots for 2020-06-29 nowcast completed
2020-06-30 09:37:58,914 INFO [manager] checklist updated with [plots] items from make_plots worker
2020-06-30 09:38:24,235 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 7491298 = 2020-07-01 04:38:00 UTC, 12.9% complete
2020-06-30 09:38:36,490 INFO [make_plots] finished rendering nemo nowcast research 30jun20 figures
2020-06-30 09:38:36,492 INFO [make_plots] nemo research plots for 2020-06-30 nowcast completed
2020-06-30 09:38:36,520 INFO [manager] checklist updated with [plots] items from make_plots worker
2020-06-30 09:41:36,089 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 38669691 = 2020-06-30 00:07:56 UTC estimated time to finish: 0000:14:18:44
2020-06-30 09:41:38,435 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15469212 = 2020-06-30 00:30:12 UTC estimated time to finish: 0000:03:39:18
2020-06-30 09:43:24,337 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 7491766 = 2020-07-01 09:50:00 UTC, 27.3% complete
2020-06-30 09:46:36,192 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 38670866 = 2020-06-30 00:15:46 UTC estimated time to finish: 0000:14:13:51
2020-06-30 09:46:38,517 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15471189 = 2020-06-30 01:03:09 UTC estimated time to finish: 0000:03:32:19
2020-06-30 09:48:24,439 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 7492230 = 2020-07-01 14:59:20 UTC, 41.6% complete
2020-06-30 09:51:36,297 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 38672136 = 2020-06-30 00:24:14 UTC estimated time to finish: 0000:14:10:49
2020-06-30 09:51:38,596 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15473001 = 2020-06-30 01:33:21 UTC estimated time to finish: 0000:03:28:01
2020-06-30 09:53:24,512 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 7492704 = 2020-07-01 20:15:20 UTC, 56.3% complete
2020-06-30 09:56:36,401 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 38673283 = 2020-06-30 00:31:53 UTC estimated time to finish: 0000:14:11:55
2020-06-30 09:56:38,678 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15474964 = 2020-06-30 02:06:04 UTC estimated time to finish: 0000:03:23:41
2020-06-30 09:58:24,613 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 7493162 = 2020-07-02 01:20:40 UTC, 70.4% complete
2020-06-30 10:01:36,504 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 38674580 = 2020-06-30 00:40:32 UTC estimated time to finish: 0000:14:02:22
2020-06-30 10:01:38,756 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15476941 = 2020-06-30 02:39:01 UTC estimated time to finish: 0000:03:18:33
2020-06-30 10:03:24,714 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 7493610 = 2020-07-02 06:19:20 UTC, 84.3% complete
2020-06-30 10:06:36,609 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 38675810 = 2020-06-30 00:48:44 UTC estimated time to finish: 0000:14:00:45
2020-06-30 10:06:38,838 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15478917 = 2020-06-30 03:11:57 UTC estimated time to finish: 0000:03:13:34
2020-06-30 10:08:24,815 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 7494077 = 2020-07-02 11:30:40 UTC, 98.7% complete
2020-06-30 10:11:36,712 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 38677270 = 2020-06-30 00:58:28 UTC estimated time to finish: 0000:13:37:58
2020-06-30 10:11:38,916 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15480840 = 2020-06-30 03:44:00 UTC estimated time to finish: 0000:03:09:11
2020-06-30 10:13:24,920 INFO [watch_NEMO] forecast NEMO run on arbutus.cloud-nowcast completed
2020-06-30 10:13:24,925 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2020-06-30 10:13:25,256 INFO [manager] launching NextWorker(module='nowcast.workers.make_turbidity_file', args=['--run-date', '2020-06-30'], host='localhost')
2020-06-30 10:13:25,262 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2020-06-30'], host='localhost')
2020-06-30 10:13:26,358 INFO [download_results] running in process 31360
2020-06-30 10:13:26,359 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-06-30 10:13:26,360 INFO [download_results] publishing log messages to tcp://*:5563
2020-06-30 10:13:26,361 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2020-06-30 10:13:26,362 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/forecast/30jun20 to /results/SalishSea/forecast.201905
2020-06-30 10:13:26,714 INFO [make_turbidity_file] running in process 31359
2020-06-30 10:13:26,715 INFO [make_turbidity_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-06-30 10:13:26,716 INFO [make_turbidity_file] publishing log messages to tcp://*:5564
2020-06-30 10:13:26,717 INFO [make_turbidity_file] connected to skookum.eos.ubc.ca port 5555
2020-06-30 10:13:26,718 INFO [make_turbidity_file] Creating Fraser River turbidity forcing file for 2020-06-30
2020-06-30 10:13:27,116 INFO [make_turbidity_file] Note: NumExpr detected 12 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
2020-06-30 10:13:27,117 INFO [make_turbidity_file] NumExpr defaulting to 8 threads.
2020-06-30 10:13:28,690 INFO [make_turbidity_file] 2020-06-30 Fraser River turbidity file creation complete
2020-06-30 10:13:28,692 INFO [manager] checklist updated with [Fraser River turbidity] items from make_turbidity_file worker
2020-06-30 10:13:28,710 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'turbidity'], host='localhost')
2020-06-30 10:13:28,713 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'turbidity'], host='localhost')
2020-06-30 10:13:28,717 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['graham-hindcast', 'turbidity'], host='localhost')
2020-06-30 10:13:28,720 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'turbidity'], host='localhost')
2020-06-30 10:13:29,652 INFO [upload_forcing] running in process 31379
2020-06-30 10:13:29,654 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-06-30 10:13:29,654 INFO [upload_forcing] publishing log messages to tcp://*:5564
2020-06-30 10:13:29,655 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2020-06-30 10:13:29,687 INFO [upload_forcing] running in process 31378
2020-06-30 10:13:29,688 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-06-30 10:13:29,689 INFO [upload_forcing] publishing log messages to tcp://*:5565
2020-06-30 10:13:29,690 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2020-06-30 10:13:29,720 INFO [upload_forcing] running in process 31380
2020-06-30 10:13:29,721 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-06-30 10:13:29,722 INFO [upload_forcing] publishing log messages to tcp://*:5566
2020-06-30 10:13:29,723 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2020-06-30 10:13:29,777 INFO [upload_forcing] running in process 31377
2020-06-30 10:13:29,778 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-06-30 10:13:29,779 INFO [upload_forcing] publishing log messages to tcp://*:5567
2020-06-30 10:13:29,780 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2020-06-30 10:13:30,168 INFO [upload_forcing] turbidity 2020-06-30 forcing files upload to optimum-hindcast completed
2020-06-30 10:13:30,174 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2020-06-30 10:13:30,344 INFO [upload_forcing] turbidity 2020-06-30 forcing files upload to orcinus-nowcast-agrif completed
2020-06-30 10:13:30,348 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2020-06-30 10:13:30,372 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['orcinus-nowcast-agrif', 'nowcast-agrif', '--run-date', '2020-06-30'], host='localhost')
2020-06-30 10:13:30,921 INFO [upload_forcing] turbidity 2020-06-30 forcing files upload to arbutus.cloud-nowcast completed
2020-06-30 10:13:30,926 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2020-06-30 10:13:30,950 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2020-06-30'], host='localhost')
2020-06-30 10:13:31,415 INFO [make_forcing_links] running in process 31403
2020-06-30 10:13:31,416 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-06-30 10:13:31,417 INFO [make_forcing_links] publishing log messages to tcp://*:5565
2020-06-30 10:13:31,418 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2020-06-30 10:13:31,978 INFO [make_forcing_links] running in process 31405
2020-06-30 10:13:31,979 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-06-30 10:13:31,980 INFO [make_forcing_links] publishing log messages to tcp://*:5566
2020-06-30 10:13:31,981 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2020-06-30 10:13:32,081 INFO [make_forcing_links] nowcast-agrif 2020-06-30 forcing file links on orcinus-nowcast-agrif created
2020-06-30 10:13:32,086 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2020-06-30 10:13:32,119 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_agrif', args=['orcinus-nowcast-agrif', 'nowcast-agrif', '--run-date', '2020-06-30'], host='localhost')
2020-06-30 10:13:32,975 INFO [upload_forcing] turbidity 2020-06-30 forcing files upload to graham-hindcast completed
2020-06-30 10:13:32,981 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2020-06-30 10:13:33,207 INFO [make_forcing_links] nowcast-green 2020-06-30 forcing file links on arbutus.cloud-nowcast created
2020-06-30 10:13:33,211 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2020-06-30 10:13:33,244 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2020-06-30'], host='arbutus.cloud-nowcast')
2020-06-30 10:13:33,370 INFO [run_NEMO_agrif] running in process 31415
2020-06-30 10:13:33,371 INFO [run_NEMO_agrif] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-06-30 10:13:33,372 INFO [run_NEMO_agrif] publishing log messages to tcp://*:5564
2020-06-30 10:13:33,373 INFO [run_NEMO_agrif] connected to skookum.eos.ubc.ca port 5555
2020-06-30 10:13:33,869 CRITICAL [run_NEMO_agrif] unhandled exception:
Traceback (most recent call last):
  File "/SalishSeaCast/NEMO_Nowcast/nemo_nowcast/worker.py", line 390, in _do_work
    checklist = self.worker_func(
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/run_NEMO_agrif.py", line 110, in run_NEMO_agrif
    prev_run_namelists_info = _get_prev_run_namelists_info(
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/run_NEMO_agrif.py", line 156, in _get_prev_run_namelists_info
    sftp_client.get(os.fspath(prev_namelist_cfg), namelist_cfg.name)
  File "/SalishSeaCast/nowcast-env/lib/python3.8/site-packages/paramiko/sftp_client.py", line 802, in get
    size = self.getfo(remotepath, fl, callback)
  File "/SalishSeaCast/nowcast-env/lib/python3.8/site-packages/paramiko/sftp_client.py", line 778, in getfo
    file_size = self.stat(remotepath).st_size
  File "/SalishSeaCast/nowcast-env/lib/python3.8/site-packages/paramiko/sftp_client.py", line 493, in stat
    t, msg = self._request(CMD_STAT, path)
  File "/SalishSeaCast/nowcast-env/lib/python3.8/site-packages/paramiko/sftp_client.py", line 813, in _request
    return self._read_response(num)
  File "/SalishSeaCast/nowcast-env/lib/python3.8/site-packages/paramiko/sftp_client.py", line 865, in _read_response
    self._convert_status(msg)
  File "/SalishSeaCast/nowcast-env/lib/python3.8/site-packages/paramiko/sftp_client.py", line 894, in _convert_status
    raise IOError(errno.ENOENT, text)
FileNotFoundError: [Errno 2] No such file
2020-06-30 10:13:35,517 INFO [run_NEMO] running in process 24832
2020-06-30 10:13:35,518 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2020-06-30 10:13:35,520 INFO [run_NEMO] publishing log messages to tcp://*:5556
2020-06-30 10:13:35,527 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2020-06-30 10:13:36,683 INFO [run_NEMO] nowcast-green: launching /nemoShare/MEOPAR/nowcast-sys/runs/30jun20nowcast-green_2020-06-30T101335.588719-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2020-06-30 10:13:36,702 INFO [run_NEMO] nowcast-green NEMO run for 2020-06-30 on arbutus.cloud-nowcast started
2020-06-30 10:13:36,709 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2020-06-30 10:13:36,745 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'nowcast-green'], host='arbutus.cloud-nowcast')
2020-06-30 10:13:38,742 INFO [watch_NEMO] running in process 25097
2020-06-30 10:13:38,743 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2020-06-30 10:13:38,744 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2020-06-30 10:13:38,746 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2020-06-30 10:13:38,864 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2020-06-30 10:15:18,262 INFO [upload_forcing] running in process 31547
2020-06-30 10:15:18,264 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-06-30 10:15:18,265 INFO [upload_forcing] publishing log messages to tcp://*:5564
2020-06-30 10:15:18,266 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2020-06-30 10:15:24,940 INFO [upload_forcing] nowcast+ 2020-06-27 forcing files upload to orcinus-nowcast-agrif completed
2020-06-30 10:15:24,945 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2020-06-30 10:15:34,943 INFO [upload_forcing] running in process 31554
2020-06-30 10:15:34,945 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-06-30 10:15:34,946 INFO [upload_forcing] publishing log messages to tcp://*:5564
2020-06-30 10:15:34,947 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2020-06-30 10:15:35,543 INFO [upload_forcing] turbidity 2020-06-27 forcing files upload to orcinus-nowcast-agrif completed
2020-06-30 10:15:35,548 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2020-06-30 10:15:35,585 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['orcinus-nowcast-agrif', 'nowcast-agrif', '--run-date', '2020-06-27'], host='localhost')
2020-06-30 10:15:36,631 INFO [make_forcing_links] running in process 31560
2020-06-30 10:15:36,632 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-06-30 10:15:36,633 INFO [make_forcing_links] publishing log messages to tcp://*:5564
2020-06-30 10:15:36,634 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2020-06-30 10:15:37,243 INFO [make_forcing_links] nowcast-agrif 2020-06-27 forcing file links on orcinus-nowcast-agrif created
2020-06-30 10:15:37,248 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2020-06-30 10:15:37,281 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_agrif', args=['orcinus-nowcast-agrif', 'nowcast-agrif', '--run-date', '2020-06-27'], host='localhost')
2020-06-30 10:15:38,409 INFO [run_NEMO_agrif] running in process 31566
2020-06-30 10:15:38,410 INFO [run_NEMO_agrif] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-06-30 10:15:38,412 INFO [run_NEMO_agrif] publishing log messages to tcp://*:5564
2020-06-30 10:15:38,413 INFO [run_NEMO_agrif] connected to skookum.eos.ubc.ca port 5555
2020-06-30 10:16:13,208 INFO [run_NEMO_agrif] job id for 27jun20nowcast-agrif: 10846020.orca2.ibb
2020-06-30 10:16:13,209 INFO [run_NEMO_agrif] NEMO nowcast-agrif run for 2020-06-27 queued on orcinus-nowcast-agrif
2020-06-30 10:16:13,216 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_agrif worker
2020-06-30 10:16:13,252 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_agrif', args=['orcinus-nowcast-agrif', '10846020.orca2.ibb'], host='localhost')
2020-06-30 10:16:14,506 INFO [watch_NEMO_agrif] running in process 31608
2020-06-30 10:16:14,508 INFO [watch_NEMO_agrif] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-06-30 10:16:14,509 INFO [watch_NEMO_agrif] publishing log messages to tcp://*:5564
2020-06-30 10:16:14,510 INFO [watch_NEMO_agrif] connected to skookum.eos.ubc.ca port 5555
2020-06-30 10:16:15,347 INFO [watch_NEMO_agrif] watching 27jun20nowcast-agrif job 10846020 on orcinus-nowcast-agrif
2020-06-30 10:16:15,585 INFO [watch_NEMO_agrif] 27jun20nowcast-agrif job 10846020 is queued on orcinus-nowcast-agrif
2020-06-30 10:16:21,662 INFO [download_results] forecast 2020-06-30 results files from arbutus.cloud-nowcast downloaded
2020-06-30 10:16:21,668 INFO [manager] checklist updated with [results files] items from download_results worker
2020-06-30 10:16:21,704 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['forecast', '--run-date', '2020-06-30'], host='localhost')
2020-06-30 10:16:24,594 INFO [make_CHS_currents_file] running in process 31614
2020-06-30 10:16:24,595 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-06-30 10:16:24,596 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5563
2020-06-30 10:16:24,597 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555
2020-06-30 10:16:36,821 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 38678690 = 2020-06-30 01:07:56 UTC estimated time to finish: 0000:13:22:14
2020-06-30 10:16:39,023 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15482611 = 2020-06-30 04:13:31 UTC estimated time to finish: 0000:03:04:56
2020-06-30 10:17:13,395 INFO [make_CHS_currents_file] Made CHS currents file for 2020-06-30 for forecast
2020-06-30 10:17:13,400 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker
2020-06-30 10:17:13,435 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['nemo', 'forecast', '--run-date', '2020-06-30'], host='localhost')
2020-06-30 10:17:14,223 INFO [update_forecast_datasets] running in process 31638
2020-06-30 10:17:14,224 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-06-30 10:17:14,225 INFO [update_forecast_datasets] publishing log messages to tcp://*:5563
2020-06-30 10:17:14,226 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2020-06-30 10:17:14,228 INFO [update_forecast_datasets] updating nemo forecast directory for forecast/30jun20 run
2020-06-30 10:17:14,657 INFO [update_forecast_datasets] updated nemo forecast directory for forecast/30jun20 run: /results/SalishSea/rolling-forecasts/nemo
2020-06-30 10:17:14,658 INFO [update_forecast_datasets] nemo 2020-06-30 forecast rolling forecast datasets updated
2020-06-30 10:17:14,662 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2020-06-30 10:17:14,696 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nemo-forecast'], host='localhost')
2020-06-30 10:17:14,700 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'forecast', 'publish', '--run-date', '2020-06-30'], host='localhost')
2020-06-30 10:17:14,704 INFO [manager] launching NextWorker(module='nowcast.workers.make_surface_current_tiles', args=['forecast', '--run-date', '2020-06-30'], host='localhost')
2020-06-30 10:17:15,539 INFO [ping_erddap] running in process 31642
2020-06-30 10:17:15,540 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-06-30 10:17:15,541 INFO [ping_erddap] publishing log messages to tcp://*:5563
2020-06-30 10:17:15,542 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2020-06-30 10:17:15,573 INFO [ping_erddap] nemo-forecast ERDDAP dataset flag file(s) created
2020-06-30 10:17:15,575 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2020-06-30 10:17:15,834 INFO [watch_NEMO_agrif] 27jun20nowcast-agrif job 10846020 is queued on orcinus-nowcast-agrif
2020-06-30 10:17:15,908 INFO [make_surface_current_tiles] running in process 31644
2020-06-30 10:17:15,909 INFO [make_surface_current_tiles] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-06-30 10:17:15,910 INFO [make_surface_current_tiles] publishing log messages to tcp://*:5563
2020-06-30 10:17:15,910 INFO [make_surface_current_tiles] connected to skookum.eos.ubc.ca port 5555
2020-06-30 10:17:15,970 INFO [make_surface_current_tiles] starting rendering of tiles for 2020-06-30 forecast in 6 processes into /results/nowcast-sys/figures/surface_currents/forecast/30jun20
2020-06-30 10:17:21,243 INFO [make_plots] running in process 31643
2020-06-30 10:17:21,245 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-06-30 10:17:21,246 INFO [make_plots] publishing log messages to tcp://*:5565
2020-06-30 10:17:21,246 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2020-06-30 10:17:22,126 INFO [make_plots] preparing render list for 2020-06-30 NEMO forecast publish figures
2020-06-30 10:17:22,387 INFO [make_plots] added Website_thumbnail figure to 2020-06-30 NEMO forecast publish render list
2020-06-30 10:17:22,388 INFO [make_plots] added Threshold_website figure to 2020-06-30 NEMO forecast publish render list
2020-06-30 10:17:22,389 INFO [make_plots] added PA_tidal_predictions figure to 2020-06-30 NEMO forecast publish render list
2020-06-30 10:17:22,390 INFO [make_plots] added Boundary Bay figure to 2020-06-30 NEMO forecast publish render list
2020-06-30 10:17:22,390 INFO [make_plots] added Campbell River figure to 2020-06-30 NEMO forecast publish render list
2020-06-30 10:17:22,391 INFO [make_plots] added Cherry Point figure to 2020-06-30 NEMO forecast publish render list
2020-06-30 10:17:22,392 INFO [make_plots] added Friday Harbor figure to 2020-06-30 NEMO forecast publish render list
2020-06-30 10:17:22,393 INFO [make_plots] added Halfmoon Bay figure to 2020-06-30 NEMO forecast publish render list
2020-06-30 10:17:22,393 INFO [make_plots] added Nanaimo figure to 2020-06-30 NEMO forecast publish render list
2020-06-30 10:17:22,394 INFO [make_plots] added Neah Bay figure to 2020-06-30 NEMO forecast publish render list
2020-06-30 10:17:22,395 INFO [make_plots] added New Westminster figure to 2020-06-30 NEMO forecast publish render list
2020-06-30 10:17:22,396 INFO [make_plots] added Patricia Bay figure to 2020-06-30 NEMO forecast publish render list
2020-06-30 10:17:22,397 INFO [make_plots] added Point Atkinson figure to 2020-06-30 NEMO forecast publish render list
2020-06-30 10:17:22,398 INFO [make_plots] added Port Renfrew figure to 2020-06-30 NEMO forecast publish render list
2020-06-30 10:17:22,398 INFO [make_plots] added Sand Heads figure to 2020-06-30 NEMO forecast publish render list
2020-06-30 10:17:22,399 INFO [make_plots] added Sandy Cove figure to 2020-06-30 NEMO forecast publish render list
2020-06-30 10:17:22,400 INFO [make_plots] added Squamish figure to 2020-06-30 NEMO forecast publish render list
2020-06-30 10:17:22,401 INFO [make_plots] added Victoria figure to 2020-06-30 NEMO forecast publish render list
2020-06-30 10:17:22,402 INFO [make_plots] added Woodwards Landing figure to 2020-06-30 NEMO forecast publish render list
2020-06-30 10:17:22,402 INFO [make_plots] starting to render nemo forecast publish 30jun20 figures
2020-06-30 10:17:52,873 INFO [make_plots] Note: NumExpr detected 12 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
2020-06-30 10:17:52,874 INFO [make_plots] NumExpr defaulting to 8 threads.
2020-06-30 10:18:16,971 INFO [watch_NEMO_agrif] 27jun20nowcast-agrif on orcinus-nowcast-agrif: time.step not found; continuing to watch...
2020-06-30 10:18:38,965 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 7488860 = 2020-06-30 01:32:40 UTC, 6.4% complete
2020-06-30 10:21:36,927 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 38679796 = 2020-06-30 01:15:18 UTC estimated time to finish: 0000:13:22:40
2020-06-30 10:21:39,083 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15484587 = 2020-06-30 04:46:27 UTC estimated time to finish: 0000:02:59:37
2020-06-30 10:23:17,518 INFO [watch_NEMO_agrif] 27jun20nowcast-agrif on orcinus-nowcast-agrif: timestep: 4611840 = 2020-06-27 02:39:20 UTC, 11.1% complete
2020-06-30 10:23:39,067 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 7489018 = 2020-06-30 03:18:00 UTC, 13.8% complete
2020-06-30 10:26:37,037 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 38681053 = 2020-06-30 01:23:41 UTC estimated time to finish: 0000:13:20:25
2020-06-30 10:26:39,161 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15486510 = 2020-06-30 05:18:30 UTC estimated time to finish: 0000:02:55:07
2020-06-30 10:28:18,062 INFO [watch_NEMO_agrif] 27jun20nowcast-agrif on orcinus-nowcast-agrif: timestep: 4612098 = 2020-06-27 05:31:20 UTC, 23.0% complete
2020-06-30 10:28:39,168 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 7489171 = 2020-06-30 05:00:00 UTC, 20.8% complete
2020-06-30 10:31:37,143 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 38682281 = 2020-06-30 01:31:52 UTC estimated time to finish: 0000:13:19:23
2020-06-30 10:31:39,247 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15488253 = 2020-06-30 05:47:33 UTC estimated time to finish: 0000:02:52:27
2020-06-30 10:33:18,643 INFO [watch_NEMO_agrif] 27jun20nowcast-agrif on orcinus-nowcast-agrif: timestep: 4612357 = 2020-06-27 08:24:00 UTC, 35.0% complete
2020-06-30 10:33:39,269 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 7489324 = 2020-06-30 06:42:00 UTC, 27.9% complete
2020-06-30 10:34:42,693 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 08074 Campbell River from 2020-06-25 00:00:00 to 2020-07-02 11:55:00
2020-06-30 10:36:37,255 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 38683538 = 2020-06-30 01:40:15 UTC estimated time to finish: 0000:13:17:49
2020-06-30 10:36:39,327 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15489780 = 2020-06-30 06:13:00 UTC estimated time to finish: 0000:02:49:54
2020-06-30 10:38:19,191 INFO [watch_NEMO_agrif] 27jun20nowcast-agrif on orcinus-nowcast-agrif: timestep: 4612614 = 2020-06-27 11:15:20 UTC, 46.9% complete
2020-06-30 10:38:36,015 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 9449424 Cherry Point from 2020-06-25 00:00:00 to 2020-07-02 11:55:00
2020-06-30 10:38:39,369 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 7489477 = 2020-06-30 08:24:00 UTC, 35.0% complete
2020-06-30 10:39:40,642 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 9449880 Friday Harbor from 2020-06-25 00:00:00 to 2020-07-02 11:55:00
2020-06-30 10:40:35,910 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07830 Halfmoon Bay from 2020-06-25 00:00:00 to 2020-07-02 11:55:00
2020-06-30 10:41:37,362 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 38684685 = 2020-06-30 01:47:54 UTC estimated time to finish: 0000:13:13:49
2020-06-30 10:41:38,331 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07917 Nanaimo from 2020-06-25 00:00:00 to 2020-07-02 11:55:00
2020-06-30 10:41:39,405 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15491553 = 2020-06-30 06:42:33 UTC estimated time to finish: 0000:02:46:32
2020-06-30 10:43:19,739 INFO [watch_NEMO_agrif] 27jun20nowcast-agrif on orcinus-nowcast-agrif: timestep: 4612869 = 2020-06-27 14:05:20 UTC, 58.7% complete
2020-06-30 10:43:39,469 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 7489631 = 2020-06-30 10:06:40 UTC, 42.1% complete
2020-06-30 10:44:34,507 INFO [make_surface_current_tiles] starting PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast/30jun20
2020-06-30 10:44:50,119 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 9443090 Neah Bay from 2020-06-25 00:00:00 to 2020-07-02 11:55:00
2020-06-30 10:45:06,827 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07654 New Westminster from 2020-06-25 00:00:00 to 2020-07-02 11:55:00
2020-06-30 10:46:37,474 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 38685913 = 2020-06-30 01:56:05 UTC estimated time to finish: 0000:13:10:39
2020-06-30 10:46:39,515 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15493256 = 2020-06-30 07:10:56 UTC estimated time to finish: 0000:02:43:07
2020-06-30 10:47:20,551 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07277 Patricia Bay from 2020-06-25 00:00:00 to 2020-07-02 11:55:00
2020-06-30 10:48:20,279 INFO [watch_NEMO_agrif] 27jun20nowcast-agrif on orcinus-nowcast-agrif: timestep: 4613128 = 2020-06-27 16:58:00 UTC, 70.7% complete
2020-06-30 10:48:39,570 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 7489786 = 2020-06-30 11:50:00 UTC, 49.3% complete
2020-06-30 10:48:49,541 INFO [make_surface_current_tiles] finished PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast/30jun20
2020-06-30 10:48:49,563 INFO [make_surface_current_tiles] finished rendering of tiles for 2020-06-30 forecast into /results/nowcast-sys/figures/surface_currents/forecast/30jun20
2020-06-30 10:48:49,564 INFO [make_surface_current_tiles] surface current tile figures for 2020-06-30 forecast completed
2020-06-30 10:48:50,044 INFO [manager] checklist updated with [surface current tiles] items from make_surface_current_tiles worker
2020-06-30 10:49:29,870 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07795 Point Atkinson from 2020-06-25 00:00:00 to 2020-07-02 11:55:00
2020-06-30 10:49:47,883 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 08525 Port Renfrew from 2020-06-25 00:00:00 to 2020-07-02 11:55:00
2020-06-30 10:51:37,583 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 38687211 = 2020-06-30 02:04:44 UTC estimated time to finish: 0000:13:06:13
2020-06-30 10:51:39,570 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15495029 = 2020-06-30 07:40:29 UTC estimated time to finish: 0000:02:39:13
2020-06-30 10:51:56,650 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07594 Sand Heads from 2020-06-25 00:00:00 to 2020-07-02 11:55:00
2020-06-30 10:53:20,803 INFO [watch_NEMO_agrif] 27jun20nowcast-agrif on orcinus-nowcast-agrif: timestep: 4613384 = 2020-06-27 19:48:40 UTC, 82.6% complete
2020-06-30 10:53:39,672 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 7489943 = 2020-06-30 13:34:40 UTC, 56.6% complete
2020-06-30 10:54:06,741 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07786 Sandy Cove from 2020-06-25 00:00:00 to 2020-07-02 11:55:00
2020-06-30 10:56:17,757 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07811 Squamish from 2020-06-25 00:00:00 to 2020-07-02 11:55:00
2020-06-30 10:56:35,952 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07120 Victoria from 2020-06-25 00:00:00 to 2020-07-02 11:55:00
2020-06-30 10:56:37,694 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 38688440 = 2020-06-30 02:12:56 UTC estimated time to finish: 0000:13:02:46
2020-06-30 10:56:39,651 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15496560 = 2020-06-30 08:06:00 UTC estimated time to finish: 0000:02:35:54
2020-06-30 10:58:21,385 INFO [watch_NEMO_agrif] 27jun20nowcast-agrif on orcinus-nowcast-agrif: timestep: 4613641 = 2020-06-27 22:40:00 UTC, 94.5% complete
2020-06-30 10:58:39,770 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 7490102 = 2020-06-30 15:20:40 UTC, 64.0% complete
2020-06-30 10:58:47,681 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07610 Woodwards Landing from 2020-06-25 00:00:00 to 2020-07-02 11:55:00
2020-06-30 11:00:54,053 INFO [make_plots] finished rendering nemo forecast publish 30jun20 figures
2020-06-30 11:00:54,055 INFO [make_plots] nemo publish plots for 2020-06-30 forecast completed
2020-06-30 11:00:54,067 INFO [manager] checklist updated with [plots] items from make_plots worker
2020-06-30 11:00:54,329 INFO [manager] launching NextWorker(module='nowcast.workers.make_feeds', args=['forecast', '--run-date', '2020-06-30'], host='localhost')
2020-06-30 11:00:58,919 INFO [make_feeds] running in process 2333
2020-06-30 11:00:58,920 INFO [make_feeds] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-06-30 11:00:58,922 INFO [make_feeds] publishing log messages to tcp://*:5563
2020-06-30 11:00:58,923 INFO [make_feeds] connected to skookum.eos.ubc.ca port 5555
2020-06-30 11:01:07,823 INFO [make_feeds] Note: NumExpr detected 12 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
2020-06-30 11:01:07,824 INFO [make_feeds] NumExpr defaulting to 8 threads.
2020-06-30 11:01:13,960 INFO [make_feeds] ATOM feeds for 2020-06-30 forecast run completed
2020-06-30 11:01:13,963 INFO [manager] checklist updated with [storm surge alert feeds] items from make_feeds worker
2020-06-30 11:01:37,804 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 38689573 = 2020-06-30 02:20:29 UTC estimated time to finish: 0000:12:59:56
2020-06-30 11:01:39,767 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15498300 = 2020-06-30 08:35:00 UTC estimated time to finish: 0000:02:31:53
2020-06-30 11:03:21,867 INFO [watch_NEMO_agrif] 27jun20nowcast-agrif on orcinus-nowcast-agrif: timestep: 4613760 = 2020-06-27 23:59:20 UTC, 100.0% complete
2020-06-30 11:03:39,871 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 7490260 = 2020-06-30 17:06:00 UTC, 71.3% complete
2020-06-30 11:06:37,925 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 38690802 = 2020-06-30 02:28:40 UTC estimated time to finish: 0000:12:56:09
2020-06-30 11:06:39,876 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15500032 = 2020-06-30 09:03:52 UTC estimated time to finish: 0000:02:27:43
2020-06-30 11:08:22,191 INFO [watch_NEMO_agrif] NEMO AGRIF run 10846020.orca2.ibb on orcinus-nowcast-agrif completed
2020-06-30 11:08:22,196 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_agrif worker
2020-06-30 11:08:22,757 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['orcinus-nowcast-agrif', 'nowcast-agrif', '--run-date', '2020-06-27'], host='localhost')
2020-06-30 11:08:24,107 INFO [download_results] running in process 2667
2020-06-30 11:08:24,109 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-06-30 11:08:24,110 INFO [download_results] publishing log messages to tcp://*:5563
2020-06-30 11:08:24,111 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2020-06-30 11:08:24,112 INFO [download_results] downloading results from orcinus-nowcast-agrif:/global/scratch/dlatorne/nowcast-agrif/27jun20 to /results/SalishSea/nowcast-agrif.201702
2020-06-30 11:08:39,973 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 7490419 = 2020-06-30 18:52:00 UTC, 78.6% complete
2020-06-30 11:09:08,350 INFO [download_results] nowcast-agrif 2020-06-27 results files from orcinus-nowcast-agrif downloaded
2020-06-30 11:09:08,363 INFO [manager] checklist updated with [results files] items from download_results worker
2020-06-30 11:09:08,633 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast-agrif', 'research', '--run-date', '2020-06-27'], host='localhost')
2020-06-30 11:09:13,254 INFO [make_plots] running in process 2676
2020-06-30 11:09:13,255 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-06-30 11:09:13,256 INFO [make_plots] publishing log messages to tcp://*:5563
2020-06-30 11:09:13,257 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2020-06-30 11:09:13,698 INFO [make_plots] preparing render list for 2020-06-27 NEMO nowcast-agrif research figures
2020-06-30 11:09:13,699 INFO [make_plots] added baynes_sound_surface figure to 2020-06-27 NEMO nowcast-agrif research render list
2020-06-30 11:09:13,700 INFO [make_plots] starting to render nemo nowcast-agrif research 27jun20 figures
2020-06-30 11:09:20,915 INFO [make_plots] finished rendering nemo nowcast-agrif research 27jun20 figures
2020-06-30 11:09:20,916 INFO [make_plots] nemo research plots for 2020-06-27 nowcast-agrif completed
2020-06-30 11:09:20,922 INFO [manager] checklist updated with [plots] items from make_plots worker
2020-06-30 11:11:38,037 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 38692071 = 2020-06-30 02:37:08 UTC estimated time to finish: 0000:12:51:49
2020-06-30 11:11:39,990 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15501763 = 2020-06-30 09:32:43 UTC estimated time to finish: 0000:02:23:30
2020-06-30 11:13:40,073 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 7490578 = 2020-06-30 20:38:00 UTC, 86.0% complete
2020-06-30 11:16:38,148 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 38693287 = 2020-06-30 02:45:14 UTC estimated time to finish: 0000:12:47:38
2020-06-30 11:16:40,009 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15503494 = 2020-06-30 10:01:34 UTC estimated time to finish: 0000:02:19:09
2020-06-30 11:18:40,174 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 7490736 = 2020-06-30 22:23:20 UTC, 93.3% complete
2020-06-30 11:21:38,265 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 38694626 = 2020-06-30 02:54:10 UTC estimated time to finish: 0000:12:41:18
2020-06-30 11:21:40,125 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15505157 = 2020-06-30 10:29:17 UTC estimated time to finish: 0000:02:14:40
2020-06-30 11:23:40,273 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 7490880 = 2020-06-30 23:59:20 UTC, 100.0% complete
2020-06-30 11:26:38,386 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 38695936 = 2020-06-30 03:02:54 UTC estimated time to finish: 0000:12:32:09
2020-06-30 11:26:40,236 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15506848 = 2020-06-30 10:57:28 UTC estimated time to finish: 0000:02:10:30
2020-06-30 11:28:40,381 INFO [watch_NEMO] nowcast-green NEMO run on arbutus.cloud-nowcast completed
2020-06-30 11:28:40,385 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2020-06-30 11:28:40,910 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_wind_file', args=['arbutus.cloud-nowcast', 'forecast'], host='arbutus.cloud-nowcast')
2020-06-30 11:28:40,918 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_current_file', args=['arbutus.cloud-nowcast', 'forecast'], host='arbutus.cloud-nowcast')
2020-06-30 11:28:40,922 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['salish-nowcast', 'nowcast+', '--shared-storage'], host='localhost')
2020-06-30 11:28:40,927 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2020-06-30'], host='localhost')
2020-06-30 11:28:41,985 INFO [make_forcing_links] running in process 3372
2020-06-30 11:28:41,986 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-06-30 11:28:41,987 INFO [make_forcing_links] publishing log messages to tcp://*:5563
2020-06-30 11:28:41,989 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2020-06-30 11:28:41,990 INFO [download_results] running in process 3373
2020-06-30 11:28:41,991 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-06-30 11:28:41,992 INFO [download_results] publishing log messages to tcp://*:5564
2020-06-30 11:28:41,994 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2020-06-30 11:28:41,995 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/nowcast-green/30jun20 to /results2/SalishSea/nowcast-green.201905
2020-06-30 11:28:42,823 INFO [make_forcing_links] nowcast+ 2020-06-30 forcing file links on salish-nowcast created
2020-06-30 11:28:42,829 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2020-06-30 11:28:43,094 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['salish-nowcast', 'nowcast-dev', '--run-date', '2020-06-30'], host='salish-nowcast')
2020-06-30 11:28:44,673 INFO [run_NEMO] running in process 25300
2020-06-30 11:28:44,674 INFO [run_NEMO] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-06-30 11:28:44,676 INFO [run_NEMO] publishing log messages to tcp://*:5556
2020-06-30 11:28:44,677 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2020-06-30 11:28:46,329 WARNING [run_NEMO] There are uncommitted changes in /SalishSeaCast/SalishSeaNowcast
2020-06-30 11:28:47,721 INFO [run_NEMO] nowcast-dev: launching /SalishSeaCast/runs/30jun20nowcast-dev_2020-06-30T112844.790753-0700/SalishSeaNEMO.sh on salish-nowcast
2020-06-30 11:28:47,828 INFO [run_NEMO] nowcast-dev NEMO run for 2020-06-30 on salish-nowcast started
2020-06-30 11:28:47,837 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2020-06-30 11:28:48,111 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['salish-nowcast', 'nowcast-dev'], host='salish-nowcast')
2020-06-30 11:28:48,932 INFO [make_ww3_wind_file] running in process 25532
2020-06-30 11:28:48,933 INFO [make_ww3_wind_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2020-06-30 11:28:48,934 INFO [make_ww3_wind_file] publishing log messages to tcp://*:5570
2020-06-30 11:28:48,945 INFO [make_ww3_wind_file] connected to skookum.eos.ubc.ca port 5555
2020-06-30 11:28:48,946 INFO [make_ww3_wind_file] Creating wwatch3 wind forcing file for 2020-06-30 forecast run
2020-06-30 11:28:49,152 INFO [make_ww3_current_file] running in process 25531
2020-06-30 11:28:49,153 INFO [make_ww3_current_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2020-06-30 11:28:49,154 INFO [make_ww3_current_file] publishing log messages to tcp://*:5571
2020-06-30 11:28:49,155 INFO [make_ww3_current_file] connected to skookum.eos.ubc.ca port 5555
2020-06-30 11:28:49,156 INFO [make_ww3_current_file] Creating wwatch3 currents forcing file for 2020-06-30 forecast run
2020-06-30 11:28:53,135 INFO [make_ww3_wind_file] wwatch3 wind forcing file created on arbutus.cloud-nowcast for 2020-06-30 forecast run
2020-06-30 11:28:53,139 INFO [manager] checklist updated with [WW3 wind forcing] items from make_ww3_wind_file worker
2020-06-30 11:29:41,218 INFO [make_ww3_current_file] wwatch3 currents forcing file created on arbutus.cloud-nowcast for 2020-06-30 forecast run
2020-06-30 11:29:41,223 INFO [manager] checklist updated with [WW3 currents forcing] items from make_ww3_current_file worker
2020-06-30 11:29:41,494 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2020-06-30'], host='arbutus.cloud-nowcast')
2020-06-30 11:29:43,683 INFO [run_ww3] running in process 25733
2020-06-30 11:29:43,769 INFO [run_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2020-06-30 11:29:43,867 INFO [run_ww3] publishing log messages to tcp://*:5572
2020-06-30 11:29:43,956 INFO [run_ww3] connected to skookum.eos.ubc.ca port 5555
2020-06-30 11:29:44,280 INFO [run_ww3] Created run directory /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/nowcast_2020-06-30T112943.680762-0700
2020-06-30 11:29:44,423 INFO [run_ww3] nowcast: launching /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/nowcast_2020-06-30T112943.680762-0700/SoGWW3.sh on arbutus.cloud-nowcast
2020-06-30 11:29:44,425 INFO [run_ww3] nowcast WaveWatch3 run for 2020-06-30 on arbutus.cloud-nowcast started
2020-06-30 11:29:44,428 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker
2020-06-30 11:29:44,429 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'nowcast'], host='arbutus.cloud-nowcast')
2020-06-30 11:29:46,333 INFO [watch_ww3] running in process 25836
2020-06-30 11:29:46,335 INFO [watch_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2020-06-30 11:29:46,335 INFO [watch_ww3] publishing log messages to tcp://*:5573
2020-06-30 11:29:46,336 INFO [watch_ww3] connected to skookum.eos.ubc.ca port 5555
2020-06-30 11:29:46,376 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2020-06-30 11:31:38,497 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 38697630 = 2020-06-30 03:14:12 UTC estimated time to finish: 0000:12:15:12
2020-06-30 11:31:40,352 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15508498 = 2020-06-30 11:24:58 UTC estimated time to finish: 0000:02:06:29
2020-06-30 11:33:51,534 INFO [download_results] nowcast-green 2020-06-30 results files from arbutus.cloud-nowcast downloaded
2020-06-30 11:33:51,544 INFO [manager] checklist updated with [results files] items from download_results worker
2020-06-30 11:33:51,821 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast-green', 'research', '--run-date', '2020-06-30'], host='localhost')
2020-06-30 11:33:51,826 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nowcast-green'], host='localhost')
2020-06-30 11:33:53,646 INFO [ping_erddap] running in process 3475
2020-06-30 11:33:53,648 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-06-30 11:33:53,649 INFO [ping_erddap] publishing log messages to tcp://*:5563
2020-06-30 11:33:53,649 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2020-06-30 11:33:53,653 INFO [ping_erddap] nowcast-green ERDDAP dataset flag file(s) created
2020-06-30 11:33:53,655 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2020-06-30 11:33:57,059 INFO [make_plots] running in process 3474
2020-06-30 11:33:57,061 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-06-30 11:33:57,062 INFO [make_plots] publishing log messages to tcp://*:5563
2020-06-30 11:33:57,063 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2020-06-30 11:33:57,500 INFO [make_plots] preparing render list for 2020-06-30 NEMO nowcast-green research figures
2020-06-30 11:34:01,318 INFO [make_plots] added temperature_thalweg_and_surface figures to 2020-06-30 NEMO nowcast-green research render list
2020-06-30 11:34:02,607 INFO [make_plots] added nitrate_thalweg_and_surface figures to 2020-06-30 NEMO nowcast-green research render list
2020-06-30 11:34:03,782 INFO [make_plots] added ammonium_thalweg_and_surface figures to 2020-06-30 NEMO nowcast-green research render list
2020-06-30 11:34:04,937 INFO [make_plots] added silicon_thalweg_and_surface figures to 2020-06-30 NEMO nowcast-green research render list
2020-06-30 11:34:06,195 INFO [make_plots] added dissolved_organic_nitrogen_thalweg_and_surface figures to 2020-06-30 NEMO nowcast-green research render list
2020-06-30 11:34:07,487 INFO [make_plots] added particulate_organic_nitrogen_thalweg_and_surface figures to 2020-06-30 NEMO nowcast-green research render list
2020-06-30 11:34:08,732 INFO [make_plots] added biogenic_silicon_thalweg_and_surface figures to 2020-06-30 NEMO nowcast-green research render list
2020-06-30 11:34:10,143 INFO [make_plots] added diatoms_thalweg_and_surface figures to 2020-06-30 NEMO nowcast-green research render list
2020-06-30 11:34:11,487 INFO [make_plots] added ciliates_thalweg_and_surface figures to 2020-06-30 NEMO nowcast-green research render list
2020-06-30 11:34:12,839 INFO [make_plots] added flagellates_thalweg_and_surface figures to 2020-06-30 NEMO nowcast-green research render list
2020-06-30 11:34:14,190 INFO [make_plots] added microzooplankton_thalweg_and_surface figures to 2020-06-30 NEMO nowcast-green research render list
2020-06-30 11:34:15,551 INFO [make_plots] added mesozooplankton_thalweg_and_surface figures to 2020-06-30 NEMO nowcast-green research render list
2020-06-30 11:34:16,839 INFO [make_plots] added Fraser_tracer_thalweg_and_surface figures to 2020-06-30 NEMO nowcast-green research render list
2020-06-30 11:34:16,927 INFO [make_plots] added temperature_salinity_timeseries figure to 2020-06-30 NEMO nowcast-green research render list
2020-06-30 11:34:16,928 INFO [make_plots] added nitrate_diatoms_timeseries figure to 2020-06-30 NEMO nowcast-green research render list
2020-06-30 11:34:16,928 INFO [make_plots] added mesodinium_flagellates_timeseries figure to 2020-06-30 NEMO nowcast-green research render list
2020-06-30 11:34:16,929 INFO [make_plots] added mesozoo_microzoo_timeseries figure to 2020-06-30 NEMO nowcast-green research render list
2020-06-30 11:34:16,930 INFO [make_plots] starting to render nemo nowcast-green research 30jun20 figures
2020-06-30 11:34:46,475 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 99, 22.9% complete
2020-06-30 11:36:38,607 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 38699253 = 2020-06-30 03:25:01 UTC estimated time to finish: 0000:12:01:01
2020-06-30 11:36:40,379 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15510160 = 2020-06-30 11:52:40 UTC estimated time to finish: 0000:02:02:18
2020-06-30 11:39:46,575 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 207, 47.9% complete
2020-06-30 11:41:38,717 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 38700744 = 2020-06-30 03:34:57 UTC estimated time to finish: 0000:11:48:16
2020-06-30 11:41:40,499 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15511688 = 2020-06-30 12:18:08 UTC estimated time to finish: 0000:01:58:18
2020-06-30 11:44:46,676 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 318, 73.6% complete
2020-06-30 11:46:38,828 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 38702409 = 2020-06-30 03:46:03 UTC estimated time to finish: 0000:11:35:05
2020-06-30 11:46:40,620 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15513391 = 2020-06-30 12:46:31 UTC estimated time to finish: 0000:01:53:51
2020-06-30 11:49:46,777 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 426, 98.6% complete
2020-06-30 11:51:38,940 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 38704061 = 2020-06-30 03:57:04 UTC estimated time to finish: 0000:11:22:43
2020-06-30 11:51:40,646 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15515082 = 2020-06-30 13:14:42 UTC estimated time to finish: 0000:01:49:23
2020-06-30 11:54:46,880 INFO [watch_ww3] nowcast WWATCH3 run on arbutus.cloud-nowcast completed
2020-06-30 11:54:46,882 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2020-06-30 11:54:47,442 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2020-06-30'], host='localhost')
2020-06-30 11:54:47,447 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2020-06-30'], host='arbutus.cloud-nowcast')
2020-06-30 11:54:48,587 INFO [download_wwatch3_results] running in process 3970
2020-06-30 11:54:48,589 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-06-30 11:54:48,590 INFO [download_wwatch3_results] publishing log messages to tcp://*:5564
2020-06-30 11:54:48,591 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2020-06-30 11:54:49,674 INFO [run_ww3] running in process 26002
2020-06-30 11:54:49,675 INFO [run_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2020-06-30 11:54:49,675 INFO [run_ww3] publishing log messages to tcp://*:5572
2020-06-30 11:54:49,687 INFO [run_ww3] connected to skookum.eos.ubc.ca port 5555
2020-06-30 11:54:49,689 INFO [run_ww3] Created run directory /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast_2020-06-30T115449.673565-0700
2020-06-30 11:54:49,690 INFO [run_ww3] forecast: launching /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast_2020-06-30T115449.673565-0700/SoGWW3.sh on arbutus.cloud-nowcast
2020-06-30 11:54:49,703 INFO [run_ww3] forecast WaveWatch3 run for 2020-06-30 on arbutus.cloud-nowcast started
2020-06-30 11:54:49,710 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker
2020-06-30 11:54:49,937 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'forecast'], host='arbutus.cloud-nowcast')
2020-06-30 11:54:56,933 INFO [watch_ww3] running in process 26192
2020-06-30 11:54:56,934 INFO [watch_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2020-06-30 11:54:56,935 INFO [watch_ww3] publishing log messages to tcp://*:5573
2020-06-30 11:54:56,936 INFO [watch_ww3] connected to skookum.eos.ubc.ca port 5555
2020-06-30 11:54:56,968 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2020-06-30 11:55:20,641 INFO [download_wwatch3_results] nowcast 2020-06-30 results files from arbutus.cloud-nowcast downloaded
2020-06-30 11:55:20,645 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2020-06-30 11:56:39,051 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 38705727 = 2020-06-30 04:08:10 UTC estimated time to finish: 0000:11:10:39
2020-06-30 11:56:40,762 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15516773 = 2020-06-30 13:42:53 UTC estimated time to finish: 0000:01:44:52
2020-06-30 11:59:16,508 INFO [upload_forcing] running in process 4071
2020-06-30 11:59:16,508 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-06-30 11:59:16,509 INFO [upload_forcing] publishing log messages to tcp://*:5564
2020-06-30 11:59:16,510 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2020-06-30 11:59:22,553 INFO [upload_forcing] nowcast+ 2020-06-28 forcing files upload to orcinus-nowcast-agrif completed
2020-06-30 11:59:22,557 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2020-06-30 11:59:30,078 INFO [upload_forcing] running in process 4078
2020-06-30 11:59:30,079 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-06-30 11:59:30,080 INFO [upload_forcing] publishing log messages to tcp://*:5564
2020-06-30 11:59:30,081 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2020-06-30 11:59:30,681 INFO [upload_forcing] turbidity 2020-06-28 forcing files upload to orcinus-nowcast-agrif completed
2020-06-30 11:59:30,686 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2020-06-30 11:59:31,028 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['orcinus-nowcast-agrif', 'nowcast-agrif', '--run-date', '2020-06-28'], host='localhost')
2020-06-30 11:59:32,255 INFO [make_forcing_links] running in process 4084
2020-06-30 11:59:32,324 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-06-30 11:59:32,325 INFO [make_forcing_links] publishing log messages to tcp://*:5564
2020-06-30 11:59:32,326 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2020-06-30 11:59:32,909 INFO [make_forcing_links] nowcast-agrif 2020-06-28 forcing file links on orcinus-nowcast-agrif created
2020-06-30 11:59:32,911 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2020-06-30 11:59:33,181 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_agrif', args=['orcinus-nowcast-agrif', 'nowcast-agrif', '--run-date', '2020-06-28'], host='localhost')
2020-06-30 11:59:34,615 INFO [run_NEMO_agrif] running in process 4090
2020-06-30 11:59:34,631 INFO [run_NEMO_agrif] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-06-30 11:59:34,632 INFO [run_NEMO_agrif] publishing log messages to tcp://*:5564
2020-06-30 11:59:34,633 INFO [run_NEMO_agrif] connected to skookum.eos.ubc.ca port 5555
2020-06-30 11:59:46,258 INFO [run_NEMO_agrif] job id for 28jun20nowcast-agrif: 10846027.orca2.ibb
2020-06-30 11:59:46,259 INFO [run_NEMO_agrif] NEMO nowcast-agrif run for 2020-06-28 queued on orcinus-nowcast-agrif
2020-06-30 11:59:46,264 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_agrif worker
2020-06-30 11:59:46,496 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_agrif', args=['orcinus-nowcast-agrif', '10846027.orca2.ibb'], host='localhost')
2020-06-30 11:59:47,643 INFO [watch_NEMO_agrif] running in process 4101
2020-06-30 11:59:47,669 INFO [watch_NEMO_agrif] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-06-30 11:59:47,670 INFO [watch_NEMO_agrif] publishing log messages to tcp://*:5564
2020-06-30 11:59:47,670 INFO [watch_NEMO_agrif] connected to skookum.eos.ubc.ca port 5555
2020-06-30 11:59:48,155 INFO [watch_NEMO_agrif] watching 28jun20nowcast-agrif job 10846027 on orcinus-nowcast-agrif
2020-06-30 11:59:48,339 INFO [watch_NEMO_agrif] 28jun20nowcast-agrif job 10846027 is queued on orcinus-nowcast-agrif
2020-06-30 11:59:57,067 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 102, 15.7% complete
2020-06-30 12:00:48,600 INFO [watch_NEMO_agrif] 28jun20nowcast-agrif job 10846027 is queued on orcinus-nowcast-agrif
2020-06-30 12:01:39,165 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 38707380 = 2020-06-30 04:19:12 UTC estimated time to finish: 0000:10:59:30
2020-06-30 12:01:40,879 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15518313 = 2020-06-30 14:08:33 UTC estimated time to finish: 0000:01:40:41
2020-06-30 12:01:48,759 INFO [watch_NEMO_agrif] 28jun20nowcast-agrif job 10846027 is queued on orcinus-nowcast-agrif
2020-06-30 12:02:48,959 INFO [watch_NEMO_agrif] 28jun20nowcast-agrif job 10846027 is queued on orcinus-nowcast-agrif
2020-06-30 12:03:49,177 INFO [watch_NEMO_agrif] 28jun20nowcast-agrif job 10846027 is queued on orcinus-nowcast-agrif
2020-06-30 12:04:49,371 INFO [watch_NEMO_agrif] 28jun20nowcast-agrif job 10846027 is queued on orcinus-nowcast-agrif
2020-06-30 12:04:57,168 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 210, 32.4% complete
2020-06-30 12:05:49,574 INFO [watch_NEMO_agrif] 28jun20nowcast-agrif job 10846027 is queued on orcinus-nowcast-agrif
2020-06-30 12:06:39,278 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 38708840 = 2020-06-30 04:28:56 UTC estimated time to finish: 0000:10:50:14
2020-06-30 12:06:41,001 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15519962 = 2020-06-30 14:36:02 UTC estimated time to finish: 0000:01:36:14
2020-06-30 12:06:49,830 INFO [watch_NEMO_agrif] 28jun20nowcast-agrif job 10846027 is queued on orcinus-nowcast-agrif
2020-06-30 12:07:50,092 INFO [watch_NEMO_agrif] 28jun20nowcast-agrif job 10846027 is queued on orcinus-nowcast-agrif
2020-06-30 12:08:50,303 INFO [watch_NEMO_agrif] 28jun20nowcast-agrif job 10846027 is queued on orcinus-nowcast-agrif
2020-06-30 12:09:50,524 INFO [watch_NEMO_agrif] 28jun20nowcast-agrif job 10846027 is queued on orcinus-nowcast-agrif
2020-06-30 12:09:57,269 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 327, 50.5% complete
2020-06-30 12:10:50,786 INFO [watch_NEMO_agrif] 28jun20nowcast-agrif job 10846027 is queued on orcinus-nowcast-agrif
2020-06-30 12:11:39,393 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 38710533 = 2020-06-30 04:40:13 UTC estimated time to finish: 0000:10:39:34
2020-06-30 12:11:41,097 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15521666 = 2020-06-30 15:04:26 UTC estimated time to finish: 0000:01:31:34
2020-06-30 12:11:51,052 INFO [watch_NEMO_agrif] 28jun20nowcast-agrif job 10846027 is queued on orcinus-nowcast-agrif
2020-06-30 12:12:51,279 INFO [watch_NEMO_agrif] 28jun20nowcast-agrif job 10846027 is queued on orcinus-nowcast-agrif
2020-06-30 12:13:51,509 INFO [watch_NEMO_agrif] 28jun20nowcast-agrif job 10846027 is queued on orcinus-nowcast-agrif
2020-06-30 12:14:51,700 INFO [watch_NEMO_agrif] 28jun20nowcast-agrif job 10846027 is queued on orcinus-nowcast-agrif
2020-06-30 12:14:57,352 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 435, 67.1% complete
2020-06-30 12:15:51,948 INFO [watch_NEMO_agrif] 28jun20nowcast-agrif job 10846027 is queued on orcinus-nowcast-agrif
2020-06-30 12:16:39,462 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 38712199 = 2020-06-30 04:51:19 UTC estimated time to finish: 0000:10:29:22
2020-06-30 12:16:41,177 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15523356 = 2020-06-30 15:32:36 UTC estimated time to finish: 0000:01:26:56
2020-06-30 12:16:52,170 INFO [watch_NEMO_agrif] 28jun20nowcast-agrif job 10846027 is queued on orcinus-nowcast-agrif
2020-06-30 12:17:52,354 INFO [watch_NEMO_agrif] 28jun20nowcast-agrif job 10846027 is queued on orcinus-nowcast-agrif
2020-06-30 12:18:52,616 INFO [watch_NEMO_agrif] 28jun20nowcast-agrif job 10846027 is queued on orcinus-nowcast-agrif
2020-06-30 12:19:52,830 INFO [watch_NEMO_agrif] 28jun20nowcast-agrif job 10846027 is queued on orcinus-nowcast-agrif
2020-06-30 12:19:57,356 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 543, 83.8% complete
2020-06-30 12:20:53,093 INFO [watch_NEMO_agrif] 28jun20nowcast-agrif job 10846027 is queued on orcinus-nowcast-agrif
2020-06-30 12:21:39,576 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 38713851 = 2020-06-30 05:02:20 UTC estimated time to finish: 0000:10:19:55
2020-06-30 12:21:41,260 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15525250 = 2020-06-30 16:04:10 UTC estimated time to finish: 0000:01:21:24
2020-06-30 12:21:54,139 INFO [watch_NEMO_agrif] 28jun20nowcast-agrif on orcinus-nowcast-agrif: time.step not found; continuing to watch...
2020-06-30 12:24:57,364 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2020-06-30 12:26:39,691 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 38715353 = 2020-06-30 05:12:21 UTC estimated time to finish: 0000:10:11:15
2020-06-30 12:26:41,338 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15526981 = 2020-06-30 16:33:01 UTC estimated time to finish: 0000:01:16:19
2020-06-30 12:26:54,664 INFO [watch_NEMO_agrif] 28jun20nowcast-agrif on orcinus-nowcast-agrif: timestep: 4614007 = 2020-06-28 02:44:00 UTC, 11.4% complete
2020-06-30 12:29:57,466 INFO [watch_ww3] forecast WWATCH3 run on arbutus.cloud-nowcast completed
2020-06-30 12:29:57,470 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2020-06-30 12:29:58,020 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2020-06-30'], host='localhost')
2020-06-30 12:29:59,429 INFO [download_wwatch3_results] running in process 4821
2020-06-30 12:29:59,430 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-06-30 12:29:59,431 INFO [download_wwatch3_results] publishing log messages to tcp://*:5565
2020-06-30 12:29:59,432 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2020-06-30 12:30:47,108 INFO [download_wwatch3_results] forecast 2020-06-30 results files from arbutus.cloud-nowcast downloaded
2020-06-30 12:30:47,112 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2020-06-30 12:30:47,435 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['wwatch3', 'forecast', '--run-date', '2020-06-30'], host='localhost')
2020-06-30 12:30:48,650 INFO [update_forecast_datasets] running in process 4864
2020-06-30 12:30:48,651 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-06-30 12:30:48,652 INFO [update_forecast_datasets] publishing log messages to tcp://*:5565
2020-06-30 12:30:48,652 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2020-06-30 12:30:48,653 INFO [update_forecast_datasets] updating wwatch3 most_recent_forecast directory from forecast/30jun20 run
2020-06-30 12:30:48,655 INFO [update_forecast_datasets] updated wwatch3 most_recent_forecast directory from forecast/30jun20 run
2020-06-30 12:30:48,655 INFO [update_forecast_datasets] updating wwatch3 forecast directory for forecast/30jun20 run
2020-06-30 12:30:48,927 INFO [update_forecast_datasets] updated wwatch3 forecast directory for forecast/30jun20 run: /results/SalishSea/rolling-forecasts/wwatch3
2020-06-30 12:30:48,928 INFO [update_forecast_datasets] wwatch3 2020-06-30 forecast rolling forecast datasets updated
2020-06-30 12:30:48,932 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2020-06-30 12:30:49,185 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['wwatch3-forecast'], host='localhost')
2020-06-30 12:30:50,312 INFO [ping_erddap] running in process 4869
2020-06-30 12:30:50,313 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-06-30 12:30:50,314 INFO [ping_erddap] publishing log messages to tcp://*:5565
2020-06-30 12:30:50,315 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2020-06-30 12:30:50,316 INFO [ping_erddap] wwatch3-forecast ERDDAP dataset flag file(s) created
2020-06-30 12:30:50,317 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2020-06-30 12:30:50,583 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['wwatch3', 'forecast', 'publish', '--run-date', '2020-06-30'], host='localhost')
2020-06-30 12:30:57,732 INFO [make_plots] running in process 4873
2020-06-30 12:30:57,733 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-06-30 12:30:57,734 INFO [make_plots] publishing log messages to tcp://*:5565
2020-06-30 12:30:57,735 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2020-06-30 12:30:57,736 INFO [make_plots] preparing render list for 2020-06-30 WaveWatch3 forecast publish figures
2020-06-30 12:30:57,737 INFO [make_plots] added Halibut Bank figure to 2020-06-30 WaveWatch3 forecast publish render list
2020-06-30 12:30:57,738 INFO [make_plots] added Sentry Shoal figure to 2020-06-30 WaveWatch3 forecast publish render list
2020-06-30 12:30:57,739 INFO [make_plots] starting to render wwatch3 forecast publish 30jun20 figures
2020-06-30 12:31:39,806 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 38717006 = 2020-06-30 05:23:22 UTC estimated time to finish: 0000:10:02:13
2020-06-30 12:31:41,418 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15528916 = 2020-06-30 17:05:16 UTC estimated time to finish: 0000:01:10:39
2020-06-30 12:31:55,213 INFO [watch_NEMO_agrif] 28jun20nowcast-agrif on orcinus-nowcast-agrif: timestep: 4614265 = 2020-06-28 05:36:00 UTC, 23.3% complete
2020-06-30 12:32:04,499 INFO [make_plots] Note: NumExpr detected 12 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
2020-06-30 12:32:04,500 INFO [make_plots] NumExpr defaulting to 8 threads.
2020-06-30 12:32:04,512 INFO [make_plots] retrieving last 45 days of real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46146 Halibut Bank
2020-06-30 12:33:13,917 INFO [make_plots] retrieving last 45 days of real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46131 Sentry Shoal
2020-06-30 12:33:15,248 INFO [make_plots] finished rendering wwatch3 forecast publish 30jun20 figures
2020-06-30 12:33:15,249 INFO [make_plots] wwatch3 publish plots for 2020-06-30 forecast completed
2020-06-30 12:33:15,255 INFO [manager] checklist updated with [plots] items from make_plots worker
2020-06-30 12:36:39,922 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 38718630 = 2020-06-30 05:34:12 UTC estimated time to finish: 0000:09:53:56
2020-06-30 12:36:41,500 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15530799 = 2020-06-30 17:36:39 UTC estimated time to finish: 0000:01:05:12
2020-06-30 12:36:55,774 INFO [watch_NEMO_agrif] 28jun20nowcast-agrif on orcinus-nowcast-agrif: timestep: 4614524 = 2020-06-28 08:28:40 UTC, 35.3% complete
2020-06-30 12:41:40,039 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 38720323 = 2020-06-30 05:45:29 UTC estimated time to finish: 0000:09:45:00
2020-06-30 12:41:41,544 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15532775 = 2020-06-30 18:09:35 UTC estimated time to finish: 0000:00:59:28
2020-06-30 12:41:56,291 INFO [watch_NEMO_agrif] 28jun20nowcast-agrif on orcinus-nowcast-agrif: timestep: 4614780 = 2020-06-28 11:19:20 UTC, 47.2% complete
2020-06-30 12:46:40,156 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 38721825 = 2020-06-30 05:55:30 UTC estimated time to finish: 0000:09:37:24
2020-06-30 12:46:41,660 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15534507 = 2020-06-30 18:38:27 UTC estimated time to finish: 0000:00:54:28
2020-06-30 12:46:56,777 INFO [watch_NEMO_agrif] 28jun20nowcast-agrif on orcinus-nowcast-agrif: timestep: 4615035 = 2020-06-28 14:09:20 UTC, 59.0% complete
2020-06-30 12:51:40,274 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 38723519 = 2020-06-30 06:06:47 UTC estimated time to finish: 0000:09:28:56
2020-06-30 12:51:41,740 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15536441 = 2020-06-30 19:10:41 UTC estimated time to finish: 0000:00:48:55
2020-06-30 12:51:57,347 INFO [watch_NEMO_agrif] 28jun20nowcast-agrif on orcinus-nowcast-agrif: timestep: 4615289 = 2020-06-28 16:58:40 UTC, 70.8% complete
2020-06-30 12:56:40,387 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 38725212 = 2020-06-30 06:18:04 UTC estimated time to finish: 0000:09:20:35
2020-06-30 12:56:41,821 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15538406 = 2020-06-30 19:43:26 UTC estimated time to finish: 0000:00:43:17
2020-06-30 12:56:57,890 INFO [watch_NEMO_agrif] 28jun20nowcast-agrif on orcinus-nowcast-agrif: timestep: 4615544 = 2020-06-28 19:48:40 UTC, 82.6% complete
2020-06-30 13:01:40,506 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 38726878 = 2020-06-30 06:29:11 UTC estimated time to finish: 0000:09:12:47
2020-06-30 13:01:41,901 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15540340 = 2020-06-30 20:15:40 UTC estimated time to finish: 0000:00:37:47
2020-06-30 13:01:58,449 INFO [watch_NEMO_agrif] 28jun20nowcast-agrif on orcinus-nowcast-agrif: timestep: 4615800 = 2020-06-28 22:39:20 UTC, 94.4% complete
2020-06-30 13:01:58,482 INFO [make_plots] finished rendering nemo nowcast-green research 30jun20 figures
2020-06-30 13:01:58,653 INFO [make_plots] nemo research plots for 2020-06-30 nowcast-green completed
2020-06-30 13:01:58,796 INFO [manager] checklist updated with [plots] items from make_plots worker
2020-06-30 13:06:40,625 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 38728530 = 2020-06-30 06:40:12 UTC estimated time to finish: 0000:09:05:10
2020-06-30 13:06:41,983 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15542072 = 2020-06-30 20:44:32 UTC estimated time to finish: 0000:00:32:53
2020-06-30 13:06:58,838 INFO [watch_NEMO_agrif] 28jun20nowcast-agrif on orcinus-nowcast-agrif: timestep: 4615920 = 2020-06-28 23:59:20 UTC, 100.0% complete
2020-06-30 13:11:40,746 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 38730031 = 2020-06-30 06:50:12 UTC estimated time to finish: 0000:08:58:19
2020-06-30 13:11:42,063 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15544007 = 2020-06-30 21:16:47 UTC estimated time to finish: 0000:00:27:25
2020-06-30 13:11:59,146 INFO [watch_NEMO_agrif] NEMO AGRIF run 10846027.orca2.ibb on orcinus-nowcast-agrif completed
2020-06-30 13:11:59,150 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_agrif worker
2020-06-30 13:11:59,755 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['orcinus-nowcast-agrif', 'nowcast-agrif', '--run-date', '2020-06-28'], host='localhost')
2020-06-30 13:12:01,807 INFO [download_results] running in process 5733
2020-06-30 13:12:01,807 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-06-30 13:12:01,808 INFO [download_results] publishing log messages to tcp://*:5563
2020-06-30 13:12:01,809 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2020-06-30 13:12:01,809 INFO [download_results] downloading results from orcinus-nowcast-agrif:/global/scratch/dlatorne/nowcast-agrif/28jun20 to /results/SalishSea/nowcast-agrif.201702
2020-06-30 13:12:43,674 INFO [download_results] nowcast-agrif 2020-06-28 results files from orcinus-nowcast-agrif downloaded
2020-06-30 13:12:43,686 INFO [manager] checklist updated with [results files] items from download_results worker
2020-06-30 13:12:43,992 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast-agrif', 'research', '--run-date', '2020-06-28'], host='localhost')
2020-06-30 13:12:50,194 INFO [make_plots] running in process 5748
2020-06-30 13:12:50,195 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-06-30 13:12:50,196 INFO [make_plots] publishing log messages to tcp://*:5563
2020-06-30 13:12:50,196 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2020-06-30 13:12:50,629 INFO [make_plots] preparing render list for 2020-06-28 NEMO nowcast-agrif research figures
2020-06-30 13:12:50,630 INFO [make_plots] added baynes_sound_surface figure to 2020-06-28 NEMO nowcast-agrif research render list
2020-06-30 13:12:50,631 INFO [make_plots] starting to render nemo nowcast-agrif research 28jun20 figures
2020-06-30 13:12:58,158 INFO [make_plots] finished rendering nemo nowcast-agrif research 28jun20 figures
2020-06-30 13:12:58,160 INFO [make_plots] nemo research plots for 2020-06-28 nowcast-agrif completed
2020-06-30 13:12:58,164 INFO [manager] checklist updated with [plots] items from make_plots worker
2020-06-30 13:16:40,865 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 38731656 = 2020-06-30 07:01:02 UTC estimated time to finish: 0000:08:51:16
2020-06-30 13:16:42,143 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15545930 = 2020-06-30 21:48:50 UTC estimated time to finish: 0000:00:22:00
2020-06-30 13:21:40,986 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 38733309 = 2020-06-30 07:12:03 UTC estimated time to finish: 0000:08:44:09
2020-06-30 13:21:42,226 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15547865 = 2020-06-30 22:21:05 UTC estimated time to finish: 0000:00:16:34
2020-06-30 13:26:41,107 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 38734961 = 2020-06-30 07:23:04 UTC estimated time to finish: 0000:08:37:02
2020-06-30 13:26:42,305 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15549801 = 2020-06-30 22:53:21 UTC estimated time to finish: 0000:00:11:09
2020-06-30 13:31:41,229 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 38736464 = 2020-06-30 07:33:05 UTC estimated time to finish: 0000:08:30:34
2020-06-30 13:31:42,386 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15551410 = 2020-06-30 23:20:10 UTC estimated time to finish: 0000:00:06:40
2020-06-30 13:36:41,350 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 38738048 = 2020-06-30 07:43:39 UTC estimated time to finish: 0000:08:24:16
2020-06-30 13:36:42,466 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15553073 = 2020-06-30 23:47:53 UTC estimated time to finish: 0000:00:02:02
2020-06-30 13:37:55,662 INFO [upload_forcing] running in process 6256
2020-06-30 13:37:55,663 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-06-30 13:37:55,663 INFO [upload_forcing] publishing log messages to tcp://*:5563
2020-06-30 13:37:55,665 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2020-06-30 13:38:02,082 INFO [upload_forcing] nowcast+ 2020-06-29 forcing files upload to orcinus-nowcast-agrif completed
2020-06-30 13:38:02,087 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2020-06-30 13:38:09,799 INFO [upload_forcing] running in process 6281
2020-06-30 13:38:09,800 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-06-30 13:38:09,801 INFO [upload_forcing] publishing log messages to tcp://*:5563
2020-06-30 13:38:09,802 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2020-06-30 13:38:10,441 INFO [upload_forcing] turbidity 2020-06-29 forcing files upload to orcinus-nowcast-agrif completed
2020-06-30 13:38:10,446 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2020-06-30 13:38:10,764 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['orcinus-nowcast-agrif', 'nowcast-agrif', '--run-date', '2020-06-29'], host='localhost')
2020-06-30 13:38:11,767 INFO [make_forcing_links] running in process 6287
2020-06-30 13:38:11,768 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-06-30 13:38:11,768 INFO [make_forcing_links] publishing log messages to tcp://*:5563
2020-06-30 13:38:11,769 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2020-06-30 13:38:12,363 INFO [make_forcing_links] nowcast-agrif 2020-06-29 forcing file links on orcinus-nowcast-agrif created
2020-06-30 13:38:12,369 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2020-06-30 13:38:12,700 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_agrif', args=['orcinus-nowcast-agrif', 'nowcast-agrif', '--run-date', '2020-06-29'], host='localhost')
2020-06-30 13:38:13,662 INFO [run_NEMO_agrif] running in process 6307
2020-06-30 13:38:13,663 INFO [run_NEMO_agrif] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-06-30 13:38:13,665 INFO [run_NEMO_agrif] publishing log messages to tcp://*:5563
2020-06-30 13:38:13,666 INFO [run_NEMO_agrif] connected to skookum.eos.ubc.ca port 5555
2020-06-30 13:38:22,782 INFO [run_NEMO_agrif] job id for 29jun20nowcast-agrif: 10846032.orca2.ibb
2020-06-30 13:38:22,783 INFO [run_NEMO_agrif] NEMO nowcast-agrif run for 2020-06-29 queued on orcinus-nowcast-agrif
2020-06-30 13:38:22,789 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_agrif worker
2020-06-30 13:38:23,113 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_agrif', args=['orcinus-nowcast-agrif', '10846032.orca2.ibb'], host='localhost')
2020-06-30 13:38:24,145 INFO [watch_NEMO_agrif] running in process 6318
2020-06-30 13:38:24,146 INFO [watch_NEMO_agrif] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-06-30 13:38:24,147 INFO [watch_NEMO_agrif] publishing log messages to tcp://*:5563
2020-06-30 13:38:24,148 INFO [watch_NEMO_agrif] connected to skookum.eos.ubc.ca port 5555
2020-06-30 13:38:24,807 INFO [watch_NEMO_agrif] watching 29jun20nowcast-agrif job 10846032 on orcinus-nowcast-agrif
2020-06-30 13:38:25,041 INFO [watch_NEMO_agrif] 29jun20nowcast-agrif job 10846032 is queued on orcinus-nowcast-agrif
2020-06-30 13:39:25,283 INFO [watch_NEMO_agrif] 29jun20nowcast-agrif job 10846032 is queued on orcinus-nowcast-agrif
2020-06-30 13:40:25,551 INFO [watch_NEMO_agrif] 29jun20nowcast-agrif job 10846032 is queued on orcinus-nowcast-agrif
2020-06-30 13:41:25,816 INFO [watch_NEMO_agrif] 29jun20nowcast-agrif job 10846032 is queued on orcinus-nowcast-agrif
2020-06-30 13:41:41,474 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 38740096 = 2020-06-30 07:57:18 UTC estimated time to finish: 0000:08:13:25
2020-06-30 13:41:42,525 INFO [watch_fvcom] x2 nowcast FVCOM VH-FR run on arbutus.cloud-nowcast completed
2020-06-30 13:41:42,531 INFO [manager] checklist updated with [FVCOM run] items from watch_fvcom worker
2020-06-30 13:41:43,147 INFO [manager] launching NextWorker(module='nowcast.workers.download_fvcom_results', args=['arbutus.cloud-nowcast', 'x2', 'nowcast', '--run-date', '2020-06-30'], host='localhost')
2020-06-30 13:41:43,152 INFO [manager] launching NextWorker(module='nowcast.workers.make_fvcom_boundary', args=['arbutus.cloud-nowcast', 'x2', 'forecast', '--run-date', '2020-06-30'], host='arbutus.cloud-nowcast')
2020-06-30 13:41:44,156 INFO [download_fvcom_results] running in process 6363
2020-06-30 13:41:44,158 INFO [download_fvcom_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-06-30 13:41:44,159 INFO [download_fvcom_results] publishing log messages to tcp://*:5564
2020-06-30 13:41:44,160 INFO [download_fvcom_results] connected to skookum.eos.ubc.ca port 5555
2020-06-30 13:41:47,974 INFO [make_fvcom_boundary] running in process 27114
2020-06-30 13:41:47,975 INFO [make_fvcom_boundary] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2020-06-30 13:41:47,977 INFO [make_fvcom_boundary] publishing log messages to tcp://*:5580
2020-06-30 13:41:47,982 INFO [make_fvcom_boundary] connected to skookum.eos.ubc.ca port 5555
2020-06-30 13:41:47,983 INFO [make_fvcom_boundary] Creating VHFR FVCOM open boundary file for x2 forecast run from 2020-06-30 NEMO run
2020-06-30 13:42:26,081 INFO [watch_NEMO_agrif] 29jun20nowcast-agrif job 10846032 is queued on orcinus-nowcast-agrif
2020-06-30 13:42:28,574 INFO [download_fvcom_results] VHFR FVCOM x2 nowcast 2020-06-30 results files from arbutus.cloud-nowcast downloaded
2020-06-30 13:42:28,582 INFO [manager] checklist updated with [VHFR FVCOM results files] items from download_fvcom_results worker
2020-06-30 13:42:28,900 INFO [manager] launching NextWorker(module='nowcast.workers.get_vfpa_hadcp', args=['--data-date', '2020-06-30'], host='localhost')
2020-06-30 13:42:28,903 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['fvcom', 'nowcast-x2', 'research', '--run-date', '2020-06-30'], host='localhost')
2020-06-30 13:42:32,554 INFO [get_vfpa_hadcp] running in process 6382
2020-06-30 13:42:32,560 INFO [get_vfpa_hadcp] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-06-30 13:42:32,560 INFO [get_vfpa_hadcp] publishing log messages to tcp://*:5564
2020-06-30 13:42:32,561 INFO [get_vfpa_hadcp] connected to skookum.eos.ubc.ca port 5555
2020-06-30 13:42:32,562 INFO [get_vfpa_hadcp] processing VFPA HADCP data from 2nd Narrows Rail Bridge for 2020-06-30
2020-06-30 13:42:33,876 INFO [get_vfpa_hadcp] added VFPA HADCP data from 2nd Narrows Rail Bridge for 2020-06-30 to /opp/observations/AISDATA/netcdf/VFPA_2ND_NARROWS_HADCP_2s_202006.nc
2020-06-30 13:42:33,877 INFO [get_vfpa_hadcp] VFPA HADCP observations added to 2020-06 netcdf file
2020-06-30 13:42:33,880 INFO [manager] checklist updated with [VFPA HADCP data] items from get_vfpa_hadcp worker
2020-06-30 13:42:34,192 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['VFPA-HADCP'], host='localhost')
2020-06-30 13:42:34,541 INFO [make_plots] running in process 6383
2020-06-30 13:42:34,542 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-06-30 13:42:34,543 INFO [make_plots] publishing log messages to tcp://*:5564
2020-06-30 13:42:34,544 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2020-06-30 13:42:34,616 INFO [make_plots] preparing render list for 2020-06-30 VHFR FVCOM nowcast-x2 research figures
2020-06-30 13:42:34,620 INFO [make_plots] added EnglishBay_surface_currents figures to 2020-06-30 VHFR FVCOM nowcast-x2 research render list
2020-06-30 13:42:34,623 INFO [make_plots] added VancouverHarbour_surface_currents figures to 2020-06-30 VHFR FVCOM nowcast-x2 research render list
2020-06-30 13:42:34,626 INFO [make_plots] added IndianArm_surface_currents figures to 2020-06-30 VHFR FVCOM nowcast-x2 research render list
2020-06-30 13:42:34,629 INFO [make_plots] added VancouverHarbour_thalweg figures to 2020-06-30 VHFR FVCOM nowcast-x2 research render list
2020-06-30 13:42:34,633 INFO [make_plots] added PortMoody_thalweg figures to 2020-06-30 VHFR FVCOM nowcast-x2 research render list
2020-06-30 13:42:34,636 INFO [make_plots] added IndianArm_thalweg figures to 2020-06-30 VHFR FVCOM nowcast-x2 research render list
2020-06-30 13:42:34,636 INFO [make_plots] starting to render fvcom nowcast-x2 research 30jun20 figures
2020-06-30 13:42:35,354 INFO [ping_erddap] running in process 6390
2020-06-30 13:42:35,355 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-06-30 13:42:35,356 INFO [ping_erddap] publishing log messages to tcp://*:5565
2020-06-30 13:42:35,356 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2020-06-30 13:42:35,358 INFO [ping_erddap] VFPA-HADCP ERDDAP dataset flag file(s) created
2020-06-30 13:42:35,359 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2020-06-30 13:42:35,592 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['fvcom', 'nowcast-x2', 'publish', '--run-date', '2020-06-30'], host='localhost')
2020-06-30 13:42:38,238 INFO [make_plots] running in process 6396
2020-06-30 13:42:38,239 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-06-30 13:42:38,240 INFO [make_plots] publishing log messages to tcp://*:5565
2020-06-30 13:42:38,240 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2020-06-30 13:42:39,548 INFO [make_plots] preparing render list for 2020-06-30 VHFR FVCOM nowcast-x2 publish figures
2020-06-30 13:42:39,549 INFO [make_plots] added Calamity Point figure to 2020-06-30 VHFR FVCOM nowcast-x2 publish render list
2020-06-30 13:42:39,550 INFO [make_plots] added Indian Arm Head figure to 2020-06-30 VHFR FVCOM nowcast-x2 publish render list
2020-06-30 13:42:39,551 INFO [make_plots] added New Westminster figure to 2020-06-30 VHFR FVCOM nowcast-x2 publish render list
2020-06-30 13:42:39,551 INFO [make_plots] added Port Moody figure to 2020-06-30 VHFR FVCOM nowcast-x2 publish render list
2020-06-30 13:42:39,552 INFO [make_plots] added Sand Heads figure to 2020-06-30 VHFR FVCOM nowcast-x2 publish render list
2020-06-30 13:42:39,553 INFO [make_plots] added Sandy Cove figure to 2020-06-30 VHFR FVCOM nowcast-x2 publish render list
2020-06-30 13:42:39,553 INFO [make_plots] added Vancouver Harbour figure to 2020-06-30 VHFR FVCOM nowcast-x2 publish render list
2020-06-30 13:42:39,554 INFO [make_plots] added Woodwards Landing figure to 2020-06-30 VHFR FVCOM nowcast-x2 publish render list
2020-06-30 13:42:39,555 INFO [make_plots] added 2ndNarrows_current figure to 2020-06-30 VHFR FVCOM nowcast-x2 publish render list
2020-06-30 13:42:39,556 INFO [make_plots] starting to render fvcom nowcast-x2 publish 30jun20 figures
2020-06-30 13:42:39,579 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07724 Calamity Point from 2020-06-29 23:55:00 to 2020-07-01 00:00:00
2020-06-30 13:42:59,926 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07724 Calamity Point from 2020-06-30 00:00:00 to 2020-07-01 00:00:00
2020-06-30 13:43:01,403 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07774 Indian Arm Head from 2020-06-29 23:55:00 to 2020-07-01 00:00:00
2020-06-30 13:43:02,201 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07774 Indian Arm Head from 2020-06-30 00:00:00 to 2020-07-01 00:00:00
2020-06-30 13:43:04,235 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07654 New Westminster from 2020-06-29 23:55:00 to 2020-07-01 00:00:00
2020-06-30 13:43:23,557 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07654 New Westminster from 2020-06-30 00:00:00 to 2020-07-01 00:00:00
2020-06-30 13:43:26,350 INFO [watch_NEMO_agrif] 29jun20nowcast-agrif job 10846032 is queued on orcinus-nowcast-agrif
2020-06-30 13:43:26,458 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07755 Port Moody from 2020-06-29 23:55:00 to 2020-07-01 00:00:00
2020-06-30 13:43:44,922 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07755 Port Moody from 2020-06-30 00:00:00 to 2020-07-01 00:00:00
2020-06-30 13:43:48,000 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07594 Sand Heads from 2020-06-29 23:55:00 to 2020-07-01 00:00:00
2020-06-30 13:44:07,694 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07594 Sand Heads from 2020-06-30 00:00:00 to 2020-07-01 00:00:00
2020-06-30 13:44:10,808 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07786 Sandy Cove from 2020-06-29 23:55:00 to 2020-07-01 00:00:00
2020-06-30 13:44:26,581 INFO [watch_NEMO_agrif] 29jun20nowcast-agrif job 10846032 is queued on orcinus-nowcast-agrif
2020-06-30 13:44:29,662 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07795 Point Atkinson from 2020-06-30 00:00:00 to 2020-07-01 00:00:00
2020-06-30 13:44:32,569 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07735 Vancouver Harbour from 2020-06-29 23:55:00 to 2020-07-01 00:00:00
2020-06-30 13:44:51,350 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07735 Vancouver Harbour from 2020-06-30 00:00:00 to 2020-07-01 00:00:00
2020-06-30 13:44:54,633 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07610 Woodwards Landing from 2020-06-29 23:55:00 to 2020-07-01 00:00:00
2020-06-30 13:45:04,532 INFO [make_fvcom_boundary] Stored VHFR FVCOM open boundary file: /nemoShare/MEOPAR/nowcast-sys/fvcom-runs/input.x2/bdy_x2_forecast_brcl_20200701.nc
2020-06-30 13:45:04,536 INFO [make_fvcom_boundary] FVCOM x2 forecast run boundary condition file for 2020-06-30 created on arbutus.cloud-nowcast
2020-06-30 13:45:04,540 INFO [manager] checklist updated with [FVCOM boundary conditions] items from make_fvcom_boundary worker
2020-06-30 13:45:04,948 INFO [manager] launching NextWorker(module='nowcast.workers.make_fvcom_atmos_forcing', args=['x2', 'forecast', '--run-date', '2020-06-30'], host='localhost')
2020-06-30 13:45:04,950 INFO [manager] launching NextWorker(module='nowcast.workers.make_fvcom_rivers_forcing', args=['arbutus.cloud-nowcast', 'x2', 'forecast', '--run-date', '2020-06-30'], host='arbutus.cloud-nowcast')
2020-06-30 13:45:05,973 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07610 Woodwards Landing from 2020-06-30 00:00:00 to 2020-07-01 00:00:00
2020-06-30 13:45:08,168 INFO [make_fvcom_rivers_forcing] running in process 27274
2020-06-30 13:45:08,248 INFO [make_fvcom_rivers_forcing] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2020-06-30 13:45:08,248 INFO [make_fvcom_rivers_forcing] publishing log messages to tcp://*:5582
2020-06-30 13:45:08,436 INFO [make_fvcom_rivers_forcing] connected to skookum.eos.ubc.ca port 5555
2020-06-30 13:45:08,615 INFO [make_fvcom_rivers_forcing] Creating VHFR FVCOM rivers forcing file for x2 forecast run from 2020-06-30 NEMO runoff forcing files
2020-06-30 13:45:08,727 INFO [make_fvcom_rivers_forcing] Stored VHFR FVCOM rivers forcing file: /nemoShare/MEOPAR/nowcast-sys/fvcom-runs/input.x2/rivers_x2_forecast_20200701.nc
2020-06-30 13:45:08,727 INFO [make_fvcom_rivers_forcing] FVCOM x2 forecast run rivers forcing file for 2020-06-30 created on arbutus.cloud-nowcast
2020-06-30 13:45:08,986 INFO [manager] checklist updated with [FVCOM rivers forcing] items from make_fvcom_rivers_forcing worker
2020-06-30 13:45:10,587 INFO [make_fvcom_atmos_forcing] running in process 6483
2020-06-30 13:45:10,643 INFO [make_fvcom_atmos_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-06-30 13:45:10,644 INFO [make_fvcom_atmos_forcing] publishing log messages to tcp://*:5566
2020-06-30 13:45:10,645 INFO [make_fvcom_atmos_forcing] connected to skookum.eos.ubc.ca port 5555
2020-06-30 13:45:10,646 INFO [make_fvcom_atmos_forcing] Creating VHFR FVCOM atmospheric forcing file for x2 forecast run from 2020-06-30 HRDPS GRIB files
2020-06-30 13:45:12,931 INFO [make_plots] finished rendering fvcom nowcast-x2 publish 30jun20 figures
2020-06-30 13:45:13,379 INFO [make_plots] fvcom publish plots for 2020-06-30 nowcast-x2 completed
2020-06-30 13:45:13,384 INFO [manager] checklist updated with [plots] items from make_plots worker
2020-06-30 13:45:26,851 INFO [watch_NEMO_agrif] 29jun20nowcast-agrif job 10846032 is queued on orcinus-nowcast-agrif
2020-06-30 13:45:45,362 INFO [make_fvcom_atmos_forcing] Stored VHFR FVCOM atmospheric forcing hfx file: /results/forcing/atmospheric/GEM2.5/vhfr-fvcom/atmos_x2_forecast_hfx_20200701.nc
2020-06-30 13:45:54,014 INFO [make_fvcom_atmos_forcing] Stored VHFR FVCOM atmospheric forcing precip file: /results/forcing/atmospheric/GEM2.5/vhfr-fvcom/atmos_x2_forecast_precip_20200701.nc
2020-06-30 13:46:07,636 INFO [make_fvcom_atmos_forcing] Stored VHFR FVCOM atmospheric forcing wnd file: /results/forcing/atmospheric/GEM2.5/vhfr-fvcom/atmos_x2_forecast_wnd_20200701.nc
2020-06-30 13:46:07,637 INFO [make_fvcom_atmos_forcing] FVCOM x2 forecast run atmospheric forcing file for 2020-06-30 created
2020-06-30 13:46:07,640 INFO [manager] checklist updated with [FVCOM atmospheric forcing] items from make_fvcom_atmos_forcing worker
2020-06-30 13:46:07,928 INFO [manager] launching NextWorker(module='nowcast.workers.upload_fvcom_atmos_forcing', args=['arbutus.cloud-nowcast', 'x2', 'forecast', '--run-date', '2020-06-30'], host='localhost')
2020-06-30 13:46:08,897 INFO [upload_fvcom_atmos_forcing] running in process 6495
2020-06-30 13:46:08,898 INFO [upload_fvcom_atmos_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-06-30 13:46:08,899 INFO [upload_fvcom_atmos_forcing] publishing log messages to tcp://*:5565
2020-06-30 13:46:08,900 INFO [upload_fvcom_atmos_forcing] connected to skookum.eos.ubc.ca port 5555
2020-06-30 13:46:08,900 INFO [upload_fvcom_atmos_forcing] Uploading VHFR FVCOM atmospheric forcing files for 2020-06-30 x2 forecast run to arbutus.cloud-nowcast
2020-06-30 13:46:13,578 INFO [upload_fvcom_atmos_forcing] FVCOM x2 forecast run atmospheric forcing files for 2020-06-30 uploaded to arbutus.cloud-nowcast
2020-06-30 13:46:13,582 INFO [manager] checklist updated with [FVCOM atmospheric forcing upload] items from upload_fvcom_atmos_forcing worker
2020-06-30 13:46:13,878 INFO [manager] launching NextWorker(module='nowcast.workers.run_fvcom', args=['arbutus.cloud-nowcast', 'x2', 'forecast', '--run-date', '2020-06-30'], host='arbutus.cloud-nowcast')
2020-06-30 13:46:15,947 INFO [run_fvcom] running in process 27625
2020-06-30 13:46:15,948 INFO [run_fvcom] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2020-06-30 13:46:15,949 INFO [run_fvcom] publishing log messages to tcp://*:5584
2020-06-30 13:46:15,950 INFO [run_fvcom] connected to skookum.eos.ubc.ca port 5555
2020-06-30 13:46:16,019 INFO [run_fvcom] forecast: launching /nemoShare/MEOPAR/nowcast-sys/fvcom-runs/20200630-20h46m15.985721s/VHFR_FVCOM.sh on arbutus.cloud-nowcast
2020-06-30 13:46:16,037 INFO [run_fvcom] x2 forecast FVCOM VH-FR run for 2020-06-30 on arbutus.cloud-nowcast started
2020-06-30 13:46:16,042 INFO [manager] checklist updated with [FVCOM run] items from run_fvcom worker
2020-06-30 13:46:16,279 INFO [manager] launching NextWorker(module='nowcast.workers.watch_fvcom', args=['arbutus.cloud-nowcast', 'x2', 'forecast'], host='arbutus.cloud-nowcast')
2020-06-30 13:46:18,173 INFO [watch_fvcom] running in process 27727
2020-06-30 13:46:18,174 INFO [watch_fvcom] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2020-06-30 13:46:18,174 INFO [watch_fvcom] publishing log messages to tcp://*:5587
2020-06-30 13:46:18,175 INFO [watch_fvcom] connected to skookum.eos.ubc.ca port 5555
2020-06-30 13:46:18,213 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: no run progress found in fvcom.log ; continuing to watch...
2020-06-30 13:46:27,074 INFO [watch_NEMO_agrif] 29jun20nowcast-agrif job 10846032 is queued on orcinus-nowcast-agrif
2020-06-30 13:46:41,597 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 38742430 = 2020-06-30 08:12:52 UTC estimated time to finish: 0000:07:59:53
2020-06-30 13:47:27,331 INFO [watch_NEMO_agrif] 29jun20nowcast-agrif job 10846032 is queued on orcinus-nowcast-agrif
2020-06-30 13:48:28,446 INFO [watch_NEMO_agrif] 29jun20nowcast-agrif on orcinus-nowcast-agrif: time.step not found; continuing to watch...
2020-06-30 13:48:35,777 CRITICAL [make_plots] unhandled exception:
Traceback (most recent call last):
  File "/SalishSeaCast/NEMO_Nowcast/nemo_nowcast/worker.py", line 390, in _do_work
    checklist = self.worker_func(
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 325, in make_plots
    checklist = _render_figures(
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1038, in _render_figures
    fig = _calc_figure(fig_func, args, kwargs)
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure
    fig = fig_func(*args, **kwargs)
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure
    plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname)
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 80, in _prep_plot_data
    date_bytes = fvcom_results_dataset.variables["Times"][time_index, :]
  File "netCDF4/_netCDF4.pyx", line 4384, in netCDF4._netCDF4.Variable.__getitem__
  File "netCDF4/_netCDF4.pyx", line 4072, in netCDF4._netCDF4.Variable.shape.__get__
  File "netCDF4/_netCDF4.pyx", line 4020, in netCDF4._netCDF4.Variable._getdims
  File "netCDF4/_netCDF4.pyx", line 1887, in netCDF4._netCDF4._ensure_nc_success
RuntimeError: NetCDF: Not a valid ID
2020-06-30 13:51:18,315 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15555543 = 2020-07-01 00:29:03 UTC estimated time to finish: 0000:05:28:20
2020-06-30 13:51:41,655 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 38744001 = 2020-06-30 08:23:20 UTC estimated time to finish: 0000:07:53:14
2020-06-30 13:53:29,005 INFO [watch_NEMO_agrif] 29jun20nowcast-agrif on orcinus-nowcast-agrif: timestep: 4616122 = 2020-06-29 02:14:00 UTC, 9.3% complete
2020-06-30 13:56:18,417 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15557507 = 2020-07-01 01:01:47 UTC estimated time to finish: 0000:05:25:50
2020-06-30 13:56:41,779 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 38745738 = 2020-06-30 08:34:55 UTC estimated time to finish: 0000:07:46:02
2020-06-30 13:58:29,586 INFO [watch_NEMO_agrif] 29jun20nowcast-agrif on orcinus-nowcast-agrif: timestep: 4616376 = 2020-06-29 05:03:20 UTC, 21.1% complete
2020-06-30 14:01:18,521 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15559523 = 2020-07-01 01:35:23 UTC estimated time to finish: 0000:05:19:52
2020-06-30 14:01:41,905 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 38747484 = 2020-06-30 08:46:33 UTC estimated time to finish: 0000:07:38:54
2020-06-30 14:03:30,071 INFO [watch_NEMO_agrif] 29jun20nowcast-agrif on orcinus-nowcast-agrif: timestep: 4616635 = 2020-06-29 07:56:00 UTC, 33.1% complete
2020-06-30 14:06:18,625 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15561487 = 2020-07-01 02:08:07 UTC estimated time to finish: 0000:05:14:54
2020-06-30 14:06:42,029 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 38749260 = 2020-06-30 08:58:24 UTC estimated time to finish: 0000:07:31:41
2020-06-30 14:08:30,523 INFO [watch_NEMO_agrif] 29jun20nowcast-agrif on orcinus-nowcast-agrif: timestep: 4616888 = 2020-06-29 10:44:40 UTC, 44.8% complete
2020-06-30 14:11:18,729 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15563260 = 2020-07-01 02:37:40 UTC estimated time to finish: 0000:05:10:20
2020-06-30 14:11:42,156 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 38750994 = 2020-06-30 09:09:57 UTC estimated time to finish: 0000:07:24:41
2020-06-30 14:13:31,082 INFO [watch_NEMO_agrif] 29jun20nowcast-agrif on orcinus-nowcast-agrif: timestep: 4617139 = 2020-06-29 13:32:00 UTC, 56.4% complete
2020-06-30 14:16:18,834 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15565276 = 2020-07-01 03:11:16 UTC estimated time to finish: 0000:05:04:49
2020-06-30 14:16:42,284 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 38752564 = 2020-06-30 09:20:25 UTC estimated time to finish: 0000:07:18:32
2020-06-30 14:18:31,601 INFO [watch_NEMO_agrif] 29jun20nowcast-agrif on orcinus-nowcast-agrif: timestep: 4617392 = 2020-06-29 16:20:40 UTC, 68.1% complete
2020-06-30 14:21:18,940 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15567240 = 2020-07-01 03:44:00 UTC estimated time to finish: 0000:04:59:38
2020-06-30 14:21:42,411 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 38754272 = 2020-06-30 09:31:48 UTC estimated time to finish: 0000:07:11:57
2020-06-30 14:23:32,105 INFO [watch_NEMO_agrif] 29jun20nowcast-agrif on orcinus-nowcast-agrif: timestep: 4617644 = 2020-06-29 19:08:40 UTC, 79.8% complete
2020-06-30 14:26:19,046 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15569215 = 2020-07-01 04:16:55 UTC estimated time to finish: 0000:04:55:13
2020-06-30 14:26:42,538 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 38756048 = 2020-06-30 09:43:39 UTC estimated time to finish: 0000:07:05:00
2020-06-30 14:28:32,617 INFO [watch_NEMO_agrif] 29jun20nowcast-agrif on orcinus-nowcast-agrif: timestep: 4617899 = 2020-06-29 21:58:40 UTC, 91.6% complete
2020-06-30 14:31:19,152 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15570987 = 2020-07-01 04:46:27 UTC estimated time to finish: 0000:04:50:48
2020-06-30 14:31:42,666 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 38757781 = 2020-06-30 09:55:12 UTC estimated time to finish: 0000:06:58:20
2020-06-30 14:33:33,090 INFO [watch_NEMO_agrif] 29jun20nowcast-agrif on orcinus-nowcast-agrif: timestep: 4618080 = 2020-06-29 23:59:20 UTC, 100.0% complete
2020-06-30 14:36:19,248 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15572910 = 2020-07-01 05:18:30 UTC estimated time to finish: 0000:04:46:12
2020-06-30 14:36:42,794 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 38759365 = 2020-06-30 10:05:46 UTC estimated time to finish: 0000:06:52:15
2020-06-30 14:37:48,118 INFO [upload_forcing] running in process 7824
2020-06-30 14:37:48,119 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-06-30 14:37:48,121 INFO [upload_forcing] publishing log messages to tcp://*:5564
2020-06-30 14:37:48,122 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2020-06-30 14:37:52,795 INFO [upload_forcing] nowcast+ 2020-06-30 forcing files upload to orcinus-nowcast-agrif completed
2020-06-30 14:37:52,800 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2020-06-30 14:37:57,690 INFO [upload_forcing] running in process 7830
2020-06-30 14:37:57,691 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-06-30 14:37:57,692 INFO [upload_forcing] publishing log messages to tcp://*:5564
2020-06-30 14:37:57,693 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2020-06-30 14:37:58,270 INFO [upload_forcing] turbidity 2020-06-30 forcing files upload to orcinus-nowcast-agrif completed
2020-06-30 14:37:58,276 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2020-06-30 14:37:58,603 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['orcinus-nowcast-agrif', 'nowcast-agrif', '--run-date', '2020-06-30'], host='localhost')
2020-06-30 14:37:59,529 INFO [make_forcing_links] running in process 7836
2020-06-30 14:37:59,531 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-06-30 14:37:59,532 INFO [make_forcing_links] publishing log messages to tcp://*:5564
2020-06-30 14:37:59,532 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2020-06-30 14:38:00,173 INFO [make_forcing_links] nowcast-agrif 2020-06-30 forcing file links on orcinus-nowcast-agrif created
2020-06-30 14:38:00,178 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2020-06-30 14:38:00,499 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_agrif', args=['orcinus-nowcast-agrif', 'nowcast-agrif', '--run-date', '2020-06-30'], host='localhost')
2020-06-30 14:38:01,474 INFO [run_NEMO_agrif] running in process 7842
2020-06-30 14:38:01,475 INFO [run_NEMO_agrif] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-06-30 14:38:01,476 INFO [run_NEMO_agrif] publishing log messages to tcp://*:5564
2020-06-30 14:38:01,476 INFO [run_NEMO_agrif] connected to skookum.eos.ubc.ca port 5555
2020-06-30 14:38:08,125 INFO [run_NEMO_agrif] job id for 30jun20nowcast-agrif: 10846033.orca2.ibb
2020-06-30 14:38:08,126 INFO [run_NEMO_agrif] NEMO nowcast-agrif run for 2020-06-30 queued on orcinus-nowcast-agrif
2020-06-30 14:38:08,133 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_agrif worker
2020-06-30 14:38:08,459 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_agrif', args=['orcinus-nowcast-agrif', '10846033.orca2.ibb'], host='localhost')
2020-06-30 14:38:09,434 INFO [watch_NEMO_agrif] running in process 7871
2020-06-30 14:38:09,436 INFO [watch_NEMO_agrif] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-06-30 14:38:09,436 INFO [watch_NEMO_agrif] publishing log messages to tcp://*:5564
2020-06-30 14:38:09,437 INFO [watch_NEMO_agrif] connected to skookum.eos.ubc.ca port 5555
2020-06-30 14:38:10,087 INFO [watch_NEMO_agrif] watching 30jun20nowcast-agrif job 10846033 on orcinus-nowcast-agrif
2020-06-30 14:38:10,319 INFO [watch_NEMO_agrif] 30jun20nowcast-agrif job 10846033 is queued on orcinus-nowcast-agrif
2020-06-30 14:38:33,373 INFO [watch_NEMO_agrif] NEMO AGRIF run 10846032.orca2.ibb on orcinus-nowcast-agrif completed
2020-06-30 14:38:33,378 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_agrif worker
2020-06-30 14:38:33,952 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['orcinus-nowcast-agrif', 'nowcast-agrif', '--run-date', '2020-06-29'], host='localhost')
2020-06-30 14:38:34,897 INFO [download_results] running in process 7891
2020-06-30 14:38:34,899 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-06-30 14:38:34,900 INFO [download_results] publishing log messages to tcp://*:5563
2020-06-30 14:38:34,902 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2020-06-30 14:38:34,903 INFO [download_results] downloading results from orcinus-nowcast-agrif:/global/scratch/dlatorne/nowcast-agrif/29jun20 to /results/SalishSea/nowcast-agrif.201702
2020-06-30 14:39:11,330 INFO [watch_NEMO_agrif] 30jun20nowcast-agrif on orcinus-nowcast-agrif: time.step not found; continuing to watch...
2020-06-30 14:39:16,447 INFO [download_results] nowcast-agrif 2020-06-29 results files from orcinus-nowcast-agrif downloaded
2020-06-30 14:39:16,459 INFO [manager] checklist updated with [results files] items from download_results worker
2020-06-30 14:39:16,762 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast-agrif', 'research', '--run-date', '2020-06-29'], host='localhost')
2020-06-30 14:39:20,100 INFO [make_plots] running in process 7934
2020-06-30 14:39:20,101 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-06-30 14:39:20,102 INFO [make_plots] publishing log messages to tcp://*:5563
2020-06-30 14:39:20,102 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2020-06-30 14:39:20,755 INFO [make_plots] preparing render list for 2020-06-29 NEMO nowcast-agrif research figures
2020-06-30 14:39:20,756 INFO [make_plots] added baynes_sound_surface figure to 2020-06-29 NEMO nowcast-agrif research render list
2020-06-30 14:39:20,757 INFO [make_plots] starting to render nemo nowcast-agrif research 29jun20 figures
2020-06-30 14:39:27,258 INFO [make_plots] finished rendering nemo nowcast-agrif research 29jun20 figures
2020-06-30 14:39:27,260 INFO [make_plots] nemo research plots for 2020-06-29 nowcast-agrif completed
2020-06-30 14:39:27,265 INFO [manager] checklist updated with [plots] items from make_plots worker
2020-06-30 14:41:19,356 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15574926 = 2020-07-01 05:52:06 UTC estimated time to finish: 0000:04:40:47
2020-06-30 14:41:42,923 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 38761140 = 2020-06-30 10:17:36 UTC estimated time to finish: 0000:06:45:27
2020-06-30 14:44:11,901 INFO [watch_NEMO_agrif] 30jun20nowcast-agrif on orcinus-nowcast-agrif: timestep: 4618307 = 2020-06-30 02:30:40 UTC, 10.5% complete
2020-06-30 14:46:19,466 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15576930 = 2020-07-01 06:25:30 UTC estimated time to finish: 0000:04:35:26
2020-06-30 14:46:43,054 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 38762917 = 2020-06-30 10:29:26 UTC estimated time to finish: 0000:06:38:43
2020-06-30 14:49:12,362 INFO [watch_NEMO_agrif] 30jun20nowcast-agrif on orcinus-nowcast-agrif: timestep: 4618563 = 2020-06-30 05:21:20 UTC, 22.3% complete
2020-06-30 14:51:19,573 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15578866 = 2020-07-01 06:57:46 UTC estimated time to finish: 0000:04:30:48
2020-06-30 14:51:43,184 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 38764651 = 2020-06-30 10:41:00 UTC estimated time to finish: 0000:06:32:20
2020-06-30 14:54:12,969 INFO [watch_NEMO_agrif] 30jun20nowcast-agrif on orcinus-nowcast-agrif: timestep: 4618822 = 2020-06-30 08:14:00 UTC, 34.3% complete
2020-06-30 14:56:19,684 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15580638 = 2020-07-01 07:27:18 UTC estimated time to finish: 0000:04:26:09
2020-06-30 14:56:43,313 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 38766235 = 2020-06-30 10:51:34 UTC estimated time to finish: 0000:06:26:26
2020-06-30 14:59:13,430 INFO [watch_NEMO_agrif] 30jun20nowcast-agrif on orcinus-nowcast-agrif: timestep: 4619076 = 2020-06-30 11:03:20 UTC, 46.1% complete
2020-06-30 15:01:19,742 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15582601 = 2020-07-01 08:00:01 UTC estimated time to finish: 0000:04:21:21
2020-06-30 15:01:43,444 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 38768010 = 2020-06-30 11:03:24 UTC estimated time to finish: 0000:06:19:51
2020-06-30 15:04:13,897 INFO [watch_NEMO_agrif] 30jun20nowcast-agrif on orcinus-nowcast-agrif: timestep: 4619334 = 2020-06-30 13:55:20 UTC, 58.0% complete
2020-06-30 15:06:19,854 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15584579 = 2020-07-01 08:32:59 UTC estimated time to finish: 0000:04:16:10
2020-06-30 15:06:43,480 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 38769745 = 2020-06-30 11:14:58 UTC estimated time to finish: 0000:06:13:32
2020-06-30 15:09:14,396 INFO [watch_NEMO_agrif] 30jun20nowcast-agrif on orcinus-nowcast-agrif: timestep: 4619589 = 2020-06-30 16:45:20 UTC, 69.8% complete
2020-06-30 15:11:19,964 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15586582 = 2020-07-01 09:06:22 UTC estimated time to finish: 0000:04:10:48
2020-06-30 15:11:43,612 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 38771520 = 2020-06-30 11:26:48 UTC estimated time to finish: 0000:06:07:03
2020-06-30 15:13:59,701 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-west/18/ to /results/forcing/atmospheric/GEM2.5/GRIB/20200630/18/
2020-06-30 15:13:59,702 INFO [collect_weather] 2020-06-30 2.5km weather forecast 18 collection complete
2020-06-30 15:13:59,706 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2020-06-30 15:14:00,318 INFO [manager] launching NextWorker(module='nowcast.workers.download_weather', args=['00', '1km', '--no-verify-certs'], host='localhost')
2020-06-30 15:14:00,341 INFO [manager] launching NextWorker(module='nowcast.workers.download_weather', args=['12', '1km', '--no-verify-certs'], host='localhost')
2020-06-30 15:14:00,439 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['00', '2.5km'], host='localhost')
2020-06-30 15:14:01,703 INFO [download_weather] running in process 8772
2020-06-30 15:14:01,704 INFO [download_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-06-30 15:14:01,705 INFO [download_weather] publishing log messages to tcp://*:5562
2020-06-30 15:14:01,706 INFO [download_weather] connected to skookum.eos.ubc.ca port 5555
2020-06-30 15:14:01,707 INFO [download_weather] downloading 00 1 km forecast GRIB2 files for 20200630
2020-06-30 15:14:01,708 INFO [download_weather] running in process 8774
2020-06-30 15:14:01,709 INFO [download_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-06-30 15:14:01,710 INFO [download_weather] publishing log messages to tcp://*:5563
2020-06-30 15:14:01,711 INFO [download_weather] connected to skookum.eos.ubc.ca port 5555
2020-06-30 15:14:01,712 INFO [download_weather] downloading 12 1 km forecast GRIB2 files for 20200630
2020-06-30 15:14:01,943 INFO [collect_weather] running in process 8775
2020-06-30 15:14:01,944 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-06-30 15:14:01,945 INFO [collect_weather] publishing log messages to tcp://*:5565
2020-06-30 15:14:01,945 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2020-06-30 15:14:02,023 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-west/00/
2020-06-30 15:14:14,959 INFO [watch_NEMO_agrif] 30jun20nowcast-agrif on orcinus-nowcast-agrif: timestep: 4619843 = 2020-06-30 19:34:40 UTC, 81.6% complete
2020-06-30 15:16:20,076 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15588395 = 2020-07-01 09:36:35 UTC estimated time to finish: 0000:04:05:44
2020-06-30 15:16:43,743 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 38773297 = 2020-06-30 11:38:38 UTC estimated time to finish: 0000:06:00:37
2020-06-30 15:19:15,484 INFO [watch_NEMO_agrif] 30jun20nowcast-agrif on orcinus-nowcast-agrif: timestep: 4620096 = 2020-06-30 22:23:20 UTC, 93.3% complete
2020-06-30 15:21:20,187 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15590371 = 2020-07-01 10:09:31 UTC estimated time to finish: 0000:04:00:36
2020-06-30 15:21:43,878 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 38774880 = 2020-06-30 11:49:12 UTC estimated time to finish: 0000:05:54:55
2020-06-30 15:24:15,869 INFO [watch_NEMO_agrif] 30jun20nowcast-agrif on orcinus-nowcast-agrif: timestep: 4620240 = 2020-06-30 23:59:20 UTC, 100.0% complete
2020-06-30 15:26:20,300 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15592376 = 2020-07-01 10:42:56 UTC estimated time to finish: 0000:03:55:20
2020-06-30 15:26:44,012 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 38776655 = 2020-06-30 12:01:02 UTC estimated time to finish: 0000:05:48:34
2020-06-30 15:29:16,142 INFO [watch_NEMO_agrif] NEMO AGRIF run 10846033.orca2.ibb on orcinus-nowcast-agrif completed
2020-06-30 15:29:16,148 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_agrif worker
2020-06-30 15:29:16,759 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['orcinus-nowcast-agrif', 'nowcast-agrif', '--run-date', '2020-06-30'], host='localhost')
2020-06-30 15:29:18,009 INFO [download_results] running in process 9197
2020-06-30 15:29:18,010 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-06-30 15:29:18,011 INFO [download_results] publishing log messages to tcp://*:5564
2020-06-30 15:29:18,012 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2020-06-30 15:29:18,013 INFO [download_results] downloading results from orcinus-nowcast-agrif:/global/scratch/dlatorne/nowcast-agrif/30jun20 to /results/SalishSea/nowcast-agrif.201702
2020-06-30 15:30:00,370 INFO [download_results] nowcast-agrif 2020-06-30 results files from orcinus-nowcast-agrif downloaded
2020-06-30 15:30:00,379 INFO [manager] checklist updated with [results files] items from download_results worker
2020-06-30 15:30:00,672 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast-agrif', 'research', '--run-date', '2020-06-30'], host='localhost')
2020-06-30 15:30:05,846 INFO [make_plots] running in process 9204
2020-06-30 15:30:05,847 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-06-30 15:30:05,848 INFO [make_plots] publishing log messages to tcp://*:5564
2020-06-30 15:30:05,849 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2020-06-30 15:30:06,460 INFO [make_plots] preparing render list for 2020-06-30 NEMO nowcast-agrif research figures
2020-06-30 15:30:06,462 INFO [make_plots] added baynes_sound_surface figure to 2020-06-30 NEMO nowcast-agrif research render list
2020-06-30 15:30:06,463 INFO [make_plots] starting to render nemo nowcast-agrif research 30jun20 figures
2020-06-30 15:30:13,304 INFO [make_plots] finished rendering nemo nowcast-agrif research 30jun20 figures
2020-06-30 15:30:13,307 INFO [make_plots] nemo research plots for 2020-06-30 nowcast-agrif completed
2020-06-30 15:30:13,311 INFO [manager] checklist updated with [plots] items from make_plots worker
2020-06-30 15:31:20,413 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15594351 = 2020-07-01 11:15:51 UTC estimated time to finish: 0000:03:50:16
2020-06-30 15:31:44,147 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 38778430 = 2020-06-30 12:12:52 UTC estimated time to finish: 0000:05:42:13
2020-06-30 15:32:15,604 INFO [download_weather] 2020-06-30 1km weather forecast 00 downloads complete
2020-06-30 15:32:15,608 INFO [manager] checklist updated with [weather forecast] items from download_weather worker
2020-06-30 15:36:20,528 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15596124 = 2020-07-01 11:45:24 UTC estimated time to finish: 0000:03:45:42
2020-06-30 15:36:44,281 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 38780206 = 2020-06-30 12:24:42 UTC estimated time to finish: 0000:05:35:57
2020-06-30 15:41:20,642 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15598129 = 2020-07-01 12:18:49 UTC estimated time to finish: 0000:03:40:25
2020-06-30 15:41:44,406 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 38781819 = 2020-06-30 12:35:27 UTC estimated time to finish: 0000:05:30:14
2020-06-30 15:46:20,758 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15600105 = 2020-07-01 12:51:45 UTC estimated time to finish: 0000:03:35:22
2020-06-30 15:46:44,541 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 38783553 = 2020-06-30 12:47:01 UTC estimated time to finish: 0000:05:24:11
2020-06-30 15:51:20,873 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15602109 = 2020-07-01 13:25:09 UTC estimated time to finish: 0000:03:29:57
2020-06-30 15:51:44,677 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 38785329 = 2020-06-30 12:58:51 UTC estimated time to finish: 0000:05:18:00
2020-06-30 15:56:20,990 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15604168 = 2020-07-01 13:59:28 UTC estimated time to finish: 0000:03:24:26
2020-06-30 15:56:44,811 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 38787116 = 2020-06-30 13:10:46 UTC estimated time to finish: 0000:05:11:48
2020-06-30 16:01:21,106 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15606008 = 2020-07-01 14:30:08 UTC estimated time to finish: 0000:03:19:29
2020-06-30 16:01:44,947 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 38788687 = 2020-06-30 13:21:14 UTC estimated time to finish: 0000:05:06:24
2020-06-30 16:06:21,226 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15608025 = 2020-07-01 15:03:45 UTC estimated time to finish: 0000:03:14:03
2020-06-30 16:06:45,085 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 38790464 = 2020-06-30 13:33:05 UTC estimated time to finish: 0000:05:00:19
2020-06-30 16:11:21,342 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15610070 = 2020-07-01 15:37:50 UTC estimated time to finish: 0000:03:08:40
2020-06-30 16:11:45,224 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 38792240 = 2020-06-30 13:44:56 UTC estimated time to finish: 0000:04:54:14
2020-06-30 16:16:21,461 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15612086 = 2020-07-01 16:11:26 UTC estimated time to finish: 0000:03:03:27
2020-06-30 16:16:45,363 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 38794014 = 2020-06-30 13:56:45 UTC estimated time to finish: 0000:04:48:11
2020-06-30 16:21:21,579 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15613899 = 2020-07-01 16:41:39 UTC estimated time to finish: 0000:02:58:44
2020-06-30 16:21:45,501 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 38795790 = 2020-06-30 14:08:36 UTC estimated time to finish: 0000:04:42:10
2020-06-30 16:26:21,706 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15615862 = 2020-07-01 17:14:22 UTC estimated time to finish: 0000:02:53:39
2020-06-30 16:26:45,639 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 38797373 = 2020-06-30 14:19:09 UTC estimated time to finish: 0000:04:36:48
2020-06-30 16:31:21,827 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15617880 = 2020-07-01 17:48:00 UTC estimated time to finish: 0000:02:48:27
2020-06-30 16:31:45,779 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 38799150 = 2020-06-30 14:31:00 UTC estimated time to finish: 0000:04:30:49
2020-06-30 16:36:21,949 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15619842 = 2020-07-01 18:20:42 UTC estimated time to finish: 0000:02:43:27
2020-06-30 16:36:45,903 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 38800925 = 2020-06-30 14:42:50 UTC estimated time to finish: 0000:04:24:54
2020-06-30 16:41:22,069 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15621656 = 2020-07-01 18:50:56 UTC estimated time to finish: 0000:02:38:47
2020-06-30 16:41:46,058 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 38802700 = 2020-06-30 14:54:40 UTC estimated time to finish: 0000:04:18:57
2020-06-30 16:46:22,192 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15623660 = 2020-07-01 19:24:20 UTC estimated time to finish: 0000:02:33:35
2020-06-30 16:46:46,198 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 38804283 = 2020-06-30 15:05:13 UTC estimated time to finish: 0000:04:13:40
2020-06-30 16:51:22,313 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15625637 = 2020-07-01 19:57:17 UTC estimated time to finish: 0000:02:28:29
2020-06-30 16:51:46,340 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 38806100 = 2020-06-30 15:17:20 UTC estimated time to finish: 0000:04:07:37
2020-06-30 16:56:22,394 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15627600 = 2020-07-01 20:30:00 UTC estimated time to finish: 0000:02:23:29
2020-06-30 16:56:46,483 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 38807876 = 2020-06-30 15:29:10 UTC estimated time to finish: 0000:04:01:44
2020-06-30 17:01:22,517 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15629617 = 2020-07-01 21:03:37 UTC estimated time to finish: 0000:02:18:19
2020-06-30 17:01:46,627 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 38809610 = 2020-06-30 15:40:44 UTC estimated time to finish: 0000:03:56:04
2020-06-30 17:06:22,613 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15631349 = 2020-07-01 21:32:29 UTC estimated time to finish: 0000:02:13:55
2020-06-30 17:06:46,769 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 38811153 = 2020-06-30 15:51:01 UTC estimated time to finish: 0000:03:51:04
2020-06-30 17:11:22,736 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15633324 = 2020-07-01 22:05:24 UTC estimated time to finish: 0000:02:08:53
2020-06-30 17:11:46,913 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 38812888 = 2020-06-30 16:02:35 UTC estimated time to finish: 0000:03:45:24
2020-06-30 17:16:22,862 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15635330 = 2020-07-01 22:38:50 UTC estimated time to finish: 0000:02:03:41
2020-06-30 17:16:47,057 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 38814663 = 2020-06-30 16:14:25 UTC estimated time to finish: 0000:03:39:37
2020-06-30 17:21:22,908 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15637374 = 2020-07-01 23:12:54 UTC estimated time to finish: 0000:01:58:22
2020-06-30 17:21:47,201 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 38816398 = 2020-06-30 16:25:59 UTC estimated time to finish: 0000:03:33:59
2020-06-30 17:26:23,034 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15639188 = 2020-07-01 23:43:08 UTC estimated time to finish: 0000:01:53:39
2020-06-30 17:26:47,345 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 38818174 = 2020-06-30 16:37:49 UTC estimated time to finish: 0000:03:28:14
2020-06-30 17:31:23,161 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15641203 = 2020-07-02 00:16:43 UTC estimated time to finish: 0000:01:48:28
2020-06-30 17:31:47,516 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 38819717 = 2020-06-30 16:48:06 UTC estimated time to finish: 0000:03:23:16
2020-06-30 17:36:23,290 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15643208 = 2020-07-02 00:50:08 UTC estimated time to finish: 0000:01:43:18
2020-06-30 17:36:47,631 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 38821451 = 2020-06-30 16:59:40 UTC estimated time to finish: 0000:03:17:41
2020-06-30 17:41:23,417 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15645184 = 2020-07-02 01:23:04 UTC estimated time to finish: 0000:01:38:12
2020-06-30 17:41:47,779 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 38823226 = 2020-06-30 17:11:30 UTC estimated time to finish: 0000:03:11:57
2020-06-30 17:46:23,484 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15647039 = 2020-07-02 01:53:59 UTC estimated time to finish: 0000:01:33:23
2020-06-30 17:46:47,927 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 38825001 = 2020-06-30 17:23:20 UTC estimated time to finish: 0000:03:06:13
2020-06-30 17:51:23,613 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15649083 = 2020-07-02 02:28:03 UTC estimated time to finish: 0000:01:28:05
2020-06-30 17:51:48,074 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 38826628 = 2020-06-30 17:34:11 UTC estimated time to finish: 0000:03:00:59
2020-06-30 17:54:00,843 ERROR [download_weather] giving up; download from https://dd.alpha.meteo.gc.ca/model_hrdps/west/1km/grib2/12/013/CMC_hrdps_west_UGRD_TGL_10_rotated_latlon0.009x0.009_20200630T12Z_P013-00.grib2 failed 43 times
2020-06-30 17:54:01,144 CRITICAL [download_weather] 2020-06-30 1km weather forecast 12 downloads failed
2020-06-30 17:56:23,697 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15651128 = 2020-07-02 03:02:08 UTC estimated time to finish: 0000:01:22:48
2020-06-30 17:56:48,222 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 38828402 = 2020-06-30 17:46:00 UTC estimated time to finish: 0000:02:55:17
2020-06-30 18:01:23,826 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15653103 = 2020-07-02 03:35:03 UTC estimated time to finish: 0000:01:17:45
2020-06-30 18:01:48,371 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 38830178 = 2020-06-30 17:57:51 UTC estimated time to finish: 0000:02:49:34
2020-06-30 18:06:23,957 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15655108 = 2020-07-02 04:08:28 UTC estimated time to finish: 0000:01:12:35
2020-06-30 18:06:48,520 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 38831953 = 2020-06-30 18:09:41 UTC estimated time to finish: 0000:02:43:54
2020-06-30 18:11:24,087 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15656921 = 2020-07-02 04:38:41 UTC estimated time to finish: 0000:01:07:56
2020-06-30 18:11:48,670 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 38833564 = 2020-06-30 18:20:25 UTC estimated time to finish: 0000:02:38:45
2020-06-30 18:16:24,220 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15658925 = 2020-07-02 05:12:05 UTC estimated time to finish: 0000:01:02:47
2020-06-30 18:16:48,821 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 38835313 = 2020-06-30 18:32:05 UTC estimated time to finish: 0000:02:33:11
2020-06-30 18:21:24,352 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15660942 = 2020-07-02 05:45:42 UTC estimated time to finish: 0000:00:57:36
2020-06-30 18:21:48,972 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 38837130 = 2020-06-30 18:44:12 UTC estimated time to finish: 0000:02:27:23
2020-06-30 18:26:24,486 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15662988 = 2020-07-02 06:19:48 UTC estimated time to finish: 0000:00:52:19
2020-06-30 18:26:49,123 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 38838904 = 2020-06-30 18:56:01 UTC estimated time to finish: 0000:02:21:45
2020-06-30 18:31:24,619 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15664841 = 2020-07-02 06:50:41 UTC estimated time to finish: 0000:00:47:33
2020-06-30 18:31:49,279 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 38840680 = 2020-06-30 19:07:52 UTC estimated time to finish: 0000:02:16:08
2020-06-30 18:36:24,735 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15666846 = 2020-07-02 07:24:06 UTC estimated time to finish: 0000:00:42:25
2020-06-30 18:36:49,358 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 38842250 = 2020-06-30 19:18:20 UTC estimated time to finish: 0000:02:11:11
2020-06-30 18:41:24,870 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15668862 = 2020-07-02 07:57:42 UTC estimated time to finish: 0000:00:37:14
2020-06-30 18:41:49,511 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 38844027 = 2020-06-30 19:30:10 UTC estimated time to finish: 0000:02:05:35
2020-06-30 18:46:25,006 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15670908 = 2020-07-02 08:31:48 UTC estimated time to finish: 0000:00:31:59
2020-06-30 18:46:49,667 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 38845733 = 2020-06-30 19:41:33 UTC estimated time to finish: 0000:02:00:14
2020-06-30 18:51:25,142 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15672761 = 2020-07-02 09:02:41 UTC estimated time to finish: 0000:00:27:14
2020-06-30 18:51:49,823 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 38847509 = 2020-06-30 19:53:23 UTC estimated time to finish: 0000:01:54:40
2020-06-30 18:56:25,226 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15674766 = 2020-07-02 09:36:06 UTC estimated time to finish: 0000:00:22:05
2020-06-30 18:56:49,979 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 38849080 = 2020-06-30 20:03:52 UTC estimated time to finish: 0000:01:49:44
2020-06-30 19:01:25,363 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15676783 = 2020-07-02 10:09:43 UTC estimated time to finish: 0000:00:16:56
2020-06-30 19:01:50,136 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 38850826 = 2020-06-30 20:15:30 UTC estimated time to finish: 0000:01:44:16
2020-06-30 19:06:25,501 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15678788 = 2020-07-02 10:43:08 UTC estimated time to finish: 0000:00:11:48
2020-06-30 19:06:50,293 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 38852561 = 2020-06-30 20:27:04 UTC estimated time to finish: 0000:01:38:51
2020-06-30 19:11:25,638 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15680804 = 2020-07-02 11:16:44 UTC estimated time to finish: 0000:00:06:38
2020-06-30 19:11:50,450 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 38854338 = 2020-06-30 20:38:55 UTC estimated time to finish: 0000:01:33:17
2020-06-30 19:16:25,716 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15682618 = 2020-07-02 11:46:58 UTC estimated time to finish: 0000:00:02:00
2020-06-30 19:16:50,609 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 38856112 = 2020-06-30 20:50:44 UTC estimated time to finish: 0000:01:27:44
2020-06-30 19:21:25,824 INFO [watch_fvcom] x2 forecast FVCOM VH-FR run on arbutus.cloud-nowcast completed
2020-06-30 19:21:25,830 INFO [manager] checklist updated with [FVCOM run] items from watch_fvcom worker
2020-06-30 19:21:26,418 INFO [manager] launching NextWorker(module='nowcast.workers.download_fvcom_results', args=['arbutus.cloud-nowcast', 'x2', 'forecast', '--run-date', '2020-06-30'], host='localhost')
2020-06-30 19:21:27,921 INFO [download_fvcom_results] running in process 14649
2020-06-30 19:21:27,921 INFO [download_fvcom_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-06-30 19:21:27,923 INFO [download_fvcom_results] publishing log messages to tcp://*:5562
2020-06-30 19:21:27,923 INFO [download_fvcom_results] connected to skookum.eos.ubc.ca port 5555
2020-06-30 19:21:50,769 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 38858243 = 2020-06-30 21:04:57 UTC estimated time to finish: 0000:01:20:53
2020-06-30 19:22:32,401 INFO [download_fvcom_results] VHFR FVCOM x2 forecast 2020-06-30 results files from arbutus.cloud-nowcast downloaded
2020-06-30 19:22:32,408 INFO [manager] checklist updated with [VHFR FVCOM results files] items from download_fvcom_results worker
2020-06-30 19:22:32,724 INFO [manager] launching NextWorker(module='nowcast.workers.get_vfpa_hadcp', args=['--data-date', '2020-06-30'], host='localhost')
2020-06-30 19:22:32,727 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['fvcom', 'forecast-x2', 'research', '--run-date', '2020-06-30'], host='localhost')
2020-06-30 19:22:32,730 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['fvcom', 'forecast', '--run-date', '2020-06-30'], host='localhost')
2020-06-30 19:22:34,045 INFO [update_forecast_datasets] running in process 14676
2020-06-30 19:22:34,046 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-06-30 19:22:34,047 INFO [update_forecast_datasets] publishing log messages to tcp://*:5562
2020-06-30 19:22:34,048 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2020-06-30 19:22:34,049 INFO [update_forecast_datasets] updating fvcom most_recent_forecast directory from forecast/30jun20 run
2020-06-30 19:22:34,050 INFO [update_forecast_datasets] updated fvcom most_recent_forecast directory from forecast/30jun20 run
2020-06-30 19:22:34,051 INFO [update_forecast_datasets] fvcom 2020-06-30 forecast rolling forecast datasets updated
2020-06-30 19:22:34,052 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2020-06-30 19:22:34,330 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['fvcom-forecast'], host='localhost')
2020-06-30 19:22:35,198 INFO [get_vfpa_hadcp] running in process 14674
2020-06-30 19:22:35,199 INFO [get_vfpa_hadcp] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-06-30 19:22:35,199 INFO [get_vfpa_hadcp] publishing log messages to tcp://*:5562
2020-06-30 19:22:35,200 INFO [get_vfpa_hadcp] connected to skookum.eos.ubc.ca port 5555
2020-06-30 19:22:35,201 INFO [get_vfpa_hadcp] processing VFPA HADCP data from 2nd Narrows Rail Bridge for 2020-06-30
2020-06-30 19:22:36,498 INFO [get_vfpa_hadcp] added VFPA HADCP data from 2nd Narrows Rail Bridge for 2020-06-30 to /opp/observations/AISDATA/netcdf/VFPA_2ND_NARROWS_HADCP_2s_202006.nc
2020-06-30 19:22:36,498 INFO [get_vfpa_hadcp] VFPA HADCP observations added to 2020-06 netcdf file
2020-06-30 19:22:36,501 INFO [manager] checklist updated with [VFPA HADCP data] items from get_vfpa_hadcp worker
2020-06-30 19:22:36,798 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['VFPA-HADCP'], host='localhost')
2020-06-30 19:22:36,817 INFO [make_plots] running in process 14675
2020-06-30 19:22:36,818 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-06-30 19:22:36,819 INFO [make_plots] publishing log messages to tcp://*:5563
2020-06-30 19:22:36,819 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2020-06-30 19:22:37,653 INFO [ping_erddap] running in process 14689
2020-06-30 19:22:37,654 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-06-30 19:22:37,655 INFO [ping_erddap] publishing log messages to tcp://*:5562
2020-06-30 19:22:37,656 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2020-06-30 19:22:37,657 INFO [ping_erddap] VFPA-HADCP ERDDAP dataset flag file(s) created
2020-06-30 19:22:37,659 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2020-06-30 19:22:37,901 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['fvcom', 'nowcast-x2', 'publish', '--run-date', '2020-06-30'], host='localhost')
2020-06-30 19:22:37,904 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['fvcom', 'forecast-x2', 'publish', '--run-date', '2020-06-30'], host='localhost')
2020-06-30 19:22:40,579 INFO [make_plots] running in process 14697
2020-06-30 19:22:40,580 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-06-30 19:22:40,581 INFO [make_plots] publishing log messages to tcp://*:5562
2020-06-30 19:22:40,582 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2020-06-30 19:22:40,658 INFO [make_plots] running in process 14696
2020-06-30 19:22:40,659 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-06-30 19:22:40,660 INFO [make_plots] publishing log messages to tcp://*:5564
2020-06-30 19:22:40,661 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2020-06-30 19:22:47,321 INFO [make_plots] preparing render list for 2020-06-30 VHFR FVCOM forecast-x2 research figures
2020-06-30 19:22:47,328 INFO [make_plots] added EnglishBay_surface_currents figures to 2020-06-30 VHFR FVCOM forecast-x2 research render list
2020-06-30 19:22:47,335 INFO [make_plots] added VancouverHarbour_surface_currents figures to 2020-06-30 VHFR FVCOM forecast-x2 research render list
2020-06-30 19:22:47,342 INFO [make_plots] added IndianArm_surface_currents figures to 2020-06-30 VHFR FVCOM forecast-x2 research render list
2020-06-30 19:22:47,348 INFO [make_plots] added VancouverHarbour_thalweg figures to 2020-06-30 VHFR FVCOM forecast-x2 research render list
2020-06-30 19:22:47,355 INFO [make_plots] added PortMoody_thalweg figures to 2020-06-30 VHFR FVCOM forecast-x2 research render list
2020-06-30 19:22:47,361 INFO [make_plots] added IndianArm_thalweg figures to 2020-06-30 VHFR FVCOM forecast-x2 research render list
2020-06-30 19:22:47,362 INFO [make_plots] starting to render fvcom forecast-x2 research 30jun20 figures
2020-06-30 19:22:48,233 INFO [make_plots] preparing render list for 2020-06-30 VHFR FVCOM forecast-x2 publish figures
2020-06-30 19:22:48,234 INFO [make_plots] added Calamity Point figure to 2020-06-30 VHFR FVCOM forecast-x2 publish render list
2020-06-30 19:22:48,235 INFO [make_plots] added Indian Arm Head figure to 2020-06-30 VHFR FVCOM forecast-x2 publish render list
2020-06-30 19:22:48,236 INFO [make_plots] added New Westminster figure to 2020-06-30 VHFR FVCOM forecast-x2 publish render list
2020-06-30 19:22:48,237 INFO [make_plots] added Port Moody figure to 2020-06-30 VHFR FVCOM forecast-x2 publish render list
2020-06-30 19:22:48,238 INFO [make_plots] added Sand Heads figure to 2020-06-30 VHFR FVCOM forecast-x2 publish render list
2020-06-30 19:22:48,238 INFO [make_plots] added Sandy Cove figure to 2020-06-30 VHFR FVCOM forecast-x2 publish render list
2020-06-30 19:22:48,239 INFO [make_plots] added Vancouver Harbour figure to 2020-06-30 VHFR FVCOM forecast-x2 publish render list
2020-06-30 19:22:48,240 INFO [make_plots] added Woodwards Landing figure to 2020-06-30 VHFR FVCOM forecast-x2 publish render list
2020-06-30 19:22:48,241 INFO [make_plots] added 2ndNarrows_current figure to 2020-06-30 VHFR FVCOM forecast-x2 publish render list
2020-06-30 19:22:48,241 INFO [make_plots] starting to render fvcom forecast-x2 publish 30jun20 figures
2020-06-30 19:22:48,257 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07724 Calamity Point from 2020-06-29 23:55:00 to 2020-07-02 12:00:00
2020-06-30 19:22:48,259 INFO [make_plots] preparing render list for 2020-06-30 VHFR FVCOM nowcast-x2 publish figures
2020-06-30 19:22:48,260 INFO [make_plots] added Calamity Point figure to 2020-06-30 VHFR FVCOM nowcast-x2 publish render list
2020-06-30 19:22:48,261 INFO [make_plots] added Indian Arm Head figure to 2020-06-30 VHFR FVCOM nowcast-x2 publish render list
2020-06-30 19:22:48,262 INFO [make_plots] added New Westminster figure to 2020-06-30 VHFR FVCOM nowcast-x2 publish render list
2020-06-30 19:22:48,262 INFO [make_plots] added Port Moody figure to 2020-06-30 VHFR FVCOM nowcast-x2 publish render list
2020-06-30 19:22:48,263 INFO [make_plots] added Sand Heads figure to 2020-06-30 VHFR FVCOM nowcast-x2 publish render list
2020-06-30 19:22:48,264 INFO [make_plots] added Sandy Cove figure to 2020-06-30 VHFR FVCOM nowcast-x2 publish render list
2020-06-30 19:22:48,265 INFO [make_plots] added Vancouver Harbour figure to 2020-06-30 VHFR FVCOM nowcast-x2 publish render list
2020-06-30 19:22:48,266 INFO [make_plots] added Woodwards Landing figure to 2020-06-30 VHFR FVCOM nowcast-x2 publish render list
2020-06-30 19:22:48,266 INFO [make_plots] added 2ndNarrows_current figure to 2020-06-30 VHFR FVCOM nowcast-x2 publish render list
2020-06-30 19:22:48,267 INFO [make_plots] starting to render fvcom nowcast-x2 publish 30jun20 figures
2020-06-30 19:22:48,280 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07724 Calamity Point from 2020-06-29 23:55:00 to 2020-07-01 00:00:00
2020-06-30 19:23:11,663 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07724 Calamity Point from 2020-06-30 00:00:00 to 2020-07-01 00:00:00
2020-06-30 19:23:13,255 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07774 Indian Arm Head from 2020-06-29 23:55:00 to 2020-07-01 00:00:00
2020-06-30 19:23:13,493 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07724 Calamity Point from 2020-06-30 00:00:00 to 2020-07-02 12:00:00
2020-06-30 19:23:13,997 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07774 Indian Arm Head from 2020-06-30 00:00:00 to 2020-07-01 00:00:00
2020-06-30 19:23:15,115 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07774 Indian Arm Head from 2020-06-29 23:55:00 to 2020-07-02 12:00:00
2020-06-30 19:23:15,552 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07654 New Westminster from 2020-06-29 23:55:00 to 2020-07-01 00:00:00
2020-06-30 19:23:15,670 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07774 Indian Arm Head from 2020-06-30 00:00:00 to 2020-07-02 12:00:00
2020-06-30 19:23:17,134 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07654 New Westminster from 2020-06-29 23:55:00 to 2020-07-02 12:00:00
2020-06-30 19:23:36,852 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07654 New Westminster from 2020-06-30 00:00:00 to 2020-07-01 00:00:00
2020-06-30 19:23:39,895 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07755 Port Moody from 2020-06-29 23:55:00 to 2020-07-01 00:00:00
2020-06-30 19:23:41,205 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07654 New Westminster from 2020-06-30 00:00:00 to 2020-07-02 12:00:00
2020-06-30 19:23:46,192 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07755 Port Moody from 2020-06-29 23:55:00 to 2020-07-02 12:00:00
2020-06-30 19:24:01,875 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07755 Port Moody from 2020-06-30 00:00:00 to 2020-07-01 00:00:00
2020-06-30 19:24:05,026 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07594 Sand Heads from 2020-06-29 23:55:00 to 2020-07-01 00:00:00
2020-06-30 19:24:09,623 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07755 Port Moody from 2020-06-30 00:00:00 to 2020-07-02 12:00:00
2020-06-30 19:24:14,844 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07594 Sand Heads from 2020-06-29 23:55:00 to 2020-07-02 12:00:00
2020-06-30 19:24:26,490 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07594 Sand Heads from 2020-06-30 00:00:00 to 2020-07-01 00:00:00
2020-06-30 19:24:29,491 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07786 Sandy Cove from 2020-06-29 23:55:00 to 2020-07-01 00:00:00
2020-06-30 19:24:38,528 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07594 Sand Heads from 2020-06-30 00:00:00 to 2020-07-02 12:00:00
2020-06-30 19:24:43,680 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07786 Sandy Cove from 2020-06-29 23:55:00 to 2020-07-02 12:00:00
2020-06-30 19:24:51,339 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07795 Point Atkinson from 2020-06-30 00:00:00 to 2020-07-01 00:00:00
2020-06-30 19:24:54,329 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07735 Vancouver Harbour from 2020-06-29 23:55:00 to 2020-07-01 00:00:00
2020-06-30 19:25:07,529 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07795 Point Atkinson from 2020-06-30 00:00:00 to 2020-07-02 12:00:00
2020-06-30 19:25:12,499 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07735 Vancouver Harbour from 2020-06-29 23:55:00 to 2020-07-02 12:00:00
2020-06-30 19:25:15,981 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07735 Vancouver Harbour from 2020-06-30 00:00:00 to 2020-07-01 00:00:00
2020-06-30 19:25:18,953 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07610 Woodwards Landing from 2020-06-29 23:55:00 to 2020-07-01 00:00:00
2020-06-30 19:25:31,835 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07610 Woodwards Landing from 2020-06-30 00:00:00 to 2020-07-01 00:00:00
2020-06-30 19:25:36,044 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07735 Vancouver Harbour from 2020-06-30 00:00:00 to 2020-07-02 12:00:00
2020-06-30 19:25:36,503 INFO [make_plots] finished rendering fvcom nowcast-x2 publish 30jun20 figures
2020-06-30 19:25:37,090 INFO [make_plots] fvcom publish plots for 2020-06-30 nowcast-x2 completed
2020-06-30 19:25:37,097 INFO [manager] checklist updated with [plots] items from make_plots worker
2020-06-30 19:25:40,737 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07610 Woodwards Landing from 2020-06-29 23:55:00 to 2020-07-02 12:00:00
2020-06-30 19:25:55,890 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07610 Woodwards Landing from 2020-06-30 00:00:00 to 2020-07-02 12:00:00
2020-06-30 19:26:00,396 INFO [make_plots] finished rendering fvcom forecast-x2 publish 30jun20 figures
2020-06-30 19:26:00,967 INFO [make_plots] fvcom publish plots for 2020-06-30 forecast-x2 completed
2020-06-30 19:26:00,975 INFO [manager] checklist updated with [plots] items from make_plots worker
2020-06-30 19:26:50,927 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 38860769 = 2020-06-30 21:21:47 UTC estimated time to finish: 0000:01:12:47
2020-06-30 19:31:51,090 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 38863297 = 2020-06-30 21:38:38 UTC estimated time to finish: 0000:01:04:44
2020-06-30 19:36:51,251 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 38865822 = 2020-06-30 21:55:28 UTC estimated time to finish: 0000:00:56:47
2020-06-30 19:36:58,706 CRITICAL [make_plots] unhandled exception:
Traceback (most recent call last):
  File "/SalishSeaCast/NEMO_Nowcast/nemo_nowcast/worker.py", line 390, in _do_work
    checklist = self.worker_func(
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 325, in make_plots
    checklist = _render_figures(
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1038, in _render_figures
    fig = _calc_figure(fig_func, args, kwargs)
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure
    fig = fig_func(*args, **kwargs)
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure
    plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname)
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 80, in _prep_plot_data
    date_bytes = fvcom_results_dataset.variables["Times"][time_index, :]
  File "netCDF4/_netCDF4.pyx", line 4384, in netCDF4._netCDF4.Variable.__getitem__
  File "netCDF4/_netCDF4.pyx", line 4072, in netCDF4._netCDF4.Variable.shape.__get__
  File "netCDF4/_netCDF4.pyx", line 4020, in netCDF4._netCDF4.Variable._getdims
  File "netCDF4/_netCDF4.pyx", line 1887, in netCDF4._netCDF4._ensure_nc_success
RuntimeError: NetCDF: Not a valid ID
2020-06-30 19:41:51,400 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 38868116 = 2020-06-30 22:10:46 UTC estimated time to finish: 0000:00:49:37
2020-06-30 19:46:51,512 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 38870640 = 2020-06-30 22:27:36 UTC estimated time to finish: 0000:00:41:48
2020-06-30 19:51:51,675 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 38873160 = 2020-06-30 22:44:24 UTC estimated time to finish: 0000:00:34:03
2020-06-30 19:56:51,840 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 38875724 = 2020-06-30 23:01:29 UTC estimated time to finish: 0000:00:26:15
2020-06-30 20:01:52,007 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 38878018 = 2020-06-30 23:16:47 UTC estimated time to finish: 0000:00:19:19
2020-06-30 20:06:52,173 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 38880505 = 2020-06-30 23:33:22 UTC estimated time to finish: 0000:00:11:51
2020-06-30 20:11:52,341 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 38882990 = 2020-06-30 23:49:56 UTC estimated time to finish: 0000:00:04:28
2020-06-30 20:16:52,457 INFO [watch_fvcom] r12 nowcast FVCOM VH-FR run on arbutus.cloud-nowcast completed
2020-06-30 20:16:52,463 INFO [manager] checklist updated with [FVCOM run] items from watch_fvcom worker
2020-06-30 20:16:53,080 INFO [manager] launching NextWorker(module='nowcast.workers.download_fvcom_results', args=['arbutus.cloud-nowcast', 'r12', 'nowcast', '--run-date', '2020-06-30'], host='localhost')
2020-06-30 20:16:54,215 INFO [download_fvcom_results] running in process 15984
2020-06-30 20:16:54,216 INFO [download_fvcom_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-06-30 20:16:54,217 INFO [download_fvcom_results] publishing log messages to tcp://*:5562
2020-06-30 20:16:54,218 INFO [download_fvcom_results] connected to skookum.eos.ubc.ca port 5555
2020-06-30 20:18:29,093 INFO [download_fvcom_results] VHFR FVCOM r12 nowcast 2020-06-30 results files from arbutus.cloud-nowcast downloaded
2020-06-30 20:18:29,099 INFO [manager] checklist updated with [VHFR FVCOM results files] items from download_fvcom_results worker
2020-06-30 20:18:29,407 INFO [manager] launching NextWorker(module='nowcast.workers.get_vfpa_hadcp', args=['--data-date', '2020-06-30'], host='localhost')
2020-06-30 20:18:29,411 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['fvcom', 'nowcast-r12', 'research', '--run-date', '2020-06-30'], host='localhost')
2020-06-30 20:18:31,431 INFO [get_vfpa_hadcp] running in process 16033
2020-06-30 20:18:31,433 INFO [get_vfpa_hadcp] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-06-30 20:18:31,433 INFO [get_vfpa_hadcp] publishing log messages to tcp://*:5562
2020-06-30 20:18:31,434 INFO [get_vfpa_hadcp] connected to skookum.eos.ubc.ca port 5555
2020-06-30 20:18:31,435 INFO [get_vfpa_hadcp] processing VFPA HADCP data from 2nd Narrows Rail Bridge for 2020-06-30
2020-06-30 20:18:32,550 INFO [get_vfpa_hadcp] added VFPA HADCP data from 2nd Narrows Rail Bridge for 2020-06-30 to /opp/observations/AISDATA/netcdf/VFPA_2ND_NARROWS_HADCP_2s_202006.nc
2020-06-30 20:18:32,551 INFO [get_vfpa_hadcp] VFPA HADCP observations added to 2020-06 netcdf file
2020-06-30 20:18:32,552 INFO [manager] checklist updated with [VFPA HADCP data] items from get_vfpa_hadcp worker
2020-06-30 20:18:32,797 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['VFPA-HADCP'], host='localhost')
2020-06-30 20:18:32,846 INFO [make_plots] running in process 16034
2020-06-30 20:18:32,847 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-06-30 20:18:32,847 INFO [make_plots] publishing log messages to tcp://*:5563
2020-06-30 20:18:32,848 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2020-06-30 20:18:32,947 INFO [make_plots] preparing render list for 2020-06-30 VHFR FVCOM nowcast-r12 research figures
2020-06-30 20:18:32,949 INFO [make_plots] added EnglishBay_surface_currents figures to 2020-06-30 VHFR FVCOM nowcast-r12 research render list
2020-06-30 20:18:32,951 INFO [make_plots] added VancouverHarbour_surface_currents figures to 2020-06-30 VHFR FVCOM nowcast-r12 research render list
2020-06-30 20:18:32,952 INFO [make_plots] added IndianArm_surface_currents figures to 2020-06-30 VHFR FVCOM nowcast-r12 research render list
2020-06-30 20:18:32,954 INFO [make_plots] added VancouverHarbour_thalweg figures to 2020-06-30 VHFR FVCOM nowcast-r12 research render list
2020-06-30 20:18:32,956 INFO [make_plots] added PortMoody_thalweg figures to 2020-06-30 VHFR FVCOM nowcast-r12 research render list
2020-06-30 20:18:32,957 INFO [make_plots] added IndianArm_thalweg figures to 2020-06-30 VHFR FVCOM nowcast-r12 research render list
2020-06-30 20:18:32,958 INFO [make_plots] starting to render fvcom nowcast-r12 research 30jun20 figures
2020-06-30 20:18:33,819 INFO [ping_erddap] running in process 16043
2020-06-30 20:18:33,820 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-06-30 20:18:33,821 INFO [ping_erddap] publishing log messages to tcp://*:5562
2020-06-30 20:18:33,822 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2020-06-30 20:18:33,823 INFO [ping_erddap] VFPA-HADCP ERDDAP dataset flag file(s) created
2020-06-30 20:18:33,825 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2020-06-30 20:18:34,084 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['fvcom', 'nowcast-r12', 'publish', '--run-date', '2020-06-30'], host='localhost')
2020-06-30 20:18:34,088 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['fvcom', 'nowcast-x2', 'publish', '--run-date', '2020-06-30'], host='localhost')
2020-06-30 20:18:34,089 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['fvcom', 'forecast-x2', 'publish', '--run-date', '2020-06-30'], host='localhost')
2020-06-30 20:18:36,771 INFO [make_plots] running in process 16048
2020-06-30 20:18:36,772 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-06-30 20:18:36,773 INFO [make_plots] publishing log messages to tcp://*:5562
2020-06-30 20:18:36,774 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2020-06-30 20:18:36,775 INFO [make_plots] running in process 16049
2020-06-30 20:18:36,776 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-06-30 20:18:36,776 INFO [make_plots] publishing log messages to tcp://*:5564
2020-06-30 20:18:36,777 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2020-06-30 20:18:36,778 INFO [make_plots] running in process 16050
2020-06-30 20:18:36,779 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-06-30 20:18:36,780 INFO [make_plots] publishing log messages to tcp://*:5566
2020-06-30 20:18:36,781 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2020-06-30 20:18:36,932 CRITICAL [make_plots] unhandled exception:
Traceback (most recent call last):
  File "/SalishSeaCast/NEMO_Nowcast/nemo_nowcast/worker.py", line 390, in _do_work
    checklist = self.worker_func(
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 281, in make_plots
    _rename_fvcom_vars(fvcom_stns_dataset_path)
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 336, in _rename_fvcom_vars
    subprocess.check_output(shlex.split(cmd))
  File "/SalishSeaCast/nowcast-env/lib/python3.8/subprocess.py", line 411, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/SalishSeaCast/nowcast-env/lib/python3.8/subprocess.py", line 512, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['ncrename', '-O', '-v', 'siglay,sigma_layer', '-v', 'siglev,sigma_level', '/tmp/vh_x2_station_timeseries.nc', '/tmp/vh_x2_station_timeseries.nc']' returned non-zero exit status 1.
2020-06-30 20:18:37,025 CRITICAL [make_plots] unhandled exception:
Traceback (most recent call last):
  File "/SalishSeaCast/NEMO_Nowcast/nemo_nowcast/worker.py", line 390, in _do_work
    checklist = self.worker_func(
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 248, in make_plots
    _rename_fvcom_vars(fvcom_stns_dataset_path)
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 336, in _rename_fvcom_vars
    subprocess.check_output(shlex.split(cmd))
  File "/SalishSeaCast/nowcast-env/lib/python3.8/subprocess.py", line 411, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/SalishSeaCast/nowcast-env/lib/python3.8/subprocess.py", line 512, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['ncrename', '-O', '-v', 'siglay,sigma_layer', '-v', 'siglev,sigma_level', '/opp/fvcom/nowcast-x2/30jun20/vh_x2_station_timeseries.nc', '/tmp/vh_x2_station_timeseries.nc']' returned non-zero exit status 1.
2020-06-30 20:18:37,727 INFO [make_plots] preparing render list for 2020-06-30 VHFR FVCOM nowcast-r12 publish figures
2020-06-30 20:18:37,727 INFO [make_plots] added Calamity Point figure to 2020-06-30 VHFR FVCOM nowcast-r12 publish render list
2020-06-30 20:18:37,728 INFO [make_plots] added Indian Arm Head figure to 2020-06-30 VHFR FVCOM nowcast-r12 publish render list
2020-06-30 20:18:37,729 INFO [make_plots] added New Westminster figure to 2020-06-30 VHFR FVCOM nowcast-r12 publish render list
2020-06-30 20:18:37,730 INFO [make_plots] added Port Moody figure to 2020-06-30 VHFR FVCOM nowcast-r12 publish render list
2020-06-30 20:18:37,730 INFO [make_plots] added Sand Heads figure to 2020-06-30 VHFR FVCOM nowcast-r12 publish render list
2020-06-30 20:18:37,731 INFO [make_plots] added Sandy Cove figure to 2020-06-30 VHFR FVCOM nowcast-r12 publish render list
2020-06-30 20:18:37,731 INFO [make_plots] added Vancouver Harbour figure to 2020-06-30 VHFR FVCOM nowcast-r12 publish render list
2020-06-30 20:18:37,732 INFO [make_plots] added Woodwards Landing figure to 2020-06-30 VHFR FVCOM nowcast-r12 publish render list
2020-06-30 20:18:37,733 INFO [make_plots] added 2ndNarrows_current figure to 2020-06-30 VHFR FVCOM nowcast-r12 publish render list
2020-06-30 20:18:37,734 INFO [make_plots] starting to render fvcom nowcast-r12 publish 30jun20 figures
2020-06-30 20:18:37,749 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07724 Calamity Point from 2020-06-29 23:55:00 to 2020-07-02 12:00:00
2020-06-30 20:19:02,267 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07724 Calamity Point from 2020-06-30 00:00:00 to 2020-07-02 12:00:00
2020-06-30 20:19:04,074 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07774 Indian Arm Head from 2020-06-29 23:55:00 to 2020-07-02 12:00:00
2020-06-30 20:19:04,627 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07774 Indian Arm Head from 2020-06-30 00:00:00 to 2020-07-02 12:00:00
2020-06-30 20:19:06,058 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07654 New Westminster from 2020-06-29 23:55:00 to 2020-07-02 12:00:00
2020-06-30 20:19:30,566 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07654 New Westminster from 2020-06-30 00:00:00 to 2020-07-02 12:00:00
2020-06-30 20:19:35,585 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07755 Port Moody from 2020-06-29 23:55:00 to 2020-07-02 12:00:00
2020-06-30 20:20:00,044 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07755 Port Moody from 2020-06-30 00:00:00 to 2020-07-02 12:00:00
2020-06-30 20:20:05,028 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07594 Sand Heads from 2020-06-29 23:55:00 to 2020-07-02 12:00:00
2020-06-30 20:20:29,343 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07594 Sand Heads from 2020-06-30 00:00:00 to 2020-07-02 12:00:00
2020-06-30 20:20:34,357 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07786 Sandy Cove from 2020-06-29 23:55:00 to 2020-07-02 12:00:00
2020-06-30 20:20:59,170 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07795 Point Atkinson from 2020-06-30 00:00:00 to 2020-07-02 12:00:00
2020-06-30 20:21:04,043 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07735 Vancouver Harbour from 2020-06-29 23:55:00 to 2020-07-02 12:00:00
2020-06-30 20:21:28,950 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07735 Vancouver Harbour from 2020-06-30 00:00:00 to 2020-07-02 12:00:00
2020-06-30 20:21:33,731 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07610 Woodwards Landing from 2020-06-29 23:55:00 to 2020-07-02 12:00:00
2020-06-30 20:21:49,578 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07610 Woodwards Landing from 2020-06-30 00:00:00 to 2020-07-02 12:00:00
2020-06-30 20:21:54,282 INFO [make_plots] finished rendering fvcom nowcast-r12 publish 30jun20 figures
2020-06-30 20:21:54,719 INFO [make_plots] fvcom publish plots for 2020-06-30 nowcast-r12 completed
2020-06-30 20:21:54,728 INFO [manager] checklist updated with [plots] items from make_plots worker
2020-06-30 20:27:14,788 CRITICAL [make_plots] unhandled exception:
Traceback (most recent call last):
  File "/SalishSeaCast/NEMO_Nowcast/nemo_nowcast/worker.py", line 390, in _do_work
    checklist = self.worker_func(
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 325, in make_plots
    checklist = _render_figures(
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1038, in _render_figures
    fig = _calc_figure(fig_func, args, kwargs)
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure
    fig = fig_func(*args, **kwargs)
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 64, in make_figure
    plot_data = _prep_plot_data(place, fvcom_results_dataset, time_index, varname)
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/research/thalweg_transect.py", line 80, in _prep_plot_data
    date_bytes = fvcom_results_dataset.variables["Times"][time_index, :]
  File "netCDF4/_netCDF4.pyx", line 4384, in netCDF4._netCDF4.Variable.__getitem__
  File "netCDF4/_netCDF4.pyx", line 4072, in netCDF4._netCDF4.Variable.shape.__get__
  File "netCDF4/_netCDF4.pyx", line 4020, in netCDF4._netCDF4.Variable._getdims
  File "netCDF4/_netCDF4.pyx", line 1887, in netCDF4._netCDF4._ensure_nc_success
RuntimeError: NetCDF: Not a valid ID
2020-06-30 21:00:39,281 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-west/00/ to /results/forcing/atmospheric/GEM2.5/GRIB/20200701/00/
2020-06-30 21:00:39,282 INFO [collect_weather] 2020-07-01 2.5km weather forecast 00 collection complete
2020-06-30 21:00:39,286 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2020-06-30 21:00:39,877 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['06', '2.5km'], host='localhost')
2020-06-30 21:00:41,103 INFO [collect_weather] running in process 17146
2020-06-30 21:00:41,104 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-06-30 21:00:41,105 INFO [collect_weather] publishing log messages to tcp://*:5562
2020-06-30 21:00:41,106 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2020-06-30 21:00:41,123 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-west/06/
2020-07-01 03:03:07,137 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-west/06/ to /results/forcing/atmospheric/GEM2.5/GRIB/20200701/06/
2020-07-01 03:03:07,143 INFO [collect_weather] 2020-07-01 2.5km weather forecast 06 collection complete
2020-07-01 03:03:07,148 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2020-07-01 03:03:07,800 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['Capilano', '--data-date', '2020-06-30'], host='localhost')
2020-07-01 03:03:07,809 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ChilliwackVedder', '--data-date', '2020-06-30'], host='localhost')
2020-07-01 03:03:07,814 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ClowhomClowhomLake', '--data-date', '2020-06-30'], host='localhost')
2020-07-01 03:03:07,817 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['Englishman', '--data-date', '2020-06-30'], host='localhost')
2020-07-01 03:03:07,821 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['Fraser', '--data-date', '2020-06-30'], host='localhost')
2020-07-01 03:03:07,825 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['HomathkoMouth', '--data-date', '2020-06-30'], host='localhost')
2020-07-01 03:03:07,830 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['SalmonSayward', '--data-date', '2020-06-30'], host='localhost')
2020-07-01 03:03:07,834 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['SanJuanPortRenfrew', '--data-date', '2020-06-30'], host='localhost')
2020-07-01 03:03:07,839 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['SquamishBrackendale', '--data-date', '2020-06-30'], host='localhost')
2020-07-01 03:03:07,853 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['TheodosiaScotty', '--data-date', '2020-06-30'], host='localhost')
2020-07-01 03:03:07,860 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['TheodosiaBypass', '--data-date', '2020-06-30'], host='localhost')
2020-07-01 03:03:07,865 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['TheodosiaDiversion', '--data-date', '2020-06-30'], host='localhost')
2020-07-01 03:03:07,974 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ctd', args=['SCVIP'], host='localhost')
2020-07-01 03:03:07,976 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ctd', args=['SEVIP'], host='localhost')
2020-07-01 03:03:07,983 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ferry', args=['TWDP'], host='localhost')
2020-07-01 03:03:07,999 INFO [manager] launching NextWorker(module='nowcast.workers.get_NeahBay_ssh', args=['forecast2'], host='localhost')
2020-07-01 03:03:08,006 INFO [manager] launching NextWorker(module='nowcast.workers.grib_to_netcdf', args=['forecast2'], host='localhost')
2020-07-01 03:03:08,011 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['12', '2.5km'], host='localhost')
2020-07-01 03:03:10,157 INFO [collect_weather] running in process 25469
2020-07-01 03:03:10,175 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-01 03:03:10,175 INFO [collect_weather] publishing log messages to tcp://*:5562
2020-07-01 03:03:10,177 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2020-07-01 03:03:10,245 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-west/12/
2020-07-01 03:03:11,097 INFO [grib_to_netcdf] running in process 25467
2020-07-01 03:03:11,097 INFO [grib_to_netcdf] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-01 03:03:11,098 INFO [grib_to_netcdf] publishing log messages to tcp://*:5563
2020-07-01 03:03:11,099 INFO [grib_to_netcdf] connected to skookum.eos.ubc.ca port 5555
2020-07-01 03:03:11,100 INFO [grib_to_netcdf] forecast section: 20200701/06
2020-07-01 03:03:11,100 INFO [grib_to_netcdf] next day forecast section: 20200701/06
2020-07-01 03:03:11,570 INFO [collect_river_data] running in process 25455
2020-07-01 03:03:11,571 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-01 03:03:11,571 INFO [collect_river_data] publishing log messages to tcp://*:5564
2020-07-01 03:03:11,572 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2020-07-01 03:03:11,573 INFO [collect_river_data] Collecting SanJuanPortRenfrew river data for 2020-06-30
2020-07-01 03:03:11,600 INFO [collect_river_data] running in process 25452
2020-07-01 03:03:11,633 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-01 03:03:11,634 INFO [collect_river_data] publishing log messages to tcp://*:5565
2020-07-01 03:03:11,635 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2020-07-01 03:03:11,635 INFO [collect_river_data] Collecting Fraser river data for 2020-06-30
2020-07-01 03:03:11,674 INFO [collect_river_data] running in process 25459
2020-07-01 03:03:11,675 INFO [collect_river_data] running in process 25448
2020-07-01 03:03:11,676 INFO [collect_river_data] running in process 25450
2020-07-01 03:03:11,677 INFO [collect_river_data] running in process 25449
2020-07-01 03:03:11,690 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-01 03:03:11,690 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-01 03:03:11,694 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-01 03:03:11,694 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-01 03:03:11,695 INFO [collect_river_data] publishing log messages to tcp://*:5566
2020-07-01 03:03:11,696 INFO [collect_river_data] publishing log messages to tcp://*:5567
2020-07-01 03:03:11,696 INFO [collect_river_data] publishing log messages to tcp://*:5568
2020-07-01 03:03:11,697 INFO [collect_river_data] publishing log messages to tcp://*:5569
2020-07-01 03:03:11,698 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2020-07-01 03:03:11,700 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2020-07-01 03:03:11,701 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2020-07-01 03:03:11,705 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2020-07-01 03:03:11,706 INFO [collect_river_data] Collecting TheodosiaScotty river data for 2020-06-30
2020-07-01 03:03:11,707 INFO [collect_river_data] Collecting Capilano river data for 2020-06-30
2020-07-01 03:03:11,707 INFO [collect_river_data] Collecting ClowhomClowhomLake river data for 2020-06-30
2020-07-01 03:03:11,708 INFO [collect_river_data] Collecting ChilliwackVedder river data for 2020-06-30
2020-07-01 03:03:11,719 INFO [collect_river_data] running in process 25460
2020-07-01 03:03:11,720 INFO [collect_river_data] running in process 25454
2020-07-01 03:03:11,721 INFO [collect_river_data] running in process 25461
2020-07-01 03:03:11,722 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-01 03:03:11,723 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-01 03:03:11,724 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-01 03:03:11,725 INFO [collect_river_data] publishing log messages to tcp://*:5572
2020-07-01 03:03:11,726 INFO [collect_river_data] publishing log messages to tcp://*:5570
2020-07-01 03:03:11,726 INFO [collect_river_data] publishing log messages to tcp://*:5571
2020-07-01 03:03:11,727 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2020-07-01 03:03:11,728 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2020-07-01 03:03:11,729 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2020-07-01 03:03:11,729 INFO [collect_river_data] Collecting TheodosiaDiversion river data for 2020-06-30
2020-07-01 03:03:11,730 INFO [collect_river_data] Collecting TheodosiaBypass river data for 2020-06-30
2020-07-01 03:03:11,730 INFO [collect_river_data] Collecting SalmonSayward river data for 2020-06-30
2020-07-01 03:03:11,731 INFO [collect_river_data] Appended SanJuanPortRenfrew river average discharge for 2020-06-30 to: /results/forcing/rivers/observations/SanJuan_PortRenfrew_flow
2020-07-01 03:03:11,732 INFO [collect_river_data] SanJuanPortRenfrew river average discharge for 2020-06-30 calculated and appended to SanJuanPortRenfrew_flow file
2020-07-01 03:03:11,735 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2020-07-01 03:03:11,741 INFO [collect_river_data] Appended Capilano river average discharge for 2020-06-30 to: /opp/observations/rivers/Capilano/Caplilano_08GA010_day_avg_flow
2020-07-01 03:03:11,742 INFO [collect_river_data] Capilano river average discharge for 2020-06-30 calculated and appended to Capilano_flow file
2020-07-01 03:03:11,920 INFO [collect_river_data] Appended TheodosiaScotty river average discharge for 2020-06-30 to: /results/forcing/rivers/observations/Theodosia_Scotty_flow
2020-07-01 03:03:11,921 INFO [collect_river_data] TheodosiaScotty river average discharge for 2020-06-30 calculated and appended to TheodosiaScotty_flow file
2020-07-01 03:03:11,958 INFO [collect_river_data] running in process 25451
2020-07-01 03:03:11,959 INFO [collect_river_data] running in process 25456
2020-07-01 03:03:11,987 INFO [collect_river_data] running in process 25453
2020-07-01 03:03:11,998 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-01 03:03:12,000 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-01 03:03:12,000 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-01 03:03:12,005 INFO [collect_river_data] publishing log messages to tcp://*:5573
2020-07-01 03:03:12,006 INFO [collect_river_data] publishing log messages to tcp://*:5574
2020-07-01 03:03:12,007 INFO [collect_river_data] publishing log messages to tcp://*:5575
2020-07-01 03:03:12,007 INFO [collect_river_data] Appended SalmonSayward river average discharge for 2020-06-30 to: /results/forcing/rivers/observations/Salmon_Sayward_flow
2020-07-01 03:03:12,009 INFO [collect_river_data] Appended TheodosiaDiversion river average discharge for 2020-06-30 to: /results/forcing/rivers/observations/Theodosia_Diversion_flow
2020-07-01 03:03:12,010 INFO [collect_river_data] Appended ChilliwackVedder river average discharge for 2020-06-30 to: /results/forcing/rivers/observations/Chilliwack_Vedder_flow
2020-07-01 03:03:12,011 INFO [collect_river_data] Appended TheodosiaBypass river average discharge for 2020-06-30 to: /results/forcing/rivers/observations/Theodosia_Bypass_flow
2020-07-01 03:03:12,012 INFO [collect_river_data] Appended ClowhomClowhomLake river average discharge for 2020-06-30 to: /results/forcing/rivers/observations/Clowhom_ClowhomLake_flow
2020-07-01 03:03:12,013 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2020-07-01 03:03:12,014 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2020-07-01 03:03:12,015 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2020-07-01 03:03:12,016 INFO [collect_river_data] SalmonSayward river average discharge for 2020-06-30 calculated and appended to SalmonSayward_flow file
2020-07-01 03:03:12,016 INFO [collect_river_data] TheodosiaDiversion river average discharge for 2020-06-30 calculated and appended to TheodosiaDiversion_flow file
2020-07-01 03:03:12,017 INFO [collect_river_data] ChilliwackVedder river average discharge for 2020-06-30 calculated and appended to ChilliwackVedder_flow file
2020-07-01 03:03:12,030 INFO [collect_river_data] TheodosiaBypass river average discharge for 2020-06-30 calculated and appended to TheodosiaBypass_flow file
2020-07-01 03:03:12,030 INFO [collect_river_data] ClowhomClowhomLake river average discharge for 2020-06-30 calculated and appended to ClowhomClowhomLake_flow file
2020-07-01 03:03:12,033 INFO [collect_river_data] Collecting Englishman river data for 2020-06-30
2020-07-01 03:03:12,034 INFO [collect_river_data] Collecting SquamishBrackendale river data for 2020-06-30
2020-07-01 03:03:12,035 INFO [collect_river_data] Collecting HomathkoMouth river data for 2020-06-30
2020-07-01 03:03:12,040 INFO [collect_river_data] Appended HomathkoMouth river average discharge for 2020-06-30 to: /results/forcing/rivers/observations/Homathko_Mouth_flow
2020-07-01 03:03:12,041 INFO [collect_river_data] HomathkoMouth river average discharge for 2020-06-30 calculated and appended to HomathkoMouth_flow file
2020-07-01 03:03:12,042 INFO [collect_river_data] Appended Fraser river average discharge for 2020-06-30 to: /data/dlatorne/SOG-projects/SOG-forcing/ECget/Fraser_flow
2020-07-01 03:03:12,044 INFO [collect_river_data] Fraser river average discharge for 2020-06-30 calculated and appended to Fraser_flow file
2020-07-01 03:03:12,052 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2020-07-01 03:03:12,055 INFO [collect_river_data] Appended SquamishBrackendale river average discharge for 2020-06-30 to: /results/forcing/rivers/observations/Squamish_Brackendale_flow
2020-07-01 03:03:12,056 INFO [collect_river_data] SquamishBrackendale river average discharge for 2020-06-30 calculated and appended to SquamishBrackendale_flow file
2020-07-01 03:03:12,118 INFO [collect_river_data] Appended Englishman river average discharge for 2020-06-30 to: /data/dlatorne/SOG-projects/SOG-forcing/ECget/Englishman_flow
2020-07-01 03:03:12,119 INFO [collect_river_data] Englishman river average discharge for 2020-06-30 calculated and appended to Englishman_flow file
2020-07-01 03:03:12,353 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2020-07-01 03:03:12,654 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2020-07-01 03:03:12,928 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2020-07-01 03:03:13,156 INFO [get_NeahBay_ssh] running in process 25466
2020-07-01 03:03:13,157 INFO [get_NeahBay_ssh] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-01 03:03:13,157 INFO [get_NeahBay_ssh] publishing log messages to tcp://*:5564
2020-07-01 03:03:13,158 INFO [get_NeahBay_ssh] connected to skookum.eos.ubc.ca port 5555
2020-07-01 03:03:13,200 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2020-07-01 03:03:13,445 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2020-07-01 03:03:13,534 INFO [get_onc_ctd] running in process 25462
2020-07-01 03:03:13,535 INFO [get_onc_ctd] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-01 03:03:13,536 INFO [get_onc_ctd] publishing log messages to tcp://*:5566
2020-07-01 03:03:13,537 INFO [get_onc_ctd] connected to skookum.eos.ubc.ca port 5555
2020-07-01 03:03:13,538 INFO [get_onc_ctd] requesting ONC SCVIP CTD T&S data for 2020-06-30
2020-07-01 03:03:13,559 INFO [get_onc_ferry] running in process 25464
2020-07-01 03:03:13,560 INFO [get_onc_ferry] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-01 03:03:13,561 INFO [get_onc_ferry] publishing log messages to tcp://*:5567
2020-07-01 03:03:13,561 INFO [get_onc_ferry] connected to skookum.eos.ubc.ca port 5555
2020-07-01 03:03:13,562 INFO [get_onc_ferry] requesting ONC TWDP.N1 NAV data for 2020-06-30
2020-07-01 03:03:13,563 INFO [get_onc_ctd] running in process 25463
2020-07-01 03:03:13,564 INFO [get_onc_ctd] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-01 03:03:13,564 INFO [get_onc_ctd] publishing log messages to tcp://*:5569
2020-07-01 03:03:13,565 INFO [get_onc_ctd] connected to skookum.eos.ubc.ca port 5555
2020-07-01 03:03:13,566 INFO [get_onc_ctd] requesting ONC SEVIP CTD T&S data for 2020-06-30
2020-07-01 03:03:13,698 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2020-07-01 03:03:13,947 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2020-07-01 03:03:14,195 INFO [manager] launching NextWorker(module='nowcast.workers.make_runoff_file', args=[], host='localhost')
2020-07-01 03:03:14,204 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2020-07-01 03:03:14,502 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2020-07-01 03:03:14,771 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2020-07-01 03:03:15,195 INFO [make_runoff_file] running in process 25579
2020-07-01 03:03:15,196 INFO [make_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-01 03:03:15,197 INFO [make_runoff_file] publishing log messages to tcp://*:5565
2020-07-01 03:03:15,198 INFO [make_runoff_file] connected to skookum.eos.ubc.ca port 5555
2020-07-01 03:03:15,719 INFO [make_runoff_file] calculating runoff forcing for bathymetry type: b201702
2020-07-01 03:03:15,806 INFO [make_runoff_file] 2020-07-01 runoff file creation from Fraser at Hope and climatology elsewhere complete
2020-07-01 03:03:15,808 INFO [manager] checklist updated with [rivers forcing] items from make_runoff_file worker
2020-07-01 03:03:17,671 INFO [get_NeahBay_ssh] Neah Bay sea surface height web scraping and open boundary file creation for forecast2 complete
2020-07-01 03:03:17,675 INFO [manager] checklist updated with [Neah Bay ssh] items from get_NeahBay_ssh worker
2020-07-01 03:03:37,527 INFO [get_onc_ctd] 2020-06-30 ONC SCVIP CTD T&S file created
2020-07-01 03:03:37,530 INFO [manager] checklist updated with [ONC CTD data] items from get_onc_ctd worker
2020-07-01 03:03:37,653 INFO [get_onc_ctd] 2020-06-30 ONC SEVIP CTD T&S file created
2020-07-01 03:03:37,826 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['SCVIP-CTD'], host='localhost')
2020-07-01 03:03:37,831 INFO [manager] checklist updated with [ONC CTD data] items from get_onc_ctd worker
2020-07-01 03:03:38,108 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['SEVIP-CTD'], host='localhost')
2020-07-01 03:03:38,413 INFO [get_onc_ferry] requesting ONC TWDP TSG data for 2020-06-30
2020-07-01 03:03:38,645 INFO [ping_erddap] running in process 25891
2020-07-01 03:03:38,646 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-01 03:03:38,646 INFO [ping_erddap] publishing log messages to tcp://*:5564
2020-07-01 03:03:38,647 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2020-07-01 03:03:38,649 INFO [ping_erddap] SCVIP-CTD ERDDAP dataset flag file(s) created
2020-07-01 03:03:38,650 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2020-07-01 03:03:38,914 INFO [ping_erddap] running in process 25893
2020-07-01 03:03:38,915 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-01 03:03:38,916 INFO [ping_erddap] publishing log messages to tcp://*:5565
2020-07-01 03:03:38,917 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2020-07-01 03:03:38,918 INFO [ping_erddap] SEVIP-CTD ERDDAP dataset flag file(s) created
2020-07-01 03:03:38,982 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2020-07-01 03:04:08,514 INFO [grib_to_netcdf] 2020-07-01 NEMO-atmos forcing file for forecast2 created
2020-07-01 03:04:08,518 INFO [manager] checklist updated with [weather forcing] items from grib_to_netcdf worker
2020-07-01 03:04:08,754 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'forecast2'], host='localhost')
2020-07-01 03:04:08,757 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'forecast2'], host='localhost')
2020-07-01 03:04:08,760 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['graham-hindcast', 'forecast2'], host='localhost')
2020-07-01 03:04:08,762 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'forecast2'], host='localhost')
2020-07-01 03:04:09,735 INFO [upload_forcing] running in process 26297
2020-07-01 03:04:09,736 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-01 03:04:09,737 INFO [upload_forcing] publishing log messages to tcp://*:5563
2020-07-01 03:04:09,737 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2020-07-01 03:04:09,738 INFO [upload_forcing] running in process 26295
2020-07-01 03:04:09,739 INFO [upload_forcing] running in process 26294
2020-07-01 03:04:09,739 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-01 03:04:09,740 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-01 03:04:09,740 INFO [upload_forcing] publishing log messages to tcp://*:5564
2020-07-01 03:04:09,741 INFO [upload_forcing] publishing log messages to tcp://*:5565
2020-07-01 03:04:09,742 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2020-07-01 03:04:09,742 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2020-07-01 03:04:09,743 INFO [upload_forcing] running in process 26296
2020-07-01 03:04:09,744 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-01 03:04:09,744 INFO [upload_forcing] publishing log messages to tcp://*:5566
2020-07-01 03:04:09,745 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2020-07-01 03:04:11,068 INFO [get_onc_ferry] requesting ONC TWDP OXYSENSOR data for 2020-06-30
2020-07-01 03:04:12,098 INFO [upload_forcing] LiveOcean boundary conditions file not found; created symlink to /results/forcing/LiveOcean/boundary_conditions/LiveOcean_v201905_y2020m06d30.nc
2020-07-01 03:04:12,220 INFO [upload_forcing] forecast2 2020-07-01 forcing files upload to optimum-hindcast completed
2020-07-01 03:04:12,224 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2020-07-01 03:04:12,425 INFO [upload_forcing] forecast2 2020-07-01 forcing files upload to orcinus-nowcast-agrif completed
2020-07-01 03:04:12,499 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2020-07-01 03:04:13,325 INFO [upload_forcing] forecast2 2020-07-01 forcing files upload to arbutus.cloud-nowcast completed
2020-07-01 03:04:13,329 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2020-07-01 03:04:13,662 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2020-07-01'], host='localhost')
2020-07-01 03:04:14,516 INFO [make_forcing_links] running in process 26318
2020-07-01 03:04:14,517 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-01 03:04:14,518 INFO [make_forcing_links] publishing log messages to tcp://*:5563
2020-07-01 03:04:14,519 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2020-07-01 03:04:15,776 INFO [make_forcing_links] forecast2 2020-07-01 forcing file links on arbutus.cloud-nowcast created
2020-07-01 03:04:15,780 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2020-07-01 03:04:16,114 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2020-07-01'], host='arbutus.cloud-nowcast')
2020-07-01 03:04:17,500 INFO [upload_forcing] forecast2 2020-07-01 forcing files upload to graham-hindcast completed
2020-07-01 03:04:17,505 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2020-07-01 03:04:18,556 INFO [run_NEMO] running in process 29217
2020-07-01 03:04:18,558 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2020-07-01 03:04:18,559 INFO [run_NEMO] publishing log messages to tcp://*:5556
2020-07-01 03:04:18,566 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2020-07-01 03:04:19,805 INFO [run_NEMO] forecast2: launching /nemoShare/MEOPAR/nowcast-sys/runs/30jun20forecast2_2020-07-01T030418.658015-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2020-07-01 03:04:19,823 INFO [run_NEMO] forecast2 NEMO run for 2020-07-01 on arbutus.cloud-nowcast started
2020-07-01 03:04:19,829 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2020-07-01 03:04:20,160 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'forecast2'], host='arbutus.cloud-nowcast')
2020-07-01 03:04:22,169 INFO [watch_NEMO] running in process 29482
2020-07-01 03:04:22,170 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2020-07-01 03:04:22,171 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2020-07-01 03:04:22,172 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2020-07-01 03:04:22,295 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2020-07-01 03:04:57,318 INFO [get_onc_ferry] requesting ONC TWDP TURBCHLFL data for 2020-06-30
2020-07-01 03:05:27,888 INFO [get_onc_ferry] requesting ONC TWDP CO2SENSOR data for 2020-06-30
2020-07-01 03:05:41,340 INFO [get_onc_ferry] requesting ONC TWDP TEMPHUMID data for 2020-06-30
2020-07-01 03:06:04,704 INFO [get_onc_ferry] requesting ONC TWDP BARPRESS data for 2020-06-30
2020-07-01 03:06:16,702 INFO [get_onc_ferry] requesting ONC TWDP PYRANOMETER data for 2020-06-30
2020-07-01 03:06:27,656 INFO [get_onc_ferry] requesting ONC TWDP PYRGEOMETER data for 2020-06-30
2020-07-01 03:06:41,740 INFO [get_onc_ferry] 2020-06-30 ONC TWDP ferry data file created
2020-07-01 03:06:41,746 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['TWDP-ferry'], host='localhost')
2020-07-01 03:06:42,862 INFO [ping_erddap] running in process 26384
2020-07-01 03:06:42,863 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-01 03:06:42,864 INFO [ping_erddap] publishing log messages to tcp://*:5563
2020-07-01 03:06:42,865 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2020-07-01 03:06:42,866 INFO [ping_erddap] TWDP-ferry ERDDAP dataset flag file(s) created
2020-07-01 03:06:42,867 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2020-07-01 03:09:22,384 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 7493488 = 2020-07-02 04:58:00 UTC, 16.6% complete
2020-07-01 03:14:22,395 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 7493991 = 2020-07-02 10:33:20 UTC, 35.2% complete
2020-07-01 03:19:22,470 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 7494494 = 2020-07-02 16:08:40 UTC, 53.8% complete
2020-07-01 03:24:22,571 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 7494986 = 2020-07-02 21:36:40 UTC, 72.1% complete
2020-07-01 03:29:22,673 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 7495474 = 2020-07-03 03:02:00 UTC, 90.1% complete
2020-07-01 03:34:22,706 INFO [watch_NEMO] forecast2 NEMO run on arbutus.cloud-nowcast completed
2020-07-01 03:34:22,711 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2020-07-01 03:34:23,323 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_wind_file', args=['arbutus.cloud-nowcast', 'forecast2'], host='arbutus.cloud-nowcast')
2020-07-01 03:34:23,335 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_current_file', args=['arbutus.cloud-nowcast', 'forecast2'], host='arbutus.cloud-nowcast')
2020-07-01 03:34:23,339 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2020-06-30'], host='localhost')
2020-07-01 03:34:24,600 INFO [download_results] running in process 26945
2020-07-01 03:34:24,601 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-01 03:34:24,602 INFO [download_results] publishing log messages to tcp://*:5563
2020-07-01 03:34:24,602 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2020-07-01 03:34:24,603 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/forecast2/30jun20 to /results/SalishSea/forecast2.201905
2020-07-01 03:34:26,598 INFO [make_ww3_current_file] running in process 29709
2020-07-01 03:34:26,599 INFO [make_ww3_current_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2020-07-01 03:34:26,600 INFO [make_ww3_current_file] publishing log messages to tcp://*:5571
2020-07-01 03:34:26,601 INFO [make_ww3_current_file] connected to skookum.eos.ubc.ca port 5555
2020-07-01 03:34:26,602 INFO [make_ww3_current_file] Creating wwatch3 currents forcing file for 2020-07-01 forecast2 run
2020-07-01 03:34:26,646 INFO [make_ww3_wind_file] running in process 29707
2020-07-01 03:34:26,647 INFO [make_ww3_wind_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2020-07-01 03:34:26,648 INFO [make_ww3_wind_file] publishing log messages to tcp://*:5570
2020-07-01 03:34:26,649 INFO [make_ww3_wind_file] connected to skookum.eos.ubc.ca port 5555
2020-07-01 03:34:26,650 INFO [make_ww3_wind_file] Creating wwatch3 wind forcing file for 2020-07-01 forecast2 run
2020-07-01 03:34:30,094 INFO [make_ww3_wind_file] wwatch3 wind forcing file created on arbutus.cloud-nowcast for 2020-07-01 forecast2 run
2020-07-01 03:34:30,099 INFO [manager] checklist updated with [WW3 wind forcing] items from make_ww3_wind_file worker
2020-07-01 03:36:19,296 INFO [make_ww3_current_file] wwatch3 currents forcing file created on arbutus.cloud-nowcast for 2020-07-01 forecast2 run
2020-07-01 03:36:19,299 INFO [manager] checklist updated with [WW3 currents forcing] items from make_ww3_current_file worker
2020-07-01 03:36:19,640 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2020-07-01'], host='arbutus.cloud-nowcast')
2020-07-01 03:36:21,657 INFO [run_ww3] running in process 29935
2020-07-01 03:36:21,658 INFO [run_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2020-07-01 03:36:21,659 INFO [run_ww3] publishing log messages to tcp://*:5572
2020-07-01 03:36:21,660 INFO [run_ww3] connected to skookum.eos.ubc.ca port 5555
2020-07-01 03:36:21,663 INFO [run_ww3] Created run directory /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast2_2020-07-01T033621.656214-0700
2020-07-01 03:36:21,665 INFO [run_ww3] forecast2: launching /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast2_2020-07-01T033621.656214-0700/SoGWW3.sh on arbutus.cloud-nowcast
2020-07-01 03:36:21,679 INFO [run_ww3] forecast2 WaveWatch3 run for 2020-07-01 on arbutus.cloud-nowcast started
2020-07-01 03:36:21,685 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker
2020-07-01 03:36:22,010 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'forecast2'], host='arbutus.cloud-nowcast')
2020-07-01 03:36:23,939 INFO [watch_ww3] running in process 30038
2020-07-01 03:36:23,940 INFO [watch_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2020-07-01 03:36:23,941 INFO [watch_ww3] publishing log messages to tcp://*:5573
2020-07-01 03:36:23,942 INFO [watch_ww3] connected to skookum.eos.ubc.ca port 5555
2020-07-01 03:36:23,975 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2020-07-01 03:37:32,265 INFO [download_results] forecast2 2020-06-30 results files from arbutus.cloud-nowcast downloaded
2020-07-01 03:37:32,272 INFO [manager] checklist updated with [results files] items from download_results worker
2020-07-01 03:37:32,606 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['forecast2', '--run-date', '2020-06-30'], host='localhost')
2020-07-01 03:37:36,337 INFO [make_CHS_currents_file] running in process 27002
2020-07-01 03:37:36,338 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-01 03:37:36,339 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5563
2020-07-01 03:37:36,340 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555
2020-07-01 03:39:12,853 INFO [make_CHS_currents_file] Made CHS currents file for 2020-06-30 for forecast2
2020-07-01 03:39:12,858 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker
2020-07-01 03:39:13,191 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['nemo', 'forecast2', '--run-date', '2020-06-30'], host='localhost')
2020-07-01 03:39:14,219 INFO [update_forecast_datasets] running in process 27112
2020-07-01 03:39:14,220 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-01 03:39:14,221 INFO [update_forecast_datasets] publishing log messages to tcp://*:5563
2020-07-01 03:39:14,222 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2020-07-01 03:39:14,223 INFO [update_forecast_datasets] updating nemo forecast directory for forecast2/30jun20 run
2020-07-01 03:41:24,076 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 90, 16.7% complete
2020-07-01 03:43:19,610 INFO [update_forecast_datasets] updated nemo forecast directory for forecast2/30jun20 run: /results/SalishSea/rolling-forecasts/nemo
2020-07-01 03:43:19,611 INFO [update_forecast_datasets] nemo 2020-06-30 forecast2 rolling forecast datasets updated
2020-07-01 03:43:19,615 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2020-07-01 03:43:19,951 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nemo-forecast'], host='localhost')
2020-07-01 03:43:19,954 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'forecast2', 'publish', '--run-date', '2020-06-30'], host='localhost')
2020-07-01 03:43:19,956 INFO [manager] launching NextWorker(module='nowcast.workers.make_surface_current_tiles', args=['forecast2', '--run-date', '2020-06-30'], host='localhost')
2020-07-01 03:43:21,039 INFO [ping_erddap] running in process 27271
2020-07-01 03:43:21,040 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-01 03:43:21,041 INFO [ping_erddap] publishing log messages to tcp://*:5563
2020-07-01 03:43:21,042 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2020-07-01 03:43:21,078 INFO [ping_erddap] nemo-forecast ERDDAP dataset flag file(s) created
2020-07-01 03:43:21,080 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2020-07-01 03:43:21,352 INFO [make_surface_current_tiles] running in process 27273
2020-07-01 03:43:21,353 INFO [make_surface_current_tiles] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-01 03:43:21,354 INFO [make_surface_current_tiles] publishing log messages to tcp://*:5564
2020-07-01 03:43:21,354 INFO [make_surface_current_tiles] connected to skookum.eos.ubc.ca port 5555
2020-07-01 03:43:21,409 INFO [make_surface_current_tiles] starting rendering of tiles for 2020-06-30 forecast2 in 6 processes into /results/nowcast-sys/figures/surface_currents/forecast2/30jun20
2020-07-01 03:43:27,442 INFO [make_plots] running in process 27272
2020-07-01 03:43:27,443 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-01 03:43:27,444 INFO [make_plots] publishing log messages to tcp://*:5563
2020-07-01 03:43:27,445 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2020-07-01 03:43:28,325 INFO [make_plots] preparing render list for 2020-06-30 NEMO forecast2 publish figures
2020-07-01 03:43:28,518 INFO [make_plots] added Website_thumbnail figure to 2020-06-30 NEMO forecast2 publish render list
2020-07-01 03:43:28,519 INFO [make_plots] added Threshold_website figure to 2020-06-30 NEMO forecast2 publish render list
2020-07-01 03:43:28,520 INFO [make_plots] added PA_tidal_predictions figure to 2020-06-30 NEMO forecast2 publish render list
2020-07-01 03:43:28,521 INFO [make_plots] added Boundary Bay figure to 2020-06-30 NEMO forecast2 publish render list
2020-07-01 03:43:28,522 INFO [make_plots] added Campbell River figure to 2020-06-30 NEMO forecast2 publish render list
2020-07-01 03:43:28,523 INFO [make_plots] added Cherry Point figure to 2020-06-30 NEMO forecast2 publish render list
2020-07-01 03:43:28,524 INFO [make_plots] added Friday Harbor figure to 2020-06-30 NEMO forecast2 publish render list
2020-07-01 03:43:28,525 INFO [make_plots] added Halfmoon Bay figure to 2020-06-30 NEMO forecast2 publish render list
2020-07-01 03:43:28,526 INFO [make_plots] added Nanaimo figure to 2020-06-30 NEMO forecast2 publish render list
2020-07-01 03:43:28,527 INFO [make_plots] added Neah Bay figure to 2020-06-30 NEMO forecast2 publish render list
2020-07-01 03:43:28,528 INFO [make_plots] added New Westminster figure to 2020-06-30 NEMO forecast2 publish render list
2020-07-01 03:43:28,529 INFO [make_plots] added Patricia Bay figure to 2020-06-30 NEMO forecast2 publish render list
2020-07-01 03:43:28,529 INFO [make_plots] added Point Atkinson figure to 2020-06-30 NEMO forecast2 publish render list
2020-07-01 03:43:28,530 INFO [make_plots] added Port Renfrew figure to 2020-06-30 NEMO forecast2 publish render list
2020-07-01 03:43:28,531 INFO [make_plots] added Sand Heads figure to 2020-06-30 NEMO forecast2 publish render list
2020-07-01 03:43:28,532 INFO [make_plots] added Sandy Cove figure to 2020-06-30 NEMO forecast2 publish render list
2020-07-01 03:43:28,532 INFO [make_plots] added Squamish figure to 2020-06-30 NEMO forecast2 publish render list
2020-07-01 03:43:28,533 INFO [make_plots] added Victoria figure to 2020-06-30 NEMO forecast2 publish render list
2020-07-01 03:43:28,534 INFO [make_plots] added Woodwards Landing figure to 2020-06-30 NEMO forecast2 publish render list
2020-07-01 03:43:28,535 INFO [make_plots] starting to render nemo forecast2 publish 30jun20 figures
2020-07-01 03:43:59,409 INFO [make_plots] Note: NumExpr detected 12 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
2020-07-01 03:43:59,409 INFO [make_plots] NumExpr defaulting to 8 threads.
2020-07-01 03:46:24,179 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 198, 36.7% complete
2020-07-01 03:51:24,278 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 306, 56.7% complete
2020-07-01 03:56:24,378 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 417, 77.2% complete
2020-07-01 04:01:13,795 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 08074 Campbell River from 2020-06-26 00:00:00 to 2020-07-03 05:55:00
2020-07-01 04:01:24,480 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 525, 97.2% complete
2020-07-01 04:04:48,301 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 9449424 Cherry Point from 2020-06-26 00:00:00 to 2020-07-03 05:55:00
2020-07-01 04:05:50,382 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 9449880 Friday Harbor from 2020-06-26 00:00:00 to 2020-07-03 05:55:00
2020-07-01 04:06:24,585 INFO [watch_ww3] forecast2 WWATCH3 run on arbutus.cloud-nowcast completed
2020-07-01 04:06:24,587 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2020-07-01 04:06:25,749 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2020-07-01'], host='localhost')
2020-07-01 04:06:27,500 INFO [download_wwatch3_results] running in process 29658
2020-07-01 04:06:27,501 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-01 04:06:27,502 INFO [download_wwatch3_results] publishing log messages to tcp://*:5565
2020-07-01 04:06:27,503 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2020-07-01 04:06:49,470 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07830 Halfmoon Bay from 2020-06-26 00:00:00 to 2020-07-03 05:55:00
2020-07-01 04:07:09,493 INFO [download_wwatch3_results] forecast2 2020-07-01 results files from arbutus.cloud-nowcast downloaded
2020-07-01 04:07:09,497 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2020-07-01 04:07:10,435 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['wwatch3', 'forecast2', '--run-date', '2020-07-01'], host='localhost')
2020-07-01 04:07:12,256 INFO [update_forecast_datasets] running in process 29778
2020-07-01 04:07:12,258 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-01 04:07:12,258 INFO [update_forecast_datasets] publishing log messages to tcp://*:5565
2020-07-01 04:07:12,260 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2020-07-01 04:07:12,260 INFO [update_forecast_datasets] updating wwatch3 most_recent_forecast directory from forecast2/01jul20 run
2020-07-01 04:07:12,262 INFO [update_forecast_datasets] updated wwatch3 most_recent_forecast directory from forecast2/01jul20 run
2020-07-01 04:07:12,263 INFO [update_forecast_datasets] updating wwatch3 forecast directory for forecast2/01jul20 run
2020-07-01 04:07:51,601 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07917 Nanaimo from 2020-06-26 00:00:00 to 2020-07-03 05:55:00
2020-07-01 04:08:12,354 INFO [update_forecast_datasets] updated wwatch3 forecast directory for forecast2/01jul20 run: /results/SalishSea/rolling-forecasts/wwatch3
2020-07-01 04:08:12,355 INFO [update_forecast_datasets] wwatch3 2020-07-01 forecast2 rolling forecast datasets updated
2020-07-01 04:08:12,360 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2020-07-01 04:08:13,271 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['wwatch3-forecast'], host='localhost')
2020-07-01 04:08:15,417 INFO [ping_erddap] running in process 29851
2020-07-01 04:08:15,418 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-01 04:08:15,419 INFO [ping_erddap] publishing log messages to tcp://*:5565
2020-07-01 04:08:15,420 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2020-07-01 04:08:15,427 INFO [ping_erddap] wwatch3-forecast ERDDAP dataset flag file(s) created
2020-07-01 04:08:15,428 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2020-07-01 04:08:16,497 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['wwatch3', 'forecast2', 'publish', '--run-date', '2020-07-01'], host='localhost')
2020-07-01 04:08:26,377 INFO [make_plots] running in process 29855
2020-07-01 04:08:26,378 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-01 04:08:26,379 INFO [make_plots] publishing log messages to tcp://*:5565
2020-07-01 04:08:26,379 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2020-07-01 04:08:26,380 INFO [make_plots] preparing render list for 2020-07-01 WaveWatch3 forecast2 publish figures
2020-07-01 04:08:26,381 INFO [make_plots] added Halibut Bank figure to 2020-07-01 WaveWatch3 forecast2 publish render list
2020-07-01 04:08:26,382 INFO [make_plots] added Sentry Shoal figure to 2020-07-01 WaveWatch3 forecast2 publish render list
2020-07-01 04:08:26,382 INFO [make_plots] starting to render wwatch3 forecast2 publish 01jul20 figures
2020-07-01 04:09:37,860 INFO [make_plots] Note: NumExpr detected 12 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
2020-07-01 04:09:37,861 INFO [make_plots] NumExpr defaulting to 8 threads.
2020-07-01 04:09:37,872 INFO [make_plots] retrieving last 45 days of real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46146 Halibut Bank
2020-07-01 04:10:49,877 INFO [make_plots] retrieving last 45 days of real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46131 Sentry Shoal
2020-07-01 04:10:53,803 INFO [make_plots] finished rendering wwatch3 forecast2 publish 01jul20 figures
2020-07-01 04:10:53,808 INFO [make_plots] wwatch3 publish plots for 2020-07-01 forecast2 completed
2020-07-01 04:10:53,810 INFO [manager] checklist updated with [plots] items from make_plots worker
2020-07-01 04:11:46,126 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 9443090 Neah Bay from 2020-06-26 00:00:00 to 2020-07-03 05:55:00
2020-07-01 04:12:34,016 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07654 New Westminster from 2020-06-26 00:00:00 to 2020-07-03 05:55:00
2020-07-01 04:12:47,486 INFO [make_surface_current_tiles] starting PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast2/30jun20
2020-07-01 04:14:42,102 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07277 Patricia Bay from 2020-06-26 00:00:00 to 2020-07-03 05:55:00
2020-07-01 04:16:48,900 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07795 Point Atkinson from 2020-06-26 00:00:00 to 2020-07-03 05:55:00
2020-07-01 04:17:07,320 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 08525 Port Renfrew from 2020-06-26 00:00:00 to 2020-07-03 05:55:00
2020-07-01 04:17:12,976 INFO [make_surface_current_tiles] finished PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast2/30jun20
2020-07-01 04:17:12,997 INFO [make_surface_current_tiles] finished rendering of tiles for 2020-06-30 forecast2 into /results/nowcast-sys/figures/surface_currents/forecast2/30jun20
2020-07-01 04:17:12,998 INFO [make_surface_current_tiles] surface current tile figures for 2020-06-30 forecast2 completed
2020-07-01 04:17:13,479 INFO [manager] checklist updated with [surface current tiles] items from make_surface_current_tiles worker
2020-07-01 04:19:15,068 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07594 Sand Heads from 2020-06-26 00:00:00 to 2020-07-03 05:55:00
2020-07-01 04:21:20,811 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07786 Sandy Cove from 2020-06-26 00:00:00 to 2020-07-03 05:55:00
2020-07-01 04:23:26,188 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07811 Squamish from 2020-06-26 00:00:00 to 2020-07-03 05:55:00
2020-07-01 04:23:43,954 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07120 Victoria from 2020-06-26 00:00:00 to 2020-07-03 05:55:00
2020-07-01 04:25:48,213 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07610 Woodwards Landing from 2020-06-26 00:00:00 to 2020-07-03 05:55:00
2020-07-01 04:27:45,362 INFO [make_plots] finished rendering nemo forecast2 publish 30jun20 figures
2020-07-01 04:27:45,364 INFO [make_plots] nemo publish plots for 2020-06-30 forecast2 completed
2020-07-01 04:27:45,374 INFO [manager] checklist updated with [plots] items from make_plots worker
2020-07-01 04:27:45,769 INFO [manager] launching NextWorker(module='nowcast.workers.make_feeds', args=['forecast2', '--run-date', '2020-06-30'], host='localhost')
2020-07-01 04:27:50,663 INFO [make_feeds] running in process 30531
2020-07-01 04:27:50,664 INFO [make_feeds] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-01 04:27:50,666 INFO [make_feeds] publishing log messages to tcp://*:5563
2020-07-01 04:27:50,667 INFO [make_feeds] connected to skookum.eos.ubc.ca port 5555
2020-07-01 04:27:58,989 INFO [make_feeds] Note: NumExpr detected 12 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
2020-07-01 04:27:58,990 INFO [make_feeds] NumExpr defaulting to 8 threads.
2020-07-01 04:28:04,894 INFO [make_feeds] ATOM feeds for 2020-06-30 forecast2 run completed
2020-07-01 04:28:04,897 INFO [manager] checklist updated with [storm surge alert feeds] items from make_feeds worker
2020-07-01 04:28:05,300 INFO [manager] launching NextWorker(module='nemo_nowcast.workers.clear_checklist', args=[], host='localhost')
2020-07-01 04:28:06,257 INFO [clear_checklist] running in process 30563
2020-07-01 04:28:06,258 INFO [clear_checklist] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-01 04:28:06,259 INFO [clear_checklist] publishing log messages to tcp://*:5563
2020-07-01 04:28:06,260 INFO [clear_checklist] connected to skookum.eos.ubc.ca port 5555
2020-07-01 04:28:06,261 INFO [clear_checklist] requesting that manager clear system state checklist
2020-07-01 04:28:06,263 INFO [manager] writing checklist to log file
2020-07-01 04:28:06,342 INFO [manager] checklist cleared
2020-07-01 04:28:06,345 INFO [clear_checklist] nowcast system checklist cleared
2020-07-01 04:28:06,348 INFO [manager] launching NextWorker(module='nemo_nowcast.workers.rotate_logs', args=[], host='localhost')
2020-07-01 04:28:07,418 INFO [rotate_logs] running in process 30567
2020-07-01 04:28:07,420 INFO [rotate_logs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2020-07-01 04:28:07,421 INFO [rotate_logs] publishing log messages to tcp://*:5563
2020-07-01 04:28:07,421 INFO [rotate_logs] connected to skookum.eos.ubc.ca port 5555
2020-07-01 04:28:07,422 INFO [rotate_logs] rotating log files