2020-07-05 04:48:26,561 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/nowcast.log 2020-07-05 04:48:26,567 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/nowcast.debug.log 2020-07-05 04:48:26,572 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/checklist.log 2020-07-05 04:48:26,573 INFO [rotate_logs] log files rotated 2020-07-05 04:48:26,577 INFO [manager] checklist updated with [log rotation] items from rotate_logs worker 2020-07-05 04:51:28,004 CRITICAL [make_plots] unhandled exception: Traceback (most recent call last): File "/SalishSeaCast/NEMO_Nowcast/nemo_nowcast/worker.py", line 390, in _do_work checklist = self.worker_func( File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 325, in make_plots checklist = _render_figures( File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1038, in _render_figures fig = _calc_figure(fig_func, args, kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/publish/tide_stn_water_level.py", line 78, in make_figure plot_data = _prep_plot_data(place, fvcom_ssh_datasets, nemo_ssh_dataset_url_tmpl) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/publish/tide_stn_water_level.py", line 144, in _prep_plot_data fvcom_residuals[model_config] = fvcom_ssh - pred.water_level File "/SalishSeaCast/nowcast-env/lib/python3.8/site-packages/xarray/core/dataarray.py", line 2588, in func self, other = align(self, other, join=align_type, copy=False) File "/SalishSeaCast/nowcast-env/lib/python3.8/site-packages/xarray/core/alignment.py", line 331, in align new_obj = obj.reindex(copy=copy, fill_value=fill_value, **valid_indexers) File "/SalishSeaCast/nowcast-env/lib/python3.8/site-packages/xarray/core/dataarray.py", line 1295, in reindex ds = self._to_temp_dataset().reindex( File "/SalishSeaCast/nowcast-env/lib/python3.8/site-packages/xarray/core/dataset.py", line 2476, in reindex return self._reindex( File "/SalishSeaCast/nowcast-env/lib/python3.8/site-packages/xarray/core/dataset.py", line 2505, in _reindex variables, indexes = alignment.reindex_variables( File "/SalishSeaCast/nowcast-env/lib/python3.8/site-packages/xarray/core/alignment.py", line 541, in reindex_variables raise ValueError( ValueError: cannot reindex or align along dimension 'time' because the index has duplicate values 2020-07-05 04:52:23,346 CRITICAL [make_plots] unhandled exception: Traceback (most recent call last): File "/SalishSeaCast/NEMO_Nowcast/nemo_nowcast/worker.py", line 390, in _do_work checklist = self.worker_func( File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 325, in make_plots checklist = _render_figures( File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1038, in _render_figures fig = _calc_figure(fig_func, args, kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1105, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/publish/tide_stn_water_level.py", line 78, in make_figure plot_data = _prep_plot_data(place, fvcom_ssh_datasets, nemo_ssh_dataset_url_tmpl) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/fvcom/publish/tide_stn_water_level.py", line 144, in _prep_plot_data fvcom_residuals[model_config] = fvcom_ssh - pred.water_level File "/SalishSeaCast/nowcast-env/lib/python3.8/site-packages/xarray/core/dataarray.py", line 2588, in func self, other = align(self, other, join=align_type, copy=False) File "/SalishSeaCast/nowcast-env/lib/python3.8/site-packages/xarray/core/alignment.py", line 331, in align new_obj = obj.reindex(copy=copy, fill_value=fill_value, **valid_indexers) File "/SalishSeaCast/nowcast-env/lib/python3.8/site-packages/xarray/core/dataarray.py", line 1295, in reindex ds = self._to_temp_dataset().reindex( File "/SalishSeaCast/nowcast-env/lib/python3.8/site-packages/xarray/core/dataset.py", line 2476, in reindex return self._reindex( File "/SalishSeaCast/nowcast-env/lib/python3.8/site-packages/xarray/core/dataset.py", line 2505, in _reindex variables, indexes = alignment.reindex_variables( File "/SalishSeaCast/nowcast-env/lib/python3.8/site-packages/xarray/core/alignment.py", line 541, in reindex_variables raise ValueError( ValueError: cannot reindex or align along dimension 'time' because the index has duplicate values 2020-07-05 05:09:51,764 INFO [make_surface_current_tiles] finished PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast2/04jul20 2020-07-05 05:09:51,783 INFO [make_surface_current_tiles] finished rendering of tiles for 2020-07-04 forecast2 into /results/nowcast-sys/figures/surface_currents/forecast2/04jul20 2020-07-05 05:09:51,785 INFO [make_surface_current_tiles] surface current tile figures for 2020-07-04 forecast2 completed 2020-07-05 05:09:52,288 INFO [manager] checklist updated with [surface current tiles] items from make_surface_current_tiles worker 2020-07-05 09:06:03,385 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-west/12/ to /results/forcing/atmospheric/GEM2.5/GRIB/20200705/12/ 2020-07-05 09:06:03,387 INFO [collect_weather] 2020-07-05 2.5km weather forecast 12 collection complete 2020-07-05 09:06:03,391 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker 2020-07-05 09:06:03,913 INFO [manager] launching NextWorker(module='nowcast.workers.get_NeahBay_ssh', args=['nowcast'], host='localhost') 2020-07-05 09:06:03,919 INFO [manager] launching NextWorker(module='nowcast.workers.grib_to_netcdf', args=['nowcast+'], host='localhost') 2020-07-05 09:06:03,923 INFO [manager] launching NextWorker(module='nowcast.workers.download_live_ocean', args=[], host='localhost') 2020-07-05 09:06:03,927 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['18', '2.5km'], host='localhost') 2020-07-05 09:06:07,783 INFO [collect_weather] running in process 26807 2020-07-05 09:06:07,823 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-05 09:06:07,824 INFO [collect_weather] publishing log messages to tcp://*:5562 2020-07-05 09:06:07,933 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555 2020-07-05 09:06:08,064 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-west/18/ 2020-07-05 09:06:08,568 INFO [download_live_ocean] running in process 26806 2020-07-05 09:06:08,585 INFO [download_live_ocean] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-05 09:06:08,608 INFO [download_live_ocean] publishing log messages to tcp://*:5564 2020-07-05 09:06:08,609 INFO [download_live_ocean] connected to skookum.eos.ubc.ca port 5555 2020-07-05 09:06:08,609 INFO [download_live_ocean] downloading Salish Sea western boundary daily averaged Live Ocean file for 2020-07-05 2020-07-05 09:06:08,661 INFO [grib_to_netcdf] running in process 26805 2020-07-05 09:06:08,662 INFO [grib_to_netcdf] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-05 09:06:08,666 INFO [grib_to_netcdf] publishing log messages to tcp://*:5567 2020-07-05 09:06:08,667 INFO [grib_to_netcdf] connected to skookum.eos.ubc.ca port 5555 2020-07-05 09:06:14,247 INFO [get_NeahBay_ssh] running in process 26804 2020-07-05 09:06:14,248 INFO [get_NeahBay_ssh] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-05 09:06:14,249 INFO [get_NeahBay_ssh] publishing log messages to tcp://*:5568 2020-07-05 09:06:14,307 INFO [get_NeahBay_ssh] connected to skookum.eos.ubc.ca port 5555 2020-07-05 09:06:22,214 INFO [download_live_ocean] Deflating in up to 6 concurrent sub-processes 2020-07-05 09:06:24,764 INFO [download_live_ocean] netCDF4 deflated /results/forcing/LiveOcean/downloaded/20200705/low_passed_UBC.nc 2020-07-05 09:06:24,765 INFO [download_live_ocean] 2020-07-05 Live Ocean file for Salish Sea western boundary downloaded 2020-07-05 09:06:24,768 INFO [manager] checklist updated with [Live Ocean products] items from download_live_ocean worker 2020-07-05 09:06:24,890 INFO [get_NeahBay_ssh] Neah Bay sea surface height web scraping and open boundary file creation for nowcast complete 2020-07-05 09:06:25,443 INFO [manager] launching NextWorker(module='nowcast.workers.make_live_ocean_files', args=['--run-date', '2020-07-05'], host='localhost') 2020-07-05 09:06:25,454 INFO [manager] checklist updated with [Neah Bay ssh] items from get_NeahBay_ssh worker 2020-07-05 09:06:32,216 INFO [make_live_ocean_files] running in process 26887 2020-07-05 09:06:32,218 INFO [make_live_ocean_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-05 09:06:32,220 INFO [make_live_ocean_files] publishing log messages to tcp://*:5564 2020-07-05 09:06:32,222 INFO [make_live_ocean_files] connected to skookum.eos.ubc.ca port 5555 2020-07-05 09:06:32,224 INFO [make_live_ocean_files] Creating T&S western boundary conditions file from 2020-07-05 Live Ocean run 2020-07-05 09:07:31,201 INFO [make_live_ocean_files] Stored T&S western boundary conditions file: / 2020-07-05 09:07:31,202 INFO [make_live_ocean_files] 2020-07-05 Live Ocean western boundary conditions files created 2020-07-05 09:07:31,205 INFO [manager] checklist updated with [Live Ocean boundary conditions] items from make_live_ocean_files worker 2020-07-05 09:11:10,058 INFO [grib_to_netcdf] 2020-07-05 NEMO-atmos forcing file for nowcast+ created 2020-07-05 09:11:10,062 INFO [manager] checklist updated with [weather forcing] items from grib_to_netcdf worker 2020-07-05 09:11:10,238 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'nowcast+'], host='localhost') 2020-07-05 09:11:10,242 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'nowcast+'], host='localhost') 2020-07-05 09:11:10,245 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['graham-hindcast', 'nowcast+'], host='localhost') 2020-07-05 09:11:10,249 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'nowcast+'], host='localhost') 2020-07-05 09:11:10,252 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['weather'], host='localhost') 2020-07-05 09:11:11,509 INFO [ping_erddap] running in process 28329 2020-07-05 09:11:11,510 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-05 09:11:11,511 INFO [ping_erddap] publishing log messages to tcp://*:5564 2020-07-05 09:11:11,512 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555 2020-07-05 09:11:11,514 INFO [ping_erddap] weather ERDDAP dataset flag file(s) created 2020-07-05 09:11:11,517 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker 2020-07-05 09:11:11,737 INFO [upload_forcing] running in process 28328 2020-07-05 09:11:11,738 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-05 09:11:11,739 INFO [upload_forcing] publishing log messages to tcp://*:5567 2020-07-05 09:11:11,740 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2020-07-05 09:11:11,761 INFO [upload_forcing] running in process 28325 2020-07-05 09:11:11,762 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-05 09:11:11,763 INFO [upload_forcing] publishing log messages to tcp://*:5568 2020-07-05 09:11:11,764 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2020-07-05 09:11:11,772 INFO [upload_forcing] running in process 28326 2020-07-05 09:11:11,773 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-05 09:11:11,774 INFO [upload_forcing] publishing log messages to tcp://*:5569 2020-07-05 09:11:11,775 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2020-07-05 09:11:11,776 INFO [upload_forcing] running in process 28327 2020-07-05 09:11:11,778 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-05 09:11:11,779 INFO [upload_forcing] publishing log messages to tcp://*:5570 2020-07-05 09:11:11,780 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2020-07-05 09:11:16,294 INFO [upload_forcing] nowcast+ 2020-07-05 forcing files upload to optimum-hindcast completed 2020-07-05 09:11:16,297 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2020-07-05 09:11:16,438 INFO [upload_forcing] nowcast+ 2020-07-05 forcing files upload to orcinus-nowcast-agrif completed 2020-07-05 09:11:16,552 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2020-07-05 09:11:16,996 INFO [upload_forcing] nowcast+ 2020-07-05 forcing files upload to arbutus.cloud-nowcast completed 2020-07-05 09:11:17,000 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2020-07-05 09:11:17,220 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'nowcast+', '--run-date', '2020-07-05'], host='localhost') 2020-07-05 09:11:18,439 INFO [make_forcing_links] running in process 28358 2020-07-05 09:11:18,441 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-05 09:11:18,442 INFO [make_forcing_links] publishing log messages to tcp://*:5564 2020-07-05 09:11:18,444 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555 2020-07-05 09:11:19,676 INFO [make_forcing_links] nowcast+ 2020-07-05 forcing file links on arbutus.cloud-nowcast created 2020-07-05 09:11:19,681 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker 2020-07-05 09:11:19,924 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2020-07-05'], host='arbutus.cloud-nowcast') 2020-07-05 09:11:22,105 INFO [run_NEMO] running in process 12096 2020-07-05 09:11:22,107 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2020-07-05 09:11:22,108 INFO [run_NEMO] publishing log messages to tcp://*:5556 2020-07-05 09:11:22,116 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555 2020-07-05 09:11:22,310 INFO [upload_forcing] nowcast+ 2020-07-05 forcing files upload to graham-hindcast completed 2020-07-05 09:11:22,316 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2020-07-05 09:11:23,118 INFO [run_NEMO] nowcast: launching /nemoShare/MEOPAR/nowcast-sys/runs/05jul20nowcast_2020-07-05T091122.170920-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast 2020-07-05 09:11:23,137 INFO [run_NEMO] nowcast NEMO run for 2020-07-05 on arbutus.cloud-nowcast started 2020-07-05 09:11:23,143 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker 2020-07-05 09:11:23,383 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'nowcast'], host='arbutus.cloud-nowcast') 2020-07-05 09:11:25,336 INFO [watch_NEMO] running in process 12361 2020-07-05 09:11:25,337 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2020-07-05 09:11:25,338 INFO [watch_NEMO] publishing log messages to tcp://*:5557 2020-07-05 09:11:25,338 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555 2020-07-05 09:11:25,444 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: time.step not found; continuing to watch... 2020-07-05 09:16:25,546 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 7499967 = 2020-07-05 04:57:20 UTC, 20.7% complete 2020-07-05 09:21:25,645 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 7500484 = 2020-07-05 10:42:00 UTC, 44.6% complete 2020-07-05 09:26:25,745 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 7500974 = 2020-07-05 16:08:40 UTC, 67.3% complete 2020-07-05 09:31:25,847 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 7501460 = 2020-07-05 21:32:40 UTC, 89.8% complete 2020-07-05 09:36:25,951 INFO [watch_NEMO] nowcast NEMO run on arbutus.cloud-nowcast completed 2020-07-05 09:36:25,955 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker 2020-07-05 09:36:26,517 INFO [manager] launching NextWorker(module='nowcast.workers.get_NeahBay_ssh', args=['forecast'], host='localhost') 2020-07-05 09:36:26,523 INFO [manager] launching NextWorker(module='nowcast.workers.make_fvcom_boundary', args=['arbutus.cloud-nowcast', 'x2', 'nowcast'], host='arbutus.cloud-nowcast') 2020-07-05 09:36:26,638 INFO [manager] launching NextWorker(module='nowcast.workers.make_fvcom_boundary', args=['arbutus.cloud-nowcast', 'r12', 'nowcast'], host='arbutus.cloud-nowcast') 2020-07-05 09:36:26,642 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2020-07-05'], host='localhost') 2020-07-05 09:36:28,139 INFO [download_results] running in process 28878 2020-07-05 09:36:28,140 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-05 09:36:28,141 INFO [download_results] publishing log messages to tcp://*:5564 2020-07-05 09:36:28,141 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2020-07-05 09:36:28,142 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/nowcast/05jul20 to /results/SalishSea/nowcast-blue.201905 2020-07-05 09:36:30,233 INFO [make_fvcom_boundary] running in process 12641 2020-07-05 09:36:30,234 INFO [make_fvcom_boundary] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2020-07-05 09:36:30,236 INFO [make_fvcom_boundary] publishing log messages to tcp://*:5580 2020-07-05 09:36:30,237 INFO [make_fvcom_boundary] connected to skookum.eos.ubc.ca port 5555 2020-07-05 09:36:30,238 INFO [make_fvcom_boundary] Creating VHFR FVCOM open boundary file for x2 nowcast run from 2020-07-05 NEMO run 2020-07-05 09:36:30,270 INFO [make_fvcom_boundary] running in process 12642 2020-07-05 09:36:30,272 INFO [make_fvcom_boundary] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2020-07-05 09:36:30,273 INFO [make_fvcom_boundary] publishing log messages to tcp://*:5581 2020-07-05 09:36:30,275 INFO [make_fvcom_boundary] connected to skookum.eos.ubc.ca port 5555 2020-07-05 09:36:30,276 INFO [make_fvcom_boundary] Creating VHFR FVCOM open boundary file for r12 nowcast run from 2020-07-05 NEMO run 2020-07-05 09:36:30,553 INFO [get_NeahBay_ssh] running in process 28875 2020-07-05 09:36:30,555 INFO [get_NeahBay_ssh] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-05 09:36:30,557 INFO [get_NeahBay_ssh] publishing log messages to tcp://*:5567 2020-07-05 09:36:30,559 INFO [get_NeahBay_ssh] connected to skookum.eos.ubc.ca port 5555 2020-07-05 09:36:37,382 INFO [get_NeahBay_ssh] Neah Bay sea surface height web scraping and open boundary file creation for forecast complete 2020-07-05 09:36:37,386 INFO [manager] checklist updated with [Neah Bay ssh] items from get_NeahBay_ssh worker 2020-07-05 09:36:37,589 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'ssh'], host='localhost') 2020-07-05 09:36:38,596 INFO [upload_forcing] running in process 28893 2020-07-05 09:36:38,597 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-05 09:36:38,599 INFO [upload_forcing] publishing log messages to tcp://*:5567 2020-07-05 09:36:38,600 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2020-07-05 09:36:41,040 INFO [upload_forcing] ssh 2020-07-05 forcing files upload to arbutus.cloud-nowcast completed 2020-07-05 09:36:41,045 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2020-07-05 09:36:41,280 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'ssh', '--run-date', '2020-07-05'], host='localhost') 2020-07-05 09:36:42,305 INFO [make_forcing_links] running in process 28899 2020-07-05 09:36:42,307 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-05 09:36:42,309 INFO [make_forcing_links] publishing log messages to tcp://*:5567 2020-07-05 09:36:42,311 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555 2020-07-05 09:36:43,423 INFO [make_forcing_links] ssh 2020-07-05 forcing file links on arbutus.cloud-nowcast created 2020-07-05 09:36:43,427 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker 2020-07-05 09:36:43,661 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2020-07-05'], host='arbutus.cloud-nowcast') 2020-07-05 09:36:45,906 INFO [run_NEMO] running in process 13003 2020-07-05 09:36:45,908 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2020-07-05 09:36:45,909 INFO [run_NEMO] publishing log messages to tcp://*:5556 2020-07-05 09:36:45,910 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555 2020-07-05 09:36:47,071 INFO [run_NEMO] forecast: launching /nemoShare/MEOPAR/nowcast-sys/runs/05jul20forecast_2020-07-05T093645.994175-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast 2020-07-05 09:36:47,090 INFO [run_NEMO] forecast NEMO run for 2020-07-05 on arbutus.cloud-nowcast started 2020-07-05 09:36:47,095 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker 2020-07-05 09:36:47,307 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'forecast'], host='arbutus.cloud-nowcast') 2020-07-05 09:36:49,277 INFO [watch_NEMO] running in process 13270 2020-07-05 09:36:49,278 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2020-07-05 09:36:49,280 INFO [watch_NEMO] publishing log messages to tcp://*:5557 2020-07-05 09:36:49,282 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555 2020-07-05 09:36:49,386 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: time.step not found; continuing to watch... 2020-07-05 09:38:50,523 INFO [make_fvcom_boundary] Stored VHFR FVCOM open boundary file: /nemoShare/MEOPAR/nowcast-sys/fvcom-runs/input.r12/bdy_r12_nowcast_brcl_20200705.nc 2020-07-05 09:38:50,528 INFO [make_fvcom_boundary] FVCOM r12 nowcast run boundary condition file for 2020-07-05 created on arbutus.cloud-nowcast 2020-07-05 09:38:50,532 INFO [manager] checklist updated with [FVCOM boundary conditions] items from make_fvcom_boundary worker 2020-07-05 09:38:50,792 INFO [manager] launching NextWorker(module='nowcast.workers.make_fvcom_atmos_forcing', args=['r12', 'nowcast', '--run-date', '2020-07-05'], host='localhost') 2020-07-05 09:38:50,798 INFO [manager] launching NextWorker(module='nowcast.workers.make_fvcom_rivers_forcing', args=['arbutus.cloud-nowcast', 'r12', 'nowcast', '--run-date', '2020-07-05'], host='arbutus.cloud-nowcast') 2020-07-05 09:38:57,734 INFO [make_fvcom_atmos_forcing] running in process 28991 2020-07-05 09:38:57,736 INFO [make_fvcom_atmos_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-05 09:38:57,737 INFO [make_fvcom_atmos_forcing] publishing log messages to tcp://*:5567 2020-07-05 09:38:57,739 INFO [make_fvcom_atmos_forcing] connected to skookum.eos.ubc.ca port 5555 2020-07-05 09:38:57,741 INFO [make_fvcom_atmos_forcing] Creating VHFR FVCOM atmospheric forcing file for r12 nowcast run from 2020-07-05 HRDPS GRIB files 2020-07-05 09:39:00,413 INFO [make_fvcom_rivers_forcing] running in process 13361 2020-07-05 09:39:00,416 INFO [make_fvcom_rivers_forcing] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2020-07-05 09:39:00,417 INFO [make_fvcom_rivers_forcing] publishing log messages to tcp://*:5582 2020-07-05 09:39:00,419 INFO [make_fvcom_rivers_forcing] connected to skookum.eos.ubc.ca port 5555 2020-07-05 09:39:00,420 INFO [make_fvcom_rivers_forcing] Creating VHFR FVCOM rivers forcing file for r12 nowcast run from 2020-07-05 NEMO runoff forcing files 2020-07-05 09:39:00,656 INFO [make_fvcom_rivers_forcing] Stored VHFR FVCOM rivers forcing file: /nemoShare/MEOPAR/nowcast-sys/fvcom-runs/input.r12/rivers_r12_nowcast_20200705.nc 2020-07-05 09:39:00,658 INFO [make_fvcom_rivers_forcing] FVCOM r12 nowcast run rivers forcing file for 2020-07-05 created on arbutus.cloud-nowcast 2020-07-05 09:39:01,441 INFO [manager] checklist updated with [FVCOM rivers forcing] items from make_fvcom_rivers_forcing worker 2020-07-05 09:39:04,333 INFO [make_fvcom_boundary] Stored VHFR FVCOM open boundary file: /nemoShare/MEOPAR/nowcast-sys/fvcom-runs/input.x2/bdy_x2_nowcast_brcl_20200705.nc 2020-07-05 09:39:04,339 INFO [make_fvcom_boundary] FVCOM x2 nowcast run boundary condition file for 2020-07-05 created on arbutus.cloud-nowcast 2020-07-05 09:39:04,346 INFO [manager] checklist updated with [FVCOM boundary conditions] items from make_fvcom_boundary worker 2020-07-05 09:39:04,627 INFO [manager] launching NextWorker(module='nowcast.workers.make_fvcom_atmos_forcing', args=['x2', 'nowcast', '--run-date', '2020-07-05'], host='localhost') 2020-07-05 09:39:04,631 INFO [manager] launching NextWorker(module='nowcast.workers.make_fvcom_rivers_forcing', args=['arbutus.cloud-nowcast', 'x2', 'nowcast', '--run-date', '2020-07-05'], host='arbutus.cloud-nowcast') 2020-07-05 09:39:07,758 INFO [make_fvcom_atmos_forcing] running in process 28997 2020-07-05 09:39:07,759 INFO [make_fvcom_atmos_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-05 09:39:07,760 INFO [make_fvcom_atmos_forcing] publishing log messages to tcp://*:5568 2020-07-05 09:39:07,762 INFO [make_fvcom_atmos_forcing] connected to skookum.eos.ubc.ca port 5555 2020-07-05 09:39:07,763 INFO [make_fvcom_atmos_forcing] Creating VHFR FVCOM atmospheric forcing file for x2 nowcast run from 2020-07-05 HRDPS GRIB files 2020-07-05 09:39:08,858 INFO [make_fvcom_rivers_forcing] running in process 13454 2020-07-05 09:39:08,859 INFO [make_fvcom_rivers_forcing] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2020-07-05 09:39:08,860 INFO [make_fvcom_rivers_forcing] publishing log messages to tcp://*:5582 2020-07-05 09:39:08,862 INFO [make_fvcom_rivers_forcing] connected to skookum.eos.ubc.ca port 5555 2020-07-05 09:39:08,863 INFO [make_fvcom_rivers_forcing] Creating VHFR FVCOM rivers forcing file for x2 nowcast run from 2020-07-05 NEMO runoff forcing files 2020-07-05 09:39:09,097 INFO [make_fvcom_rivers_forcing] Stored VHFR FVCOM rivers forcing file: /nemoShare/MEOPAR/nowcast-sys/fvcom-runs/input.x2/rivers_x2_nowcast_20200705.nc 2020-07-05 09:39:09,099 INFO [make_fvcom_rivers_forcing] FVCOM x2 nowcast run rivers forcing file for 2020-07-05 created on arbutus.cloud-nowcast 2020-07-05 09:39:09,104 INFO [manager] checklist updated with [FVCOM rivers forcing] items from make_fvcom_rivers_forcing worker 2020-07-05 09:40:31,730 INFO [make_fvcom_atmos_forcing] Stored VHFR FVCOM atmospheric forcing hfx file: /results/forcing/atmospheric/GEM2.5/vhfr-fvcom/atmos_x2_nowcast_hfx_20200705.nc 2020-07-05 09:40:32,005 INFO [make_fvcom_atmos_forcing] Stored VHFR FVCOM atmospheric forcing hfx file: /results/forcing/atmospheric/GEM2.5/vhfr-fvcom/atmos_r12_nowcast_hfx_20200705.nc 2020-07-05 09:40:48,737 INFO [make_fvcom_atmos_forcing] Stored VHFR FVCOM atmospheric forcing precip file: /results/forcing/atmospheric/GEM2.5/vhfr-fvcom/atmos_x2_nowcast_precip_20200705.nc 2020-07-05 09:40:49,004 INFO [make_fvcom_atmos_forcing] Stored VHFR FVCOM atmospheric forcing precip file: /results/forcing/atmospheric/GEM2.5/vhfr-fvcom/atmos_r12_nowcast_precip_20200705.nc 2020-07-05 09:41:27,673 INFO [make_fvcom_atmos_forcing] Stored VHFR FVCOM atmospheric forcing wnd file: /results/forcing/atmospheric/GEM2.5/vhfr-fvcom/atmos_x2_nowcast_wnd_20200705.nc 2020-07-05 09:41:27,673 INFO [make_fvcom_atmos_forcing] FVCOM x2 nowcast run atmospheric forcing file for 2020-07-05 created 2020-07-05 09:41:27,678 INFO [manager] checklist updated with [FVCOM atmospheric forcing] items from make_fvcom_atmos_forcing worker 2020-07-05 09:41:27,909 INFO [manager] launching NextWorker(module='nowcast.workers.upload_fvcom_atmos_forcing', args=['arbutus.cloud-nowcast', 'x2', 'nowcast', '--run-date', '2020-07-05'], host='localhost') 2020-07-05 09:41:28,240 INFO [make_fvcom_atmos_forcing] Stored VHFR FVCOM atmospheric forcing wnd file: /results/forcing/atmospheric/GEM2.5/vhfr-fvcom/atmos_r12_nowcast_wnd_20200705.nc 2020-07-05 09:41:28,241 INFO [make_fvcom_atmos_forcing] FVCOM r12 nowcast run atmospheric forcing file for 2020-07-05 created 2020-07-05 09:41:28,248 INFO [manager] checklist updated with [FVCOM atmospheric forcing] items from make_fvcom_atmos_forcing worker 2020-07-05 09:41:28,444 INFO [manager] launching NextWorker(module='nowcast.workers.upload_fvcom_atmos_forcing', args=['arbutus.cloud-nowcast', 'r12', 'nowcast', '--run-date', '2020-07-05'], host='localhost') 2020-07-05 09:41:29,742 INFO [upload_fvcom_atmos_forcing] running in process 29060 2020-07-05 09:41:29,744 INFO [upload_fvcom_atmos_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-05 09:41:29,745 INFO [upload_fvcom_atmos_forcing] publishing log messages to tcp://*:5567 2020-07-05 09:41:29,747 INFO [upload_fvcom_atmos_forcing] connected to skookum.eos.ubc.ca port 5555 2020-07-05 09:41:29,748 INFO [upload_fvcom_atmos_forcing] Uploading VHFR FVCOM atmospheric forcing files for 2020-07-05 x2 nowcast run to arbutus.cloud-nowcast 2020-07-05 09:41:29,750 INFO [upload_fvcom_atmos_forcing] running in process 29062 2020-07-05 09:41:29,751 INFO [upload_fvcom_atmos_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-05 09:41:29,752 INFO [upload_fvcom_atmos_forcing] publishing log messages to tcp://*:5568 2020-07-05 09:41:29,753 INFO [upload_fvcom_atmos_forcing] connected to skookum.eos.ubc.ca port 5555 2020-07-05 09:41:29,754 INFO [upload_fvcom_atmos_forcing] Uploading VHFR FVCOM atmospheric forcing files for 2020-07-05 r12 nowcast run to arbutus.cloud-nowcast 2020-07-05 09:41:34,729 INFO [upload_fvcom_atmos_forcing] FVCOM x2 nowcast run atmospheric forcing files for 2020-07-05 uploaded to arbutus.cloud-nowcast 2020-07-05 09:41:34,732 INFO [manager] checklist updated with [FVCOM atmospheric forcing upload] items from upload_fvcom_atmos_forcing worker 2020-07-05 09:41:34,919 INFO [manager] launching NextWorker(module='nowcast.workers.run_fvcom', args=['arbutus.cloud-nowcast', 'x2', 'nowcast', '--run-date', '2020-07-05'], host='arbutus.cloud-nowcast') 2020-07-05 09:41:35,870 INFO [upload_fvcom_atmos_forcing] FVCOM r12 nowcast run atmospheric forcing files for 2020-07-05 uploaded to arbutus.cloud-nowcast 2020-07-05 09:41:35,876 INFO [manager] checklist updated with [FVCOM atmospheric forcing upload] items from upload_fvcom_atmos_forcing worker 2020-07-05 09:41:36,126 INFO [manager] launching NextWorker(module='nowcast.workers.run_fvcom', args=['arbutus.cloud-nowcast', 'r12', 'nowcast', '--run-date', '2020-07-05'], host='arbutus.cloud-nowcast') 2020-07-05 09:41:37,593 INFO [run_fvcom] running in process 14063 2020-07-05 09:41:37,595 INFO [run_fvcom] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2020-07-05 09:41:37,596 INFO [run_fvcom] publishing log messages to tcp://*:5584 2020-07-05 09:41:37,598 INFO [run_fvcom] connected to skookum.eos.ubc.ca port 5555 2020-07-05 09:41:37,654 INFO [run_fvcom] nowcast: launching /nemoShare/MEOPAR/nowcast-sys/fvcom-runs/20200705-16h41m37.639728s/VHFR_FVCOM.sh on arbutus.cloud-nowcast 2020-07-05 09:41:37,675 INFO [run_fvcom] x2 nowcast FVCOM VH-FR run for 2020-07-05 on arbutus.cloud-nowcast started 2020-07-05 09:41:37,681 INFO [manager] checklist updated with [FVCOM run] items from run_fvcom worker 2020-07-05 09:41:37,896 INFO [manager] launching NextWorker(module='nowcast.workers.watch_fvcom', args=['arbutus.cloud-nowcast', 'x2', 'nowcast'], host='arbutus.cloud-nowcast') 2020-07-05 09:41:38,539 INFO [run_fvcom] running in process 14151 2020-07-05 09:41:38,541 INFO [run_fvcom] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2020-07-05 09:41:38,542 INFO [run_fvcom] publishing log messages to tcp://*:5584 2020-07-05 09:41:38,543 INFO [run_fvcom] connected to skookum.eos.ubc.ca port 5555 2020-07-05 09:41:38,596 INFO [run_fvcom] nowcast: launching /nemoShare/MEOPAR/nowcast-sys/fvcom-runs/20200705-16h41m38.585479s/VHFR_FVCOM.sh on arbutus.cloud-nowcast 2020-07-05 09:41:38,626 INFO [run_fvcom] r12 nowcast FVCOM VH-FR run for 2020-07-05 on arbutus.cloud-nowcast started 2020-07-05 09:41:38,631 INFO [manager] checklist updated with [FVCOM run] items from run_fvcom worker 2020-07-05 09:41:38,847 INFO [manager] launching NextWorker(module='nowcast.workers.watch_fvcom', args=['arbutus.cloud-nowcast', 'r12', 'nowcast'], host='arbutus.cloud-nowcast') 2020-07-05 09:41:40,291 INFO [watch_fvcom] running in process 14269 2020-07-05 09:41:40,292 INFO [watch_fvcom] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2020-07-05 09:41:40,293 INFO [watch_fvcom] publishing log messages to tcp://*:5586 2020-07-05 09:41:40,294 INFO [watch_fvcom] connected to skookum.eos.ubc.ca port 5555 2020-07-05 09:41:40,352 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: no run progress found in fvcom.log ; continuing to watch... 2020-07-05 09:41:46,172 INFO [watch_fvcom] running in process 14359 2020-07-05 09:41:46,173 INFO [watch_fvcom] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2020-07-05 09:41:46,174 INFO [watch_fvcom] publishing log messages to tcp://*:5587 2020-07-05 09:41:46,175 INFO [watch_fvcom] connected to skookum.eos.ubc.ca port 5555 2020-07-05 09:41:46,224 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: no run progress found in fvcom.log ; continuing to watch... 2020-07-05 09:41:49,485 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 7502111 = 2020-07-06 04:46:40 UTC, 13.3% complete 2020-07-05 09:46:40,454 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15901061 = 2020-07-05 00:27:41 UTC estimated time to finish: 0000:03:44:57 2020-07-05 09:46:46,326 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39749760 = 2020-07-05 00:08:24 UTC estimated time to finish: 0000:13:25:47 2020-07-05 09:46:49,587 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 7502580 = 2020-07-06 09:59:20 UTC, 27.8% complete 2020-07-05 09:46:54,565 INFO [download_results] nowcast 2020-07-05 results files from arbutus.cloud-nowcast downloaded 2020-07-05 09:46:54,572 INFO [manager] checklist updated with [results files] items from download_results worker 2020-07-05 09:46:54,822 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast', 'research', '--run-date', '2020-07-05'], host='localhost') 2020-07-05 09:46:54,827 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast', 'comparison', '--run-date', '2020-07-04'], host='localhost') 2020-07-05 09:46:54,831 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['nowcast', '--run-date', '2020-07-05'], host='localhost') 2020-07-05 09:46:59,240 INFO [make_CHS_currents_file] running in process 29245 2020-07-05 09:46:59,241 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-05 09:46:59,242 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5564 2020-07-05 09:46:59,243 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555 2020-07-05 09:47:01,545 INFO [make_plots] running in process 29243 2020-07-05 09:47:01,546 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-05 09:47:01,547 INFO [make_plots] publishing log messages to tcp://*:5567 2020-07-05 09:47:01,549 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555 2020-07-05 09:47:01,557 INFO [make_plots] running in process 29244 2020-07-05 09:47:01,691 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-05 09:47:01,692 INFO [make_plots] publishing log messages to tcp://*:5568 2020-07-05 09:47:01,700 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555 2020-07-05 09:47:02,318 INFO [make_plots] preparing render list for 2020-07-05 NEMO nowcast-blue research figures 2020-07-05 09:47:02,321 INFO [make_plots] preparing render list for 2020-07-04 NEMO nowcast-blue comparison figures 2020-07-05 09:47:10,252 INFO [make_plots] added salinity_thalweg_and_surface figures to 2020-07-05 NEMO nowcast-blue research render list 2020-07-05 09:47:16,722 INFO [make_plots] added temperature_thalweg_and_surface figures to 2020-07-05 NEMO nowcast-blue research render list 2020-07-05 09:47:16,723 INFO [make_plots] added Currents_sections_and_surface figure to 2020-07-05 NEMO nowcast-blue research render list 2020-07-05 09:47:16,725 INFO [make_plots] added Currents_at_VENUS_Central figure to 2020-07-05 NEMO nowcast-blue research render list 2020-07-05 09:47:16,727 INFO [make_plots] added Currents_at_VENUS_East figure to 2020-07-05 NEMO nowcast-blue research render list 2020-07-05 09:47:16,728 INFO [make_plots] starting to render nemo nowcast research 05jul20 figures 2020-07-05 09:47:25,596 INFO [make_plots] added SH_wind figure to 2020-07-04 NEMO nowcast-blue comparison render list 2020-07-05 09:47:25,598 INFO [make_plots] added Compare_VENUS_East figure to 2020-07-04 NEMO nowcast-blue comparison render list 2020-07-05 09:47:25,599 INFO [make_plots] added Compare_VENUS_Central figure to 2020-07-04 NEMO nowcast-blue comparison render list 2020-07-05 09:47:25,600 INFO [make_plots] added Compare_VENUS_Delta_DDL figure to 2020-07-04 NEMO nowcast-blue comparison render list 2020-07-05 09:47:25,601 INFO [make_plots] added Central_ADCP figure to 2020-07-04 NEMO nowcast-blue comparison render list 2020-07-05 09:47:25,602 INFO [make_plots] added Central_depavADCP figure to 2020-07-04 NEMO nowcast-blue comparison render list 2020-07-05 09:47:25,604 INFO [make_plots] added Central_timeavADCP figure to 2020-07-04 NEMO nowcast-blue comparison render list 2020-07-05 09:47:25,605 INFO [make_plots] added East_ADCP figure to 2020-07-04 NEMO nowcast-blue comparison render list 2020-07-05 09:47:25,605 INFO [make_plots] added East_depavADCP figure to 2020-07-04 NEMO nowcast-blue comparison render list 2020-07-05 09:47:25,606 INFO [make_plots] added East_timeavADCP figure to 2020-07-04 NEMO nowcast-blue comparison render list 2020-07-05 09:47:25,607 INFO [make_plots] added ddl_ADCP figure to 2020-07-04 NEMO nowcast-blue comparison render list 2020-07-05 09:47:25,608 INFO [make_plots] added ddl_depavADCP figure to 2020-07-04 NEMO nowcast-blue comparison render list 2020-07-05 09:47:25,608 INFO [make_plots] added ddl_timeavADCP figure to 2020-07-04 NEMO nowcast-blue comparison render list 2020-07-05 09:47:25,609 INFO [make_plots] starting to render nemo nowcast comparison 04jul20 figures 2020-07-05 09:50:11,408 INFO [make_CHS_currents_file] Made CHS currents file for 2020-07-05 for nowcast 2020-07-05 09:50:11,411 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker 2020-07-05 09:51:40,558 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15903025 = 2020-07-05 01:00:25 UTC estimated time to finish: 0000:03:40:42 2020-07-05 09:51:46,430 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39751111 = 2020-07-05 00:17:24 UTC estimated time to finish: 0000:13:00:05 2020-07-05 09:51:49,688 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 7503057 = 2020-07-06 15:17:20 UTC, 42.5% complete 2020-07-05 09:53:36,680 INFO [make_plots] finished rendering nemo nowcast research 05jul20 figures 2020-07-05 09:53:36,683 INFO [make_plots] nemo research plots for 2020-07-05 nowcast completed 2020-07-05 09:53:36,707 INFO [manager] checklist updated with [plots] items from make_plots worker 2020-07-05 09:53:59,619 WARNING [make_plots] VENUS Delta DDL node CTD comparison figure failed: No observations available 2020-07-05 09:54:01,650 WARNING [make_plots] VENUS Central ADCP comparison figure failed: No observations available 2020-07-05 09:54:02,242 WARNING [make_plots] VENUS Central ADCP comparison figure failed: No observations available 2020-07-05 09:54:02,798 WARNING [make_plots] VENUS Central ADCP comparison figure failed: No observations available 2020-07-05 09:54:04,048 WARNING [make_plots] VENUS East ADCP comparison figure failed: No observations available 2020-07-05 09:54:04,602 WARNING [make_plots] VENUS East ADCP comparison figure failed: No observations available 2020-07-05 09:54:05,167 WARNING [make_plots] VENUS East ADCP comparison figure failed: No observations available 2020-07-05 09:54:09,051 WARNING [make_plots] VENUS ddl ADCP comparison figure failed: No observations available 2020-07-05 09:54:09,688 WARNING [make_plots] VENUS ddl ADCP comparison figure failed: No observations available 2020-07-05 09:54:10,261 WARNING [make_plots] VENUS ddl ADCP comparison figure failed: No observations available 2020-07-05 09:54:10,267 INFO [make_plots] finished rendering nemo nowcast comparison 04jul20 figures 2020-07-05 09:54:10,363 INFO [make_plots] nemo comparison plots for 2020-07-04 nowcast completed 2020-07-05 09:54:10,367 INFO [manager] checklist updated with [plots] items from make_plots worker 2020-07-05 09:56:40,566 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15904960 = 2020-07-05 01:32:40 UTC estimated time to finish: 0000:03:35:31 2020-07-05 09:56:46,533 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39752531 = 2020-07-05 00:26:52 UTC estimated time to finish: 0000:12:52:45 2020-07-05 09:56:49,789 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 7503538 = 2020-07-06 20:38:00 UTC, 57.3% complete 2020-07-05 10:01:40,670 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15906732 = 2020-07-05 02:02:12 UTC estimated time to finish: 0000:03:31:24 2020-07-05 10:01:46,638 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39753789 = 2020-07-05 00:35:15 UTC estimated time to finish: 0000:12:53:46 2020-07-05 10:01:49,891 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 7503998 = 2020-07-07 01:44:40 UTC, 71.5% complete 2020-07-05 10:06:40,681 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15908669 = 2020-07-05 02:34:29 UTC estimated time to finish: 0000:03:25:58 2020-07-05 10:06:46,744 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39755210 = 2020-07-05 00:44:44 UTC estimated time to finish: 0000:12:54:09 2020-07-05 10:06:49,992 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 7504451 = 2020-07-07 06:46:40 UTC, 85.5% complete 2020-07-05 10:11:40,789 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15910590 = 2020-07-05 03:06:30 UTC estimated time to finish: 0000:03:20:46 2020-07-05 10:11:46,848 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39756600 = 2020-07-05 00:54:00 UTC estimated time to finish: 0000:12:46:11 2020-07-05 10:11:50,094 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 7504920 = 2020-07-07 11:59:20 UTC, 100.0% complete 2020-07-05 10:16:40,888 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15912294 = 2020-07-05 03:34:54 UTC estimated time to finish: 0000:03:16:35 2020-07-05 10:16:46,951 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39758131 = 2020-07-05 01:04:12 UTC estimated time to finish: 0000:12:21:00 2020-07-05 10:16:50,198 INFO [watch_NEMO] forecast NEMO run on arbutus.cloud-nowcast completed 2020-07-05 10:16:50,203 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker 2020-07-05 10:16:50,723 INFO [manager] launching NextWorker(module='nowcast.workers.make_turbidity_file', args=['--run-date', '2020-07-05'], host='localhost') 2020-07-05 10:16:50,730 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2020-07-05'], host='localhost') 2020-07-05 10:16:55,053 INFO [download_results] running in process 29863 2020-07-05 10:16:55,054 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-05 10:16:55,055 INFO [download_results] publishing log messages to tcp://*:5564 2020-07-05 10:16:55,056 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2020-07-05 10:16:55,056 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/forecast/05jul20 to /results/SalishSea/forecast.201905 2020-07-05 10:16:55,793 INFO [make_turbidity_file] running in process 29862 2020-07-05 10:16:55,795 INFO [make_turbidity_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-05 10:16:55,797 INFO [make_turbidity_file] publishing log messages to tcp://*:5567 2020-07-05 10:16:55,798 INFO [make_turbidity_file] connected to skookum.eos.ubc.ca port 5555 2020-07-05 10:16:55,800 INFO [make_turbidity_file] Creating Fraser River turbidity forcing file for 2020-07-05 2020-07-05 10:17:03,454 INFO [make_turbidity_file] Note: NumExpr detected 12 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8. 2020-07-05 10:17:03,455 INFO [make_turbidity_file] NumExpr defaulting to 8 threads. 2020-07-05 10:17:05,994 INFO [make_turbidity_file] 2020-07-05 Fraser River turbidity file creation complete 2020-07-05 10:17:05,997 INFO [manager] checklist updated with [Fraser River turbidity] items from make_turbidity_file worker 2020-07-05 10:17:06,225 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'turbidity'], host='localhost') 2020-07-05 10:17:06,228 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'turbidity'], host='localhost') 2020-07-05 10:17:06,231 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['graham-hindcast', 'turbidity'], host='localhost') 2020-07-05 10:17:06,234 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'turbidity'], host='localhost') 2020-07-05 10:17:07,146 INFO [upload_forcing] running in process 29891 2020-07-05 10:17:07,147 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-05 10:17:07,148 INFO [upload_forcing] publishing log messages to tcp://*:5567 2020-07-05 10:17:07,149 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2020-07-05 10:17:07,196 INFO [upload_forcing] running in process 29892 2020-07-05 10:17:07,197 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-05 10:17:07,198 INFO [upload_forcing] publishing log messages to tcp://*:5568 2020-07-05 10:17:07,199 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2020-07-05 10:17:07,269 INFO [upload_forcing] running in process 29893 2020-07-05 10:17:07,270 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-05 10:17:07,272 INFO [upload_forcing] publishing log messages to tcp://*:5569 2020-07-05 10:17:07,273 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2020-07-05 10:17:07,302 INFO [upload_forcing] running in process 29894 2020-07-05 10:17:07,303 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-05 10:17:07,304 INFO [upload_forcing] publishing log messages to tcp://*:5570 2020-07-05 10:17:07,305 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2020-07-05 10:17:12,832 INFO [upload_forcing] turbidity 2020-07-05 forcing files upload to optimum-hindcast completed 2020-07-05 10:17:12,836 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2020-07-05 10:17:12,883 INFO [upload_forcing] turbidity 2020-07-05 forcing files upload to orcinus-nowcast-agrif completed 2020-07-05 10:17:13,061 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2020-07-05 10:17:13,092 INFO [upload_forcing] turbidity 2020-07-05 forcing files upload to arbutus.cloud-nowcast completed 2020-07-05 10:17:13,255 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['orcinus-nowcast-agrif', 'nowcast-agrif', '--run-date', '2020-07-05'], host='localhost') 2020-07-05 10:17:13,261 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2020-07-05 10:17:13,456 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2020-07-05'], host='localhost') 2020-07-05 10:17:13,734 INFO [upload_forcing] turbidity 2020-07-05 forcing files upload to graham-hindcast completed 2020-07-05 10:17:13,738 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2020-07-05 10:17:14,503 INFO [make_forcing_links] running in process 29916 2020-07-05 10:17:14,505 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-05 10:17:14,506 INFO [make_forcing_links] publishing log messages to tcp://*:5567 2020-07-05 10:17:14,508 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555 2020-07-05 10:17:14,560 INFO [make_forcing_links] running in process 29917 2020-07-05 10:17:14,562 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-05 10:17:14,564 INFO [make_forcing_links] publishing log messages to tcp://*:5568 2020-07-05 10:17:14,566 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555 2020-07-05 10:17:15,180 INFO [make_forcing_links] nowcast-agrif 2020-07-05 forcing file links on orcinus-nowcast-agrif created 2020-07-05 10:17:15,184 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker 2020-07-05 10:17:15,438 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_agrif', args=['orcinus-nowcast-agrif', 'nowcast-agrif', '--run-date', '2020-07-05'], host='localhost') 2020-07-05 10:17:15,801 INFO [make_forcing_links] nowcast-green 2020-07-05 forcing file links on arbutus.cloud-nowcast created 2020-07-05 10:17:15,805 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker 2020-07-05 10:17:16,077 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2020-07-05'], host='arbutus.cloud-nowcast') 2020-07-05 10:17:16,625 INFO [run_NEMO_agrif] running in process 29928 2020-07-05 10:17:16,626 INFO [run_NEMO_agrif] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-05 10:17:16,627 INFO [run_NEMO_agrif] publishing log messages to tcp://*:5567 2020-07-05 10:17:16,629 INFO [run_NEMO_agrif] connected to skookum.eos.ubc.ca port 5555 2020-07-05 10:17:17,145 CRITICAL [run_NEMO_agrif] unhandled exception: Traceback (most recent call last): File "/SalishSeaCast/NEMO_Nowcast/nemo_nowcast/worker.py", line 390, in _do_work checklist = self.worker_func( File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/run_NEMO_agrif.py", line 110, in run_NEMO_agrif prev_run_namelists_info = _get_prev_run_namelists_info( File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/run_NEMO_agrif.py", line 156, in _get_prev_run_namelists_info sftp_client.get(os.fspath(prev_namelist_cfg), namelist_cfg.name) File "/SalishSeaCast/nowcast-env/lib/python3.8/site-packages/paramiko/sftp_client.py", line 802, in get size = self.getfo(remotepath, fl, callback) File "/SalishSeaCast/nowcast-env/lib/python3.8/site-packages/paramiko/sftp_client.py", line 778, in getfo file_size = self.stat(remotepath).st_size File "/SalishSeaCast/nowcast-env/lib/python3.8/site-packages/paramiko/sftp_client.py", line 493, in stat t, msg = self._request(CMD_STAT, path) File "/SalishSeaCast/nowcast-env/lib/python3.8/site-packages/paramiko/sftp_client.py", line 813, in _request return self._read_response(num) File "/SalishSeaCast/nowcast-env/lib/python3.8/site-packages/paramiko/sftp_client.py", line 865, in _read_response self._convert_status(msg) File "/SalishSeaCast/nowcast-env/lib/python3.8/site-packages/paramiko/sftp_client.py", line 894, in _convert_status raise IOError(errno.ENOENT, text) FileNotFoundError: [Errno 2] No such file 2020-07-05 10:17:18,366 INFO [run_NEMO] running in process 14844 2020-07-05 10:17:18,367 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2020-07-05 10:17:18,368 INFO [run_NEMO] publishing log messages to tcp://*:5556 2020-07-05 10:17:18,376 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555 2020-07-05 10:17:19,514 INFO [run_NEMO] nowcast-green: launching /nemoShare/MEOPAR/nowcast-sys/runs/05jul20nowcast-green_2020-07-05T101718.434495-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast 2020-07-05 10:17:19,533 INFO [run_NEMO] nowcast-green NEMO run for 2020-07-05 on arbutus.cloud-nowcast started 2020-07-05 10:17:19,537 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker 2020-07-05 10:17:19,773 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'nowcast-green'], host='arbutus.cloud-nowcast') 2020-07-05 10:17:21,755 INFO [watch_NEMO] running in process 15108 2020-07-05 10:17:21,757 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2020-07-05 10:17:21,758 INFO [watch_NEMO] publishing log messages to tcp://*:5557 2020-07-05 10:17:21,760 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555 2020-07-05 10:17:21,864 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: time.step not found; continuing to watch... 2020-07-05 10:21:41,001 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15914216 = 2020-07-05 04:06:56 UTC estimated time to finish: 0000:03:11:15 2020-07-05 10:21:47,060 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39759480 = 2020-07-05 01:13:12 UTC estimated time to finish: 0000:12:22:15 2020-07-05 10:22:21,967 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 7499656 = 2020-07-05 01:30:00 UTC, 6.3% complete 2020-07-05 10:26:41,074 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15916151 = 2020-07-05 04:39:11 UTC estimated time to finish: 0000:03:05:51 2020-07-05 10:26:47,167 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39760780 = 2020-07-05 01:21:52 UTC estimated time to finish: 0000:12:22:50 2020-07-05 10:27:22,069 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 7499812 = 2020-07-05 03:14:00 UTC, 13.5% complete 2020-07-05 10:31:41,186 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15918086 = 2020-07-05 05:11:26 UTC estimated time to finish: 0000:03:00:40 2020-07-05 10:31:47,281 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39762000 = 2020-07-05 01:30:00 UTC estimated time to finish: 0000:12:20:24 2020-07-05 10:32:22,170 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 7499969 = 2020-07-05 04:58:40 UTC, 20.8% complete 2020-07-05 10:36:41,295 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15919736 = 2020-07-05 05:38:56 UTC estimated time to finish: 0000:02:57:02 2020-07-05 10:36:47,396 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39763347 = 2020-07-05 01:38:58 UTC estimated time to finish: 0000:12:19:37 2020-07-05 10:37:22,272 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 7500123 = 2020-07-05 06:41:20 UTC, 27.9% complete 2020-07-05 10:37:43,307 INFO [download_results] forecast 2020-07-05 results files from arbutus.cloud-nowcast downloaded 2020-07-05 10:37:43,314 INFO [manager] checklist updated with [results files] items from download_results worker 2020-07-05 10:37:43,573 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['forecast', '--run-date', '2020-07-05'], host='localhost') 2020-07-05 10:37:54,786 INFO [make_CHS_currents_file] running in process 30320 2020-07-05 10:37:54,787 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-05 10:37:54,788 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5564 2020-07-05 10:37:54,789 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555 2020-07-05 10:41:41,417 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15921385 = 2020-07-05 06:06:25 UTC estimated time to finish: 0000:02:54:24 2020-07-05 10:41:47,502 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39764685 = 2020-07-05 01:47:54 UTC estimated time to finish: 0000:12:15:38 2020-07-05 10:42:03,094 INFO [make_CHS_currents_file] Made CHS currents file for 2020-07-05 for forecast 2020-07-05 10:42:03,097 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker 2020-07-05 10:42:03,279 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['nemo', 'forecast', '--run-date', '2020-07-05'], host='localhost') 2020-07-05 10:42:04,989 INFO [update_forecast_datasets] running in process 30408 2020-07-05 10:42:04,990 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-05 10:42:04,992 INFO [update_forecast_datasets] publishing log messages to tcp://*:5564 2020-07-05 10:42:04,993 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555 2020-07-05 10:42:04,994 INFO [update_forecast_datasets] updating nemo forecast directory for forecast/05jul20 run 2020-07-05 10:42:17,107 INFO [update_forecast_datasets] updated nemo forecast directory for forecast/05jul20 run: /results/SalishSea/rolling-forecasts/nemo 2020-07-05 10:42:17,109 INFO [update_forecast_datasets] nemo 2020-07-05 forecast rolling forecast datasets updated 2020-07-05 10:42:17,112 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker 2020-07-05 10:42:17,377 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nemo-forecast'], host='localhost') 2020-07-05 10:42:17,380 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'forecast', 'publish', '--run-date', '2020-07-05'], host='localhost') 2020-07-05 10:42:17,383 INFO [manager] launching NextWorker(module='nowcast.workers.make_surface_current_tiles', args=['forecast', '--run-date', '2020-07-05'], host='localhost') 2020-07-05 10:42:18,915 INFO [ping_erddap] running in process 30413 2020-07-05 10:42:18,918 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-05 10:42:18,920 INFO [ping_erddap] publishing log messages to tcp://*:5564 2020-07-05 10:42:18,921 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555 2020-07-05 10:42:20,144 INFO [make_surface_current_tiles] running in process 30415 2020-07-05 10:42:20,145 INFO [make_surface_current_tiles] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-05 10:42:20,147 INFO [make_surface_current_tiles] publishing log messages to tcp://*:5567 2020-07-05 10:42:20,148 INFO [make_surface_current_tiles] connected to skookum.eos.ubc.ca port 5555 2020-07-05 10:42:20,392 INFO [ping_erddap] nemo-forecast ERDDAP dataset flag file(s) created 2020-07-05 10:42:20,394 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker 2020-07-05 10:42:22,359 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 7500280 = 2020-07-05 08:26:00 UTC, 35.2% complete 2020-07-05 10:42:23,671 INFO [make_surface_current_tiles] starting rendering of tiles for 2020-07-05 forecast in 6 processes into /results/nowcast-sys/figures/surface_currents/forecast/05jul20 2020-07-05 10:42:30,063 INFO [make_plots] running in process 30414 2020-07-05 10:42:30,064 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-05 10:42:30,065 INFO [make_plots] publishing log messages to tcp://*:5564 2020-07-05 10:42:30,086 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555 2020-07-05 10:42:30,647 INFO [make_plots] preparing render list for 2020-07-05 NEMO forecast publish figures 2020-07-05 10:42:36,374 INFO [make_plots] added Website_thumbnail figure to 2020-07-05 NEMO forecast publish render list 2020-07-05 10:42:36,377 INFO [make_plots] added Threshold_website figure to 2020-07-05 NEMO forecast publish render list 2020-07-05 10:42:36,380 INFO [make_plots] added PA_tidal_predictions figure to 2020-07-05 NEMO forecast publish render list 2020-07-05 10:42:36,382 INFO [make_plots] added Boundary Bay figure to 2020-07-05 NEMO forecast publish render list 2020-07-05 10:42:36,383 INFO [make_plots] added Campbell River figure to 2020-07-05 NEMO forecast publish render list 2020-07-05 10:42:36,384 INFO [make_plots] added Cherry Point figure to 2020-07-05 NEMO forecast publish render list 2020-07-05 10:42:36,386 INFO [make_plots] added Friday Harbor figure to 2020-07-05 NEMO forecast publish render list 2020-07-05 10:42:36,387 INFO [make_plots] added Halfmoon Bay figure to 2020-07-05 NEMO forecast publish render list 2020-07-05 10:42:36,388 INFO [make_plots] added Nanaimo figure to 2020-07-05 NEMO forecast publish render list 2020-07-05 10:42:36,390 INFO [make_plots] added Neah Bay figure to 2020-07-05 NEMO forecast publish render list 2020-07-05 10:42:36,392 INFO [make_plots] added New Westminster figure to 2020-07-05 NEMO forecast publish render list 2020-07-05 10:42:36,394 INFO [make_plots] added Patricia Bay figure to 2020-07-05 NEMO forecast publish render list 2020-07-05 10:42:36,397 INFO [make_plots] added Point Atkinson figure to 2020-07-05 NEMO forecast publish render list 2020-07-05 10:42:36,401 INFO [make_plots] added Port Renfrew figure to 2020-07-05 NEMO forecast publish render list 2020-07-05 10:42:36,404 INFO [make_plots] added Sand Heads figure to 2020-07-05 NEMO forecast publish render list 2020-07-05 10:42:36,406 INFO [make_plots] added Sandy Cove figure to 2020-07-05 NEMO forecast publish render list 2020-07-05 10:42:36,408 INFO [make_plots] added Squamish figure to 2020-07-05 NEMO forecast publish render list 2020-07-05 10:42:36,410 INFO [make_plots] added Victoria figure to 2020-07-05 NEMO forecast publish render list 2020-07-05 10:42:36,412 INFO [make_plots] added Woodwards Landing figure to 2020-07-05 NEMO forecast publish render list 2020-07-05 10:42:36,414 INFO [make_plots] starting to render nemo forecast publish 05jul20 figures 2020-07-05 10:42:57,950 INFO [make_plots] Note: NumExpr detected 12 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8. 2020-07-05 10:42:57,951 INFO [make_plots] NumExpr defaulting to 8 threads. 2020-07-05 10:46:41,532 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15923199 = 2020-07-05 06:36:39 UTC estimated time to finish: 0000:02:50:08 2020-07-05 10:46:47,551 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39765913 = 2020-07-05 01:56:05 UTC estimated time to finish: 0000:12:17:55 2020-07-05 10:47:22,460 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 7500437 = 2020-07-05 10:10:40 UTC, 42.4% complete 2020-07-05 10:51:41,641 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15925052 = 2020-07-05 07:07:32 UTC estimated time to finish: 0000:02:45:16 2020-07-05 10:51:47,658 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39767048 = 2020-07-05 02:03:39 UTC estimated time to finish: 0000:12:18:54 2020-07-05 10:52:22,562 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 7500593 = 2020-07-05 11:54:40 UTC, 49.7% complete 2020-07-05 10:54:25,801 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 08074 Campbell River from 2020-06-30 00:00:00 to 2020-07-07 11:55:00 2020-07-05 10:56:41,764 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15926552 = 2020-07-05 07:32:32 UTC estimated time to finish: 0000:02:42:15 2020-07-05 10:56:47,764 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39768399 = 2020-07-05 02:12:39 UTC estimated time to finish: 0000:12:14:49 2020-07-05 10:57:22,663 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 7500743 = 2020-07-05 13:34:40 UTC, 56.6% complete 2020-07-05 10:58:15,574 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 9449424 Cherry Point from 2020-06-30 00:00:00 to 2020-07-07 11:55:00 2020-07-05 10:59:09,878 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 9449880 Friday Harbor from 2020-06-30 00:00:00 to 2020-07-07 11:55:00 2020-07-05 10:59:45,706 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07830 Halfmoon Bay from 2020-06-30 00:00:00 to 2020-07-07 11:55:00 2020-07-05 11:00:28,630 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07917 Nanaimo from 2020-06-30 00:00:00 to 2020-07-07 11:55:00 2020-07-05 11:01:41,869 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15928283 = 2020-07-05 08:01:23 UTC estimated time to finish: 0000:02:38:26 2020-07-05 11:01:47,873 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39769737 = 2020-07-05 02:21:34 UTC estimated time to finish: 0000:12:10:14 2020-07-05 11:02:22,764 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 7500896 = 2020-07-05 15:16:40 UTC, 63.7% complete 2020-07-05 11:04:32,001 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 9443090 Neah Bay from 2020-06-30 00:00:00 to 2020-07-07 11:55:00 2020-07-05 11:05:11,066 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07654 New Westminster from 2020-06-30 00:00:00 to 2020-07-07 11:55:00 2020-07-05 11:06:41,987 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15929946 = 2020-07-05 08:29:06 UTC estimated time to finish: 0000:02:34:47 2020-07-05 11:06:47,987 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39771034 = 2020-07-05 02:30:13 UTC estimated time to finish: 0000:12:07:07 2020-07-05 11:07:22,864 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 7501051 = 2020-07-05 17:00:00 UTC, 70.9% complete 2020-07-05 11:09:24,903 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07277 Patricia Bay from 2020-06-30 00:00:00 to 2020-07-07 11:55:00 2020-07-05 11:11:42,105 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15931637 = 2020-07-05 08:57:17 UTC estimated time to finish: 0000:02:30:45 2020-07-05 11:11:48,107 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39772222 = 2020-07-05 02:38:08 UTC estimated time to finish: 0000:12:04:05 2020-07-05 11:12:22,966 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 7501208 = 2020-07-05 18:44:40 UTC, 78.1% complete 2020-07-05 11:13:53,562 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07795 Point Atkinson from 2020-06-30 00:00:00 to 2020-07-07 11:55:00 2020-07-05 11:14:35,066 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 08525 Port Renfrew from 2020-06-30 00:00:00 to 2020-07-07 11:55:00 2020-07-05 11:16:42,214 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15933327 = 2020-07-05 09:25:27 UTC estimated time to finish: 0000:02:26:41 2020-07-05 11:16:48,222 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39773560 = 2020-07-05 02:47:04 UTC estimated time to finish: 0000:11:59:51 2020-07-05 11:17:23,067 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 7501364 = 2020-07-05 20:28:40 UTC, 85.4% complete 2020-07-05 11:19:02,478 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07594 Sand Heads from 2020-06-30 00:00:00 to 2020-07-07 11:55:00 2020-07-05 11:21:42,334 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15935100 = 2020-07-05 09:55:00 UTC estimated time to finish: 0000:02:21:22 2020-07-05 11:21:48,337 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39774952 = 2020-07-05 02:56:20 UTC estimated time to finish: 0000:11:53:33 2020-07-05 11:22:23,169 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 7501522 = 2020-07-05 22:14:00 UTC, 92.7% complete 2020-07-05 11:23:23,000 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07786 Sandy Cove from 2020-06-30 00:00:00 to 2020-07-07 11:55:00 2020-07-05 11:24:13,042 INFO [make_surface_current_tiles] starting PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast/05jul20 2020-07-05 11:26:42,446 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15937034 = 2020-07-05 10:27:14 UTC estimated time to finish: 0000:02:15:38 2020-07-05 11:26:45,017 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07811 Squamish from 2020-06-30 00:00:00 to 2020-07-07 11:55:00 2020-07-05 11:26:48,446 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39776331 = 2020-07-05 03:05:32 UTC estimated time to finish: 0000:11:48:08 2020-07-05 11:27:11,472 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07120 Victoria from 2020-06-30 00:00:00 to 2020-07-07 11:55:00 2020-07-05 11:27:23,270 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 7501678 = 2020-07-05 23:58:00 UTC, 99.9% complete 2020-07-05 11:30:38,040 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07610 Woodwards Landing from 2020-06-30 00:00:00 to 2020-07-07 11:55:00 2020-07-05 11:31:42,556 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15938930 = 2020-07-05 10:58:50 UTC estimated time to finish: 0000:02:10:02 2020-07-05 11:31:48,555 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39777902 = 2020-07-05 03:16:00 UTC estimated time to finish: 0000:11:36:54 2020-07-05 11:32:23,378 INFO [watch_NEMO] nowcast-green NEMO run on arbutus.cloud-nowcast completed 2020-07-05 11:32:23,382 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker 2020-07-05 11:32:23,926 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_wind_file', args=['arbutus.cloud-nowcast', 'forecast'], host='arbutus.cloud-nowcast') 2020-07-05 11:32:24,204 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_current_file', args=['arbutus.cloud-nowcast', 'forecast'], host='arbutus.cloud-nowcast') 2020-07-05 11:32:24,207 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['salish-nowcast', 'nowcast+', '--shared-storage'], host='localhost') 2020-07-05 11:32:24,212 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2020-07-05'], host='localhost') 2020-07-05 11:32:29,273 INFO [download_results] running in process 1165 2020-07-05 11:32:29,274 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-05 11:32:29,276 INFO [download_results] publishing log messages to tcp://*:5568 2020-07-05 11:32:29,277 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2020-07-05 11:32:29,279 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/nowcast-green/05jul20 to /results2/SalishSea/nowcast-green.201905 2020-07-05 11:32:29,282 INFO [make_forcing_links] running in process 1164 2020-07-05 11:32:29,284 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-05 11:32:29,286 INFO [make_forcing_links] publishing log messages to tcp://*:5569 2020-07-05 11:32:29,287 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555 2020-07-05 11:32:29,470 INFO [make_ww3_wind_file] running in process 15520 2020-07-05 11:32:29,472 INFO [make_ww3_wind_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2020-07-05 11:32:29,473 INFO [make_ww3_wind_file] publishing log messages to tcp://*:5570 2020-07-05 11:32:29,480 INFO [make_ww3_wind_file] connected to skookum.eos.ubc.ca port 5555 2020-07-05 11:32:29,481 INFO [make_ww3_wind_file] Creating wwatch3 wind forcing file for 2020-07-05 forecast run 2020-07-05 11:32:29,694 INFO [make_ww3_current_file] running in process 15519 2020-07-05 11:32:29,696 INFO [make_ww3_current_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2020-07-05 11:32:29,698 INFO [make_ww3_current_file] publishing log messages to tcp://*:5571 2020-07-05 11:32:29,700 INFO [make_ww3_current_file] connected to skookum.eos.ubc.ca port 5555 2020-07-05 11:32:29,702 INFO [make_ww3_current_file] Creating wwatch3 currents forcing file for 2020-07-05 forecast run 2020-07-05 11:32:33,434 INFO [make_ww3_wind_file] wwatch3 wind forcing file created on arbutus.cloud-nowcast for 2020-07-05 forecast run 2020-07-05 11:32:33,439 INFO [manager] checklist updated with [WW3 wind forcing] items from make_ww3_wind_file worker 2020-07-05 11:32:42,150 INFO [make_forcing_links] nowcast+ 2020-07-05 forcing file links on salish-nowcast created 2020-07-05 11:32:42,155 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker 2020-07-05 11:32:42,452 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['salish-nowcast', 'nowcast-dev', '--run-date', '2020-07-05'], host='salish-nowcast') 2020-07-05 11:32:44,090 INFO [run_NEMO] running in process 26911 2020-07-05 11:32:44,091 INFO [run_NEMO] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-05 11:32:44,092 INFO [run_NEMO] publishing log messages to tcp://*:5556 2020-07-05 11:32:44,093 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555 2020-07-05 11:32:47,036 WARNING [run_NEMO] There are uncommitted changes in /SalishSeaCast/SalishSeaNowcast 2020-07-05 11:32:47,838 INFO [run_NEMO] nowcast-dev: launching /SalishSeaCast/runs/05jul20nowcast-dev_2020-07-05T113245.874414-0700/SalishSeaNEMO.sh on salish-nowcast 2020-07-05 11:32:47,942 INFO [run_NEMO] nowcast-dev NEMO run for 2020-07-05 on salish-nowcast started 2020-07-05 11:32:47,951 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker 2020-07-05 11:32:48,230 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['salish-nowcast', 'nowcast-dev'], host='salish-nowcast') 2020-07-05 11:32:49,540 INFO [watch_NEMO] running in process 27128 2020-07-05 11:32:49,541 INFO [watch_NEMO] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-05 11:32:49,542 INFO [watch_NEMO] publishing log messages to tcp://*:5557 2020-07-05 11:32:49,543 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555 2020-07-05 11:32:49,817 INFO [watch_NEMO] nowcast-dev on salish-nowcast: time.step not found; continuing to watch... 2020-07-05 11:33:21,928 INFO [make_ww3_current_file] wwatch3 currents forcing file created on arbutus.cloud-nowcast for 2020-07-05 forecast run 2020-07-05 11:33:21,935 INFO [manager] checklist updated with [WW3 currents forcing] items from make_ww3_current_file worker 2020-07-05 11:33:22,196 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2020-07-05'], host='arbutus.cloud-nowcast') 2020-07-05 11:33:24,357 INFO [run_ww3] running in process 15739 2020-07-05 11:33:24,359 INFO [run_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2020-07-05 11:33:24,360 INFO [run_ww3] publishing log messages to tcp://*:5572 2020-07-05 11:33:24,361 INFO [run_ww3] connected to skookum.eos.ubc.ca port 5555 2020-07-05 11:33:24,365 INFO [run_ww3] Created run directory /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/nowcast_2020-07-05T113324.355403-0700 2020-07-05 11:33:24,367 INFO [run_ww3] nowcast: launching /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/nowcast_2020-07-05T113324.355403-0700/SoGWW3.sh on arbutus.cloud-nowcast 2020-07-05 11:33:24,391 INFO [run_ww3] nowcast WaveWatch3 run for 2020-07-05 on arbutus.cloud-nowcast started 2020-07-05 11:33:24,400 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker 2020-07-05 11:33:24,664 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'nowcast'], host='arbutus.cloud-nowcast') 2020-07-05 11:33:26,620 INFO [watch_ww3] running in process 15843 2020-07-05 11:33:26,622 INFO [watch_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2020-07-05 11:33:26,624 INFO [watch_ww3] publishing log messages to tcp://*:5573 2020-07-05 11:33:26,625 INFO [watch_ww3] connected to skookum.eos.ubc.ca port 5555 2020-07-05 11:33:26,657 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch... 2020-07-05 11:33:37,262 INFO [make_plots] finished rendering nemo forecast publish 05jul20 figures 2020-07-05 11:33:37,264 INFO [make_plots] nemo publish plots for 2020-07-05 forecast completed 2020-07-05 11:33:37,275 INFO [manager] checklist updated with [plots] items from make_plots worker 2020-07-05 11:33:37,536 INFO [manager] launching NextWorker(module='nowcast.workers.make_feeds', args=['forecast', '--run-date', '2020-07-05'], host='localhost') 2020-07-05 11:33:48,622 INFO [make_feeds] running in process 1215 2020-07-05 11:33:48,624 INFO [make_feeds] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-05 11:33:48,625 INFO [make_feeds] publishing log messages to tcp://*:5564 2020-07-05 11:33:48,626 INFO [make_feeds] connected to skookum.eos.ubc.ca port 5555 2020-07-05 11:33:57,904 INFO [make_feeds] Note: NumExpr detected 12 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8. 2020-07-05 11:33:57,905 INFO [make_feeds] NumExpr defaulting to 8 threads. 2020-07-05 11:34:06,975 INFO [make_feeds] ATOM feeds for 2020-07-05 forecast run completed 2020-07-05 11:34:07,196 INFO [manager] checklist updated with [storm surge alert feeds] items from make_feeds worker 2020-07-05 11:36:42,669 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15940811 = 2020-07-05 11:30:11 UTC estimated time to finish: 0000:02:04:44 2020-07-05 11:36:48,665 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39779404 = 2020-07-05 03:26:01 UTC estimated time to finish: 0000:11:25:27 2020-07-05 11:37:49,757 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4628898 = 2020-07-05 00:11:20 UTC, 0.8% complete 2020-07-05 11:38:26,757 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 93, 21.5% complete 2020-07-05 11:41:42,783 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15942556 = 2020-07-05 11:59:16 UTC estimated time to finish: 0000:01:59:45 2020-07-05 11:41:48,689 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39781140 = 2020-07-05 03:37:36 UTC estimated time to finish: 0000:11:11:58 2020-07-05 11:42:49,859 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4628946 = 2020-07-05 00:43:20 UTC, 3.0% complete 2020-07-05 11:43:26,857 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 201, 46.5% complete 2020-07-05 11:46:42,896 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15944438 = 2020-07-05 12:30:38 UTC estimated time to finish: 0000:01:54:25 2020-07-05 11:46:48,803 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39782846 = 2020-07-05 03:48:58 UTC estimated time to finish: 0000:10:59:40 2020-07-05 11:47:49,960 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4628997 = 2020-07-05 01:17:20 UTC, 5.4% complete 2020-07-05 11:48:26,958 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 309, 71.5% complete 2020-07-05 11:51:43,011 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15946250 = 2020-07-05 13:00:50 UTC estimated time to finish: 0000:01:49:32 2020-07-05 11:51:48,914 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39784499 = 2020-07-05 03:59:59 UTC estimated time to finish: 0000:10:48:41 2020-07-05 11:52:50,061 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4629050 = 2020-07-05 01:52:40 UTC, 7.8% complete 2020-07-05 11:53:27,002 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 417, 96.5% complete 2020-07-05 11:55:54,218 INFO [download_results] nowcast-green 2020-07-05 results files from arbutus.cloud-nowcast downloaded 2020-07-05 11:55:54,227 INFO [manager] checklist updated with [results files] items from download_results worker 2020-07-05 11:55:54,427 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast-green', 'research', '--run-date', '2020-07-05'], host='localhost') 2020-07-05 11:55:54,432 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nowcast-green'], host='localhost') 2020-07-05 11:55:59,225 INFO [ping_erddap] running in process 1644 2020-07-05 11:55:59,226 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-05 11:55:59,228 INFO [ping_erddap] publishing log messages to tcp://*:5564 2020-07-05 11:55:59,229 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555 2020-07-05 11:55:59,235 INFO [ping_erddap] nowcast-green ERDDAP dataset flag file(s) created 2020-07-05 11:55:59,238 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker 2020-07-05 11:56:12,847 INFO [make_plots] running in process 1643 2020-07-05 11:56:12,852 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-05 11:56:12,854 INFO [make_plots] publishing log messages to tcp://*:5564 2020-07-05 11:56:12,855 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555 2020-07-05 11:56:13,734 INFO [make_plots] preparing render list for 2020-07-05 NEMO nowcast-green research figures 2020-07-05 11:56:29,904 INFO [make_plots] added temperature_thalweg_and_surface figures to 2020-07-05 NEMO nowcast-green research render list 2020-07-05 11:56:38,090 INFO [make_plots] added nitrate_thalweg_and_surface figures to 2020-07-05 NEMO nowcast-green research render list 2020-07-05 11:56:43,126 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15947913 = 2020-07-05 13:28:33 UTC estimated time to finish: 0000:01:45:20 2020-07-05 11:56:45,644 INFO [make_plots] added ammonium_thalweg_and_surface figures to 2020-07-05 NEMO nowcast-green research render list 2020-07-05 11:56:49,026 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39786040 = 2020-07-05 04:10:16 UTC estimated time to finish: 0000:10:39:14 2020-07-05 11:56:54,948 INFO [make_plots] added silicon_thalweg_and_surface figures to 2020-07-05 NEMO nowcast-green research render list 2020-07-05 11:57:01,180 INFO [make_plots] added dissolved_organic_nitrogen_thalweg_and_surface figures to 2020-07-05 NEMO nowcast-green research render list 2020-07-05 11:57:09,730 INFO [make_plots] added particulate_organic_nitrogen_thalweg_and_surface figures to 2020-07-05 NEMO nowcast-green research render list 2020-07-05 11:57:18,257 INFO [make_plots] added biogenic_silicon_thalweg_and_surface figures to 2020-07-05 NEMO nowcast-green research render list 2020-07-05 11:57:28,104 INFO [make_plots] added diatoms_thalweg_and_surface figures to 2020-07-05 NEMO nowcast-green research render list 2020-07-05 11:57:36,880 INFO [make_plots] added ciliates_thalweg_and_surface figures to 2020-07-05 NEMO nowcast-green research render list 2020-07-05 11:57:45,290 INFO [make_plots] added flagellates_thalweg_and_surface figures to 2020-07-05 NEMO nowcast-green research render list 2020-07-05 11:57:50,132 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4629104 = 2020-07-05 02:28:40 UTC, 10.3% complete 2020-07-05 11:57:53,352 INFO [make_plots] added microzooplankton_thalweg_and_surface figures to 2020-07-05 NEMO nowcast-green research render list 2020-07-05 11:58:02,324 INFO [make_plots] added mesozooplankton_thalweg_and_surface figures to 2020-07-05 NEMO nowcast-green research render list 2020-07-05 11:58:09,870 INFO [make_plots] added Fraser_tracer_thalweg_and_surface figures to 2020-07-05 NEMO nowcast-green research render list 2020-07-05 11:58:10,125 INFO [make_plots] added temperature_salinity_timeseries figure to 2020-07-05 NEMO nowcast-green research render list 2020-07-05 11:58:10,126 INFO [make_plots] added nitrate_diatoms_timeseries figure to 2020-07-05 NEMO nowcast-green research render list 2020-07-05 11:58:10,126 INFO [make_plots] added mesodinium_flagellates_timeseries figure to 2020-07-05 NEMO nowcast-green research render list 2020-07-05 11:58:10,127 INFO [make_plots] added mesozoo_microzoo_timeseries figure to 2020-07-05 NEMO nowcast-green research render list 2020-07-05 11:58:10,127 INFO [make_plots] starting to render nemo nowcast-green research 05jul20 figures 2020-07-05 11:58:27,104 INFO [watch_ww3] nowcast WWATCH3 run on arbutus.cloud-nowcast completed 2020-07-05 11:58:27,107 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker 2020-07-05 11:58:27,638 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2020-07-05'], host='localhost') 2020-07-05 11:58:27,645 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2020-07-05'], host='arbutus.cloud-nowcast') 2020-07-05 11:58:29,702 INFO [run_ww3] running in process 16007 2020-07-05 11:58:29,704 INFO [run_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2020-07-05 11:58:29,706 INFO [run_ww3] publishing log messages to tcp://*:5572 2020-07-05 11:58:29,706 INFO [run_ww3] connected to skookum.eos.ubc.ca port 5555 2020-07-05 11:58:29,710 INFO [run_ww3] Created run directory /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast_2020-07-05T115829.702152-0700 2020-07-05 11:58:29,711 INFO [run_ww3] forecast: launching /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast_2020-07-05T115829.702152-0700/SoGWW3.sh on arbutus.cloud-nowcast 2020-07-05 11:58:29,727 INFO [run_ww3] forecast WaveWatch3 run for 2020-07-05 on arbutus.cloud-nowcast started 2020-07-05 11:58:29,731 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker 2020-07-05 11:58:29,889 INFO [download_wwatch3_results] running in process 1748 2020-07-05 11:58:29,890 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-05 11:58:29,891 INFO [download_wwatch3_results] publishing log messages to tcp://*:5568 2020-07-05 11:58:29,892 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555 2020-07-05 11:58:29,973 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'forecast'], host='arbutus.cloud-nowcast') 2020-07-05 11:58:31,903 INFO [watch_ww3] running in process 16177 2020-07-05 11:58:31,905 INFO [watch_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2020-07-05 11:58:31,907 INFO [watch_ww3] publishing log messages to tcp://*:5573 2020-07-05 11:58:31,908 INFO [watch_ww3] connected to skookum.eos.ubc.ca port 5555 2020-07-05 11:58:31,938 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch... 2020-07-05 11:59:52,038 INFO [download_wwatch3_results] nowcast 2020-07-05 results files from arbutus.cloud-nowcast downloaded 2020-07-05 11:59:52,042 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker 2020-07-05 12:01:43,243 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15949413 = 2020-07-05 13:53:33 UTC estimated time to finish: 0000:01:41:27 2020-07-05 12:01:49,138 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39787693 = 2020-07-05 04:21:17 UTC estimated time to finish: 0000:10:29:26 2020-07-05 12:02:50,232 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4629159 = 2020-07-05 03:05:20 UTC, 12.9% complete 2020-07-05 12:03:32,039 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 93, 14.4% complete 2020-07-05 12:06:43,359 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15951062 = 2020-07-05 14:21:02 UTC estimated time to finish: 0000:01:37:13 2020-07-05 12:06:49,251 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39789318 = 2020-07-05 04:32:07 UTC estimated time to finish: 0000:10:21:00 2020-07-05 12:07:49,896 INFO [make_surface_current_tiles] finished PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast/05jul20 2020-07-05 12:07:49,911 INFO [make_surface_current_tiles] finished rendering of tiles for 2020-07-05 forecast into /results/nowcast-sys/figures/surface_currents/forecast/05jul20 2020-07-05 12:07:49,912 INFO [make_surface_current_tiles] surface current tile figures for 2020-07-05 forecast completed 2020-07-05 12:07:50,334 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4629214 = 2020-07-05 03:42:00 UTC, 15.4% complete 2020-07-05 12:07:50,434 INFO [manager] checklist updated with [surface current tiles] items from make_surface_current_tiles worker 2020-07-05 12:08:32,141 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 201, 31.0% complete 2020-07-05 12:11:43,472 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15952712 = 2020-07-05 14:48:32 UTC estimated time to finish: 0000:01:32:55 2020-07-05 12:11:49,366 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39790970 = 2020-07-05 04:43:08 UTC estimated time to finish: 0000:10:11:48 2020-07-05 12:12:50,435 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4629269 = 2020-07-05 04:18:40 UTC, 18.0% complete 2020-07-05 12:13:32,242 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 309, 47.7% complete 2020-07-05 12:16:43,590 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15954374 = 2020-07-05 15:16:14 UTC estimated time to finish: 0000:01:28:31 2020-07-05 12:16:49,480 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39792513 = 2020-07-05 04:53:25 UTC estimated time to finish: 0000:10:03:13 2020-07-05 12:17:50,536 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4629324 = 2020-07-05 04:55:20 UTC, 20.5% complete 2020-07-05 12:18:32,343 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 417, 64.4% complete 2020-07-05 12:21:43,707 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15956024 = 2020-07-05 15:43:44 UTC estimated time to finish: 0000:01:24:07 2020-07-05 12:21:49,593 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39794207 = 2020-07-05 05:04:42 UTC estimated time to finish: 0000:09:54:07 2020-07-05 12:22:50,637 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4629379 = 2020-07-05 05:32:00 UTC, 23.1% complete 2020-07-05 12:23:32,372 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 525, 81.0% complete 2020-07-05 12:26:43,825 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15957523 = 2020-07-05 16:08:43 UTC estimated time to finish: 0000:01:20:02 2020-07-05 12:26:49,707 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39795859 = 2020-07-05 05:15:43 UTC estimated time to finish: 0000:09:46:05 2020-07-05 12:27:50,641 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4629434 = 2020-07-05 06:08:40 UTC, 25.6% complete 2020-07-05 12:28:32,473 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 633, 97.7% complete 2020-07-05 12:31:43,943 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15959213 = 2020-07-05 16:36:53 UTC estimated time to finish: 0000:01:15:25 2020-07-05 12:31:49,823 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39797565 = 2020-07-05 05:27:06 UTC estimated time to finish: 0000:09:37:40 2020-07-05 12:32:50,742 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4629488 = 2020-07-05 06:44:40 UTC, 28.1% complete 2020-07-05 12:33:32,545 INFO [watch_ww3] forecast WWATCH3 run on arbutus.cloud-nowcast completed 2020-07-05 12:33:32,562 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker 2020-07-05 12:33:33,336 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2020-07-05'], host='localhost') 2020-07-05 12:33:40,720 INFO [download_wwatch3_results] running in process 2494 2020-07-05 12:33:40,721 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-05 12:33:40,722 INFO [download_wwatch3_results] publishing log messages to tcp://*:5567 2020-07-05 12:33:40,723 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555 2020-07-05 12:35:31,429 INFO [download_wwatch3_results] forecast 2020-07-05 results files from arbutus.cloud-nowcast downloaded 2020-07-05 12:35:31,434 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker 2020-07-05 12:35:32,076 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['wwatch3', 'forecast', '--run-date', '2020-07-05'], host='localhost') 2020-07-05 12:35:38,242 INFO [update_forecast_datasets] running in process 2514 2020-07-05 12:35:38,330 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-05 12:35:38,331 INFO [update_forecast_datasets] publishing log messages to tcp://*:5567 2020-07-05 12:35:38,332 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555 2020-07-05 12:35:38,333 INFO [update_forecast_datasets] updating wwatch3 most_recent_forecast directory from forecast/05jul20 run 2020-07-05 12:35:38,335 INFO [update_forecast_datasets] updated wwatch3 most_recent_forecast directory from forecast/05jul20 run 2020-07-05 12:35:38,336 INFO [update_forecast_datasets] updating wwatch3 forecast directory for forecast/05jul20 run 2020-07-05 12:35:50,642 INFO [update_forecast_datasets] updated wwatch3 forecast directory for forecast/05jul20 run: /results/SalishSea/rolling-forecasts/wwatch3 2020-07-05 12:35:51,072 INFO [update_forecast_datasets] wwatch3 2020-07-05 forecast rolling forecast datasets updated 2020-07-05 12:35:51,074 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker 2020-07-05 12:35:51,476 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['wwatch3-forecast'], host='localhost') 2020-07-05 12:35:54,299 INFO [ping_erddap] running in process 2520 2020-07-05 12:35:54,588 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-05 12:35:54,588 INFO [ping_erddap] publishing log messages to tcp://*:5567 2020-07-05 12:35:54,589 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555 2020-07-05 12:35:54,590 INFO [ping_erddap] wwatch3-forecast ERDDAP dataset flag file(s) created 2020-07-05 12:35:54,591 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker 2020-07-05 12:35:54,983 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['wwatch3', 'forecast', 'publish', '--run-date', '2020-07-05'], host='localhost') 2020-07-05 12:36:26,368 INFO [make_plots] running in process 2524 2020-07-05 12:36:26,537 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-05 12:36:26,537 INFO [make_plots] publishing log messages to tcp://*:5567 2020-07-05 12:36:26,538 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555 2020-07-05 12:36:26,539 INFO [make_plots] preparing render list for 2020-07-05 WaveWatch3 forecast publish figures 2020-07-05 12:36:26,539 INFO [make_plots] added Halibut Bank figure to 2020-07-05 WaveWatch3 forecast publish render list 2020-07-05 12:36:26,540 INFO [make_plots] added Sentry Shoal figure to 2020-07-05 WaveWatch3 forecast publish render list 2020-07-05 12:36:26,541 INFO [make_plots] starting to render wwatch3 forecast publish 05jul20 figures 2020-07-05 12:36:44,061 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15960835 = 2020-07-05 17:03:55 UTC estimated time to finish: 0000:01:10:59 2020-07-05 12:36:49,937 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39799218 = 2020-07-05 05:38:07 UTC estimated time to finish: 0000:09:29:41 2020-07-05 12:37:50,843 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4629542 = 2020-07-05 07:20:40 UTC, 30.6% complete 2020-07-05 12:41:44,180 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15962526 = 2020-07-05 17:32:06 UTC estimated time to finish: 0000:01:06:19 2020-07-05 12:41:50,052 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39800720 = 2020-07-05 05:48:08 UTC estimated time to finish: 0000:09:22:45 2020-07-05 12:42:50,944 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4629597 = 2020-07-05 07:57:20 UTC, 33.2% complete 2020-07-05 12:43:32,780 INFO [make_plots] Note: NumExpr detected 12 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8. 2020-07-05 12:43:32,781 INFO [make_plots] NumExpr defaulting to 8 threads. 2020-07-05 12:43:32,797 INFO [make_plots] retrieving last 45 days of real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46146 Halibut Bank 2020-07-05 12:46:13,037 INFO [make_plots] retrieving last 45 days of real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46131 Sentry Shoal 2020-07-05 12:46:18,091 INFO [make_plots] finished rendering wwatch3 forecast publish 05jul20 figures 2020-07-05 12:46:18,091 INFO [make_plots] wwatch3 publish plots for 2020-07-05 forecast completed 2020-07-05 12:46:18,093 INFO [manager] checklist updated with [plots] items from make_plots worker 2020-07-05 12:46:44,299 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15964025 = 2020-07-05 17:57:05 UTC estimated time to finish: 0000:01:02:08 2020-07-05 12:46:50,169 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39802372 = 2020-07-05 05:59:08 UTC estimated time to finish: 0000:09:15:21 2020-07-05 12:47:51,009 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4629651 = 2020-07-05 08:33:20 UTC, 35.7% complete 2020-07-05 12:51:44,420 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15965715 = 2020-07-05 18:25:15 UTC estimated time to finish: 0000:00:57:26 2020-07-05 12:51:50,285 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39804078 = 2020-07-05 06:10:31 UTC estimated time to finish: 0000:09:07:39 2020-07-05 12:52:51,110 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4629704 = 2020-07-05 09:08:40 UTC, 38.1% complete 2020-07-05 12:56:44,540 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15967379 = 2020-07-05 18:52:59 UTC estimated time to finish: 0000:00:52:45 2020-07-05 12:56:50,402 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39805730 = 2020-07-05 06:21:32 UTC estimated time to finish: 0000:09:00:33 2020-07-05 12:57:51,211 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4629759 = 2020-07-05 09:45:20 UTC, 40.7% complete 2020-07-05 13:01:44,584 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15969028 = 2020-07-05 19:20:28 UTC estimated time to finish: 0000:00:48:08 2020-07-05 13:01:50,520 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39807191 = 2020-07-05 06:31:16 UTC estimated time to finish: 0000:08:54:28 2020-07-05 13:02:51,312 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4629813 = 2020-07-05 10:21:20 UTC, 43.2% complete 2020-07-05 13:06:44,705 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15970528 = 2020-07-05 19:45:28 UTC estimated time to finish: 0000:00:43:53 2020-07-05 13:06:50,638 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39808844 = 2020-07-05 06:42:17 UTC estimated time to finish: 0000:08:47:31 2020-07-05 13:07:51,413 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4629867 = 2020-07-05 10:57:20 UTC, 45.7% complete 2020-07-05 13:11:44,816 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15972260 = 2020-07-05 20:14:20 UTC estimated time to finish: 0000:00:38:57 2020-07-05 13:11:50,757 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39810550 = 2020-07-05 06:53:40 UTC estimated time to finish: 0000:08:40:08 2020-07-05 13:12:51,513 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4629922 = 2020-07-05 11:34:00 UTC, 48.2% complete 2020-07-05 13:16:44,939 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15974153 = 2020-07-05 20:45:53 UTC estimated time to finish: 0000:00:33:27 2020-07-05 13:16:50,876 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39812244 = 2020-07-05 07:04:57 UTC estimated time to finish: 0000:08:32:59 2020-07-05 13:17:51,614 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4629975 = 2020-07-05 12:09:20 UTC, 50.7% complete 2020-07-05 13:21:45,062 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15976048 = 2020-07-05 21:17:28 UTC estimated time to finish: 0000:00:27:58 2020-07-05 13:21:51,000 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39813786 = 2020-07-05 07:15:14 UTC estimated time to finish: 0000:08:26:24 2020-07-05 13:22:51,715 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4630030 = 2020-07-05 12:46:00 UTC, 53.2% complete 2020-07-05 13:26:45,186 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15977971 = 2020-07-05 21:49:31 UTC estimated time to finish: 0000:00:22:24 2020-07-05 13:26:51,117 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39815520 = 2020-07-05 07:26:48 UTC estimated time to finish: 0000:08:19:13 2020-07-05 13:27:51,815 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4630084 = 2020-07-05 13:22:00 UTC, 55.7% complete 2020-07-05 13:31:45,312 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15979674 = 2020-07-05 22:17:54 UTC estimated time to finish: 0000:00:17:31 2020-07-05 13:31:51,237 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39817174 = 2020-07-05 07:37:49 UTC estimated time to finish: 0000:08:12:39 2020-07-05 13:32:51,916 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4630139 = 2020-07-05 13:58:40 UTC, 58.3% complete 2020-07-05 13:36:45,436 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15981598 = 2020-07-05 22:49:58 UTC estimated time to finish: 0000:00:11:59 2020-07-05 13:36:51,358 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39818840 = 2020-07-05 07:48:56 UTC estimated time to finish: 0000:08:06:05 2020-07-05 13:37:52,004 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4630193 = 2020-07-05 14:34:40 UTC, 60.8% complete 2020-07-05 13:41:45,561 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15983491 = 2020-07-05 23:21:31 UTC estimated time to finish: 0000:00:06:35 2020-07-05 13:41:51,480 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39820532 = 2020-07-05 08:00:12 UTC estimated time to finish: 0000:07:59:23 2020-07-05 13:42:52,105 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4630247 = 2020-07-05 15:10:40 UTC, 63.3% complete 2020-07-05 13:46:45,687 INFO [watch_fvcom] x2 nowcast on arbutus.cloud-nowcast: timestep: 15985387 = 2020-07-05 23:53:07 UTC estimated time to finish: 0000:00:01:10 2020-07-05 13:46:51,553 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39822075 = 2020-07-05 08:10:30 UTC estimated time to finish: 0000:07:53:20 2020-07-05 13:47:52,148 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4630302 = 2020-07-05 15:47:20 UTC, 65.8% complete 2020-07-05 13:51:45,795 INFO [watch_fvcom] x2 nowcast FVCOM VH-FR run on arbutus.cloud-nowcast completed 2020-07-05 13:51:45,801 INFO [manager] checklist updated with [FVCOM run] items from watch_fvcom worker 2020-07-05 13:51:46,814 INFO [manager] launching NextWorker(module='nowcast.workers.download_fvcom_results', args=['arbutus.cloud-nowcast', 'x2', 'nowcast', '--run-date', '2020-07-05'], host='localhost') 2020-07-05 13:51:46,818 INFO [manager] launching NextWorker(module='nowcast.workers.make_fvcom_boundary', args=['arbutus.cloud-nowcast', 'x2', 'forecast', '--run-date', '2020-07-05'], host='arbutus.cloud-nowcast') 2020-07-05 13:51:51,677 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39824439 = 2020-07-05 08:26:15 UTC estimated time to finish: 0000:07:40:26 2020-07-05 13:51:52,925 INFO [make_fvcom_boundary] running in process 16661 2020-07-05 13:51:53,060 INFO [make_fvcom_boundary] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2020-07-05 13:51:53,060 INFO [make_fvcom_boundary] publishing log messages to tcp://*:5580 2020-07-05 13:51:53,061 INFO [make_fvcom_boundary] connected to skookum.eos.ubc.ca port 5555 2020-07-05 13:51:53,062 INFO [make_fvcom_boundary] Creating VHFR FVCOM open boundary file for x2 forecast run from 2020-07-05 NEMO run 2020-07-05 13:51:55,225 INFO [download_fvcom_results] running in process 4319 2020-07-05 13:51:55,227 INFO [download_fvcom_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-05 13:51:55,228 INFO [download_fvcom_results] publishing log messages to tcp://*:5567 2020-07-05 13:51:55,229 INFO [download_fvcom_results] connected to skookum.eos.ubc.ca port 5555 2020-07-05 13:52:52,219 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4630356 = 2020-07-05 16:23:20 UTC, 68.3% complete 2020-07-05 13:53:30,699 INFO [download_fvcom_results] VHFR FVCOM x2 nowcast 2020-07-05 results files from arbutus.cloud-nowcast downloaded 2020-07-05 13:53:30,706 INFO [manager] checklist updated with [VHFR FVCOM results files] items from download_fvcom_results worker 2020-07-05 13:53:31,217 INFO [manager] launching NextWorker(module='nowcast.workers.get_vfpa_hadcp', args=['--data-date', '2020-07-05'], host='localhost') 2020-07-05 13:53:31,221 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['fvcom', 'nowcast-x2', 'research', '--run-date', '2020-07-05'], host='localhost') 2020-07-05 13:53:51,066 INFO [get_vfpa_hadcp] running in process 4366 2020-07-05 13:53:51,293 INFO [get_vfpa_hadcp] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-05 13:53:51,293 INFO [get_vfpa_hadcp] publishing log messages to tcp://*:5567 2020-07-05 13:53:51,294 INFO [get_vfpa_hadcp] connected to skookum.eos.ubc.ca port 5555 2020-07-05 13:53:51,295 INFO [get_vfpa_hadcp] processing VFPA HADCP data from 2nd Narrows Rail Bridge for 2020-07-05 2020-07-05 13:53:55,593 INFO [get_vfpa_hadcp] added VFPA HADCP data from 2nd Narrows Rail Bridge for 2020-07-05 to /opp/observations/AISDATA/netcdf/VFPA_2ND_NARROWS_HADCP_2s_202007.nc 2020-07-05 13:53:55,594 INFO [get_vfpa_hadcp] VFPA HADCP observations added to 2020-07 netcdf file 2020-07-05 13:53:55,597 INFO [manager] checklist updated with [VFPA HADCP data] items from get_vfpa_hadcp worker 2020-07-05 13:53:56,062 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['VFPA-HADCP'], host='localhost') 2020-07-05 13:54:01,285 INFO [ping_erddap] running in process 4376 2020-07-05 13:54:01,286 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-05 13:54:01,288 INFO [ping_erddap] publishing log messages to tcp://*:5567 2020-07-05 13:54:01,289 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555 2020-07-05 13:54:01,291 INFO [ping_erddap] VFPA-HADCP ERDDAP dataset flag file(s) created 2020-07-05 13:54:01,294 INFO [make_plots] running in process 4367 2020-07-05 13:54:01,296 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker 2020-07-05 13:54:01,297 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-05 13:54:01,298 INFO [make_plots] publishing log messages to tcp://*:5568 2020-07-05 13:54:01,300 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555 2020-07-05 13:54:01,301 INFO [make_plots] preparing render list for 2020-07-05 VHFR FVCOM nowcast-x2 research figures 2020-07-05 13:54:01,303 INFO [make_plots] added EnglishBay_surface_currents figures to 2020-07-05 VHFR FVCOM nowcast-x2 research render list 2020-07-05 13:54:01,305 INFO [make_plots] added VancouverHarbour_surface_currents figures to 2020-07-05 VHFR FVCOM nowcast-x2 research render list 2020-07-05 13:54:01,306 INFO [make_plots] added IndianArm_surface_currents figures to 2020-07-05 VHFR FVCOM nowcast-x2 research render list 2020-07-05 13:54:01,308 INFO [make_plots] added VancouverHarbour_thalweg figures to 2020-07-05 VHFR FVCOM nowcast-x2 research render list 2020-07-05 13:54:01,310 INFO [make_plots] added PortMoody_thalweg figures to 2020-07-05 VHFR FVCOM nowcast-x2 research render list 2020-07-05 13:54:01,312 INFO [make_plots] added IndianArm_thalweg figures to 2020-07-05 VHFR FVCOM nowcast-x2 research render list 2020-07-05 13:54:01,314 INFO [make_plots] starting to render fvcom nowcast-x2 research 05jul20 figures 2020-07-05 13:54:01,858 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['fvcom', 'nowcast-x2', 'publish', '--run-date', '2020-07-05'], host='localhost') 2020-07-05 13:54:23,930 INFO [make_plots] running in process 4403 2020-07-05 13:54:24,141 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-05 13:54:24,142 INFO [make_plots] publishing log messages to tcp://*:5567 2020-07-05 13:54:24,143 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555 2020-07-05 13:54:28,334 INFO [make_plots] preparing render list for 2020-07-05 VHFR FVCOM nowcast-x2 publish figures 2020-07-05 13:54:28,477 INFO [make_plots] added Calamity Point figure to 2020-07-05 VHFR FVCOM nowcast-x2 publish render list 2020-07-05 13:54:28,478 INFO [make_plots] added Indian Arm Head figure to 2020-07-05 VHFR FVCOM nowcast-x2 publish render list 2020-07-05 13:54:28,478 INFO [make_plots] added New Westminster figure to 2020-07-05 VHFR FVCOM nowcast-x2 publish render list 2020-07-05 13:54:28,479 INFO [make_plots] added Port Moody figure to 2020-07-05 VHFR FVCOM nowcast-x2 publish render list 2020-07-05 13:54:28,480 INFO [make_plots] added Sand Heads figure to 2020-07-05 VHFR FVCOM nowcast-x2 publish render list 2020-07-05 13:54:28,481 INFO [make_plots] added Sandy Cove figure to 2020-07-05 VHFR FVCOM nowcast-x2 publish render list 2020-07-05 13:54:28,482 INFO [make_plots] added Vancouver Harbour figure to 2020-07-05 VHFR FVCOM nowcast-x2 publish render list 2020-07-05 13:54:28,483 INFO [make_plots] added Woodwards Landing figure to 2020-07-05 VHFR FVCOM nowcast-x2 publish render list 2020-07-05 13:54:28,484 INFO [make_plots] added 2ndNarrows_current figure to 2020-07-05 VHFR FVCOM nowcast-x2 publish render list 2020-07-05 13:54:28,485 INFO [make_plots] starting to render fvcom nowcast-x2 publish 05jul20 figures 2020-07-05 13:54:28,561 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07724 Calamity Point from 2020-07-04 23:55:00 to 2020-07-06 00:00:00 2020-07-05 13:54:57,314 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07724 Calamity Point from 2020-07-05 00:00:00 to 2020-07-06 00:00:00 2020-07-05 13:55:01,616 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07774 Indian Arm Head from 2020-07-04 23:55:00 to 2020-07-06 00:00:00 2020-07-05 13:55:02,469 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07774 Indian Arm Head from 2020-07-05 00:00:00 to 2020-07-06 00:00:00 2020-07-05 13:55:07,691 INFO [make_fvcom_boundary] Stored VHFR FVCOM open boundary file: /nemoShare/MEOPAR/nowcast-sys/fvcom-runs/input.x2/bdy_x2_forecast_brcl_20200706.nc 2020-07-05 13:55:07,694 INFO [make_fvcom_boundary] FVCOM x2 forecast run boundary condition file for 2020-07-05 created on arbutus.cloud-nowcast 2020-07-05 13:55:07,699 INFO [manager] checklist updated with [FVCOM boundary conditions] items from make_fvcom_boundary worker 2020-07-05 13:55:08,460 INFO [manager] launching NextWorker(module='nowcast.workers.make_fvcom_atmos_forcing', args=['x2', 'forecast', '--run-date', '2020-07-05'], host='localhost') 2020-07-05 13:55:08,465 INFO [manager] launching NextWorker(module='nowcast.workers.make_fvcom_rivers_forcing', args=['arbutus.cloud-nowcast', 'x2', 'forecast', '--run-date', '2020-07-05'], host='arbutus.cloud-nowcast') 2020-07-05 13:55:12,048 INFO [make_fvcom_rivers_forcing] running in process 16843 2020-07-05 13:55:12,049 INFO [make_fvcom_rivers_forcing] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2020-07-05 13:55:12,049 INFO [make_fvcom_rivers_forcing] publishing log messages to tcp://*:5582 2020-07-05 13:55:12,050 INFO [make_fvcom_rivers_forcing] connected to skookum.eos.ubc.ca port 5555 2020-07-05 13:55:12,050 INFO [make_fvcom_rivers_forcing] Creating VHFR FVCOM rivers forcing file for x2 forecast run from 2020-07-05 NEMO runoff forcing files 2020-07-05 13:55:12,246 INFO [make_fvcom_rivers_forcing] Stored VHFR FVCOM rivers forcing file: /nemoShare/MEOPAR/nowcast-sys/fvcom-runs/input.x2/rivers_x2_forecast_20200706.nc 2020-07-05 13:55:12,246 INFO [make_fvcom_rivers_forcing] FVCOM x2 forecast run rivers forcing file for 2020-07-05 created on arbutus.cloud-nowcast 2020-07-05 13:55:12,252 INFO [manager] checklist updated with [FVCOM rivers forcing] items from make_fvcom_rivers_forcing worker 2020-07-05 13:55:17,420 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07654 New Westminster from 2020-07-04 23:55:00 to 2020-07-06 00:00:00 2020-07-05 13:55:31,707 INFO [make_fvcom_atmos_forcing] running in process 4429 2020-07-05 13:55:31,971 INFO [make_fvcom_atmos_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-05 13:55:31,972 INFO [make_fvcom_atmos_forcing] publishing log messages to tcp://*:5569 2020-07-05 13:55:31,973 INFO [make_fvcom_atmos_forcing] connected to skookum.eos.ubc.ca port 5555 2020-07-05 13:55:31,973 INFO [make_fvcom_atmos_forcing] Creating VHFR FVCOM atmospheric forcing file for x2 forecast run from 2020-07-05 HRDPS GRIB files 2020-07-05 13:55:45,779 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07654 New Westminster from 2020-07-05 00:00:00 to 2020-07-06 00:00:00 2020-07-05 13:55:49,086 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07755 Port Moody from 2020-07-04 23:55:00 to 2020-07-06 00:00:00 2020-07-05 13:56:12,614 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07755 Port Moody from 2020-07-05 00:00:00 to 2020-07-06 00:00:00 2020-07-05 13:56:21,227 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07594 Sand Heads from 2020-07-04 23:55:00 to 2020-07-06 00:00:00 2020-07-05 13:56:47,777 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07594 Sand Heads from 2020-07-05 00:00:00 to 2020-07-06 00:00:00 2020-07-05 13:56:51,857 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39827007 = 2020-07-05 08:43:22 UTC estimated time to finish: 0000:07:26:13 2020-07-05 13:56:54,394 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07786 Sandy Cove from 2020-07-04 23:55:00 to 2020-07-06 00:00:00 2020-07-05 13:57:18,252 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07795 Point Atkinson from 2020-07-05 00:00:00 to 2020-07-06 00:00:00 2020-07-05 13:57:22,697 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07735 Vancouver Harbour from 2020-07-04 23:55:00 to 2020-07-06 00:00:00 2020-07-05 13:57:48,974 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07735 Vancouver Harbour from 2020-07-05 00:00:00 to 2020-07-06 00:00:00 2020-07-05 13:57:52,320 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4630411 = 2020-07-05 17:00:00 UTC, 70.9% complete 2020-07-05 13:57:55,161 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07610 Woodwards Landing from 2020-07-04 23:55:00 to 2020-07-06 00:00:00 2020-07-05 13:58:11,234 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07610 Woodwards Landing from 2020-07-05 00:00:00 to 2020-07-06 00:00:00 2020-07-05 13:58:16,726 INFO [make_plots] finished rendering fvcom nowcast-x2 publish 05jul20 figures 2020-07-05 13:58:17,179 INFO [make_plots] fvcom publish plots for 2020-07-05 nowcast-x2 completed 2020-07-05 13:58:17,185 INFO [manager] checklist updated with [plots] items from make_plots worker 2020-07-05 14:00:19,757 INFO [make_fvcom_atmos_forcing] Stored VHFR FVCOM atmospheric forcing hfx file: /results/forcing/atmospheric/GEM2.5/vhfr-fvcom/atmos_x2_forecast_hfx_20200706.nc 2020-07-05 14:01:44,412 INFO [make_fvcom_atmos_forcing] Stored VHFR FVCOM atmospheric forcing precip file: /results/forcing/atmospheric/GEM2.5/vhfr-fvcom/atmos_x2_forecast_precip_20200706.nc 2020-07-05 14:01:51,926 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39829573 = 2020-07-05 09:00:29 UTC estimated time to finish: 0000:07:12:29 2020-07-05 14:02:52,421 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4630465 = 2020-07-05 17:36:00 UTC, 73.4% complete 2020-07-05 14:03:35,651 INFO [make_fvcom_atmos_forcing] Stored VHFR FVCOM atmospheric forcing wnd file: /results/forcing/atmospheric/GEM2.5/vhfr-fvcom/atmos_x2_forecast_wnd_20200706.nc 2020-07-05 14:03:35,652 INFO [make_fvcom_atmos_forcing] FVCOM x2 forecast run atmospheric forcing file for 2020-07-05 created 2020-07-05 14:03:35,657 INFO [manager] checklist updated with [FVCOM atmospheric forcing] items from make_fvcom_atmos_forcing worker 2020-07-05 14:03:36,073 INFO [manager] launching NextWorker(module='nowcast.workers.upload_fvcom_atmos_forcing', args=['arbutus.cloud-nowcast', 'x2', 'forecast', '--run-date', '2020-07-05'], host='localhost') 2020-07-05 14:03:38,319 INFO [upload_fvcom_atmos_forcing] running in process 4700 2020-07-05 14:03:38,320 INFO [upload_fvcom_atmos_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-05 14:03:38,321 INFO [upload_fvcom_atmos_forcing] publishing log messages to tcp://*:5567 2020-07-05 14:03:38,322 INFO [upload_fvcom_atmos_forcing] connected to skookum.eos.ubc.ca port 5555 2020-07-05 14:03:38,323 INFO [upload_fvcom_atmos_forcing] Uploading VHFR FVCOM atmospheric forcing files for 2020-07-05 x2 forecast run to arbutus.cloud-nowcast 2020-07-05 14:03:44,457 INFO [upload_fvcom_atmos_forcing] FVCOM x2 forecast run atmospheric forcing files for 2020-07-05 uploaded to arbutus.cloud-nowcast 2020-07-05 14:03:44,458 INFO [manager] checklist updated with [FVCOM atmospheric forcing upload] items from upload_fvcom_atmos_forcing worker 2020-07-05 14:03:44,922 INFO [manager] launching NextWorker(module='nowcast.workers.run_fvcom', args=['arbutus.cloud-nowcast', 'x2', 'forecast', '--run-date', '2020-07-05'], host='arbutus.cloud-nowcast') 2020-07-05 14:03:47,130 INFO [run_fvcom] running in process 17197 2020-07-05 14:03:47,131 INFO [run_fvcom] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2020-07-05 14:03:47,132 INFO [run_fvcom] publishing log messages to tcp://*:5584 2020-07-05 14:03:47,133 INFO [run_fvcom] connected to skookum.eos.ubc.ca port 5555 2020-07-05 14:03:47,338 INFO [run_fvcom] forecast: launching /nemoShare/MEOPAR/nowcast-sys/fvcom-runs/20200705-21h03m47.166698s/VHFR_FVCOM.sh on arbutus.cloud-nowcast 2020-07-05 14:03:47,339 INFO [run_fvcom] x2 forecast FVCOM VH-FR run for 2020-07-05 on arbutus.cloud-nowcast started 2020-07-05 14:03:47,340 INFO [manager] checklist updated with [FVCOM run] items from run_fvcom worker 2020-07-05 14:03:47,815 INFO [manager] launching NextWorker(module='nowcast.workers.watch_fvcom', args=['arbutus.cloud-nowcast', 'x2', 'forecast'], host='arbutus.cloud-nowcast') 2020-07-05 14:03:49,710 INFO [watch_fvcom] running in process 17299 2020-07-05 14:03:49,712 INFO [watch_fvcom] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2020-07-05 14:03:49,712 INFO [watch_fvcom] publishing log messages to tcp://*:5586 2020-07-05 14:03:49,713 INFO [watch_fvcom] connected to skookum.eos.ubc.ca port 5555 2020-07-05 14:03:49,749 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: no run progress found in fvcom.log ; continuing to watch... 2020-07-05 14:06:52,051 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39831430 = 2020-07-05 09:12:52 UTC estimated time to finish: 0000:07:04:22 2020-07-05 14:07:52,521 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4630519 = 2020-07-05 18:12:00 UTC, 75.9% complete 2020-07-05 14:08:49,850 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15987584 = 2020-07-06 00:29:44 UTC estimated time to finish: 0000:05:42:38 2020-07-05 14:11:52,177 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39833083 = 2020-07-05 09:23:53 UTC estimated time to finish: 0000:06:58:50 2020-07-05 14:12:52,622 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4630574 = 2020-07-05 18:48:40 UTC, 78.4% complete 2020-07-05 14:13:49,952 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15989357 = 2020-07-06 00:59:17 UTC estimated time to finish: 0000:05:35:47 2020-07-05 14:16:52,303 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39834778 = 2020-07-05 09:35:11 UTC estimated time to finish: 0000:06:52:56 2020-07-05 14:17:52,723 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4630628 = 2020-07-05 19:24:40 UTC, 80.9% complete 2020-07-05 14:18:50,055 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15991238 = 2020-07-06 01:30:38 UTC estimated time to finish: 0000:05:31:57 2020-07-05 14:21:52,430 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39836484 = 2020-07-05 09:46:33 UTC estimated time to finish: 0000:06:47:06 2020-07-05 14:22:52,824 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4630682 = 2020-07-05 20:00:40 UTC, 83.4% complete 2020-07-05 14:23:50,159 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15993132 = 2020-07-06 02:02:12 UTC estimated time to finish: 0000:05:27:24 2020-07-05 14:26:52,557 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39838014 = 2020-07-05 09:56:45 UTC estimated time to finish: 0000:06:41:52 2020-07-05 14:26:57,519 INFO [make_plots] finished rendering fvcom nowcast-x2 research 05jul20 figures 2020-07-05 14:26:57,520 INFO [make_plots] fvcom research plots for 2020-07-05 nowcast-x2 completed 2020-07-05 14:26:57,657 INFO [manager] checklist updated with [plots] items from make_plots worker 2020-07-05 14:27:52,924 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4630737 = 2020-07-05 20:37:20 UTC, 86.0% complete 2020-07-05 14:28:50,265 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15995069 = 2020-07-06 02:34:29 UTC estimated time to finish: 0000:05:22:23 2020-07-05 14:31:52,684 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39839680 = 2020-07-05 10:07:52 UTC estimated time to finish: 0000:06:36:26 2020-07-05 14:32:53,024 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4630791 = 2020-07-05 21:13:20 UTC, 88.5% complete 2020-07-05 14:33:50,368 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15996568 = 2020-07-06 02:59:28 UTC estimated time to finish: 0000:05:25:06 2020-07-05 14:36:52,811 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39841332 = 2020-07-05 10:18:52 UTC estimated time to finish: 0000:06:30:55 2020-07-05 14:37:53,125 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4630846 = 2020-07-05 21:50:00 UTC, 91.0% complete 2020-07-05 14:38:50,473 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15998217 = 2020-07-06 03:26:57 UTC estimated time to finish: 0000:05:25:45 2020-07-05 14:41:52,939 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39843027 = 2020-07-05 10:30:10 UTC estimated time to finish: 0000:06:25:11 2020-07-05 14:42:53,224 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4630900 = 2020-07-05 22:26:00 UTC, 93.5% complete 2020-07-05 14:43:50,579 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 15999867 = 2020-07-06 03:54:27 UTC estimated time to finish: 0000:05:25:11 2020-07-05 14:44:10,716 INFO [make_plots] finished rendering nemo nowcast-green research 05jul20 figures 2020-07-05 14:44:10,917 INFO [make_plots] nemo research plots for 2020-07-05 nowcast-green completed 2020-07-05 14:44:11,093 INFO [manager] checklist updated with [plots] items from make_plots worker 2020-07-05 14:46:53,068 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39844733 = 2020-07-05 10:41:33 UTC estimated time to finish: 0000:06:19:24 2020-07-05 14:47:53,228 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4630955 = 2020-07-05 23:02:40 UTC, 96.1% complete 2020-07-05 14:48:50,684 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 16001570 = 2020-07-06 04:22:50 UTC estimated time to finish: 0000:05:23:22 2020-07-05 14:51:53,197 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39846263 = 2020-07-05 10:51:45 UTC estimated time to finish: 0000:06:14:20 2020-07-05 14:52:53,328 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4631006 = 2020-07-05 23:36:40 UTC, 98.4% complete 2020-07-05 14:53:50,791 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 16003219 = 2020-07-06 04:50:19 UTC estimated time to finish: 0000:05:21:07 2020-07-05 14:56:53,326 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39847888 = 2020-07-05 11:02:35 UTC estimated time to finish: 0000:06:09:07 2020-07-05 14:57:53,429 INFO [watch_NEMO] nowcast-dev on salish-nowcast: timestep: 4631040 = 2020-07-05 23:59:20 UTC, 100.0% complete 2020-07-05 14:58:50,898 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 16004718 = 2020-07-06 05:15:18 UTC estimated time to finish: 0000:05:18:49 2020-07-05 15:01:53,455 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39849540 = 2020-07-05 11:13:36 UTC estimated time to finish: 0000:06:03:46 2020-07-05 15:02:53,540 INFO [watch_NEMO] nowcast-dev NEMO run on salish-nowcast completed 2020-07-05 15:02:53,546 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker 2020-07-05 15:03:51,005 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 16006368 = 2020-07-06 05:42:48 UTC estimated time to finish: 0000:05:15:59 2020-07-05 15:05:44,632 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-west/18/ to /results/forcing/atmospheric/GEM2.5/GRIB/20200705/18/ 2020-07-05 15:05:44,634 INFO [collect_weather] 2020-07-05 2.5km weather forecast 18 collection complete 2020-07-05 15:05:44,637 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker 2020-07-05 15:05:45,479 INFO [manager] launching NextWorker(module='nowcast.workers.download_weather', args=['00', '1km', '--no-verify-certs'], host='localhost') 2020-07-05 15:05:45,484 INFO [manager] launching NextWorker(module='nowcast.workers.download_weather', args=['12', '1km', '--no-verify-certs'], host='localhost') 2020-07-05 15:05:45,488 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['00', '2.5km'], host='localhost') 2020-07-05 15:05:46,823 INFO [download_weather] running in process 5859 2020-07-05 15:05:46,824 INFO [download_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-05 15:05:46,825 INFO [download_weather] publishing log messages to tcp://*:5562 2020-07-05 15:05:46,826 INFO [download_weather] connected to skookum.eos.ubc.ca port 5555 2020-07-05 15:05:46,827 INFO [download_weather] downloading 12 1 km forecast GRIB2 files for 20200705 2020-07-05 15:05:46,855 INFO [collect_weather] running in process 5860 2020-07-05 15:05:46,856 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-05 15:05:46,857 INFO [collect_weather] publishing log messages to tcp://*:5564 2020-07-05 15:05:46,858 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555 2020-07-05 15:05:46,874 INFO [download_weather] running in process 5858 2020-07-05 15:05:46,875 INFO [download_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-05 15:05:46,876 INFO [download_weather] publishing log messages to tcp://*:5567 2020-07-05 15:05:46,877 INFO [download_weather] connected to skookum.eos.ubc.ca port 5555 2020-07-05 15:05:46,878 INFO [download_weather] downloading 00 1 km forecast GRIB2 files for 20200705 2020-07-05 15:05:46,917 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-west/00/ 2020-07-05 15:06:53,585 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39851206 = 2020-07-05 11:24:42 UTC estimated time to finish: 0000:05:58:26 2020-07-05 15:08:51,113 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 16008030 = 2020-07-06 06:10:30 UTC estimated time to finish: 0000:05:12:40 2020-07-05 15:11:53,714 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39852628 = 2020-07-05 11:34:11 UTC estimated time to finish: 0000:05:53:56 2020-07-05 15:13:51,221 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 16009680 = 2020-07-06 06:38:00 UTC estimated time to finish: 0000:05:09:17 2020-07-05 15:16:53,846 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39854240 = 2020-07-05 11:44:56 UTC estimated time to finish: 0000:05:48:53 2020-07-05 15:18:51,331 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 16011180 = 2020-07-06 07:03:00 UTC estimated time to finish: 0000:05:05:45 2020-07-05 15:21:53,976 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39855904 = 2020-07-05 11:56:01 UTC estimated time to finish: 0000:05:43:30 2020-07-05 15:23:51,428 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 16012830 = 2020-07-06 07:30:30 UTC estimated time to finish: 0000:05:02:05 2020-07-05 15:26:54,108 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39857557 = 2020-07-05 12:07:02 UTC estimated time to finish: 0000:05:38:09 2020-07-05 15:27:00,688 INFO [download_weather] 2020-07-05 1km weather forecast 12 downloads complete 2020-07-05 15:27:00,693 INFO [manager] checklist updated with [weather forecast] items from download_weather worker 2020-07-05 15:27:10,051 INFO [download_weather] 2020-07-05 1km weather forecast 00 downloads complete 2020-07-05 15:27:10,055 INFO [manager] checklist updated with [weather forecast] items from download_weather worker 2020-07-05 15:28:51,538 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 16014451 = 2020-07-06 07:57:31 UTC estimated time to finish: 0000:04:58:15 2020-07-05 15:31:54,242 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39859018 = 2020-07-05 12:16:47 UTC estimated time to finish: 0000:05:33:27 2020-07-05 15:33:51,648 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 16016100 = 2020-07-06 08:25:00 UTC estimated time to finish: 0000:04:54:18 2020-07-05 15:36:54,376 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39860671 = 2020-07-05 12:27:48 UTC estimated time to finish: 0000:05:28:10 2020-07-05 15:38:51,759 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 16017600 = 2020-07-06 08:50:00 UTC estimated time to finish: 0000:04:50:25 2020-07-05 15:41:54,509 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39862338 = 2020-07-05 12:38:55 UTC estimated time to finish: 0000:05:22:46 2020-07-05 15:43:51,871 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 16019263 = 2020-07-06 09:17:43 UTC estimated time to finish: 0000:04:46:07 2020-07-05 15:46:54,644 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39863990 = 2020-07-05 12:49:56 UTC estimated time to finish: 0000:05:17:28 2020-07-05 15:48:51,983 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 16020953 = 2020-07-06 09:45:53 UTC estimated time to finish: 0000:04:41:20 2020-07-05 15:51:54,778 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39865655 = 2020-07-05 13:01:02 UTC estimated time to finish: 0000:05:12:06 2020-07-05 15:53:52,097 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 16022603 = 2020-07-06 10:13:23 UTC estimated time to finish: 0000:04:36:48 2020-07-05 15:56:54,913 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39867185 = 2020-07-05 13:11:14 UTC estimated time to finish: 0000:05:07:05 2020-07-05 15:58:52,210 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 16024266 = 2020-07-06 10:41:06 UTC estimated time to finish: 0000:04:32:22 2020-07-05 16:01:55,049 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39868810 = 2020-07-05 13:22:04 UTC estimated time to finish: 0000:05:01:59 2020-07-05 16:03:52,324 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 16025724 = 2020-07-06 11:05:24 UTC estimated time to finish: 0000:04:28:26 2020-07-05 16:06:55,185 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39870464 = 2020-07-05 13:33:05 UTC estimated time to finish: 0000:04:56:41 2020-07-05 16:08:52,438 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 16027416 = 2020-07-06 11:33:36 UTC estimated time to finish: 0000:04:23:33 2020-07-05 16:11:55,321 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39872160 = 2020-07-05 13:44:24 UTC estimated time to finish: 0000:04:51:08 2020-07-05 16:13:52,553 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 16029065 = 2020-07-06 12:01:05 UTC estimated time to finish: 0000:04:18:56 2020-07-05 16:16:55,456 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39873660 = 2020-07-05 13:54:24 UTC estimated time to finish: 0000:04:46:21 2020-07-05 16:18:52,668 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 16030728 = 2020-07-06 12:28:48 UTC estimated time to finish: 0000:04:14:08 2020-07-05 16:21:55,568 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39875324 = 2020-07-05 14:05:29 UTC estimated time to finish: 0000:04:40:58 2020-07-05 16:23:52,783 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 16032228 = 2020-07-06 12:53:48 UTC estimated time to finish: 0000:04:09:53 2020-07-05 16:26:55,704 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39876977 = 2020-07-05 14:16:30 UTC estimated time to finish: 0000:04:35:42 2020-07-05 16:28:52,899 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 16033877 = 2020-07-06 13:21:17 UTC estimated time to finish: 0000:04:05:14 2020-07-05 16:31:55,841 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39878671 = 2020-07-05 14:27:48 UTC estimated time to finish: 0000:04:30:15 2020-07-05 16:33:53,015 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 16035487 = 2020-07-06 13:48:07 UTC estimated time to finish: 0000:04:00:40 2020-07-05 16:36:55,978 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39880174 = 2020-07-05 14:37:49 UTC estimated time to finish: 0000:04:25:28 2020-07-05 16:38:53,132 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 16037190 = 2020-07-06 14:16:30 UTC estimated time to finish: 0000:03:55:40 2020-07-05 16:41:56,116 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39881840 = 2020-07-05 14:48:56 UTC estimated time to finish: 0000:04:20:07 2020-07-05 16:43:53,250 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 16038689 = 2020-07-06 14:41:29 UTC estimated time to finish: 0000:03:51:18 2020-07-05 16:46:56,255 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39883492 = 2020-07-05 14:59:56 UTC estimated time to finish: 0000:04:14:52 2020-07-05 16:48:53,367 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 16040339 = 2020-07-06 15:08:59 UTC estimated time to finish: 0000:03:46:25 2020-07-05 16:51:56,394 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39885144 = 2020-07-05 15:10:57 UTC estimated time to finish: 0000:04:09:37 2020-07-05 16:53:53,493 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 16042029 = 2020-07-06 15:37:09 UTC estimated time to finish: 0000:03:41:26 2020-07-05 16:56:56,532 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39886810 = 2020-07-05 15:22:04 UTC estimated time to finish: 0000:04:04:20 2020-07-05 16:58:53,613 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 16043650 = 2020-07-06 16:04:10 UTC estimated time to finish: 0000:03:36:45 2020-07-05 17:01:56,672 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39888231 = 2020-07-05 15:31:32 UTC estimated time to finish: 0000:03:59:55 2020-07-05 17:03:53,733 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 16045300 = 2020-07-06 16:31:40 UTC estimated time to finish: 0000:03:31:59 2020-07-05 17:06:56,811 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39889843 = 2020-07-05 15:42:17 UTC estimated time to finish: 0000:03:54:54 2020-07-05 17:08:53,852 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 16046799 = 2020-07-06 16:56:39 UTC estimated time to finish: 0000:03:27:32 2020-07-05 17:11:56,949 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39891468 = 2020-07-05 15:53:07 UTC estimated time to finish: 0000:03:49:45 2020-07-05 17:13:53,973 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 16048449 = 2020-07-06 17:24:09 UTC estimated time to finish: 0000:03:22:43 2020-07-05 17:16:57,090 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39893120 = 2020-07-05 16:04:08 UTC estimated time to finish: 0000:03:44:33 2020-07-05 17:18:54,093 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 16050111 = 2020-07-06 17:51:51 UTC estimated time to finish: 0000:03:17:45 2020-07-05 17:21:57,230 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39894622 = 2020-07-05 16:14:08 UTC estimated time to finish: 0000:03:39:45 2020-07-05 17:23:54,215 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 16051801 = 2020-07-06 18:20:01 UTC estimated time to finish: 0000:03:12:38 2020-07-05 17:26:57,371 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39896316 = 2020-07-05 16:25:26 UTC estimated time to finish: 0000:03:34:20 2020-07-05 17:28:54,336 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 16053301 = 2020-07-06 18:45:01 UTC estimated time to finish: 0000:03:08:09 2020-07-05 17:31:57,512 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39897941 = 2020-07-05 16:36:16 UTC estimated time to finish: 0000:03:29:11 2020-07-05 17:33:54,458 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 16054951 = 2020-07-06 19:12:31 UTC estimated time to finish: 0000:03:03:14 2020-07-05 17:36:57,655 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39899594 = 2020-07-05 16:47:17 UTC estimated time to finish: 0000:03:24:00 2020-07-05 17:38:54,580 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 16056614 = 2020-07-06 19:40:14 UTC estimated time to finish: 0000:02:58:15 2020-07-05 17:41:57,799 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39901096 = 2020-07-05 16:57:18 UTC estimated time to finish: 0000:03:19:13 2020-07-05 17:43:54,704 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 16058305 = 2020-07-06 20:08:25 UTC estimated time to finish: 0000:02:53:09 2020-07-05 17:46:57,956 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39902760 = 2020-07-05 17:08:24 UTC estimated time to finish: 0000:03:13:59 2020-07-05 17:48:54,826 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 16059960 = 2020-07-06 20:36:00 UTC estimated time to finish: 0000:02:47:50 2020-07-05 17:51:58,109 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39904414 = 2020-07-05 17:19:25 UTC estimated time to finish: 0000:03:08:44 2020-07-05 17:53:54,951 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 16061849 = 2020-07-06 21:07:29 UTC estimated time to finish: 0000:02:41:41 2020-07-05 17:56:58,254 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39906068 = 2020-07-05 17:30:27 UTC estimated time to finish: 0000:03:03:32 2020-07-05 17:58:55,076 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 16063744 = 2020-07-06 21:39:04 UTC estimated time to finish: 0000:02:35:31 2020-07-05 18:01:58,399 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39907692 = 2020-07-05 17:41:16 UTC estimated time to finish: 0000:02:58:25 2020-07-05 18:03:55,202 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 16065516 = 2020-07-06 22:08:36 UTC estimated time to finish: 0000:02:30:04 2020-07-05 18:06:58,544 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39909153 = 2020-07-05 17:51:01 UTC estimated time to finish: 0000:02:53:51 2020-07-05 18:08:55,328 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 16067166 = 2020-07-06 22:36:06 UTC estimated time to finish: 0000:02:25:10 2020-07-05 18:11:58,690 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39910806 = 2020-07-05 18:02:02 UTC estimated time to finish: 0000:02:48:37 2020-07-05 18:13:55,447 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 16068706 = 2020-07-06 23:01:46 UTC estimated time to finish: 0000:02:20:33 2020-07-05 18:16:58,838 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39912471 = 2020-07-05 18:13:08 UTC estimated time to finish: 0000:02:43:21 2020-07-05 18:18:55,574 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 16070400 = 2020-07-06 23:30:00 UTC estimated time to finish: 0000:02:15:29 2020-07-05 18:21:58,985 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39914124 = 2020-07-05 18:24:09 UTC estimated time to finish: 0000:02:38:08 2020-07-05 18:23:55,702 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 16072060 = 2020-07-06 23:57:40 UTC estimated time to finish: 0000:02:10:31 2020-07-05 18:26:59,133 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39915628 = 2020-07-05 18:34:11 UTC estimated time to finish: 0000:02:33:23 2020-07-05 18:28:55,830 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 16073749 = 2020-07-07 00:25:49 UTC estimated time to finish: 0000:02:05:28 2020-07-05 18:31:59,281 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39917320 = 2020-07-05 18:45:28 UTC estimated time to finish: 0000:02:28:02 2020-07-05 18:33:55,957 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 16075208 = 2020-07-07 00:50:08 UTC estimated time to finish: 0000:02:01:08 2020-07-05 18:36:59,428 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39918973 = 2020-07-05 18:56:29 UTC estimated time to finish: 0000:02:22:50 2020-07-05 18:38:56,085 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 16076870 = 2020-07-07 01:17:50 UTC estimated time to finish: 0000:01:56:12 2020-07-05 18:41:59,576 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39920598 = 2020-07-05 19:07:19 UTC estimated time to finish: 0000:02:17:43 2020-07-05 18:43:56,214 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 16078520 = 2020-07-07 01:45:20 UTC estimated time to finish: 0000:01:51:15 2020-07-05 18:46:59,727 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39922100 = 2020-07-05 19:17:20 UTC estimated time to finish: 0000:02:12:59 2020-07-05 18:48:56,344 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 16080211 = 2020-07-07 02:13:31 UTC estimated time to finish: 0000:01:46:11 2020-07-05 18:51:59,876 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39923794 = 2020-07-05 19:28:37 UTC estimated time to finish: 0000:02:07:38 2020-07-05 18:53:56,474 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 16081710 = 2020-07-07 02:38:30 UTC estimated time to finish: 0000:01:41:42 2020-07-05 18:57:00,027 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39925420 = 2020-07-05 19:39:28 UTC estimated time to finish: 0000:02:02:32 2020-07-05 18:58:56,604 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 16083360 = 2020-07-07 03:06:00 UTC estimated time to finish: 0000:01:36:45 2020-07-05 19:02:00,178 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39927030 = 2020-07-05 19:50:12 UTC estimated time to finish: 0000:01:57:31 2020-07-05 19:03:56,736 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 16084981 = 2020-07-07 03:33:01 UTC estimated time to finish: 0000:01:31:55 2020-07-05 19:07:00,329 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39928614 = 2020-07-05 20:00:45 UTC estimated time to finish: 0000:01:52:34 2020-07-05 19:08:56,867 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 16086630 = 2020-07-07 04:00:30 UTC estimated time to finish: 0000:01:26:58 2020-07-05 19:12:00,481 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39930144 = 2020-07-05 20:10:57 UTC estimated time to finish: 0000:01:47:44 2020-07-05 19:13:57,002 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 16088293 = 2020-07-07 04:28:13 UTC estimated time to finish: 0000:01:21:58 2020-07-05 19:17:00,639 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39931810 = 2020-07-05 20:22:04 UTC estimated time to finish: 0000:01:42:29 2020-07-05 19:18:57,132 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 16089862 = 2020-07-07 04:54:22 UTC estimated time to finish: 0000:01:17:11 2020-07-05 19:22:00,792 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39933505 = 2020-07-05 20:33:22 UTC estimated time to finish: 0000:01:37:08 2020-07-05 19:23:57,265 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 16091797 = 2020-07-07 05:26:37 UTC estimated time to finish: 0000:01:11:11 2020-07-05 19:27:00,945 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39935240 = 2020-07-05 20:44:56 UTC estimated time to finish: 0000:01:31:39 2020-07-05 19:28:57,398 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 16093692 = 2020-07-07 05:58:12 UTC estimated time to finish: 0000:01:05:20 2020-07-05 19:32:01,099 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39936700 = 2020-07-05 20:54:40 UTC estimated time to finish: 0000:01:27:04 2020-07-05 19:33:57,534 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 16095615 = 2020-07-07 06:30:15 UTC estimated time to finish: 0000:00:59:27 2020-07-05 19:37:01,250 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39938400 = 2020-07-05 21:06:00 UTC estimated time to finish: 0000:01:21:44 2020-07-05 19:38:57,669 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 16097360 = 2020-07-07 06:59:20 UTC estimated time to finish: 0000:00:54:06 2020-07-05 19:42:01,403 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39940100 = 2020-07-05 21:17:20 UTC estimated time to finish: 0000:01:16:22 2020-07-05 19:43:57,804 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 16099240 = 2020-07-07 07:30:40 UTC estimated time to finish: 0000:00:48:22 2020-07-05 19:47:01,558 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39941794 = 2020-07-05 21:28:37 UTC estimated time to finish: 0000:01:11:03 2020-07-05 19:48:57,941 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 16101176 = 2020-07-07 08:02:56 UTC estimated time to finish: 0000:00:42:30 2020-07-05 19:52:01,712 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39943448 = 2020-07-05 21:39:39 UTC estimated time to finish: 0000:01:05:52 2020-07-05 19:53:58,078 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 16103111 = 2020-07-07 08:35:11 UTC estimated time to finish: 0000:00:36:39 2020-07-05 19:57:01,869 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39944950 = 2020-07-05 21:49:40 UTC estimated time to finish: 0000:01:01:10 2020-07-05 19:58:58,215 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 16104802 = 2020-07-07 09:03:22 UTC estimated time to finish: 0000:00:31:34 2020-07-05 20:02:02,025 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39946532 = 2020-07-05 22:00:12 UTC estimated time to finish: 0000:00:56:13 2020-07-05 20:03:58,353 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 16106493 = 2020-07-07 09:31:33 UTC estimated time to finish: 0000:00:26:32 2020-07-05 20:07:02,183 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39948144 = 2020-07-05 22:10:57 UTC estimated time to finish: 0000:00:51:11 2020-07-05 20:08:58,489 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 16108156 = 2020-07-07 09:59:16 UTC estimated time to finish: 0000:00:21:35 2020-07-05 20:12:02,340 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39949769 = 2020-07-05 22:21:47 UTC estimated time to finish: 0000:00:46:06 2020-07-05 20:13:58,628 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 16109805 = 2020-07-07 10:26:45 UTC estimated time to finish: 0000:00:16:41 2020-07-05 20:17:02,500 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39951272 = 2020-07-05 22:31:48 UTC estimated time to finish: 0000:00:41:23 2020-07-05 20:18:58,766 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 16111578 = 2020-07-07 10:56:18 UTC estimated time to finish: 0000:00:11:23 2020-07-05 20:22:02,655 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39952966 = 2020-07-05 22:43:06 UTC estimated time to finish: 0000:00:36:05 2020-07-05 20:23:58,905 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 16113269 = 2020-07-07 11:24:29 UTC estimated time to finish: 0000:00:06:20 2020-07-05 20:27:02,816 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39954619 = 2020-07-05 22:54:07 UTC estimated time to finish: 0000:00:30:54 2020-07-05 20:28:59,043 INFO [watch_fvcom] x2 forecast on arbutus.cloud-nowcast: timestep: 16115204 = 2020-07-07 11:56:44 UTC estimated time to finish: 0000:00:00:34 2020-07-05 20:32:02,976 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39956760 = 2020-07-05 23:08:24 UTC estimated time to finish: 0000:00:24:09 2020-07-05 20:33:59,153 INFO [watch_fvcom] x2 forecast FVCOM VH-FR run on arbutus.cloud-nowcast completed 2020-07-05 20:33:59,159 INFO [manager] checklist updated with [FVCOM run] items from watch_fvcom worker 2020-07-05 20:33:59,974 INFO [manager] launching NextWorker(module='nowcast.workers.download_fvcom_results', args=['arbutus.cloud-nowcast', 'x2', 'forecast', '--run-date', '2020-07-05'], host='localhost') 2020-07-05 20:34:01,582 INFO [download_fvcom_results] running in process 12719 2020-07-05 20:34:01,584 INFO [download_fvcom_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-05 20:34:01,585 INFO [download_fvcom_results] publishing log messages to tcp://*:5562 2020-07-05 20:34:01,586 INFO [download_fvcom_results] connected to skookum.eos.ubc.ca port 5555 2020-07-05 20:35:08,693 INFO [download_fvcom_results] VHFR FVCOM x2 forecast 2020-07-05 results files from arbutus.cloud-nowcast downloaded 2020-07-05 20:35:08,700 INFO [manager] checklist updated with [VHFR FVCOM results files] items from download_fvcom_results worker 2020-07-05 20:35:09,243 INFO [manager] launching NextWorker(module='nowcast.workers.get_vfpa_hadcp', args=['--data-date', '2020-07-05'], host='localhost') 2020-07-05 20:35:09,247 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['fvcom', 'forecast-x2', 'research', '--run-date', '2020-07-05'], host='localhost') 2020-07-05 20:35:09,249 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['fvcom', 'forecast', '--run-date', '2020-07-05'], host='localhost') 2020-07-05 20:35:10,686 INFO [update_forecast_datasets] running in process 12730 2020-07-05 20:35:10,687 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-05 20:35:10,688 INFO [update_forecast_datasets] publishing log messages to tcp://*:5562 2020-07-05 20:35:10,689 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555 2020-07-05 20:35:10,690 INFO [update_forecast_datasets] updating fvcom most_recent_forecast directory from forecast/05jul20 run 2020-07-05 20:35:10,691 INFO [update_forecast_datasets] updated fvcom most_recent_forecast directory from forecast/05jul20 run 2020-07-05 20:35:10,692 INFO [update_forecast_datasets] fvcom 2020-07-05 forecast rolling forecast datasets updated 2020-07-05 20:35:10,695 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker 2020-07-05 20:35:11,257 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['fvcom-forecast'], host='localhost') 2020-07-05 20:35:13,416 INFO [get_vfpa_hadcp] running in process 12728 2020-07-05 20:35:13,416 INFO [get_vfpa_hadcp] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-05 20:35:13,417 INFO [get_vfpa_hadcp] publishing log messages to tcp://*:5562 2020-07-05 20:35:13,418 INFO [get_vfpa_hadcp] connected to skookum.eos.ubc.ca port 5555 2020-07-05 20:35:13,419 INFO [get_vfpa_hadcp] processing VFPA HADCP data from 2nd Narrows Rail Bridge for 2020-07-05 2020-07-05 20:35:14,786 INFO [get_vfpa_hadcp] added VFPA HADCP data from 2nd Narrows Rail Bridge for 2020-07-05 to /opp/observations/AISDATA/netcdf/VFPA_2ND_NARROWS_HADCP_2s_202007.nc 2020-07-05 20:35:14,786 INFO [get_vfpa_hadcp] VFPA HADCP observations added to 2020-07 netcdf file 2020-07-05 20:35:14,790 INFO [manager] checklist updated with [VFPA HADCP data] items from get_vfpa_hadcp worker 2020-07-05 20:35:15,341 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['VFPA-HADCP'], host='localhost') 2020-07-05 20:35:15,433 INFO [make_plots] running in process 12729 2020-07-05 20:35:15,435 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-05 20:35:15,436 INFO [make_plots] publishing log messages to tcp://*:5567 2020-07-05 20:35:15,438 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555 2020-07-05 20:35:19,966 INFO [ping_erddap] running in process 12749 2020-07-05 20:35:19,967 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-05 20:35:19,968 INFO [ping_erddap] publishing log messages to tcp://*:5562 2020-07-05 20:35:19,969 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555 2020-07-05 20:35:19,970 INFO [ping_erddap] VFPA-HADCP ERDDAP dataset flag file(s) created 2020-07-05 20:35:19,972 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker 2020-07-05 20:35:20,531 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['fvcom', 'nowcast-x2', 'publish', '--run-date', '2020-07-05'], host='localhost') 2020-07-05 20:35:20,535 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['fvcom', 'forecast-x2', 'publish', '--run-date', '2020-07-05'], host='localhost') 2020-07-05 20:35:24,888 INFO [make_plots] running in process 12755 2020-07-05 20:35:24,889 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-05 20:35:24,890 INFO [make_plots] publishing log messages to tcp://*:5562 2020-07-05 20:35:24,893 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555 2020-07-05 20:35:24,894 INFO [make_plots] running in process 12756 2020-07-05 20:35:24,895 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-05 20:35:24,896 INFO [make_plots] publishing log messages to tcp://*:5568 2020-07-05 20:35:24,897 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555 2020-07-05 20:35:27,683 INFO [make_plots] preparing render list for 2020-07-05 VHFR FVCOM forecast-x2 research figures 2020-07-05 20:35:27,690 INFO [make_plots] added EnglishBay_surface_currents figures to 2020-07-05 VHFR FVCOM forecast-x2 research render list 2020-07-05 20:35:27,697 INFO [make_plots] added VancouverHarbour_surface_currents figures to 2020-07-05 VHFR FVCOM forecast-x2 research render list 2020-07-05 20:35:27,704 INFO [make_plots] added IndianArm_surface_currents figures to 2020-07-05 VHFR FVCOM forecast-x2 research render list 2020-07-05 20:35:27,711 INFO [make_plots] added VancouverHarbour_thalweg figures to 2020-07-05 VHFR FVCOM forecast-x2 research render list 2020-07-05 20:35:27,718 INFO [make_plots] added PortMoody_thalweg figures to 2020-07-05 VHFR FVCOM forecast-x2 research render list 2020-07-05 20:35:27,724 INFO [make_plots] added IndianArm_thalweg figures to 2020-07-05 VHFR FVCOM forecast-x2 research render list 2020-07-05 20:35:27,725 INFO [make_plots] starting to render fvcom forecast-x2 research 05jul20 figures 2020-07-05 20:35:29,365 INFO [make_plots] preparing render list for 2020-07-05 VHFR FVCOM nowcast-x2 publish figures 2020-07-05 20:35:29,366 INFO [make_plots] added Calamity Point figure to 2020-07-05 VHFR FVCOM nowcast-x2 publish render list 2020-07-05 20:35:29,366 INFO [make_plots] added Indian Arm Head figure to 2020-07-05 VHFR FVCOM nowcast-x2 publish render list 2020-07-05 20:35:29,367 INFO [make_plots] added New Westminster figure to 2020-07-05 VHFR FVCOM nowcast-x2 publish render list 2020-07-05 20:35:29,368 INFO [make_plots] added Port Moody figure to 2020-07-05 VHFR FVCOM nowcast-x2 publish render list 2020-07-05 20:35:29,369 INFO [make_plots] added Sand Heads figure to 2020-07-05 VHFR FVCOM nowcast-x2 publish render list 2020-07-05 20:35:29,370 INFO [make_plots] added Sandy Cove figure to 2020-07-05 VHFR FVCOM nowcast-x2 publish render list 2020-07-05 20:35:29,370 INFO [make_plots] added Vancouver Harbour figure to 2020-07-05 VHFR FVCOM nowcast-x2 publish render list 2020-07-05 20:35:29,371 INFO [make_plots] added Woodwards Landing figure to 2020-07-05 VHFR FVCOM nowcast-x2 publish render list 2020-07-05 20:35:29,372 INFO [make_plots] added 2ndNarrows_current figure to 2020-07-05 VHFR FVCOM nowcast-x2 publish render list 2020-07-05 20:35:29,372 INFO [make_plots] starting to render fvcom nowcast-x2 publish 05jul20 figures 2020-07-05 20:35:29,433 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07724 Calamity Point from 2020-07-04 23:55:00 to 2020-07-06 00:00:00 2020-07-05 20:35:29,434 INFO [make_plots] preparing render list for 2020-07-05 VHFR FVCOM forecast-x2 publish figures 2020-07-05 20:35:29,436 INFO [make_plots] added Calamity Point figure to 2020-07-05 VHFR FVCOM forecast-x2 publish render list 2020-07-05 20:35:29,437 INFO [make_plots] added Indian Arm Head figure to 2020-07-05 VHFR FVCOM forecast-x2 publish render list 2020-07-05 20:35:29,438 INFO [make_plots] added New Westminster figure to 2020-07-05 VHFR FVCOM forecast-x2 publish render list 2020-07-05 20:35:29,439 INFO [make_plots] added Port Moody figure to 2020-07-05 VHFR FVCOM forecast-x2 publish render list 2020-07-05 20:35:29,439 INFO [make_plots] added Sand Heads figure to 2020-07-05 VHFR FVCOM forecast-x2 publish render list 2020-07-05 20:35:29,440 INFO [make_plots] added Sandy Cove figure to 2020-07-05 VHFR FVCOM forecast-x2 publish render list 2020-07-05 20:35:29,441 INFO [make_plots] added Vancouver Harbour figure to 2020-07-05 VHFR FVCOM forecast-x2 publish render list 2020-07-05 20:35:29,442 INFO [make_plots] added Woodwards Landing figure to 2020-07-05 VHFR FVCOM forecast-x2 publish render list 2020-07-05 20:35:29,443 INFO [make_plots] added 2ndNarrows_current figure to 2020-07-05 VHFR FVCOM forecast-x2 publish render list 2020-07-05 20:35:29,443 INFO [make_plots] starting to render fvcom forecast-x2 publish 05jul20 figures 2020-07-05 20:35:29,459 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07724 Calamity Point from 2020-07-04 23:55:00 to 2020-07-07 12:00:00 2020-07-05 20:35:54,078 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07724 Calamity Point from 2020-07-05 00:00:00 to 2020-07-06 00:00:00 2020-07-05 20:35:55,609 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07774 Indian Arm Head from 2020-07-04 23:55:00 to 2020-07-06 00:00:00 2020-07-05 20:35:56,380 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07774 Indian Arm Head from 2020-07-05 00:00:00 to 2020-07-06 00:00:00 2020-07-05 20:35:57,083 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07724 Calamity Point from 2020-07-05 00:00:00 to 2020-07-07 12:00:00 2020-07-05 20:35:57,858 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07654 New Westminster from 2020-07-04 23:55:00 to 2020-07-06 00:00:00 2020-07-05 20:35:58,586 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07774 Indian Arm Head from 2020-07-04 23:55:00 to 2020-07-07 12:00:00 2020-07-05 20:35:59,135 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07774 Indian Arm Head from 2020-07-05 00:00:00 to 2020-07-07 12:00:00 2020-07-05 20:36:00,652 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07654 New Westminster from 2020-07-04 23:55:00 to 2020-07-07 12:00:00 2020-07-05 20:36:19,986 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07654 New Westminster from 2020-07-05 00:00:00 to 2020-07-06 00:00:00 2020-07-05 20:36:22,882 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07755 Port Moody from 2020-07-04 23:55:00 to 2020-07-06 00:00:00 2020-07-05 20:36:25,445 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07654 New Westminster from 2020-07-05 00:00:00 to 2020-07-07 12:00:00 2020-07-05 20:36:30,320 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07755 Port Moody from 2020-07-04 23:55:00 to 2020-07-07 12:00:00 2020-07-05 20:36:44,956 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07755 Port Moody from 2020-07-05 00:00:00 to 2020-07-06 00:00:00 2020-07-05 20:36:47,930 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07594 Sand Heads from 2020-07-04 23:55:00 to 2020-07-06 00:00:00 2020-07-05 20:36:54,039 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07755 Port Moody from 2020-07-05 00:00:00 to 2020-07-07 12:00:00 2020-07-05 20:36:58,896 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07594 Sand Heads from 2020-07-04 23:55:00 to 2020-07-07 12:00:00 2020-07-05 20:37:03,135 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39959042 = 2020-07-05 23:23:36 UTC estimated time to finish: 0000:00:16:58 2020-07-05 20:37:10,445 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07594 Sand Heads from 2020-07-05 00:00:00 to 2020-07-06 00:00:00 2020-07-05 20:37:13,752 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07786 Sandy Cove from 2020-07-04 23:55:00 to 2020-07-06 00:00:00 2020-07-05 20:37:23,150 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07594 Sand Heads from 2020-07-05 00:00:00 to 2020-07-07 12:00:00 2020-07-05 20:37:28,217 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07786 Sandy Cove from 2020-07-04 23:55:00 to 2020-07-07 12:00:00 2020-07-05 20:37:35,732 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07795 Point Atkinson from 2020-07-05 00:00:00 to 2020-07-06 00:00:00 2020-07-05 20:37:38,875 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07735 Vancouver Harbour from 2020-07-04 23:55:00 to 2020-07-06 00:00:00 2020-07-05 20:37:53,214 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07795 Point Atkinson from 2020-07-05 00:00:00 to 2020-07-07 12:00:00 2020-07-05 20:37:58,180 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07735 Vancouver Harbour from 2020-07-04 23:55:00 to 2020-07-07 12:00:00 2020-07-05 20:38:00,232 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07735 Vancouver Harbour from 2020-07-05 00:00:00 to 2020-07-06 00:00:00 2020-07-05 20:38:03,472 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07610 Woodwards Landing from 2020-07-04 23:55:00 to 2020-07-06 00:00:00 2020-07-05 20:38:21,432 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07610 Woodwards Landing from 2020-07-05 00:00:00 to 2020-07-06 00:00:00 2020-07-05 20:38:22,764 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07735 Vancouver Harbour from 2020-07-05 00:00:00 to 2020-07-07 12:00:00 2020-07-05 20:38:25,959 INFO [make_plots] finished rendering fvcom nowcast-x2 publish 05jul20 figures 2020-07-05 20:38:26,475 INFO [make_plots] fvcom publish plots for 2020-07-05 nowcast-x2 completed 2020-07-05 20:38:26,485 INFO [manager] checklist updated with [plots] items from make_plots worker 2020-07-05 20:38:27,676 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07610 Woodwards Landing from 2020-07-04 23:55:00 to 2020-07-07 12:00:00 2020-07-05 20:38:48,071 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07610 Woodwards Landing from 2020-07-05 00:00:00 to 2020-07-07 12:00:00 2020-07-05 20:38:52,799 INFO [make_plots] finished rendering fvcom forecast-x2 publish 05jul20 figures 2020-07-05 20:38:53,365 INFO [make_plots] fvcom publish plots for 2020-07-05 forecast-x2 completed 2020-07-05 20:38:53,370 INFO [manager] checklist updated with [plots] items from make_plots worker 2020-07-05 20:42:03,325 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39961610 = 2020-07-05 23:40:44 UTC estimated time to finish: 0000:00:08:56 2020-07-05 20:47:03,458 INFO [watch_fvcom] r12 nowcast on arbutus.cloud-nowcast: timestep: 39964178 = 2020-07-05 23:57:51 UTC estimated time to finish: 0000:00:00:59 2020-07-05 20:52:03,544 INFO [watch_fvcom] r12 nowcast FVCOM VH-FR run on arbutus.cloud-nowcast completed 2020-07-05 20:52:03,550 INFO [manager] checklist updated with [FVCOM run] items from watch_fvcom worker 2020-07-05 20:52:04,392 INFO [manager] launching NextWorker(module='nowcast.workers.download_fvcom_results', args=['arbutus.cloud-nowcast', 'r12', 'nowcast', '--run-date', '2020-07-05'], host='localhost') 2020-07-05 20:52:05,386 INFO [download_fvcom_results] running in process 13236 2020-07-05 20:52:05,387 INFO [download_fvcom_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-05 20:52:05,388 INFO [download_fvcom_results] publishing log messages to tcp://*:5562 2020-07-05 20:52:05,388 INFO [download_fvcom_results] connected to skookum.eos.ubc.ca port 5555 2020-07-05 20:53:36,913 INFO [download_fvcom_results] VHFR FVCOM r12 nowcast 2020-07-05 results files from arbutus.cloud-nowcast downloaded 2020-07-05 20:53:36,919 INFO [manager] checklist updated with [VHFR FVCOM results files] items from download_fvcom_results worker 2020-07-05 20:53:37,441 INFO [manager] launching NextWorker(module='nowcast.workers.get_vfpa_hadcp', args=['--data-date', '2020-07-05'], host='localhost') 2020-07-05 20:53:37,444 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['fvcom', 'nowcast-r12', 'research', '--run-date', '2020-07-05'], host='localhost') 2020-07-05 20:53:39,743 INFO [get_vfpa_hadcp] running in process 13318 2020-07-05 20:53:39,744 INFO [get_vfpa_hadcp] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-05 20:53:39,744 INFO [get_vfpa_hadcp] publishing log messages to tcp://*:5562 2020-07-05 20:53:39,745 INFO [get_vfpa_hadcp] connected to skookum.eos.ubc.ca port 5555 2020-07-05 20:53:39,746 INFO [get_vfpa_hadcp] processing VFPA HADCP data from 2nd Narrows Rail Bridge for 2020-07-05 2020-07-05 20:53:40,907 INFO [get_vfpa_hadcp] added VFPA HADCP data from 2nd Narrows Rail Bridge for 2020-07-05 to /opp/observations/AISDATA/netcdf/VFPA_2ND_NARROWS_HADCP_2s_202007.nc 2020-07-05 20:53:40,908 INFO [get_vfpa_hadcp] VFPA HADCP observations added to 2020-07 netcdf file 2020-07-05 20:53:40,909 INFO [manager] checklist updated with [VFPA HADCP data] items from get_vfpa_hadcp worker 2020-07-05 20:53:41,173 INFO [make_plots] running in process 13319 2020-07-05 20:53:41,174 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-05 20:53:41,175 INFO [make_plots] publishing log messages to tcp://*:5568 2020-07-05 20:53:41,175 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555 2020-07-05 20:53:41,177 INFO [make_plots] preparing render list for 2020-07-05 VHFR FVCOM nowcast-r12 research figures 2020-07-05 20:53:41,179 INFO [make_plots] added EnglishBay_surface_currents figures to 2020-07-05 VHFR FVCOM nowcast-r12 research render list 2020-07-05 20:53:41,180 INFO [make_plots] added VancouverHarbour_surface_currents figures to 2020-07-05 VHFR FVCOM nowcast-r12 research render list 2020-07-05 20:53:41,182 INFO [make_plots] added IndianArm_surface_currents figures to 2020-07-05 VHFR FVCOM nowcast-r12 research render list 2020-07-05 20:53:41,183 INFO [make_plots] added VancouverHarbour_thalweg figures to 2020-07-05 VHFR FVCOM nowcast-r12 research render list 2020-07-05 20:53:41,185 INFO [make_plots] added PortMoody_thalweg figures to 2020-07-05 VHFR FVCOM nowcast-r12 research render list 2020-07-05 20:53:41,187 INFO [make_plots] added IndianArm_thalweg figures to 2020-07-05 VHFR FVCOM nowcast-r12 research render list 2020-07-05 20:53:41,187 INFO [make_plots] starting to render fvcom nowcast-r12 research 05jul20 figures 2020-07-05 20:53:41,417 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['VFPA-HADCP'], host='localhost') 2020-07-05 20:53:42,292 INFO [ping_erddap] running in process 13327 2020-07-05 20:53:42,293 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-05 20:53:42,294 INFO [ping_erddap] publishing log messages to tcp://*:5562 2020-07-05 20:53:42,295 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555 2020-07-05 20:53:42,296 INFO [ping_erddap] VFPA-HADCP ERDDAP dataset flag file(s) created 2020-07-05 20:53:42,298 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker 2020-07-05 20:53:42,861 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['fvcom', 'nowcast-x2', 'publish', '--run-date', '2020-07-05'], host='localhost') 2020-07-05 20:53:42,864 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['fvcom', 'nowcast-r12', 'publish', '--run-date', '2020-07-05'], host='localhost') 2020-07-05 20:53:42,866 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['fvcom', 'forecast-x2', 'publish', '--run-date', '2020-07-05'], host='localhost') 2020-07-05 20:53:45,872 INFO [make_plots] running in process 13332 2020-07-05 20:53:45,873 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-05 20:53:45,874 INFO [make_plots] publishing log messages to tcp://*:5562 2020-07-05 20:53:45,874 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555 2020-07-05 20:53:45,909 INFO [make_plots] running in process 13331 2020-07-05 20:53:45,909 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-05 20:53:45,910 INFO [make_plots] publishing log messages to tcp://*:5569 2020-07-05 20:53:45,911 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555 2020-07-05 20:53:45,941 INFO [make_plots] running in process 13333 2020-07-05 20:53:45,942 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-05 20:53:45,943 INFO [make_plots] publishing log messages to tcp://*:5570 2020-07-05 20:53:45,944 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555 2020-07-05 20:53:46,878 INFO [make_plots] preparing render list for 2020-07-05 VHFR FVCOM nowcast-r12 publish figures 2020-07-05 20:53:46,880 INFO [make_plots] added Calamity Point figure to 2020-07-05 VHFR FVCOM nowcast-r12 publish render list 2020-07-05 20:53:46,881 INFO [make_plots] added Indian Arm Head figure to 2020-07-05 VHFR FVCOM nowcast-r12 publish render list 2020-07-05 20:53:46,881 INFO [make_plots] added New Westminster figure to 2020-07-05 VHFR FVCOM nowcast-r12 publish render list 2020-07-05 20:53:46,882 INFO [make_plots] added Port Moody figure to 2020-07-05 VHFR FVCOM nowcast-r12 publish render list 2020-07-05 20:53:46,883 INFO [make_plots] added Sand Heads figure to 2020-07-05 VHFR FVCOM nowcast-r12 publish render list 2020-07-05 20:53:46,884 INFO [make_plots] added Sandy Cove figure to 2020-07-05 VHFR FVCOM nowcast-r12 publish render list 2020-07-05 20:53:46,884 INFO [make_plots] added Vancouver Harbour figure to 2020-07-05 VHFR FVCOM nowcast-r12 publish render list 2020-07-05 20:53:46,885 INFO [make_plots] added Woodwards Landing figure to 2020-07-05 VHFR FVCOM nowcast-r12 publish render list 2020-07-05 20:53:46,885 INFO [make_plots] added 2ndNarrows_current figure to 2020-07-05 VHFR FVCOM nowcast-r12 publish render list 2020-07-05 20:53:46,886 INFO [make_plots] starting to render fvcom nowcast-r12 publish 05jul20 figures 2020-07-05 20:53:46,897 INFO [make_plots] preparing render list for 2020-07-05 VHFR FVCOM nowcast-x2 publish figures 2020-07-05 20:53:46,898 INFO [make_plots] added Calamity Point figure to 2020-07-05 VHFR FVCOM nowcast-x2 publish render list 2020-07-05 20:53:46,898 INFO [make_plots] added Indian Arm Head figure to 2020-07-05 VHFR FVCOM nowcast-x2 publish render list 2020-07-05 20:53:46,899 INFO [make_plots] added New Westminster figure to 2020-07-05 VHFR FVCOM nowcast-x2 publish render list 2020-07-05 20:53:46,900 INFO [make_plots] added Port Moody figure to 2020-07-05 VHFR FVCOM nowcast-x2 publish render list 2020-07-05 20:53:46,901 INFO [make_plots] added Sand Heads figure to 2020-07-05 VHFR FVCOM nowcast-x2 publish render list 2020-07-05 20:53:46,901 INFO [make_plots] added Sandy Cove figure to 2020-07-05 VHFR FVCOM nowcast-x2 publish render list 2020-07-05 20:53:46,902 INFO [make_plots] added Vancouver Harbour figure to 2020-07-05 VHFR FVCOM nowcast-x2 publish render list 2020-07-05 20:53:46,903 INFO [make_plots] added Woodwards Landing figure to 2020-07-05 VHFR FVCOM nowcast-x2 publish render list 2020-07-05 20:53:46,904 INFO [make_plots] added 2ndNarrows_current figure to 2020-07-05 VHFR FVCOM nowcast-x2 publish render list 2020-07-05 20:53:46,905 INFO [make_plots] starting to render fvcom nowcast-x2 publish 05jul20 figures 2020-07-05 20:53:46,907 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07724 Calamity Point from 2020-07-04 23:55:00 to 2020-07-06 00:00:00 2020-07-05 20:53:46,923 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07724 Calamity Point from 2020-07-04 23:55:00 to 2020-07-06 00:00:00 2020-07-05 20:53:47,006 INFO [make_plots] preparing render list for 2020-07-05 VHFR FVCOM forecast-x2 publish figures 2020-07-05 20:53:47,007 INFO [make_plots] added Calamity Point figure to 2020-07-05 VHFR FVCOM forecast-x2 publish render list 2020-07-05 20:53:47,008 INFO [make_plots] added Indian Arm Head figure to 2020-07-05 VHFR FVCOM forecast-x2 publish render list 2020-07-05 20:53:47,008 INFO [make_plots] added New Westminster figure to 2020-07-05 VHFR FVCOM forecast-x2 publish render list 2020-07-05 20:53:47,009 INFO [make_plots] added Port Moody figure to 2020-07-05 VHFR FVCOM forecast-x2 publish render list 2020-07-05 20:53:47,010 INFO [make_plots] added Sand Heads figure to 2020-07-05 VHFR FVCOM forecast-x2 publish render list 2020-07-05 20:53:47,010 INFO [make_plots] added Sandy Cove figure to 2020-07-05 VHFR FVCOM forecast-x2 publish render list 2020-07-05 20:53:47,011 INFO [make_plots] added Vancouver Harbour figure to 2020-07-05 VHFR FVCOM forecast-x2 publish render list 2020-07-05 20:53:47,012 INFO [make_plots] added Woodwards Landing figure to 2020-07-05 VHFR FVCOM forecast-x2 publish render list 2020-07-05 20:53:47,012 INFO [make_plots] added 2ndNarrows_current figure to 2020-07-05 VHFR FVCOM forecast-x2 publish render list 2020-07-05 20:53:47,013 INFO [make_plots] starting to render fvcom forecast-x2 publish 05jul20 figures 2020-07-05 20:53:47,055 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07724 Calamity Point from 2020-07-04 23:55:00 to 2020-07-07 12:00:00 2020-07-05 20:54:09,745 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07724 Calamity Point from 2020-07-05 00:00:00 to 2020-07-06 00:00:00 2020-07-05 20:54:10,129 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07724 Calamity Point from 2020-07-05 00:00:00 to 2020-07-06 00:00:00 2020-07-05 20:54:11,180 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07774 Indian Arm Head from 2020-07-04 23:55:00 to 2020-07-06 00:00:00 2020-07-05 20:54:11,465 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07774 Indian Arm Head from 2020-07-04 23:55:00 to 2020-07-06 00:00:00 2020-07-05 20:54:11,921 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07774 Indian Arm Head from 2020-07-05 00:00:00 to 2020-07-06 00:00:00 2020-07-05 20:54:12,198 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07774 Indian Arm Head from 2020-07-05 00:00:00 to 2020-07-06 00:00:00 2020-07-05 20:54:12,533 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07724 Calamity Point from 2020-07-05 00:00:00 to 2020-07-07 12:00:00 2020-07-05 20:54:13,299 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07654 New Westminster from 2020-07-04 23:55:00 to 2020-07-06 00:00:00 2020-07-05 20:54:13,596 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07654 New Westminster from 2020-07-04 23:55:00 to 2020-07-06 00:00:00 2020-07-05 20:54:14,181 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07774 Indian Arm Head from 2020-07-04 23:55:00 to 2020-07-07 12:00:00 2020-07-05 20:54:14,761 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07774 Indian Arm Head from 2020-07-05 00:00:00 to 2020-07-07 12:00:00 2020-07-05 20:54:16,304 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07654 New Westminster from 2020-07-04 23:55:00 to 2020-07-07 12:00:00 2020-07-05 20:54:35,690 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07654 New Westminster from 2020-07-05 00:00:00 to 2020-07-06 00:00:00 2020-07-05 20:54:35,798 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07654 New Westminster from 2020-07-05 00:00:00 to 2020-07-06 00:00:00 2020-07-05 20:54:38,952 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07755 Port Moody from 2020-07-04 23:55:00 to 2020-07-06 00:00:00 2020-07-05 20:54:38,953 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07755 Port Moody from 2020-07-04 23:55:00 to 2020-07-06 00:00:00 2020-07-05 20:54:41,377 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07654 New Westminster from 2020-07-05 00:00:00 to 2020-07-07 12:00:00 2020-07-05 20:54:46,473 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07755 Port Moody from 2020-07-04 23:55:00 to 2020-07-07 12:00:00 2020-07-05 20:55:01,276 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07755 Port Moody from 2020-07-05 00:00:00 to 2020-07-06 00:00:00 2020-07-05 20:55:01,584 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07755 Port Moody from 2020-07-05 00:00:00 to 2020-07-06 00:00:00 2020-07-05 20:55:04,282 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07594 Sand Heads from 2020-07-04 23:55:00 to 2020-07-06 00:00:00 2020-07-05 20:55:04,494 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07594 Sand Heads from 2020-07-04 23:55:00 to 2020-07-06 00:00:00 2020-07-05 20:55:11,613 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07755 Port Moody from 2020-07-05 00:00:00 to 2020-07-07 12:00:00 2020-07-05 20:55:16,724 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07594 Sand Heads from 2020-07-04 23:55:00 to 2020-07-07 12:00:00 2020-07-05 20:55:26,828 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07594 Sand Heads from 2020-07-05 00:00:00 to 2020-07-06 00:00:00 2020-07-05 20:55:27,194 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07594 Sand Heads from 2020-07-05 00:00:00 to 2020-07-06 00:00:00 2020-07-05 20:55:29,897 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07786 Sandy Cove from 2020-07-04 23:55:00 to 2020-07-06 00:00:00 2020-07-05 20:55:30,161 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07786 Sandy Cove from 2020-07-04 23:55:00 to 2020-07-06 00:00:00 2020-07-05 20:55:42,564 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07594 Sand Heads from 2020-07-05 00:00:00 to 2020-07-07 12:00:00 2020-07-05 20:55:47,709 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07786 Sandy Cove from 2020-07-04 23:55:00 to 2020-07-07 12:00:00 2020-07-05 20:55:53,148 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07795 Point Atkinson from 2020-07-05 00:00:00 to 2020-07-06 00:00:00 2020-07-05 20:55:53,225 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07795 Point Atkinson from 2020-07-05 00:00:00 to 2020-07-06 00:00:00 2020-07-05 20:55:56,359 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07735 Vancouver Harbour from 2020-07-04 23:55:00 to 2020-07-06 00:00:00 2020-07-05 20:55:56,359 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07735 Vancouver Harbour from 2020-07-04 23:55:00 to 2020-07-06 00:00:00 2020-07-05 20:56:13,285 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07795 Point Atkinson from 2020-07-05 00:00:00 to 2020-07-07 12:00:00 2020-07-05 20:56:18,445 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07735 Vancouver Harbour from 2020-07-04 23:55:00 to 2020-07-07 12:00:00 2020-07-05 20:56:19,125 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07735 Vancouver Harbour from 2020-07-05 00:00:00 to 2020-07-06 00:00:00 2020-07-05 20:56:19,182 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07735 Vancouver Harbour from 2020-07-05 00:00:00 to 2020-07-06 00:00:00 2020-07-05 20:56:22,093 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07610 Woodwards Landing from 2020-07-04 23:55:00 to 2020-07-06 00:00:00 2020-07-05 20:56:22,115 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07610 Woodwards Landing from 2020-07-04 23:55:00 to 2020-07-06 00:00:00 2020-07-05 20:56:40,890 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07610 Woodwards Landing from 2020-07-05 00:00:00 to 2020-07-06 00:00:00 2020-07-05 20:56:40,967 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07610 Woodwards Landing from 2020-07-05 00:00:00 to 2020-07-06 00:00:00 2020-07-05 20:56:43,618 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07735 Vancouver Harbour from 2020-07-05 00:00:00 to 2020-07-07 12:00:00 2020-07-05 20:56:46,137 INFO [make_plots] finished rendering fvcom nowcast-x2 publish 05jul20 figures 2020-07-05 20:56:46,309 INFO [make_plots] finished rendering fvcom nowcast-r12 publish 05jul20 figures 2020-07-05 20:56:46,569 INFO [make_plots] fvcom publish plots for 2020-07-05 nowcast-x2 completed 2020-07-05 20:56:46,575 INFO [manager] checklist updated with [plots] items from make_plots worker 2020-07-05 20:56:46,735 INFO [make_plots] fvcom publish plots for 2020-07-05 nowcast-r12 completed 2020-07-05 20:56:47,069 INFO [manager] checklist updated with [plots] items from make_plots worker 2020-07-05 20:56:48,844 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07610 Woodwards Landing from 2020-07-04 23:55:00 to 2020-07-07 12:00:00 2020-07-05 20:57:10,114 INFO [make_plots] retrieving pred water level data from https://ws-shc.qc.dfo-mpo.gc.ca/predictions?WSDL for station 07610 Woodwards Landing from 2020-07-05 00:00:00 to 2020-07-07 12:00:00 2020-07-05 20:57:15,382 INFO [make_plots] finished rendering fvcom forecast-x2 publish 05jul20 figures 2020-07-05 20:57:15,994 INFO [make_plots] fvcom publish plots for 2020-07-05 forecast-x2 completed 2020-07-05 20:57:16,002 INFO [manager] checklist updated with [plots] items from make_plots worker 2020-07-05 21:05:52,786 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-west/00/ to /results/forcing/atmospheric/GEM2.5/GRIB/20200706/00/ 2020-07-05 21:05:52,787 INFO [collect_weather] 2020-07-06 2.5km weather forecast 00 collection complete 2020-07-05 21:05:52,789 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker 2020-07-05 21:05:53,635 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['06', '2.5km'], host='localhost') 2020-07-05 21:05:54,590 INFO [collect_weather] running in process 13693 2020-07-05 21:05:54,591 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-05 21:05:54,592 INFO [collect_weather] publishing log messages to tcp://*:5562 2020-07-05 21:05:54,593 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555 2020-07-05 21:05:54,610 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-west/06/ 2020-07-05 21:49:58,142 INFO [make_plots] finished rendering fvcom forecast-x2 research 05jul20 figures 2020-07-05 21:49:58,143 INFO [make_plots] fvcom research plots for 2020-07-05 forecast-x2 completed 2020-07-05 21:49:58,399 INFO [manager] checklist updated with [plots] items from make_plots worker 2020-07-05 22:14:56,164 INFO [make_plots] finished rendering fvcom nowcast-r12 research 05jul20 figures 2020-07-05 22:14:56,165 INFO [make_plots] fvcom research plots for 2020-07-05 nowcast-r12 completed 2020-07-05 22:14:56,265 INFO [manager] checklist updated with [plots] items from make_plots worker 2020-07-06 03:02:35,376 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-west/06/ to /results/forcing/atmospheric/GEM2.5/GRIB/20200706/06/ 2020-07-06 03:02:35,377 INFO [collect_weather] 2020-07-06 2.5km weather forecast 06 collection complete 2020-07-06 03:02:35,379 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker 2020-07-06 03:02:36,372 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['Capilano', '--data-date', '2020-07-05'], host='localhost') 2020-07-06 03:02:36,541 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ChilliwackVedder', '--data-date', '2020-07-05'], host='localhost') 2020-07-06 03:02:36,546 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ClowhomClowhomLake', '--data-date', '2020-07-05'], host='localhost') 2020-07-06 03:02:36,550 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['Englishman', '--data-date', '2020-07-05'], host='localhost') 2020-07-06 03:02:36,555 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['Fraser', '--data-date', '2020-07-05'], host='localhost') 2020-07-06 03:02:36,560 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['HomathkoMouth', '--data-date', '2020-07-05'], host='localhost') 2020-07-06 03:02:36,564 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['SalmonSayward', '--data-date', '2020-07-05'], host='localhost') 2020-07-06 03:02:36,569 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['SanJuanPortRenfrew', '--data-date', '2020-07-05'], host='localhost') 2020-07-06 03:02:36,575 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['SquamishBrackendale', '--data-date', '2020-07-05'], host='localhost') 2020-07-06 03:02:36,580 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['TheodosiaScotty', '--data-date', '2020-07-05'], host='localhost') 2020-07-06 03:02:36,588 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['TheodosiaBypass', '--data-date', '2020-07-05'], host='localhost') 2020-07-06 03:02:36,592 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['TheodosiaDiversion', '--data-date', '2020-07-05'], host='localhost') 2020-07-06 03:02:36,596 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ctd', args=['SCVIP'], host='localhost') 2020-07-06 03:02:36,603 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ctd', args=['SEVIP'], host='localhost') 2020-07-06 03:02:36,608 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ferry', args=['TWDP'], host='localhost') 2020-07-06 03:02:36,613 INFO [manager] launching NextWorker(module='nowcast.workers.get_NeahBay_ssh', args=['forecast2'], host='localhost') 2020-07-06 03:02:36,618 INFO [manager] launching NextWorker(module='nowcast.workers.grib_to_netcdf', args=['forecast2'], host='localhost') 2020-07-06 03:02:36,623 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['12', '2.5km'], host='localhost') 2020-07-06 03:02:38,619 INFO [collect_weather] running in process 21519 2020-07-06 03:02:38,620 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-06 03:02:38,620 INFO [collect_weather] publishing log messages to tcp://*:5562 2020-07-06 03:02:38,621 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555 2020-07-06 03:02:38,643 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-west/12/ 2020-07-06 03:02:39,914 INFO [grib_to_netcdf] running in process 21518 2020-07-06 03:02:39,916 INFO [grib_to_netcdf] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-06 03:02:39,917 INFO [grib_to_netcdf] publishing log messages to tcp://*:5564 2020-07-06 03:02:39,918 INFO [grib_to_netcdf] connected to skookum.eos.ubc.ca port 5555 2020-07-06 03:02:39,919 INFO [grib_to_netcdf] forecast section: 20200706/06 2020-07-06 03:02:39,920 INFO [grib_to_netcdf] next day forecast section: 20200706/06 2020-07-06 03:02:40,300 INFO [collect_river_data] running in process 21507 2020-07-06 03:02:40,309 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-06 03:02:40,310 INFO [collect_river_data] publishing log messages to tcp://*:5567 2020-07-06 03:02:40,311 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2020-07-06 03:02:40,312 INFO [collect_river_data] Collecting TheodosiaDiversion river data for 2020-07-05 2020-07-06 03:02:40,313 INFO [collect_river_data] running in process 21500 2020-07-06 03:02:40,314 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-06 03:02:40,315 INFO [collect_river_data] publishing log messages to tcp://*:5568 2020-07-06 03:02:40,315 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2020-07-06 03:02:40,316 INFO [collect_river_data] Collecting Fraser river data for 2020-07-05 2020-07-06 03:02:40,340 INFO [collect_river_data] running in process 21502 2020-07-06 03:02:40,340 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-06 03:02:40,341 INFO [collect_river_data] publishing log messages to tcp://*:5569 2020-07-06 03:02:40,342 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2020-07-06 03:02:40,343 INFO [collect_river_data] Collecting SalmonSayward river data for 2020-07-05 2020-07-06 03:02:40,352 INFO [collect_river_data] running in process 21498 2020-07-06 03:02:40,353 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-06 03:02:40,354 INFO [collect_river_data] publishing log messages to tcp://*:5570 2020-07-06 03:02:40,354 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2020-07-06 03:02:40,355 INFO [collect_river_data] Collecting ClowhomClowhomLake river data for 2020-07-05 2020-07-06 03:02:40,364 INFO [collect_river_data] running in process 21497 2020-07-06 03:02:40,365 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-06 03:02:40,366 INFO [collect_river_data] publishing log messages to tcp://*:5571 2020-07-06 03:02:40,367 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2020-07-06 03:02:40,367 INFO [collect_river_data] Collecting ChilliwackVedder river data for 2020-07-05 2020-07-06 03:02:40,378 INFO [collect_river_data] running in process 21505 2020-07-06 03:02:40,379 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-06 03:02:40,379 INFO [collect_river_data] publishing log messages to tcp://*:5572 2020-07-06 03:02:40,380 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2020-07-06 03:02:40,381 INFO [collect_river_data] Collecting TheodosiaScotty river data for 2020-07-05 2020-07-06 03:02:40,386 INFO [collect_river_data] running in process 21506 2020-07-06 03:02:40,387 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-06 03:02:40,387 INFO [collect_river_data] publishing log messages to tcp://*:5573 2020-07-06 03:02:40,388 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2020-07-06 03:02:40,389 INFO [collect_river_data] Collecting TheodosiaBypass river data for 2020-07-05 2020-07-06 03:02:40,445 INFO [collect_river_data] Appended Fraser river average discharge for 2020-07-05 to: /data/dlatorne/SOG-projects/SOG-forcing/ECget/Fraser_flow 2020-07-06 03:02:40,445 INFO [collect_river_data] Fraser river average discharge for 2020-07-05 calculated and appended to Fraser_flow file 2020-07-06 03:02:40,447 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2020-07-06 03:02:40,451 INFO [collect_river_data] running in process 21499 2020-07-06 03:02:40,451 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-06 03:02:40,452 INFO [collect_river_data] publishing log messages to tcp://*:5574 2020-07-06 03:02:40,453 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2020-07-06 03:02:40,454 INFO [collect_river_data] Collecting Englishman river data for 2020-07-05 2020-07-06 03:02:40,495 INFO [collect_river_data] Appended ChilliwackVedder river average discharge for 2020-07-05 to: /results/forcing/rivers/observations/Chilliwack_Vedder_flow 2020-07-06 03:02:40,495 INFO [collect_river_data] ChilliwackVedder river average discharge for 2020-07-05 calculated and appended to ChilliwackVedder_flow file 2020-07-06 03:02:40,496 INFO [collect_river_data] Appended TheodosiaDiversion river average discharge for 2020-07-05 to: /results/forcing/rivers/observations/Theodosia_Diversion_flow 2020-07-06 03:02:40,497 INFO [collect_river_data] TheodosiaDiversion river average discharge for 2020-07-05 calculated and appended to TheodosiaDiversion_flow file 2020-07-06 03:02:40,519 INFO [collect_river_data] Appended SalmonSayward river average discharge for 2020-07-05 to: /results/forcing/rivers/observations/Salmon_Sayward_flow 2020-07-06 03:02:40,520 INFO [collect_river_data] SalmonSayward river average discharge for 2020-07-05 calculated and appended to SalmonSayward_flow file 2020-07-06 03:02:40,521 INFO [collect_river_data] Appended ClowhomClowhomLake river average discharge for 2020-07-05 to: /results/forcing/rivers/observations/Clowhom_ClowhomLake_flow 2020-07-06 03:02:40,522 INFO [collect_river_data] ClowhomClowhomLake river average discharge for 2020-07-05 calculated and appended to ClowhomClowhomLake_flow file 2020-07-06 03:02:40,535 INFO [collect_river_data] Appended TheodosiaScotty river average discharge for 2020-07-05 to: /results/forcing/rivers/observations/Theodosia_Scotty_flow 2020-07-06 03:02:40,544 INFO [collect_river_data] TheodosiaScotty river average discharge for 2020-07-05 calculated and appended to TheodosiaScotty_flow file 2020-07-06 03:02:40,546 INFO [collect_river_data] Appended TheodosiaBypass river average discharge for 2020-07-05 to: /results/forcing/rivers/observations/Theodosia_Bypass_flow 2020-07-06 03:02:40,547 INFO [collect_river_data] TheodosiaBypass river average discharge for 2020-07-05 calculated and appended to TheodosiaBypass_flow file 2020-07-06 03:02:40,560 INFO [collect_river_data] running in process 21503 2020-07-06 03:02:40,561 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-06 03:02:40,562 INFO [collect_river_data] publishing log messages to tcp://*:5575 2020-07-06 03:02:40,563 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2020-07-06 03:02:40,563 INFO [collect_river_data] Collecting SanJuanPortRenfrew river data for 2020-07-05 2020-07-06 03:02:40,564 INFO [collect_river_data] Appended Englishman river average discharge for 2020-07-05 to: /data/dlatorne/SOG-projects/SOG-forcing/ECget/Englishman_flow 2020-07-06 03:02:40,565 INFO [collect_river_data] Englishman river average discharge for 2020-07-05 calculated and appended to Englishman_flow file 2020-07-06 03:02:40,587 INFO [collect_river_data] running in process 21495 2020-07-06 03:02:40,587 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-06 03:02:40,591 INFO [collect_river_data] publishing log messages to tcp://*:5576 2020-07-06 03:02:40,652 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2020-07-06 03:02:40,727 INFO [collect_river_data] Collecting Capilano river data for 2020-07-05 2020-07-06 03:02:40,787 INFO [collect_river_data] running in process 21501 2020-07-06 03:02:40,788 INFO [collect_river_data] running in process 21504 2020-07-06 03:02:40,789 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-06 03:02:40,789 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-06 03:02:40,791 INFO [collect_river_data] publishing log messages to tcp://*:5577 2020-07-06 03:02:40,791 INFO [collect_river_data] publishing log messages to tcp://*:5578 2020-07-06 03:02:40,792 INFO [collect_river_data] Appended SanJuanPortRenfrew river average discharge for 2020-07-05 to: /results/forcing/rivers/observations/SanJuan_PortRenfrew_flow 2020-07-06 03:02:40,793 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2020-07-06 03:02:40,794 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2020-07-06 03:02:40,795 INFO [collect_river_data] SanJuanPortRenfrew river average discharge for 2020-07-05 calculated and appended to SanJuanPortRenfrew_flow file 2020-07-06 03:02:40,796 INFO [collect_river_data] Collecting HomathkoMouth river data for 2020-07-05 2020-07-06 03:02:40,796 INFO [collect_river_data] Collecting SquamishBrackendale river data for 2020-07-05 2020-07-06 03:02:40,799 INFO [collect_river_data] Appended HomathkoMouth river average discharge for 2020-07-05 to: /results/forcing/rivers/observations/Homathko_Mouth_flow 2020-07-06 03:02:40,799 INFO [collect_river_data] HomathkoMouth river average discharge for 2020-07-05 calculated and appended to HomathkoMouth_flow file 2020-07-06 03:02:40,811 INFO [collect_river_data] Appended Capilano river average discharge for 2020-07-05 to: /opp/observations/rivers/Capilano/Caplilano_08GA010_day_avg_flow 2020-07-06 03:02:40,812 INFO [collect_river_data] Capilano river average discharge for 2020-07-05 calculated and appended to Capilano_flow file 2020-07-06 03:02:40,827 INFO [collect_river_data] Appended SquamishBrackendale river average discharge for 2020-07-05 to: /results/forcing/rivers/observations/Squamish_Brackendale_flow 2020-07-06 03:02:40,854 INFO [collect_river_data] SquamishBrackendale river average discharge for 2020-07-05 calculated and appended to SquamishBrackendale_flow file 2020-07-06 03:02:41,097 INFO [manager] launching NextWorker(module='nowcast.workers.make_runoff_file', args=[], host='localhost') 2020-07-06 03:02:41,119 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2020-07-06 03:02:41,811 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2020-07-06 03:02:42,219 INFO [get_onc_ctd] running in process 21512 2020-07-06 03:02:42,220 INFO [get_onc_ctd] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-06 03:02:42,221 INFO [get_onc_ctd] publishing log messages to tcp://*:5568 2020-07-06 03:02:42,222 INFO [get_onc_ctd] connected to skookum.eos.ubc.ca port 5555 2020-07-06 03:02:42,222 INFO [get_onc_ctd] requesting ONC SEVIP CTD T&S data for 2020-07-05 2020-07-06 03:02:42,243 INFO [get_onc_ctd] running in process 21508 2020-07-06 03:02:42,244 INFO [get_onc_ctd] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-06 03:02:42,245 INFO [get_onc_ctd] publishing log messages to tcp://*:5571 2020-07-06 03:02:42,246 INFO [get_onc_ctd] connected to skookum.eos.ubc.ca port 5555 2020-07-06 03:02:42,246 INFO [get_onc_ctd] requesting ONC SCVIP CTD T&S data for 2020-07-05 2020-07-06 03:02:42,260 INFO [get_onc_ferry] running in process 21515 2020-07-06 03:02:42,260 INFO [get_onc_ferry] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-06 03:02:42,261 INFO [get_onc_ferry] publishing log messages to tcp://*:5579 2020-07-06 03:02:42,262 INFO [get_onc_ferry] connected to skookum.eos.ubc.ca port 5555 2020-07-06 03:02:42,262 INFO [get_onc_ferry] requesting ONC TWDP.N1 NAV data for 2020-07-05 2020-07-06 03:02:42,339 INFO [get_NeahBay_ssh] running in process 21517 2020-07-06 03:02:42,340 INFO [get_NeahBay_ssh] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-06 03:02:42,341 INFO [get_NeahBay_ssh] publishing log messages to tcp://*:5590 2020-07-06 03:02:42,342 INFO [get_NeahBay_ssh] connected to skookum.eos.ubc.ca port 5555 2020-07-06 03:02:42,411 INFO [make_runoff_file] running in process 21584 2020-07-06 03:02:42,413 INFO [make_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-06 03:02:42,413 INFO [make_runoff_file] publishing log messages to tcp://*:5591 2020-07-06 03:02:42,414 INFO [make_runoff_file] connected to skookum.eos.ubc.ca port 5555 2020-07-06 03:02:42,440 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2020-07-06 03:02:42,738 INFO [make_runoff_file] calculating runoff forcing for bathymetry type: b201702 2020-07-06 03:02:42,836 INFO [make_runoff_file] 2020-07-06 runoff file creation from Fraser at Hope and climatology elsewhere complete 2020-07-06 03:02:42,990 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2020-07-06 03:02:43,591 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2020-07-06 03:02:44,158 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2020-07-06 03:02:44,740 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2020-07-06 03:02:45,288 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2020-07-06 03:02:45,850 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2020-07-06 03:02:46,405 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2020-07-06 03:02:46,471 INFO [get_NeahBay_ssh] Neah Bay sea surface height web scraping and open boundary file creation for forecast2 complete 2020-07-06 03:02:46,962 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2020-07-06 03:02:47,537 INFO [manager] checklist updated with [rivers forcing] items from make_runoff_file worker 2020-07-06 03:02:48,126 INFO [manager] checklist updated with [Neah Bay ssh] items from get_NeahBay_ssh worker 2020-07-06 03:03:05,854 INFO [get_onc_ctd] 2020-07-05 ONC SCVIP CTD T&S file created 2020-07-06 03:03:05,856 INFO [manager] checklist updated with [ONC CTD data] items from get_onc_ctd worker 2020-07-06 03:03:06,240 INFO [get_onc_ctd] 2020-07-05 ONC SEVIP CTD T&S file created 2020-07-06 03:03:06,260 INFO [get_onc_ferry] requesting ONC TWDP TSG data for 2020-07-05 2020-07-06 03:03:06,432 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['SCVIP-CTD'], host='localhost') 2020-07-06 03:03:06,436 INFO [manager] checklist updated with [ONC CTD data] items from get_onc_ctd worker 2020-07-06 03:03:06,996 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['SEVIP-CTD'], host='localhost') 2020-07-06 03:03:07,236 INFO [ping_erddap] running in process 21959 2020-07-06 03:03:07,237 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-06 03:03:07,237 INFO [ping_erddap] publishing log messages to tcp://*:5567 2020-07-06 03:03:07,238 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555 2020-07-06 03:03:07,258 INFO [ping_erddap] SCVIP-CTD ERDDAP dataset flag file(s) created 2020-07-06 03:03:07,261 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker 2020-07-06 03:03:07,824 INFO [ping_erddap] running in process 21963 2020-07-06 03:03:07,825 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-06 03:03:07,826 INFO [ping_erddap] publishing log messages to tcp://*:5568 2020-07-06 03:03:07,827 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555 2020-07-06 03:03:07,828 INFO [ping_erddap] SEVIP-CTD ERDDAP dataset flag file(s) created 2020-07-06 03:03:07,832 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker 2020-07-06 03:03:35,614 INFO [grib_to_netcdf] 2020-07-06 NEMO-atmos forcing file for forecast2 created 2020-07-06 03:03:35,617 INFO [manager] checklist updated with [weather forcing] items from grib_to_netcdf worker 2020-07-06 03:03:36,123 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'forecast2'], host='localhost') 2020-07-06 03:03:36,126 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'forecast2'], host='localhost') 2020-07-06 03:03:36,129 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['graham-hindcast', 'forecast2'], host='localhost') 2020-07-06 03:03:36,132 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'forecast2'], host='localhost') 2020-07-06 03:03:37,032 INFO [upload_forcing] running in process 22361 2020-07-06 03:03:37,033 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-06 03:03:37,034 INFO [upload_forcing] publishing log messages to tcp://*:5564 2020-07-06 03:03:37,035 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2020-07-06 03:03:37,050 INFO [upload_forcing] running in process 22362 2020-07-06 03:03:37,051 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-06 03:03:37,052 INFO [upload_forcing] publishing log messages to tcp://*:5567 2020-07-06 03:03:37,052 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2020-07-06 03:03:37,053 INFO [upload_forcing] running in process 22364 2020-07-06 03:03:37,054 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-06 03:03:37,055 INFO [upload_forcing] publishing log messages to tcp://*:5568 2020-07-06 03:03:37,055 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2020-07-06 03:03:37,076 INFO [upload_forcing] running in process 22363 2020-07-06 03:03:37,077 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-06 03:03:37,078 INFO [upload_forcing] publishing log messages to tcp://*:5569 2020-07-06 03:03:37,079 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2020-07-06 03:03:38,949 INFO [get_onc_ferry] requesting ONC TWDP OXYSENSOR data for 2020-07-05 2020-07-06 03:03:39,391 INFO [upload_forcing] LiveOcean boundary conditions file not found; created symlink to /results/forcing/LiveOcean/boundary_conditions/LiveOcean_v201905_y2020m07d05.nc 2020-07-06 03:03:39,503 INFO [upload_forcing] forecast2 2020-07-06 forcing files upload to optimum-hindcast completed 2020-07-06 03:03:39,507 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2020-07-06 03:03:39,930 INFO [upload_forcing] forecast2 2020-07-06 forcing files upload to orcinus-nowcast-agrif completed 2020-07-06 03:03:40,108 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2020-07-06 03:03:40,345 INFO [upload_forcing] forecast2 2020-07-06 forcing files upload to arbutus.cloud-nowcast completed 2020-07-06 03:03:40,652 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2020-07-06 03:03:41,188 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2020-07-06'], host='localhost') 2020-07-06 03:03:42,218 INFO [make_forcing_links] running in process 22385 2020-07-06 03:03:42,219 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-06 03:03:42,220 INFO [make_forcing_links] publishing log messages to tcp://*:5564 2020-07-06 03:03:42,221 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555 2020-07-06 03:03:43,459 INFO [make_forcing_links] forecast2 2020-07-06 forcing file links on arbutus.cloud-nowcast created 2020-07-06 03:03:43,464 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker 2020-07-06 03:03:44,120 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2020-07-06'], host='arbutus.cloud-nowcast') 2020-07-06 03:03:44,873 INFO [upload_forcing] forecast2 2020-07-06 forcing files upload to graham-hindcast completed 2020-07-06 03:03:44,879 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2020-07-06 03:03:46,380 INFO [run_NEMO] running in process 18802 2020-07-06 03:03:46,381 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2020-07-06 03:03:46,382 INFO [run_NEMO] publishing log messages to tcp://*:5556 2020-07-06 03:03:46,391 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555 2020-07-06 03:03:47,619 INFO [run_NEMO] forecast2: launching /nemoShare/MEOPAR/nowcast-sys/runs/05jul20forecast2_2020-07-06T030346.478662-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast 2020-07-06 03:03:47,636 INFO [run_NEMO] forecast2 NEMO run for 2020-07-06 on arbutus.cloud-nowcast started 2020-07-06 03:03:47,642 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker 2020-07-06 03:03:48,343 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'forecast2'], host='arbutus.cloud-nowcast') 2020-07-06 03:03:50,377 INFO [watch_NEMO] running in process 19071 2020-07-06 03:03:50,378 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2020-07-06 03:03:50,379 INFO [watch_NEMO] publishing log messages to tcp://*:5557 2020-07-06 03:03:50,380 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555 2020-07-06 03:03:50,491 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: time.step not found; continuing to watch... 2020-07-06 03:04:14,502 INFO [get_onc_ferry] requesting ONC TWDP TURBCHLFL data for 2020-07-05 2020-07-06 03:04:48,953 INFO [get_onc_ferry] requesting ONC TWDP CO2SENSOR data for 2020-07-05 2020-07-06 03:05:02,120 INFO [get_onc_ferry] requesting ONC TWDP TEMPHUMID data for 2020-07-05 2020-07-06 03:05:26,632 INFO [get_onc_ferry] requesting ONC TWDP BARPRESS data for 2020-07-05 2020-07-06 03:05:38,605 INFO [get_onc_ferry] requesting ONC TWDP PYRANOMETER data for 2020-07-05 2020-07-06 03:05:49,369 INFO [get_onc_ferry] requesting ONC TWDP PYRGEOMETER data for 2020-07-05 2020-07-06 03:06:03,664 INFO [get_onc_ferry] 2020-07-05 ONC TWDP ferry data file created 2020-07-06 03:06:03,670 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['TWDP-ferry'], host='localhost') 2020-07-06 03:06:04,722 INFO [ping_erddap] running in process 22429 2020-07-06 03:06:04,723 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-06 03:06:04,724 INFO [ping_erddap] publishing log messages to tcp://*:5564 2020-07-06 03:06:04,725 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555 2020-07-06 03:06:04,727 INFO [ping_erddap] TWDP-ferry ERDDAP dataset flag file(s) created 2020-07-06 03:06:04,729 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker 2020-07-06 03:08:50,592 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 7504301 = 2020-07-07 05:06:40 UTC, 17.0% complete 2020-07-06 03:13:50,694 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 7504823 = 2020-07-07 10:54:40 UTC, 36.4% complete 2020-07-06 03:18:50,795 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 7505341 = 2020-07-07 16:40:00 UTC, 55.6% complete 2020-07-06 03:23:50,897 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 7505856 = 2020-07-07 22:23:20 UTC, 74.7% complete 2020-07-06 03:28:51,000 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 7506347 = 2020-07-08 03:50:40 UTC, 92.8% complete 2020-07-06 03:33:51,102 INFO [watch_NEMO] forecast2 NEMO run on arbutus.cloud-nowcast completed 2020-07-06 03:33:51,107 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker 2020-07-06 03:33:52,062 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_wind_file', args=['arbutus.cloud-nowcast', 'forecast2'], host='arbutus.cloud-nowcast') 2020-07-06 03:33:52,070 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_current_file', args=['arbutus.cloud-nowcast', 'forecast2'], host='arbutus.cloud-nowcast') 2020-07-06 03:33:52,074 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2020-07-05'], host='localhost') 2020-07-06 03:33:53,260 INFO [download_results] running in process 23097 2020-07-06 03:33:53,261 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-06 03:33:53,261 INFO [download_results] publishing log messages to tcp://*:5564 2020-07-06 03:33:53,262 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2020-07-06 03:33:53,263 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/forecast2/05jul20 to /results/SalishSea/forecast2.201905 2020-07-06 03:33:57,545 INFO [make_ww3_wind_file] running in process 19845 2020-07-06 03:33:57,546 INFO [make_ww3_wind_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2020-07-06 03:33:57,547 INFO [make_ww3_wind_file] publishing log messages to tcp://*:5570 2020-07-06 03:33:57,548 INFO [make_ww3_wind_file] connected to skookum.eos.ubc.ca port 5555 2020-07-06 03:33:57,550 INFO [make_ww3_wind_file] Creating wwatch3 wind forcing file for 2020-07-06 forecast2 run 2020-07-06 03:33:57,888 INFO [make_ww3_current_file] running in process 19846 2020-07-06 03:33:57,889 INFO [make_ww3_current_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2020-07-06 03:33:57,890 INFO [make_ww3_current_file] publishing log messages to tcp://*:5571 2020-07-06 03:33:57,891 INFO [make_ww3_current_file] connected to skookum.eos.ubc.ca port 5555 2020-07-06 03:33:57,893 INFO [make_ww3_current_file] Creating wwatch3 currents forcing file for 2020-07-06 forecast2 run 2020-07-06 03:34:01,301 INFO [make_ww3_wind_file] wwatch3 wind forcing file created on arbutus.cloud-nowcast for 2020-07-06 forecast2 run 2020-07-06 03:34:01,305 INFO [manager] checklist updated with [WW3 wind forcing] items from make_ww3_wind_file worker 2020-07-06 03:35:52,510 INFO [make_ww3_current_file] wwatch3 currents forcing file created on arbutus.cloud-nowcast for 2020-07-06 forecast2 run 2020-07-06 03:35:52,515 INFO [manager] checklist updated with [WW3 currents forcing] items from make_ww3_current_file worker 2020-07-06 03:35:53,206 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2020-07-06'], host='arbutus.cloud-nowcast') 2020-07-06 03:35:55,166 INFO [run_ww3] running in process 19987 2020-07-06 03:35:55,167 INFO [run_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2020-07-06 03:35:55,168 INFO [run_ww3] publishing log messages to tcp://*:5572 2020-07-06 03:35:55,169 INFO [run_ww3] connected to skookum.eos.ubc.ca port 5555 2020-07-06 03:35:55,173 INFO [run_ww3] Created run directory /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast2_2020-07-06T033555.164945-0700 2020-07-06 03:35:55,174 INFO [run_ww3] forecast2: launching /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast2_2020-07-06T033555.164945-0700/SoGWW3.sh on arbutus.cloud-nowcast 2020-07-06 03:35:55,188 INFO [run_ww3] forecast2 WaveWatch3 run for 2020-07-06 on arbutus.cloud-nowcast started 2020-07-06 03:35:55,195 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker 2020-07-06 03:35:55,892 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'forecast2'], host='arbutus.cloud-nowcast') 2020-07-06 03:35:57,827 INFO [watch_ww3] running in process 20091 2020-07-06 03:35:57,829 INFO [watch_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2020-07-06 03:35:57,830 INFO [watch_ww3] publishing log messages to tcp://*:5573 2020-07-06 03:35:57,831 INFO [watch_ww3] connected to skookum.eos.ubc.ca port 5555 2020-07-06 03:35:57,865 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch... 2020-07-06 03:36:32,363 INFO [download_results] forecast2 2020-07-05 results files from arbutus.cloud-nowcast downloaded 2020-07-06 03:36:32,371 INFO [manager] checklist updated with [results files] items from download_results worker 2020-07-06 03:36:33,049 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['forecast2', '--run-date', '2020-07-05'], host='localhost') 2020-07-06 03:36:36,630 INFO [make_CHS_currents_file] running in process 23186 2020-07-06 03:36:36,631 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-06 03:36:36,632 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5564 2020-07-06 03:36:36,633 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555 2020-07-06 03:38:59,704 INFO [make_CHS_currents_file] Made CHS currents file for 2020-07-05 for forecast2 2020-07-06 03:38:59,707 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker 2020-07-06 03:39:00,276 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['nemo', 'forecast2', '--run-date', '2020-07-05'], host='localhost') 2020-07-06 03:39:01,440 INFO [update_forecast_datasets] running in process 23236 2020-07-06 03:39:01,441 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-06 03:39:01,442 INFO [update_forecast_datasets] publishing log messages to tcp://*:5564 2020-07-06 03:39:01,443 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555 2020-07-06 03:39:01,444 INFO [update_forecast_datasets] updating nemo forecast directory for forecast2/05jul20 run 2020-07-06 03:40:57,965 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 93, 17.2% complete 2020-07-06 03:43:09,029 INFO [update_forecast_datasets] updated nemo forecast directory for forecast2/05jul20 run: /results/SalishSea/rolling-forecasts/nemo 2020-07-06 03:43:09,030 INFO [update_forecast_datasets] nemo 2020-07-05 forecast2 rolling forecast datasets updated 2020-07-06 03:43:09,033 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker 2020-07-06 03:43:09,559 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nemo-forecast'], host='localhost') 2020-07-06 03:43:09,562 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'forecast2', 'publish', '--run-date', '2020-07-05'], host='localhost') 2020-07-06 03:43:09,565 INFO [manager] launching NextWorker(module='nowcast.workers.make_surface_current_tiles', args=['forecast2', '--run-date', '2020-07-05'], host='localhost') 2020-07-06 03:43:10,508 INFO [ping_erddap] running in process 23469 2020-07-06 03:43:10,509 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-06 03:43:10,510 INFO [ping_erddap] publishing log messages to tcp://*:5564 2020-07-06 03:43:10,510 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555 2020-07-06 03:43:10,519 INFO [ping_erddap] nemo-forecast ERDDAP dataset flag file(s) created 2020-07-06 03:43:10,520 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker 2020-07-06 03:43:10,768 INFO [make_surface_current_tiles] running in process 23471 2020-07-06 03:43:10,768 INFO [make_surface_current_tiles] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-06 03:43:10,769 INFO [make_surface_current_tiles] publishing log messages to tcp://*:5567 2020-07-06 03:43:10,769 INFO [make_surface_current_tiles] connected to skookum.eos.ubc.ca port 5555 2020-07-06 03:43:10,815 INFO [make_surface_current_tiles] starting rendering of tiles for 2020-07-05 forecast2 in 6 processes into /results/nowcast-sys/figures/surface_currents/forecast2/05jul20 2020-07-06 03:43:16,754 INFO [make_plots] running in process 23470 2020-07-06 03:43:16,755 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-06 03:43:16,756 INFO [make_plots] publishing log messages to tcp://*:5564 2020-07-06 03:43:16,757 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555 2020-07-06 03:43:18,839 INFO [make_plots] preparing render list for 2020-07-05 NEMO forecast2 publish figures 2020-07-06 03:43:19,146 INFO [make_plots] added Website_thumbnail figure to 2020-07-05 NEMO forecast2 publish render list 2020-07-06 03:43:19,147 INFO [make_plots] added Threshold_website figure to 2020-07-05 NEMO forecast2 publish render list 2020-07-06 03:43:19,148 INFO [make_plots] added PA_tidal_predictions figure to 2020-07-05 NEMO forecast2 publish render list 2020-07-06 03:43:19,149 INFO [make_plots] added Boundary Bay figure to 2020-07-05 NEMO forecast2 publish render list 2020-07-06 03:43:19,149 INFO [make_plots] added Campbell River figure to 2020-07-05 NEMO forecast2 publish render list 2020-07-06 03:43:19,150 INFO [make_plots] added Cherry Point figure to 2020-07-05 NEMO forecast2 publish render list 2020-07-06 03:43:19,151 INFO [make_plots] added Friday Harbor figure to 2020-07-05 NEMO forecast2 publish render list 2020-07-06 03:43:19,152 INFO [make_plots] added Halfmoon Bay figure to 2020-07-05 NEMO forecast2 publish render list 2020-07-06 03:43:19,153 INFO [make_plots] added Nanaimo figure to 2020-07-05 NEMO forecast2 publish render list 2020-07-06 03:43:19,154 INFO [make_plots] added Neah Bay figure to 2020-07-05 NEMO forecast2 publish render list 2020-07-06 03:43:19,155 INFO [make_plots] added New Westminster figure to 2020-07-05 NEMO forecast2 publish render list 2020-07-06 03:43:19,155 INFO [make_plots] added Patricia Bay figure to 2020-07-05 NEMO forecast2 publish render list 2020-07-06 03:43:19,156 INFO [make_plots] added Point Atkinson figure to 2020-07-05 NEMO forecast2 publish render list 2020-07-06 03:43:19,157 INFO [make_plots] added Port Renfrew figure to 2020-07-05 NEMO forecast2 publish render list 2020-07-06 03:43:19,158 INFO [make_plots] added Sand Heads figure to 2020-07-05 NEMO forecast2 publish render list 2020-07-06 03:43:19,159 INFO [make_plots] added Sandy Cove figure to 2020-07-05 NEMO forecast2 publish render list 2020-07-06 03:43:19,159 INFO [make_plots] added Squamish figure to 2020-07-05 NEMO forecast2 publish render list 2020-07-06 03:43:19,160 INFO [make_plots] added Victoria figure to 2020-07-05 NEMO forecast2 publish render list 2020-07-06 03:43:19,161 INFO [make_plots] added Woodwards Landing figure to 2020-07-05 NEMO forecast2 publish render list 2020-07-06 03:43:19,162 INFO [make_plots] starting to render nemo forecast2 publish 05jul20 figures 2020-07-06 03:43:50,236 INFO [make_plots] Note: NumExpr detected 12 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8. 2020-07-06 03:43:50,237 INFO [make_plots] NumExpr defaulting to 8 threads. 2020-07-06 03:45:58,066 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 198, 36.7% complete 2020-07-06 03:50:58,167 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 306, 56.7% complete 2020-07-06 03:55:58,268 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 414, 76.7% complete 2020-07-06 04:00:58,369 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 516, 95.6% complete 2020-07-06 04:01:30,054 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 08074 Campbell River from 2020-07-01 00:00:00 to 2020-07-08 05:55:00 2020-07-06 04:05:25,251 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 9449424 Cherry Point from 2020-07-01 00:00:00 to 2020-07-08 05:55:00 2020-07-06 04:05:58,471 INFO [watch_ww3] forecast2 WWATCH3 run on arbutus.cloud-nowcast completed 2020-07-06 04:05:58,529 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker 2020-07-06 04:06:00,357 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2020-07-06'], host='localhost') 2020-07-06 04:06:03,012 INFO [download_wwatch3_results] running in process 25689 2020-07-06 04:06:03,013 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-06 04:06:03,013 INFO [download_wwatch3_results] publishing log messages to tcp://*:5568 2020-07-06 04:06:03,014 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555 2020-07-06 04:06:29,111 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 9449880 Friday Harbor from 2020-07-01 00:00:00 to 2020-07-08 05:55:00 2020-07-06 04:06:37,193 INFO [download_wwatch3_results] forecast2 2020-07-06 results files from arbutus.cloud-nowcast downloaded 2020-07-06 04:06:37,198 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker 2020-07-06 04:06:39,379 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['wwatch3', 'forecast2', '--run-date', '2020-07-06'], host='localhost') 2020-07-06 04:06:41,119 INFO [update_forecast_datasets] running in process 25731 2020-07-06 04:06:41,121 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-06 04:06:41,121 INFO [update_forecast_datasets] publishing log messages to tcp://*:5568 2020-07-06 04:06:41,122 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555 2020-07-06 04:06:41,123 INFO [update_forecast_datasets] updating wwatch3 most_recent_forecast directory from forecast2/06jul20 run 2020-07-06 04:06:41,125 INFO [update_forecast_datasets] updated wwatch3 most_recent_forecast directory from forecast2/06jul20 run 2020-07-06 04:06:41,126 INFO [update_forecast_datasets] updating wwatch3 forecast directory for forecast2/06jul20 run 2020-07-06 04:07:36,745 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07830 Halfmoon Bay from 2020-07-01 00:00:00 to 2020-07-08 05:55:00 2020-07-06 04:07:42,740 INFO [update_forecast_datasets] updated wwatch3 forecast directory for forecast2/06jul20 run: /results/SalishSea/rolling-forecasts/wwatch3 2020-07-06 04:07:42,741 INFO [update_forecast_datasets] wwatch3 2020-07-06 forecast2 rolling forecast datasets updated 2020-07-06 04:07:42,744 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker 2020-07-06 04:07:45,060 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['wwatch3-forecast'], host='localhost') 2020-07-06 04:07:47,822 INFO [ping_erddap] running in process 25876 2020-07-06 04:07:47,823 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-06 04:07:47,824 INFO [ping_erddap] publishing log messages to tcp://*:5568 2020-07-06 04:07:47,825 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555 2020-07-06 04:07:47,853 INFO [ping_erddap] wwatch3-forecast ERDDAP dataset flag file(s) created 2020-07-06 04:07:47,856 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker 2020-07-06 04:07:49,988 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['wwatch3', 'forecast2', 'publish', '--run-date', '2020-07-06'], host='localhost') 2020-07-06 04:07:58,891 INFO [make_plots] running in process 25882 2020-07-06 04:07:58,892 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-06 04:07:58,892 INFO [make_plots] publishing log messages to tcp://*:5568 2020-07-06 04:07:58,893 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555 2020-07-06 04:07:58,894 INFO [make_plots] preparing render list for 2020-07-06 WaveWatch3 forecast2 publish figures 2020-07-06 04:07:58,895 INFO [make_plots] added Halibut Bank figure to 2020-07-06 WaveWatch3 forecast2 publish render list 2020-07-06 04:07:58,896 INFO [make_plots] added Sentry Shoal figure to 2020-07-06 WaveWatch3 forecast2 publish render list 2020-07-06 04:07:58,896 INFO [make_plots] starting to render wwatch3 forecast2 publish 06jul20 figures 2020-07-06 04:08:45,041 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07917 Nanaimo from 2020-07-01 00:00:00 to 2020-07-08 05:55:00 2020-07-06 04:09:08,668 INFO [make_plots] Note: NumExpr detected 12 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8. 2020-07-06 04:09:08,669 INFO [make_plots] NumExpr defaulting to 8 threads. 2020-07-06 04:09:08,679 INFO [make_plots] retrieving last 45 days of real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46146 Halibut Bank 2020-07-06 04:10:20,889 INFO [make_plots] retrieving last 45 days of real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46131 Sentry Shoal 2020-07-06 04:10:23,671 INFO [make_plots] finished rendering wwatch3 forecast2 publish 06jul20 figures 2020-07-06 04:10:23,671 INFO [make_plots] wwatch3 publish plots for 2020-07-06 forecast2 completed 2020-07-06 04:10:23,680 INFO [manager] checklist updated with [plots] items from make_plots worker 2020-07-06 04:12:48,147 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 9443090 Neah Bay from 2020-07-01 00:00:00 to 2020-07-08 05:55:00 2020-07-06 04:13:02,481 INFO [make_surface_current_tiles] starting PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast2/05jul20 2020-07-06 04:13:13,405 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07654 New Westminster from 2020-07-01 00:00:00 to 2020-07-08 05:55:00 2020-07-06 04:15:25,412 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07277 Patricia Bay from 2020-07-01 00:00:00 to 2020-07-08 05:55:00 2020-07-06 04:17:30,595 INFO [make_surface_current_tiles] finished PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast2/05jul20 2020-07-06 04:17:30,607 INFO [make_surface_current_tiles] finished rendering of tiles for 2020-07-05 forecast2 into /results/nowcast-sys/figures/surface_currents/forecast2/05jul20 2020-07-06 04:17:30,608 INFO [make_surface_current_tiles] surface current tile figures for 2020-07-05 forecast2 completed 2020-07-06 04:17:31,103 INFO [manager] checklist updated with [surface current tiles] items from make_surface_current_tiles worker 2020-07-06 04:17:39,753 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07795 Point Atkinson from 2020-07-01 00:00:00 to 2020-07-08 05:55:00 2020-07-06 04:17:59,108 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 08525 Port Renfrew from 2020-07-01 00:00:00 to 2020-07-08 05:55:00 2020-07-06 04:20:10,646 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07594 Sand Heads from 2020-07-01 00:00:00 to 2020-07-08 05:55:00 2020-07-06 04:22:23,008 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07786 Sandy Cove from 2020-07-01 00:00:00 to 2020-07-08 05:55:00 2020-07-06 04:24:36,427 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07811 Squamish from 2020-07-01 00:00:00 to 2020-07-08 05:55:00 2020-07-06 04:24:55,938 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07120 Victoria from 2020-07-01 00:00:00 to 2020-07-08 05:55:00 2020-07-06 04:27:05,968 INFO [make_plots] retrieving obs water level data from https://ws-shc.qc.dfo-mpo.gc.ca/observations?WSDL for station 07610 Woodwards Landing from 2020-07-01 00:00:00 to 2020-07-08 05:55:00 2020-07-06 04:29:11,875 INFO [make_plots] finished rendering nemo forecast2 publish 05jul20 figures 2020-07-06 04:29:11,878 INFO [make_plots] nemo publish plots for 2020-07-05 forecast2 completed 2020-07-06 04:29:11,887 INFO [manager] checklist updated with [plots] items from make_plots worker 2020-07-06 04:29:12,394 INFO [manager] launching NextWorker(module='nowcast.workers.make_feeds', args=['forecast2', '--run-date', '2020-07-05'], host='localhost') 2020-07-06 04:29:16,297 INFO [make_feeds] running in process 26608 2020-07-06 04:29:16,299 INFO [make_feeds] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-06 04:29:16,300 INFO [make_feeds] publishing log messages to tcp://*:5564 2020-07-06 04:29:16,301 INFO [make_feeds] connected to skookum.eos.ubc.ca port 5555 2020-07-06 04:29:25,347 INFO [make_feeds] Note: NumExpr detected 12 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8. 2020-07-06 04:29:25,347 INFO [make_feeds] NumExpr defaulting to 8 threads. 2020-07-06 04:29:31,420 INFO [make_feeds] ATOM feeds for 2020-07-05 forecast2 run completed 2020-07-06 04:29:31,424 INFO [manager] checklist updated with [storm surge alert feeds] items from make_feeds worker 2020-07-06 04:29:31,982 INFO [manager] launching NextWorker(module='nemo_nowcast.workers.clear_checklist', args=[], host='localhost') 2020-07-06 04:29:32,944 INFO [clear_checklist] running in process 26620 2020-07-06 04:29:32,945 INFO [clear_checklist] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-06 04:29:32,946 INFO [clear_checklist] publishing log messages to tcp://*:5564 2020-07-06 04:29:32,947 INFO [clear_checklist] connected to skookum.eos.ubc.ca port 5555 2020-07-06 04:29:32,948 INFO [clear_checklist] requesting that manager clear system state checklist 2020-07-06 04:29:32,950 INFO [manager] writing checklist to log file 2020-07-06 04:29:33,051 INFO [manager] checklist cleared 2020-07-06 04:29:33,054 INFO [clear_checklist] nowcast system checklist cleared 2020-07-06 04:29:33,058 INFO [manager] launching NextWorker(module='nemo_nowcast.workers.rotate_logs', args=[], host='localhost') 2020-07-06 04:29:34,128 INFO [rotate_logs] running in process 26624 2020-07-06 04:29:34,130 INFO [rotate_logs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2020-07-06 04:29:34,131 INFO [rotate_logs] publishing log messages to tcp://*:5564 2020-07-06 04:29:34,131 INFO [rotate_logs] connected to skookum.eos.ubc.ca port 5555 2020-07-06 04:29:34,132 INFO [rotate_logs] rotating log files