2026-11-06 10:39:15,851 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/nowcast.log 2026-11-06 10:39:15,859 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/nowcast.debug.log 2026-11-06 10:39:15,864 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/checklist.log 2026-11-06 10:39:15,865 INFO [rotate_logs] log files rotated 2026-11-06 10:39:15,871 INFO [manager] checklist updated with [log rotation] items from rotate_logs worker 2026-11-06 10:40:30,169 INFO [make_surface_current_tiles] starting PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast2/05nov26 2026-11-06 10:45:14,780 INFO [make_surface_current_tiles] finished PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast2/05nov26 2026-11-06 10:45:14,797 INFO [make_surface_current_tiles] finished rendering of tiles for 2026-11-05 forecast2 into /results/nowcast-sys/figures/surface_currents/forecast2/05nov26 2026-11-06 10:45:14,798 INFO [make_surface_current_tiles] surface current tile figures for 2026-11-05 forecast2 completed 2026-11-06 10:45:15,061 INFO [manager] checklist updated with [surface current tiles] items from make_surface_current_tiles worker 2026-11-06 10:50:09,240 INFO [collect_weather] termination signal (SIGTERM) received; shutting down 2026-11-06 10:50:36,324 INFO [collect_weather] running in process 819130 2026-11-06 10:50:36,325 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2026-11-06 10:50:36,326 INFO [collect_weather] publishing log messages to tcp://*:5562 2026-11-06 10:50:36,327 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555 2026-11-06 10:50:36,345 INFO [collect_weather] starting to move 2026-11-06 files from /SalishSeaCast/datamart/hrdps-continental/12/ 2026-11-06 10:50:49,065 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/12/ to /results/forcing/atmospheric/continental2.5/GRIB/20261106/12/ 2026-11-06 10:50:49,066 INFO [collect_weather] 2026-11-06 2.5km weather forecast 12 collection complete 2026-11-06 10:50:49,070 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker 2026-11-06 10:50:49,367 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'SkagitMountVernon', '--data-date', '2026-11-05'], host='localhost') 2026-11-06 10:50:49,372 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'SnohomishMonroe', '--data-date', '2026-11-05'], host='localhost') 2026-11-06 10:50:49,378 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'NisquallyMcKenna', '--data-date', '2026-11-05'], host='localhost') 2026-11-06 10:50:49,383 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'GreenwaterGreenwater', '--data-date', '2026-11-05'], host='localhost') 2026-11-06 10:50:49,388 INFO [manager] launching NextWorker(module='nowcast.workers.make_turbidity_file', args=[], host='localhost') 2026-11-06 10:50:49,393 INFO [manager] launching NextWorker(module='nowcast.workers.collect_NeahBay_ssh', args=['06'], host='localhost') 2026-11-06 10:50:49,399 INFO [manager] launching NextWorker(module='nowcast.workers.download_live_ocean', args=[], host='localhost') 2026-11-06 10:50:49,404 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['18', '2.5km'], host='localhost') 2026-11-06 10:50:49,410 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['18'], host='localhost') 2026-11-06 10:50:50,941 INFO [collect_NeahBay_ssh] running in process 819191 2026-11-06 10:50:50,942 INFO [collect_NeahBay_ssh] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2026-11-06 10:50:50,944 INFO [collect_NeahBay_ssh] publishing log messages to tcp://*:5562 2026-11-06 10:50:50,945 INFO [collect_NeahBay_ssh] connected to skookum.eos.ubc.ca port 5555 2026-11-06 10:50:50,946 INFO [collect_NeahBay_ssh] collecting Neah Bay ssh 06Z obs/forecast for 2026-11-06 2026-11-06 10:50:51,007 INFO [collect_weather] running in process 819193 2026-11-06 10:50:51,008 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2026-11-06 10:50:51,009 INFO [collect_weather] publishing log messages to tcp://*:5563 2026-11-06 10:50:51,009 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555 2026-11-06 10:50:51,025 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/18/ 2026-11-06 10:50:51,155 INFO [download_live_ocean] running in process 819192 2026-11-06 10:50:51,155 INFO [download_live_ocean] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2026-11-06 10:50:51,157 INFO [download_live_ocean] publishing log messages to tcp://*:5564 2026-11-06 10:50:51,160 INFO [download_live_ocean] connected to skookum.eos.ubc.ca port 5555 2026-11-06 10:50:51,161 INFO [download_live_ocean] downloading Salish Sea western boundary day-averaged LiveOcean file for 2026-11-06 2026-11-06 10:50:51,599 INFO [collect_river_data] running in process 819188 2026-11-06 10:50:51,601 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2026-11-06 10:50:51,602 INFO [collect_river_data] publishing log messages to tcp://*:5565 2026-11-06 10:50:51,603 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2026-11-06 10:50:51,604 INFO [collect_river_data] Collecting USGS NisquallyMcKenna river data for 2026-11-05 2026-11-06 10:50:51,605 INFO [collect_river_data] running in process 819185 2026-11-06 10:50:51,606 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2026-11-06 10:50:51,607 INFO [collect_river_data] publishing log messages to tcp://*:5566 2026-11-06 10:50:51,608 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2026-11-06 10:50:51,609 INFO [collect_river_data] Collecting USGS SkagitMountVernon river data for 2026-11-05 2026-11-06 10:50:51,632 INFO [collect_river_data] running in process 819187 2026-11-06 10:50:51,633 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2026-11-06 10:50:51,637 INFO [collect_river_data] publishing log messages to tcp://*:5567 2026-11-06 10:50:51,638 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2026-11-06 10:50:51,647 INFO [collect_river_data] Collecting USGS SnohomishMonroe river data for 2026-11-05 2026-11-06 10:50:51,684 INFO [collect_river_data] running in process 819189 2026-11-06 10:50:51,690 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2026-11-06 10:50:51,698 INFO [collect_river_data] publishing log messages to tcp://*:5568 2026-11-06 10:50:51,706 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2026-11-06 10:50:51,715 INFO [collect_river_data] Collecting USGS GreenwaterGreenwater river data for 2026-11-05 2026-11-06 10:50:51,729 INFO [download_live_ocean] downloaded 10164162 bytes from https://s3.kopah.uw.edu/liveocean-share/f2026.11.06/ubc.nc to /results/forcing/LiveOcean/downloaded/20261106/low_passed_UBC.nc 2026-11-06 10:50:51,732 INFO [download_live_ocean] Deflating in up to 1 concurrent sub-processes 2026-11-06 10:50:51,743 INFO [make_turbidity_file] running in process 819190 2026-11-06 10:50:51,747 INFO [make_turbidity_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2026-11-06 10:50:51,749 INFO [make_turbidity_file] publishing log messages to tcp://*:5569 2026-11-06 10:50:51,751 INFO [make_turbidity_file] connected to skookum.eos.ubc.ca port 5555 2026-11-06 10:50:51,752 INFO [make_turbidity_file] Creating Fraser River turbidity forcing file for 2026-11-06 2026-11-06 10:50:52,246 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json¶meterCd=00060&sites=12097500&startDT=2026-11-05&endDT=2026-11-05 "HTTP/1.1 200 " 2026-11-06 10:50:52,249 INFO [crop_gribs] running in process 819194 2026-11-06 10:50:52,252 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2026-11-06 10:50:52,253 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json¶meterCd=00060&sites=12200500&startDT=2026-11-05&endDT=2026-11-05 "HTTP/1.1 200 " 2026-11-06 10:50:52,254 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json¶meterCd=00060&sites=12150800&startDT=2026-11-05&endDT=2026-11-05 "HTTP/1.1 200 " 2026-11-06 10:50:52,255 INFO [crop_gribs] publishing log messages to tcp://*:5570 2026-11-06 10:50:52,256 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555 2026-11-06 10:50:52,258 INFO [crop_gribs] cropping 2026-11-06 ECCC HRDPS 2.5km continental 18Z GRIB files to SalishSeaCast subdomain 2026-11-06 10:50:52,262 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20261106/18/ 2026-11-06 10:50:52,264 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json¶meterCd=00060&sites=12089500&startDT=2026-11-05&endDT=2026-11-05 "HTTP/1.1 200 " 2026-11-06 10:50:52,277 INFO [collect_river_data] Appended USGS GreenwaterGreenwater river average discharge for 2026-11-05 to: /results/forcing/rivers/observations/Greenwater_Greenwater_flow 2026-11-06 10:50:52,277 INFO [collect_river_data] USGS GreenwaterGreenwater river data collection for 2026-11-05 completed 2026-11-06 10:50:52,280 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2026-11-06 10:50:52,282 INFO [collect_river_data] Appended USGS SnohomishMonroe river average discharge for 2026-11-05 to: /results/forcing/rivers/observations/Snohomish_Monroe_flow 2026-11-06 10:50:52,283 INFO [collect_river_data] USGS SnohomishMonroe river data collection for 2026-11-05 completed 2026-11-06 10:50:52,285 INFO [collect_river_data] Appended USGS SkagitMountVernon river average discharge for 2026-11-05 to: /results/forcing/rivers/observations/Skagit_MountVernon_flow 2026-11-06 10:50:52,285 INFO [collect_river_data] USGS SkagitMountVernon river data collection for 2026-11-05 completed 2026-11-06 10:50:52,315 INFO [collect_river_data] Appended USGS NisquallyMcKenna river average discharge for 2026-11-05 to: /results/forcing/rivers/observations/Nisqually_McKenna_flow 2026-11-06 10:50:52,316 INFO [collect_river_data] USGS NisquallyMcKenna river data collection for 2026-11-05 completed 2026-11-06 10:50:52,428 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2026-11-06 10:50:52,501 INFO [collect_NeahBay_ssh] 2026-11-06T00:00:00-07:00 Neah Bay ssh 06Z obs/forecast data collection complete 2026-11-06 10:50:52,565 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2026-11-06 10:50:52,693 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2026-11-06 10:50:52,744 INFO [download_live_ocean] netCDF4 deflated /results/forcing/LiveOcean/downloaded/20261106/low_passed_UBC.nc 2026-11-06 10:50:52,745 INFO [download_live_ocean] 2026-11-06 Live Ocean file for Salish Sea western boundary downloaded 2026-11-06 10:50:52,874 INFO [manager] checklist updated with [Neah Bay ssh data] items from collect_NeahBay_ssh worker 2026-11-06 10:50:53,009 INFO [manager] launching NextWorker(module='nowcast.workers.make_ssh_files', args=['nowcast', '--run-date', '2026-11-06'], host='localhost') 2026-11-06 10:50:53,017 INFO [manager] checklist updated with [Live Ocean products] items from download_live_ocean worker 2026-11-06 10:50:53,335 INFO [manager] launching NextWorker(module='nowcast.workers.make_live_ocean_files', args=['--run-date', '2026-11-06'], host='localhost') 2026-11-06 10:50:54,566 INFO [make_turbidity_file] 2026-11-06 Fraser River turbidity file creation complete 2026-11-06 10:50:54,567 INFO [manager] checklist updated with [Fraser River turbidity] items from make_turbidity_file worker 2026-11-06 10:50:55,911 INFO [make_live_ocean_files] running in process 819337 2026-11-06 10:50:55,914 INFO [make_live_ocean_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2026-11-06 10:50:55,915 INFO [make_live_ocean_files] publishing log messages to tcp://*:5562 2026-11-06 10:50:55,916 INFO [make_live_ocean_files] connected to skookum.eos.ubc.ca port 5555 2026-11-06 10:50:55,916 INFO [make_live_ocean_files] Creating T&S western boundary conditions file from 2026-11-06 Live Ocean run 2026-11-06 10:50:55,992 INFO [make_ssh_files] running in process 819324 2026-11-06 10:50:55,993 INFO [make_ssh_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2026-11-06 10:50:55,994 INFO [make_ssh_files] publishing log messages to tcp://*:5564 2026-11-06 10:50:55,995 INFO [make_ssh_files] connected to skookum.eos.ubc.ca port 5555 2026-11-06 10:50:55,996 INFO [make_ssh_files] building nowcast sea surface height boundary conditions file(s) from 2026-11-06 NOAA Neah Bay 06Z observation and forecast values 2026-11-06 10:50:57,758 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2026m11d02.nc 2026-11-06 10:50:57,941 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2026m11d03.nc 2026-11-06 10:50:58,125 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2026m11d04.nc 2026-11-06 10:50:58,303 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2026m11d05.nc 2026-11-06 10:50:58,523 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2026m11d06.nc 2026-11-06 10:50:58,722 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2026m11d07.nc 2026-11-06 10:50:58,932 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2026m11d08.nc 2026-11-06 10:50:59,129 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2026m11d09.nc 2026-11-06 10:50:59,258 INFO [make_ssh_files] sea surface height boundary file for 2026-11-06 nowcast run created 2026-11-06 10:50:59,262 INFO [manager] checklist updated with [sea surface height forcing] items from make_ssh_files worker 2026-11-06 10:50:59,408 INFO [manager] launching NextWorker(module='nowcast.workers.make_runoff_file', args=['v202108'], host='localhost') 2026-11-06 10:51:01,494 INFO [make_runoff_file] running in process 819414 2026-11-06 10:51:01,495 INFO [make_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2026-11-06 10:51:01,496 INFO [make_runoff_file] publishing log messages to tcp://*:5564 2026-11-06 10:51:01,497 INFO [make_runoff_file] connected to skookum.eos.ubc.ca port 5555 2026-11-06 10:51:01,499 INFO [make_runoff_file] calculating NEMO runoff forcing for v202108 bathymetry for 2026-11-05 2026-11-06 10:51:04,450 INFO [make_runoff_file] stored NEMO runoff forcing for v202108 bathymetry for 2026-11-05: /results/forcing/rivers/R202108Dailies_y2026m11d05.nc 2026-11-06 10:51:04,451 INFO [make_runoff_file] 2026-11-05 runoff file creation completed 2026-11-06 10:51:04,454 INFO [manager] checklist updated with [rivers forcing] items from make_runoff_file worker 2026-11-06 10:51:33,067 INFO [make_live_ocean_files] Stored T&S western boundary conditions file: /results/forcing/LiveOcean/boundary_conditions/LiveOcean_v201905_y2026m11d06.nc 2026-11-06 10:51:33,067 INFO [make_live_ocean_files] 2026-11-06 Live Ocean western boundary conditions files created 2026-11-06 10:51:33,070 INFO [manager] checklist updated with [Live Ocean boundary conditions] items from make_live_ocean_files worker 2026-11-06 11:01:24,326 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20261106/12/ 2026-11-06 11:01:24,327 INFO [crop_gribs] 2026-11-06 12 GRIBs cropping complete 2026-11-06 11:01:24,330 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker 2026-11-06 11:01:24,608 INFO [manager] launching NextWorker(module='nowcast.workers.grib_to_netcdf', args=['nowcast+'], host='localhost') 2026-11-06 11:01:26,693 INFO [grib_to_netcdf] running in process 820249 2026-11-06 11:01:26,694 INFO [grib_to_netcdf] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2026-11-06 11:01:26,695 INFO [grib_to_netcdf] publishing log messages to tcp://*:5562 2026-11-06 11:01:26,697 INFO [grib_to_netcdf] connected to skookum.eos.ubc.ca port 5555 2026-11-06 11:01:26,698 INFO [grib_to_netcdf] creating NEMO-atmos forcing files for 2026-11-06 nowcast and forecast runs 2026-11-06 11:01:27,316 CRITICAL [grib_to_netcdf] unhandled exception: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/.pixi/envs/default/lib/python3.14/site-packages/nemo_nowcast/worker.py", line 388, in _do_work checklist = self.worker_func( self._parsed_args, self.config, self.tell_manager ) File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/grib_to_netcdf.py", line 123, in grib_to_netcdf nemo_ds_18 = _calc_nemo_ds( var_names, ...<5 lines>... run_date_offset=-1, ) File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/grib_to_netcdf.py", line 268, in _calc_nemo_ds nemo_datasets[nemo_var] = _calc_nemo_var_ds( ~~~~~~~~~~~~~~~~~^ msc_var, ^^^^^^^^ ...<6 lines>... config, ^^^^^^^ ) ^ File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/grib_to_netcdf.py", line 383, in _calc_nemo_var_ds grib_ds = xarray.open_mfdataset( grib_files, ...<4 lines>... backend_kwargs={"indexpath": ""}, ) File "/SalishSeaCast/SalishSeaNowcast/.pixi/envs/default/lib/python3.14/site-packages/xarray/backends/api.py", line 1788, in open_mfdataset ds = open_(p, **open_kwargs) File "/SalishSeaCast/SalishSeaNowcast/.pixi/envs/default/lib/python3.14/site-packages/xarray/backends/api.py", line 760, in open_dataset backend_ds = backend.open_dataset( filename_or_obj, ...<2 lines>... **kwargs, ) File "/SalishSeaCast/SalishSeaNowcast/.pixi/envs/default/lib/python3.14/site-packages/cfgrib/xarray_plugin.py", line 113, in open_dataset store = CfGribDataStore( filename_or_obj, ...<12 lines>... values_dtype=values_dtype, ) File "/SalishSeaCast/SalishSeaNowcast/.pixi/envs/default/lib/python3.14/site-packages/cfgrib/xarray_plugin.py", line 41, in __init__ self.ds = opener(filename, **backend_kwargs) ~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/SalishSeaNowcast/.pixi/envs/default/lib/python3.14/site-packages/cfgrib/dataset.py", line 828, in open_file index = open_fileindex(stream, indexpath, index_keys, ignore_keys=ignore_keys, filter_by_keys=filter_by_keys) File "/SalishSeaCast/SalishSeaNowcast/.pixi/envs/default/lib/python3.14/site-packages/cfgrib/dataset.py", line 807, in open_fileindex index = messages.FileIndex.from_indexpath_or_filestream( stream, index_keys, indexpath=indexpath, computed_keys=computed_keys ) File "/SalishSeaCast/SalishSeaNowcast/.pixi/envs/default/lib/python3.14/site-packages/cfgrib/messages.py", line 533, in from_indexpath_or_filestream return cls.from_fieldset(filestream, index_keys, computed_keys) ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/SalishSeaNowcast/.pixi/envs/default/lib/python3.14/site-packages/cfgrib/messages.py", line 379, in from_fieldset return cls.from_fieldset_and_iteritems(fieldset, iteritems, index_keys, computed_keys) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/SalishSeaNowcast/.pixi/envs/default/lib/python3.14/site-packages/cfgrib/messages.py", line 392, in from_fieldset_and_iteritems for field_id, raw_field in iteritems: ^^^^^^^^^ File "/SalishSeaCast/SalishSeaNowcast/.pixi/envs/default/lib/python3.14/site-packages/cfgrib/messages.py", line 292, in __iter__ for message in self.itervalues(): ~~~~~~~~~~~~~~~^^ File "/SalishSeaCast/SalishSeaNowcast/.pixi/envs/default/lib/python3.14/site-packages/cfgrib/messages.py", line 268, in itervalues with open(self.filestream.path, "rb") as file: ~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^ FileNotFoundError: [Errno 2] No such file or directory: '/results/forcing/atmospheric/continental2.5/GRIB/20261105/18/005/20261105T18Z_MSC_HRDPS_UGRD_AGL-10m_RLatLon0.0225_PT005H_SSC.grib2' 2026-11-06 11:01:27,774 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'nowcast+'], host='localhost') 2026-11-06 11:01:27,775 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.nibi', 'nowcast+'], host='localhost') 2026-11-06 11:01:27,776 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'nowcast+'], host='localhost') 2026-11-06 11:01:28,953 INFO [upload_forcing] running in process 820277 2026-11-06 11:01:28,955 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2026-11-06 11:01:28,956 INFO [upload_forcing] publishing log messages to tcp://*:5562 2026-11-06 11:01:28,957 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2026-11-06 11:01:28,965 INFO [upload_forcing] running in process 820275 2026-11-06 11:01:28,966 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2026-11-06 11:01:28,967 INFO [upload_forcing] publishing log messages to tcp://*:5564 2026-11-06 11:01:28,968 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2026-11-06 11:01:28,972 INFO [upload_forcing] running in process 820276 2026-11-06 11:01:28,973 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2026-11-06 11:01:28,973 INFO [upload_forcing] publishing log messages to tcp://*:5565 2026-11-06 11:01:28,974 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2026-11-06 11:01:29,653 CRITICAL [upload_forcing] unhandled exception: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/.pixi/envs/default/lib/python3.14/site-packages/nemo_nowcast/worker.py", line 388, in _do_work checklist = self.worker_func( self._parsed_args, self.config, self.tell_manager ) File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/upload_forcing.py", line 122, in upload_forcing _upload_weather(sftp_client, run_type, run_date, config, host_name, host_config) ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/upload_forcing.py", line 227, in _upload_weather ssh_sftp.upload_file(sftp_client, host_name, localpath, remotepath, logger) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/SalishSeaNowcast/nowcast/ssh_sftp.py", line 167, in upload_file sftp_client.put(os.fspath(localpath), os.fspath(remotepath)) ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/SalishSeaNowcast/.pixi/envs/default/lib/python3.14/site-packages/paramiko/sftp_client.py", line 757, in put file_size = os.stat(localpath).st_size ~~~~~~~^^^^^^^^^^^ FileNotFoundError: [Errno 2] No such file or directory: '/results/forcing/atmospheric/continental2.5/nemo_forcing/hrdps_y2026m11d06.nc' 2026-11-06 11:01:30,307 CRITICAL [upload_forcing] unhandled exception: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/.pixi/envs/default/lib/python3.14/site-packages/nemo_nowcast/worker.py", line 388, in _do_work checklist = self.worker_func( self._parsed_args, self.config, self.tell_manager ) File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/upload_forcing.py", line 122, in upload_forcing _upload_weather(sftp_client, run_type, run_date, config, host_name, host_config) ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/upload_forcing.py", line 227, in _upload_weather ssh_sftp.upload_file(sftp_client, host_name, localpath, remotepath, logger) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/SalishSeaNowcast/nowcast/ssh_sftp.py", line 167, in upload_file sftp_client.put(os.fspath(localpath), os.fspath(remotepath)) ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/SalishSeaNowcast/.pixi/envs/default/lib/python3.14/site-packages/paramiko/sftp_client.py", line 757, in put file_size = os.stat(localpath).st_size ~~~~~~~^^^^^^^^^^^ FileNotFoundError: [Errno 2] No such file or directory: '/results/forcing/atmospheric/continental2.5/nemo_forcing/hrdps_y2026m11d06.nc' 2026-11-06 11:01:32,171 CRITICAL [upload_forcing] unhandled exception: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/.pixi/envs/default/lib/python3.14/site-packages/nemo_nowcast/worker.py", line 388, in _do_work checklist = self.worker_func( self._parsed_args, self.config, self.tell_manager ) File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/upload_forcing.py", line 122, in upload_forcing _upload_weather(sftp_client, run_type, run_date, config, host_name, host_config) ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/upload_forcing.py", line 227, in _upload_weather ssh_sftp.upload_file(sftp_client, host_name, localpath, remotepath, logger) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/SalishSeaNowcast/nowcast/ssh_sftp.py", line 167, in upload_file sftp_client.put(os.fspath(localpath), os.fspath(remotepath)) ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/SalishSeaNowcast/.pixi/envs/default/lib/python3.14/site-packages/paramiko/sftp_client.py", line 757, in put file_size = os.stat(localpath).st_size ~~~~~~~^^^^^^^^^^^ FileNotFoundError: [Errno 2] No such file or directory: '/results/forcing/atmospheric/continental2.5/nemo_forcing/hrdps_y2026m11d06.nc' 2026-11-06 12:08:11,925 INFO [grib_to_netcdf] running in process 826577 2026-11-06 12:08:11,926 INFO [grib_to_netcdf] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2026-11-06 12:08:11,927 INFO [grib_to_netcdf] publishing log messages to tcp://*:5562 2026-11-06 12:08:11,928 INFO [grib_to_netcdf] connected to skookum.eos.ubc.ca port 5555 2026-11-06 12:08:11,929 INFO [grib_to_netcdf] creating NEMO-atmos forcing files for 2026-11-06 nowcast and forecast runs 2026-11-06 12:08:25,878 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/hrdps_y2026m11d06.nc 2026-11-06 12:08:36,178 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2026m11d07.nc 2026-11-06 12:08:42,177 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2026m11d08.nc 2026-11-06 12:08:42,191 INFO [grib_to_netcdf] 2026-11-06 NEMO-atmos forcing file for nowcast+ created 2026-11-06 12:08:42,194 INFO [manager] checklist updated with [weather forcing] items from grib_to_netcdf worker 2026-11-06 12:08:42,342 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['weather'], host='localhost') 2026-11-06 12:08:43,553 INFO [ping_erddap] running in process 826724 2026-11-06 12:08:43,554 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2026-11-06 12:08:43,556 INFO [ping_erddap] publishing log messages to tcp://*:5562 2026-11-06 12:08:43,557 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555 2026-11-06 12:08:43,558 INFO [ping_erddap] weather ERDDAP dataset flag file(s) created 2026-11-06 12:08:43,560 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker 2026-11-06 12:09:12,546 INFO [upload_forcing] running in process 826847 2026-11-06 12:09:12,547 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2026-11-06 12:09:12,549 INFO [upload_forcing] publishing log messages to tcp://*:5562 2026-11-06 12:09:12,550 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2026-11-06 12:09:15,522 INFO [upload_forcing] nowcast+ 2026-11-06 forcing files upload to arbutus.cloud-nowcast completed 2026-11-06 12:09:15,525 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2026-11-06 12:09:15,670 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'nowcast+', '--run-date', '2026-11-06'], host='localhost') 2026-11-06 12:09:16,950 INFO [make_forcing_links] running in process 826883 2026-11-06 12:09:16,951 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2026-11-06 12:09:16,952 INFO [make_forcing_links] publishing log messages to tcp://*:5562 2026-11-06 12:09:16,952 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555 2026-11-06 12:09:17,478 INFO [make_forcing_links] nowcast+ 2026-11-06 forcing file links on arbutus.cloud-nowcast created 2026-11-06 12:09:17,482 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker 2026-11-06 12:09:17,627 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2026-11-06'], host='arbutus.cloud-nowcast') 2026-11-06 12:09:18,836 INFO [run_NEMO] running in process 224309 2026-11-06 12:09:18,837 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2026-11-06 12:09:18,838 INFO [run_NEMO] publishing log messages to tcp://*:5556 2026-11-06 12:09:18,899 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555 2026-11-06 12:09:21,010 INFO [run_NEMO] nowcast: launching /nemoShare/MEOPAR/nowcast-sys/runs/06nov26nowcast_2026-11-06T120918.905016-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast 2026-11-06 12:09:21,048 INFO [run_NEMO] nowcast NEMO run for 2026-11-06 on arbutus.cloud-nowcast started 2026-11-06 12:09:21,053 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker 2026-11-06 12:09:21,197 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'nowcast'], host='arbutus.cloud-nowcast') 2026-11-06 12:09:22,024 INFO [watch_NEMO] running in process 224452 2026-11-06 12:09:22,027 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2026-11-06 12:09:22,028 INFO [watch_NEMO] publishing log messages to tcp://*:5557 2026-11-06 12:09:22,029 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555 2026-11-06 12:09:22,122 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: time.step not found; continuing to watch... 2026-11-06 12:09:24,741 INFO [upload_forcing] running in process 826993 2026-11-06 12:09:24,742 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2026-11-06 12:09:24,744 INFO [upload_forcing] publishing log messages to tcp://*:5562 2026-11-06 12:09:24,745 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2026-11-06 12:09:26,364 INFO [upload_forcing] nowcast+ 2026-11-06 forcing files upload to optimum-hindcast completed 2026-11-06 12:09:26,367 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2026-11-06 12:09:36,056 INFO [upload_forcing] running in process 827123 2026-11-06 12:09:36,057 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2026-11-06 12:09:36,058 INFO [upload_forcing] publishing log messages to tcp://*:5562 2026-11-06 12:09:36,060 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2026-11-06 12:09:45,854 INFO [upload_forcing] nowcast+ 2026-11-06 forcing files upload to robot.nibi completed 2026-11-06 12:09:45,858 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2026-11-06 12:14:22,125 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 19609556 = 2026-11-06 11:56:40 UTC, 49.8% complete 2026-11-06 12:19:22,127 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 19610640 = 2026-11-06 23:59:20 UTC, 100.0% complete 2026-11-06 12:24:22,133 INFO [watch_NEMO] nowcast NEMO run on arbutus.cloud-nowcast completed 2026-11-06 12:24:22,193 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker 2026-11-06 12:24:22,689 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'ssh', '--run-date', '2026-11-06'], host='localhost') 2026-11-06 12:24:22,699 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2026-11-06'], host='localhost') 2026-11-06 12:24:26,708 INFO [download_results] running in process 827528 2026-11-06 12:24:26,709 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2026-11-06 12:24:26,711 INFO [download_results] publishing log messages to tcp://*:5562 2026-11-06 12:24:26,712 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2026-11-06 12:24:26,713 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/nowcast/06nov26 to /results/SalishSea/nowcast-blue.202111 2026-11-06 12:24:26,987 INFO [make_forcing_links] running in process 827527 2026-11-06 12:24:27,004 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2026-11-06 12:24:27,010 INFO [make_forcing_links] publishing log messages to tcp://*:5564 2026-11-06 12:24:27,014 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555 2026-11-06 12:24:28,127 INFO [make_forcing_links] ssh 2026-11-06 forcing file links on arbutus.cloud-nowcast created 2026-11-06 12:24:28,172 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker 2026-11-06 12:24:28,315 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2026-11-06'], host='arbutus.cloud-nowcast') 2026-11-06 12:24:29,341 INFO [run_NEMO] running in process 224837 2026-11-06 12:24:29,367 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2026-11-06 12:24:29,388 INFO [run_NEMO] publishing log messages to tcp://*:5556 2026-11-06 12:24:29,403 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555 2026-11-06 12:24:31,547 INFO [run_NEMO] forecast: launching /nemoShare/MEOPAR/nowcast-sys/runs/06nov26forecast_2026-11-06T122429.437913-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast 2026-11-06 12:24:31,637 INFO [run_NEMO] forecast NEMO run for 2026-11-06 on arbutus.cloud-nowcast started 2026-11-06 12:24:31,713 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker 2026-11-06 12:24:31,807 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'forecast'], host='arbutus.cloud-nowcast') 2026-11-06 12:24:32,556 INFO [watch_NEMO] running in process 224980 2026-11-06 12:24:32,557 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2026-11-06 12:24:32,559 INFO [watch_NEMO] publishing log messages to tcp://*:5557 2026-11-06 12:24:32,560 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555 2026-11-06 12:24:32,663 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: time.step not found; continuing to watch... 2026-11-06 12:25:20,781 INFO [download_results] nowcast 2026-11-06 results files from arbutus.cloud-nowcast downloaded 2026-11-06 12:25:20,808 INFO [manager] checklist updated with [results files] items from download_results worker 2026-11-06 12:25:20,957 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast', 'research', '--run-date', '2026-11-06'], host='localhost') 2026-11-06 12:25:20,990 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast', 'comparison', '--run-date', '2026-11-05'], host='localhost') 2026-11-06 12:25:21,026 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['nowcast', '--run-date', '2026-11-06'], host='localhost') 2026-11-06 12:25:28,173 INFO [make_CHS_currents_file] running in process 827567 2026-11-06 12:25:28,174 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2026-11-06 12:25:28,175 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5562 2026-11-06 12:25:28,180 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555 2026-11-06 12:25:32,390 INFO [make_plots] running in process 827566 2026-11-06 12:25:32,390 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2026-11-06 12:25:32,392 INFO [make_plots] publishing log messages to tcp://*:5564 2026-11-06 12:25:32,393 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555 2026-11-06 12:25:32,407 INFO [make_plots] running in process 827564 2026-11-06 12:25:32,435 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2026-11-06 12:25:32,463 INFO [make_plots] publishing log messages to tcp://*:5565 2026-11-06 12:25:32,486 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555 2026-11-06 12:25:33,182 INFO [make_plots] preparing render list for 2026-11-06 NEMO nowcast-blue research figures 2026-11-06 12:25:33,198 INFO [make_plots] preparing render list for 2026-11-05 NEMO nowcast-blue comparison figures 2026-11-06 12:25:33,312 INFO [make_plots] added SH_wind figure to 2026-11-05 NEMO nowcast-blue comparison render list 2026-11-06 12:25:33,313 INFO [make_plots] added Compare_VENUS_East figure to 2026-11-05 NEMO nowcast-blue comparison render list 2026-11-06 12:25:33,314 INFO [make_plots] added Compare_VENUS_Central figure to 2026-11-05 NEMO nowcast-blue comparison render list 2026-11-06 12:25:33,315 INFO [make_plots] starting to render nemo nowcast comparison 05nov26 figures 2026-11-06 12:25:34,645 INFO [make_plots] added salinity_thalweg_and_surface figures to 2026-11-06 NEMO nowcast-blue research render list 2026-11-06 12:25:35,832 INFO [make_plots] added temperature_thalweg_and_surface figures to 2026-11-06 NEMO nowcast-blue research render list 2026-11-06 12:25:35,851 INFO [make_plots] added Currents_sections_and_surface figure to 2026-11-06 NEMO nowcast-blue research render list 2026-11-06 12:25:35,853 INFO [make_plots] added Currents_at_VENUS_Central figure to 2026-11-06 NEMO nowcast-blue research render list 2026-11-06 12:25:35,854 INFO [make_plots] added Currents_at_VENUS_East figure to 2026-11-06 NEMO nowcast-blue research render list 2026-11-06 12:25:35,856 INFO [make_plots] starting to render nemo nowcast research 06nov26 figures 2026-11-06 12:26:09,073 INFO [make_CHS_currents_file] Made CHS currents file for 2026-11-06 for nowcast 2026-11-06 12:26:09,075 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker 2026-11-06 12:26:48,779 INFO [make_plots] finished rendering nemo nowcast research 06nov26 figures 2026-11-06 12:26:48,782 INFO [make_plots] nemo research plots for 2026-11-06 nowcast completed 2026-11-06 12:26:48,809 INFO [manager] checklist updated with [plots] items from make_plots worker 2026-11-06 12:27:12,883 INFO [make_plots] finished rendering nemo nowcast comparison 05nov26 figures 2026-11-06 12:27:12,886 INFO [make_plots] nemo comparison plots for 2026-11-05 nowcast completed 2026-11-06 12:27:12,889 INFO [manager] checklist updated with [plots] items from make_plots worker 2026-11-06 12:29:32,654 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 19611675 = 2026-11-07 11:29:20 UTC, 31.9% complete 2026-11-06 12:34:32,657 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 19612814 = 2026-11-08 00:08:40 UTC, 67.1% complete 2026-11-06 12:39:32,657 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 19613880 = 2026-11-08 11:59:20 UTC, 100.0% complete 2026-11-06 12:44:32,663 INFO [watch_NEMO] forecast NEMO run on arbutus.cloud-nowcast completed 2026-11-06 12:44:32,715 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker 2026-11-06 12:44:33,053 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_wind_file', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2026-11-06'], host='arbutus.cloud-nowcast') 2026-11-06 12:44:33,063 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_current_file', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2026-11-06'], host='arbutus.cloud-nowcast') 2026-11-06 12:44:33,067 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'turbidity', '--run-date', '2026-11-06'], host='localhost') 2026-11-06 12:44:33,076 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.nibi', 'turbidity', '--run-date', '2026-11-06'], host='localhost') 2026-11-06 12:44:33,081 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'turbidity', '--run-date', '2026-11-06'], host='localhost') 2026-11-06 12:44:33,085 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2026-11-06'], host='localhost') 2026-11-06 12:44:34,309 INFO [make_ww3_current_file] running in process 225298 2026-11-06 12:44:34,318 INFO [make_ww3_current_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2026-11-06 12:44:34,336 INFO [make_ww3_current_file] publishing log messages to tcp://*:5571 2026-11-06 12:44:34,349 INFO [make_ww3_current_file] connected to skookum.eos.ubc.ca port 5555 2026-11-06 12:44:34,349 INFO [make_ww3_current_file] Creating wwatch3 currents forcing file for 2026-11-06 forecast run 2026-11-06 12:44:34,354 INFO [make_ww3_wind_file] running in process 225299 2026-11-06 12:44:34,355 INFO [make_ww3_wind_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2026-11-06 12:44:34,356 INFO [make_ww3_wind_file] publishing log messages to tcp://*:5570 2026-11-06 12:44:34,357 INFO [make_ww3_wind_file] connected to skookum.eos.ubc.ca port 5555 2026-11-06 12:44:34,358 INFO [make_ww3_wind_file] Creating wwatch3 wind forcing file for 2026-11-06 forecast run 2026-11-06 12:44:36,459 INFO [download_results] running in process 828142 2026-11-06 12:44:36,460 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2026-11-06 12:44:36,461 INFO [download_results] publishing log messages to tcp://*:5562 2026-11-06 12:44:36,462 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2026-11-06 12:44:36,463 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/forecast/06nov26 to /results/SalishSea/forecast.202111 2026-11-06 12:44:36,739 INFO [upload_forcing] running in process 828141 2026-11-06 12:44:36,741 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2026-11-06 12:44:36,742 INFO [upload_forcing] publishing log messages to tcp://*:5564 2026-11-06 12:44:36,744 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2026-11-06 12:44:36,753 INFO [upload_forcing] running in process 828139 2026-11-06 12:44:36,754 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2026-11-06 12:44:36,779 INFO [upload_forcing] publishing log messages to tcp://*:5565 2026-11-06 12:44:36,800 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2026-11-06 12:44:36,820 INFO [upload_forcing] running in process 828140 2026-11-06 12:44:36,842 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2026-11-06 12:44:36,847 INFO [upload_forcing] publishing log messages to tcp://*:5566 2026-11-06 12:44:36,849 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2026-11-06 12:44:38,678 INFO [upload_forcing] turbidity 2026-11-06 forcing files upload to optimum-hindcast completed 2026-11-06 12:44:38,681 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2026-11-06 12:44:38,995 INFO [upload_forcing] turbidity 2026-11-06 forcing files upload to arbutus.cloud-nowcast completed 2026-11-06 12:44:39,000 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2026-11-06 12:44:39,194 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2026-11-06'], host='localhost') 2026-11-06 12:44:40,633 INFO [upload_forcing] turbidity 2026-11-06 forcing files upload to robot.nibi completed 2026-11-06 12:44:40,696 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2026-11-06 12:44:42,116 INFO [make_ww3_wind_file] wwatch3 wind forcing file created on arbutus.cloud-nowcast for 2026-11-06 forecast run 2026-11-06 12:44:42,151 INFO [manager] checklist updated with [WW3 wind forcing] items from make_ww3_wind_file worker 2026-11-06 12:44:44,478 INFO [make_forcing_links] running in process 828171 2026-11-06 12:44:44,481 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2026-11-06 12:44:44,482 INFO [make_forcing_links] publishing log messages to tcp://*:5564 2026-11-06 12:44:44,484 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555 2026-11-06 12:44:45,309 INFO [make_forcing_links] nowcast-green 2026-11-06 forcing file links on arbutus.cloud-nowcast created 2026-11-06 12:44:45,349 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker 2026-11-06 12:44:45,492 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2026-11-06'], host='arbutus.cloud-nowcast') 2026-11-06 12:44:46,491 INFO [run_NEMO] running in process 225772 2026-11-06 12:44:46,492 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2026-11-06 12:44:46,494 INFO [run_NEMO] publishing log messages to tcp://*:5556 2026-11-06 12:44:46,496 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555 2026-11-06 12:44:48,570 INFO [run_NEMO] nowcast-green: launching /nemoShare/MEOPAR/nowcast-sys/runs/06nov26nowcast-green_2026-11-06T124446.541449-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast 2026-11-06 12:44:48,639 INFO [run_NEMO] nowcast-green NEMO run for 2026-11-06 on arbutus.cloud-nowcast started 2026-11-06 12:44:48,684 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker 2026-11-06 12:44:48,839 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'nowcast-green'], host='arbutus.cloud-nowcast') 2026-11-06 12:44:49,766 INFO [watch_NEMO] running in process 225915 2026-11-06 12:44:49,836 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2026-11-06 12:44:49,880 INFO [watch_NEMO] publishing log messages to tcp://*:5557 2026-11-06 12:44:49,899 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555 2026-11-06 12:44:49,922 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: time.step not found; continuing to watch... 2026-11-06 12:45:57,540 INFO [download_results] forecast 2026-11-06 results files from arbutus.cloud-nowcast downloaded 2026-11-06 12:45:57,580 INFO [manager] checklist updated with [results files] items from download_results worker 2026-11-06 12:45:57,813 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['forecast', '--run-date', '2026-11-06'], host='localhost') 2026-11-06 12:46:05,655 INFO [make_CHS_currents_file] running in process 828209 2026-11-06 12:46:05,656 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2026-11-06 12:46:05,703 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5562 2026-11-06 12:46:05,730 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555 2026-11-06 12:46:43,786 INFO [make_ww3_current_file] wwatch3 currents forcing file created on arbutus.cloud-nowcast for 2026-11-06 forecast run 2026-11-06 12:46:43,818 INFO [manager] checklist updated with [WW3 currents forcing] items from make_ww3_current_file worker 2026-11-06 12:46:44,002 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2026-11-06'], host='arbutus.cloud-nowcast') 2026-11-06 12:46:44,815 INFO [run_ww3] running in process 225950 2026-11-06 12:46:44,816 INFO [run_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2026-11-06 12:46:44,817 INFO [run_ww3] publishing log messages to tcp://*:5572 2026-11-06 12:46:44,817 INFO [run_ww3] connected to skookum.eos.ubc.ca port 5555 2026-11-06 12:46:44,820 INFO [run_ww3] Created run directory /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/nowcast_2026-11-06T124644.812411-0700 2026-11-06 12:46:44,821 INFO [run_ww3] nowcast: launching /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/nowcast_2026-11-06T124644.812411-0700/SoGWW3.sh on arbutus.cloud-nowcast 2026-11-06 12:46:44,861 INFO [run_ww3] nowcast WaveWatch3 run for 2026-11-06 on arbutus.cloud-nowcast started 2026-11-06 12:46:44,865 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker 2026-11-06 12:46:45,045 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'nowcast'], host='arbutus.cloud-nowcast') 2026-11-06 12:46:45,846 INFO [watch_ww3] running in process 225967 2026-11-06 12:46:45,847 INFO [watch_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2026-11-06 12:46:45,848 INFO [watch_ww3] publishing log messages to tcp://*:5573 2026-11-06 12:46:45,848 INFO [watch_ww3] connected to skookum.eos.ubc.ca port 5555 2026-11-06 12:46:45,919 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch... 2026-11-06 12:47:04,381 INFO [make_CHS_currents_file] Made CHS currents file for 2026-11-06 for forecast 2026-11-06 12:47:04,405 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker 2026-11-06 12:47:04,632 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['nemo', 'forecast', '--run-date', '2026-11-06'], host='localhost') 2026-11-06 12:47:07,124 INFO [update_forecast_datasets] running in process 828248 2026-11-06 12:47:07,125 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2026-11-06 12:47:07,126 INFO [update_forecast_datasets] publishing log messages to tcp://*:5562 2026-11-06 12:47:07,128 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555 2026-11-06 12:47:07,129 INFO [update_forecast_datasets] updating nemo forecast directory for forecast/06nov26 run 2026-11-06 12:47:07,577 INFO [update_forecast_datasets] updated nemo forecast directory for forecast/06nov26 run: /results/SalishSea/rolling-forecasts/nemo 2026-11-06 12:47:07,578 INFO [update_forecast_datasets] nemo 2026-11-06 forecast rolling forecast datasets updated 2026-11-06 12:47:07,581 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker 2026-11-06 12:47:07,765 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nemo-forecast'], host='localhost') 2026-11-06 12:47:07,795 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'forecast', 'publish', '--run-date', '2026-11-06'], host='localhost') 2026-11-06 12:47:07,802 INFO [manager] launching NextWorker(module='nowcast.workers.make_surface_current_tiles', args=['forecast', '--run-date', '2026-11-06'], host='localhost') 2026-11-06 12:47:09,959 INFO [ping_erddap] running in process 828254 2026-11-06 12:47:09,978 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2026-11-06 12:47:09,979 INFO [ping_erddap] publishing log messages to tcp://*:5562 2026-11-06 12:47:09,980 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555 2026-11-06 12:47:09,993 INFO [ping_erddap] nemo-forecast ERDDAP dataset flag file(s) created 2026-11-06 12:47:09,994 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker 2026-11-06 12:47:13,184 INFO [make_surface_current_tiles] running in process 828257 2026-11-06 12:47:13,185 INFO [make_surface_current_tiles] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2026-11-06 12:47:13,186 INFO [make_surface_current_tiles] publishing log messages to tcp://*:5562 2026-11-06 12:47:13,187 INFO [make_surface_current_tiles] connected to skookum.eos.ubc.ca port 5555 2026-11-06 12:47:13,190 INFO [make_surface_current_tiles] starting rendering of tiles for 2026-11-06 forecast in 6 processes into /results/nowcast-sys/figures/surface_currents/forecast/06nov26 2026-11-06 12:47:17,544 INFO [make_plots] running in process 828256 2026-11-06 12:47:17,545 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2026-11-06 12:47:17,567 INFO [make_plots] publishing log messages to tcp://*:5564 2026-11-06 12:47:17,579 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555 2026-11-06 12:47:18,181 INFO [make_plots] preparing render list for 2026-11-06 NEMO forecast publish figures 2026-11-06 12:47:18,280 INFO [make_plots] added Website_thumbnail figure to 2026-11-06 NEMO forecast publish render list 2026-11-06 12:47:18,300 INFO [make_plots] added Threshold_website figure to 2026-11-06 NEMO forecast publish render list 2026-11-06 12:47:18,318 INFO [make_plots] added PA_tidal_predictions figure to 2026-11-06 NEMO forecast publish render list 2026-11-06 12:47:18,336 INFO [make_plots] added Boundary Bay figure to 2026-11-06 NEMO forecast publish render list 2026-11-06 12:47:18,342 INFO [make_plots] added Campbell River figure to 2026-11-06 NEMO forecast publish render list 2026-11-06 12:47:18,344 INFO [make_plots] added Cherry Point figure to 2026-11-06 NEMO forecast publish render list 2026-11-06 12:47:18,345 INFO [make_plots] added Friday Harbor figure to 2026-11-06 NEMO forecast publish render list 2026-11-06 12:47:18,350 INFO [make_plots] added Halfmoon Bay figure to 2026-11-06 NEMO forecast publish render list 2026-11-06 12:47:18,352 INFO [make_plots] added Nanaimo figure to 2026-11-06 NEMO forecast publish render list 2026-11-06 12:47:18,352 INFO [make_plots] added Neah Bay figure to 2026-11-06 NEMO forecast publish render list 2026-11-06 12:47:18,354 INFO [make_plots] added New Westminster figure to 2026-11-06 NEMO forecast publish render list 2026-11-06 12:47:18,354 INFO [make_plots] added Patricia Bay figure to 2026-11-06 NEMO forecast publish render list 2026-11-06 12:47:18,355 INFO [make_plots] added Point Atkinson figure to 2026-11-06 NEMO forecast publish render list 2026-11-06 12:47:18,355 INFO [make_plots] added Port Renfrew figure to 2026-11-06 NEMO forecast publish render list 2026-11-06 12:47:18,356 INFO [make_plots] added Sand Heads figure to 2026-11-06 NEMO forecast publish render list 2026-11-06 12:47:18,357 INFO [make_plots] added Sandy Cove figure to 2026-11-06 NEMO forecast publish render list 2026-11-06 12:47:18,358 INFO [make_plots] added Squamish figure to 2026-11-06 NEMO forecast publish render list 2026-11-06 12:47:18,359 INFO [make_plots] added Victoria figure to 2026-11-06 NEMO forecast publish render list 2026-11-06 12:47:18,359 INFO [make_plots] added Woodwards Landing figure to 2026-11-06 NEMO forecast publish render list 2026-11-06 12:47:18,360 INFO [make_plots] starting to render nemo forecast publish 06nov26 figures 2026-11-06 12:49:49,929 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19608920 = 2026-11-06 04:52:40 UTC, 20.3% complete 2026-11-06 12:51:45,906 INFO [watch_ww3] nowcast WWATCH3 run on arbutus.cloud-nowcast completed 2026-11-06 12:51:45,911 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker 2026-11-06 12:51:46,303 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2026-11-06'], host='localhost') 2026-11-06 12:51:46,311 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2026-11-06'], host='arbutus.cloud-nowcast') 2026-11-06 12:51:47,185 INFO [run_ww3] running in process 226146 2026-11-06 12:51:47,195 INFO [run_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2026-11-06 12:51:47,211 INFO [run_ww3] publishing log messages to tcp://*:5572 2026-11-06 12:51:47,213 INFO [run_ww3] connected to skookum.eos.ubc.ca port 5555 2026-11-06 12:51:47,216 INFO [run_ww3] Created run directory /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast_2026-11-06T125147.181467-0700 2026-11-06 12:51:47,217 INFO [run_ww3] forecast: launching /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast_2026-11-06T125147.181467-0700/SoGWW3.sh on arbutus.cloud-nowcast 2026-11-06 12:51:47,220 INFO [run_ww3] forecast WaveWatch3 run for 2026-11-06 on arbutus.cloud-nowcast started 2026-11-06 12:51:47,225 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker 2026-11-06 12:51:47,440 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'forecast'], host='arbutus.cloud-nowcast') 2026-11-06 12:51:48,286 INFO [watch_ww3] running in process 226163 2026-11-06 12:51:48,288 INFO [watch_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2026-11-06 12:51:48,289 INFO [watch_ww3] publishing log messages to tcp://*:5573 2026-11-06 12:51:48,292 INFO [watch_ww3] connected to skookum.eos.ubc.ca port 5555 2026-11-06 12:51:48,333 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch... 2026-11-06 12:51:49,468 INFO [download_wwatch3_results] running in process 828951 2026-11-06 12:51:49,470 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2026-11-06 12:51:49,470 INFO [download_wwatch3_results] publishing log messages to tcp://*:5565 2026-11-06 12:51:49,477 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555 2026-11-06 12:52:04,182 INFO [download_wwatch3_results] nowcast 2026-11-06 results files from arbutus.cloud-nowcast downloaded 2026-11-06 12:52:04,186 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker 2026-11-06 12:54:49,930 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19609398 = 2026-11-06 10:11:20 UTC, 42.5% complete 2026-11-06 12:56:48,333 INFO [watch_ww3] forecast WWATCH3 run on arbutus.cloud-nowcast completed 2026-11-06 12:56:48,338 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker 2026-11-06 12:56:48,902 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2026-11-06'], host='localhost') 2026-11-06 12:56:51,854 INFO [download_wwatch3_results] running in process 829799 2026-11-06 12:56:51,856 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2026-11-06 12:56:51,857 INFO [download_wwatch3_results] publishing log messages to tcp://*:5565 2026-11-06 12:56:51,859 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555 2026-11-06 12:57:05,740 INFO [download_wwatch3_results] forecast 2026-11-06 results files from arbutus.cloud-nowcast downloaded 2026-11-06 12:57:05,742 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker 2026-11-06 12:57:06,030 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['wwatch3', 'forecast', '--run-date', '2026-11-06'], host='localhost') 2026-11-06 12:57:08,764 INFO [update_forecast_datasets] running in process 829869 2026-11-06 12:57:08,793 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2026-11-06 12:57:08,803 INFO [update_forecast_datasets] publishing log messages to tcp://*:5565 2026-11-06 12:57:08,807 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555 2026-11-06 12:57:08,808 INFO [update_forecast_datasets] updating wwatch3 most_recent_forecast directory from forecast/06nov26 run 2026-11-06 12:57:08,809 INFO [update_forecast_datasets] updated wwatch3 most_recent_forecast directory from forecast/06nov26 run 2026-11-06 12:57:08,810 INFO [update_forecast_datasets] updating wwatch3 forecast directory for forecast/06nov26 run 2026-11-06 12:57:08,854 INFO [update_forecast_datasets] updated wwatch3 forecast directory for forecast/06nov26 run: /results/SalishSea/rolling-forecasts/wwatch3 2026-11-06 12:57:08,888 INFO [update_forecast_datasets] wwatch3 2026-11-06 forecast rolling forecast datasets updated 2026-11-06 12:57:08,918 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker 2026-11-06 12:57:09,055 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['wwatch3-forecast'], host='localhost') 2026-11-06 12:57:11,464 INFO [ping_erddap] running in process 829889 2026-11-06 12:57:11,464 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2026-11-06 12:57:11,466 INFO [ping_erddap] publishing log messages to tcp://*:5565 2026-11-06 12:57:11,467 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555 2026-11-06 12:57:11,469 INFO [ping_erddap] wwatch3-forecast ERDDAP dataset flag file(s) created 2026-11-06 12:57:11,471 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker 2026-11-06 12:57:11,758 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['wwatch3', 'forecast', 'publish', '--run-date', '2026-11-06'], host='localhost') 2026-11-06 12:57:23,610 INFO [make_plots] running in process 829901 2026-11-06 12:57:23,611 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2026-11-06 12:57:23,629 INFO [make_plots] publishing log messages to tcp://*:5565 2026-11-06 12:57:23,644 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555 2026-11-06 12:57:23,645 INFO [make_plots] preparing render list for 2026-11-06 WaveWatch3 forecast publish figures 2026-11-06 12:57:23,646 INFO [make_plots] added Halibut Bank figure to 2026-11-06 WaveWatch3 forecast publish render list 2026-11-06 12:57:23,647 INFO [make_plots] added Sentry Shoal figure to 2026-11-06 WaveWatch3 forecast publish render list 2026-11-06 12:57:23,647 INFO [make_plots] starting to render wwatch3 forecast publish 06nov26 figures 2026-11-06 12:57:25,701 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46146 Halibut Bank 2026-11-06 12:57:31,763 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46131 Sentry Shoal 2026-11-06 12:57:35,027 INFO [make_plots] finished rendering wwatch3 forecast publish 06nov26 figures 2026-11-06 12:57:35,028 INFO [make_plots] wwatch3 publish plots for 2026-11-06 forecast completed 2026-11-06 12:57:35,032 INFO [manager] checklist updated with [plots] items from make_plots worker 2026-11-06 12:57:58,667 WARNING [make_plots] invalid station number for Boundary Bay station: None 2026-11-06 12:57:58,667 WARNING [make_plots] can't resolve a valid CHS station code for Boundary Bay 2026-11-06 12:58:31,307 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb996/data for station 08074 Campbell River from 2026-11-01 00:00:00Z to 2026-11-08 11:55:00Z 2026-11-06 12:59:41,058 WARNING [make_plots] invalid station number for Halfmoon Bay station: None 2026-11-06 12:59:41,073 WARNING [make_plots] can't resolve a valid CHS station code for Halfmoon Bay 2026-11-06 12:59:49,931 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19609880 = 2026-11-06 15:32:40 UTC, 64.8% complete 2026-11-06 13:00:03,948 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb96d/data for station 07917 Nanaimo from 2026-11-01 00:00:00Z to 2026-11-08 11:55:00Z 2026-11-06 13:00:46,670 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e13d0f4a073c4bbf93/data for station 07654 New Westminster from 2026-11-01 00:00:00Z to 2026-11-08 11:55:00Z 2026-11-06 13:01:11,873 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd2d/data for station 07277 Patricia Bay from 2026-11-01 00:00:00Z to 2026-11-08 11:55:00Z 2026-11-06 13:01:37,120 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94c/data for station 07795 Point Atkinson from 2026-11-01 00:00:00Z to 2026-11-08 11:55:00Z 2026-11-06 13:02:02,044 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e23d0f4a073c4bc060/data for station 08525 Port Renfrew from 2026-11-01 00:00:00Z to 2026-11-08 11:55:00Z 2026-11-06 13:02:22,849 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb93c/data for station 07594 Sand Heads from 2026-11-01 00:00:00Z to 2026-11-08 11:55:00Z 2026-11-06 13:02:50,554 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc43b/data for station 07786 Sandy Cove from 2026-11-01 00:00:00Z to 2026-11-08 11:55:00Z 2026-11-06 13:03:17,341 WARNING [make_plots] invalid station number for Squamish station: None 2026-11-06 13:03:17,349 WARNING [make_plots] can't resolve a valid CHS station code for Squamish 2026-11-06 13:03:18,459 INFO [make_surface_current_tiles] starting PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast/06nov26 2026-11-06 13:03:35,994 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd1e/data for station 07120 Victoria from 2026-11-01 00:00:00Z to 2026-11-08 11:55:00Z 2026-11-06 13:03:57,003 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc42d/data for station 07610 Woodwards Landing from 2026-11-01 00:00:00Z to 2026-11-08 11:55:00Z 2026-11-06 13:03:57,521 INFO [make_plots] no obs water level data available from 5cebf1e43d0f4a073c4bc42d during 2026-11-01 00:00:00Z to 2026-11-08 11:55:00Z 2026-11-06 13:04:16,483 INFO [make_plots] finished rendering nemo forecast publish 06nov26 figures 2026-11-06 13:04:16,492 INFO [make_plots] nemo publish plots for 2026-11-06 forecast completed 2026-11-06 13:04:16,643 INFO [manager] checklist updated with [plots] items from make_plots worker 2026-11-06 13:04:16,901 INFO [manager] launching NextWorker(module='nowcast.workers.make_feeds', args=['forecast', '--run-date', '2026-11-06'], host='localhost') 2026-11-06 13:04:25,508 INFO [make_feeds] running in process 830652 2026-11-06 13:04:25,509 INFO [make_feeds] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2026-11-06 13:04:25,527 INFO [make_feeds] publishing log messages to tcp://*:5564 2026-11-06 13:04:25,545 INFO [make_feeds] connected to skookum.eos.ubc.ca port 5555 2026-11-06 13:04:42,363 INFO [make_feeds] ATOM feeds for 2026-11-06 forecast run completed 2026-11-06 13:04:42,366 INFO [manager] checklist updated with [storm surge alert feeds] items from make_feeds worker 2026-11-06 13:04:49,932 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19610370 = 2026-11-06 20:59:20 UTC, 87.5% complete 2026-11-06 13:09:31,184 INFO [make_surface_current_tiles] finished PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast/06nov26 2026-11-06 13:09:31,203 INFO [make_surface_current_tiles] finished rendering of tiles for 2026-11-06 forecast into /results/nowcast-sys/figures/surface_currents/forecast/06nov26 2026-11-06 13:09:31,222 INFO [make_surface_current_tiles] surface current tile figures for 2026-11-06 forecast completed 2026-11-06 13:09:31,508 INFO [manager] checklist updated with [surface current tiles] items from make_surface_current_tiles worker 2026-11-06 13:09:49,933 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19610640 = 2026-11-06 23:59:20 UTC, 100.0% complete 2026-11-06 13:14:49,942 INFO [watch_NEMO] nowcast-green NEMO run on arbutus.cloud-nowcast completed 2026-11-06 13:14:49,947 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker 2026-11-06 13:14:50,541 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2026-11-06'], host='localhost') 2026-11-06 13:14:54,707 INFO [download_results] running in process 830942 2026-11-06 13:14:54,708 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2026-11-06 13:14:54,709 INFO [download_results] publishing log messages to tcp://*:5562 2026-11-06 13:14:54,719 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2026-11-06 13:14:54,720 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/nowcast-green/06nov26 to /results2/SalishSea/nowcast-green.202111 2026-11-06 13:17:55,030 INFO [download_results] nowcast-green 2026-11-06 results files from arbutus.cloud-nowcast downloaded 2026-11-06 13:17:55,036 INFO [manager] checklist updated with [results files] items from download_results worker 2026-11-06 13:17:55,318 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast-green', 'research', '--run-date', '2026-11-06'], host='localhost') 2026-11-06 13:17:55,323 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nowcast-green'], host='localhost') 2026-11-06 13:17:55,330 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'biology', '--run-date', '2026-11-06'], host='localhost') 2026-11-06 13:17:55,336 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'physics', '--run-date', '2026-11-06'], host='localhost') 2026-11-06 13:17:55,342 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'chemistry', '--run-date', '2026-11-06'], host='localhost') 2026-11-06 13:17:58,385 INFO [ping_erddap] running in process 831024 2026-11-06 13:17:58,386 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2026-11-06 13:17:58,387 INFO [ping_erddap] publishing log messages to tcp://*:5562 2026-11-06 13:17:58,388 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555 2026-11-06 13:17:58,534 INFO [ping_erddap] nowcast-green ERDDAP dataset flag file(s) created 2026-11-06 13:17:58,536 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker 2026-11-06 13:18:02,612 INFO [make_averaged_dataset] running in process 831026 2026-11-06 13:18:02,643 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2026-11-06 13:18:02,661 INFO [make_averaged_dataset] publishing log messages to tcp://*:5562 2026-11-06 13:18:02,671 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555 2026-11-06 13:18:02,672 INFO [make_averaged_dataset] creating day-averaged dataset for 06-Nov-2026 physics 2026-11-06 13:18:02,673 INFO [make_averaged_dataset] running in process 831025 2026-11-06 13:18:02,674 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2026-11-06 13:18:02,675 INFO [make_averaged_dataset] publishing log messages to tcp://*:5564 2026-11-06 13:18:02,676 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555 2026-11-06 13:18:02,677 INFO [make_averaged_dataset] creating day-averaged dataset for 06-Nov-2026 biology 2026-11-06 13:18:02,685 INFO [make_averaged_dataset] running in process 831027 2026-11-06 13:18:02,688 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2026-11-06 13:18:02,688 INFO [make_averaged_dataset] publishing log messages to tcp://*:5565 2026-11-06 13:18:02,689 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555 2026-11-06 13:18:02,691 INFO [make_averaged_dataset] creating day-averaged dataset for 06-Nov-2026 chemistry 2026-11-06 13:18:06,225 INFO [make_plots] running in process 831022 2026-11-06 13:18:06,226 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2026-11-06 13:18:06,230 INFO [make_plots] publishing log messages to tcp://*:5566 2026-11-06 13:18:06,230 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555 2026-11-06 13:18:07,189 INFO [make_plots] preparing render list for 2026-11-06 NEMO nowcast-green research figures 2026-11-06 13:18:18,182 INFO [make_plots] added temperature_thalweg_and_surface figures to 2026-11-06 NEMO nowcast-green research render list 2026-11-06 13:18:19,077 INFO [make_plots] added nitrate_thalweg_and_surface figures to 2026-11-06 NEMO nowcast-green research render list 2026-11-06 13:18:19,935 INFO [make_plots] added ammonium_thalweg_and_surface figures to 2026-11-06 NEMO nowcast-green research render list 2026-11-06 13:18:20,790 INFO [make_plots] added silicon_thalweg_and_surface figures to 2026-11-06 NEMO nowcast-green research render list 2026-11-06 13:18:21,657 INFO [make_plots] added dissolved_organic_nitrogen_thalweg_and_surface figures to 2026-11-06 NEMO nowcast-green research render list 2026-11-06 13:18:22,547 INFO [make_plots] added particulate_organic_nitrogen_thalweg_and_surface figures to 2026-11-06 NEMO nowcast-green research render list 2026-11-06 13:18:23,425 INFO [make_plots] added biogenic_silicon_thalweg_and_surface figures to 2026-11-06 NEMO nowcast-green research render list 2026-11-06 13:18:24,394 INFO [make_plots] added diatoms_thalweg_and_surface figures to 2026-11-06 NEMO nowcast-green research render list 2026-11-06 13:18:25,343 INFO [make_plots] added flagellates_thalweg_and_surface figures to 2026-11-06 NEMO nowcast-green research render list 2026-11-06 13:18:26,243 INFO [make_plots] added microzooplankton_thalweg_and_surface figures to 2026-11-06 NEMO nowcast-green research render list 2026-11-06 13:18:27,202 INFO [make_plots] added mesozooplankton_thalweg_and_surface figures to 2026-11-06 NEMO nowcast-green research render list 2026-11-06 13:18:28,097 INFO [make_plots] added turbidity_thalweg_and_surface figures to 2026-11-06 NEMO nowcast-green research render list 2026-11-06 13:18:29,934 INFO [make_plots] added temperature_salinity_timeseries figure to 2026-11-06 NEMO nowcast-green research render list 2026-11-06 13:18:29,934 INFO [make_plots] added nitrate_diatoms_timeseries figure to 2026-11-06 NEMO nowcast-green research render list 2026-11-06 13:18:29,954 INFO [make_plots] added diatoms_flagellates_timeseries figure to 2026-11-06 NEMO nowcast-green research render list 2026-11-06 13:18:29,955 INFO [make_plots] added z1_z2_zooplankton_timeseries figure to 2026-11-06 NEMO nowcast-green research render list 2026-11-06 13:18:29,956 INFO [make_plots] starting to render nemo nowcast-green research 06nov26 figures 2026-11-06 13:19:07,549 INFO [make_averaged_dataset] day-averaged dataset for 06-Nov-2026 chemistry created 2026-11-06 13:19:07,576 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker 2026-11-06 13:19:08,701 INFO [make_averaged_dataset] day-averaged dataset for 06-Nov-2026 physics created 2026-11-06 13:19:08,704 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker 2026-11-06 13:19:22,121 INFO [make_averaged_dataset] day-averaged dataset for 06-Nov-2026 biology created 2026-11-06 13:19:22,124 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker 2026-11-06 14:29:38,626 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/18/ to /results/forcing/atmospheric/continental2.5/GRIB/20261106/18/ 2026-11-06 14:29:38,627 INFO [collect_weather] 2026-11-06 2.5km weather forecast 18 collection complete 2026-11-06 14:29:38,638 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker 2026-11-06 14:29:39,408 INFO [manager] launching NextWorker(module='nowcast.workers.download_weather', args=['00', '1km'], host='localhost') 2026-11-06 14:29:39,416 INFO [manager] launching NextWorker(module='nowcast.workers.download_weather', args=['12', '1km'], host='localhost') 2026-11-06 14:29:39,422 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['00', '2.5km'], host='localhost') 2026-11-06 14:29:39,426 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['00', '--fcst-date', '2026-11-07'], host='localhost') 2026-11-06 14:29:40,414 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20261106/18/ 2026-11-06 14:29:40,415 INFO [crop_gribs] 2026-11-06 18 GRIBs cropping complete 2026-11-06 14:29:40,468 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker 2026-11-06 14:29:45,341 INFO [download_weather] running in process 832725 2026-11-06 14:29:45,342 INFO [download_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2026-11-06 14:29:45,343 INFO [download_weather] publishing log messages to tcp://*:5562 2026-11-06 14:29:45,344 INFO [download_weather] connected to skookum.eos.ubc.ca port 5555 2026-11-06 14:29:45,345 INFO [download_weather] downloading 12 1 km forecast GRIB2 files for 20261106 2026-11-06 14:29:45,346 INFO [collect_weather] running in process 832726 2026-11-06 14:29:45,347 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2026-11-06 14:29:45,348 INFO [collect_weather] publishing log messages to tcp://*:5563 2026-11-06 14:29:45,350 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555 2026-11-06 14:29:45,351 INFO [download_weather] running in process 832723 2026-11-06 14:29:45,361 INFO [download_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2026-11-06 14:29:45,362 INFO [download_weather] publishing log messages to tcp://*:5564 2026-11-06 14:29:45,363 INFO [download_weather] connected to skookum.eos.ubc.ca port 5555 2026-11-06 14:29:45,363 INFO [download_weather] downloading 00 1 km forecast GRIB2 files for 20261106 2026-11-06 14:29:45,613 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/00/ 2026-11-06 14:29:52,136 INFO [crop_gribs] running in process 832727 2026-11-06 14:29:52,186 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2026-11-06 14:29:52,191 INFO [crop_gribs] publishing log messages to tcp://*:5565 2026-11-06 14:29:52,192 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555 2026-11-06 14:29:52,194 INFO [crop_gribs] cropping 2026-11-07 ECCC HRDPS 2.5km continental 00Z GRIB files to SalishSeaCast subdomain 2026-11-06 14:29:52,196 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20261107/00/ 2026-11-06 14:33:58,964 INFO [download_weather] 2026-11-06 1km weather forecast 12 downloads complete 2026-11-06 14:33:59,015 INFO [manager] checklist updated with [weather forecast] items from download_weather worker 2026-11-06 14:34:15,984 INFO [download_weather] 2026-11-06 1km weather forecast 00 downloads complete 2026-11-06 14:34:16,052 INFO [manager] checklist updated with [weather forecast] items from download_weather worker 2026-11-06 14:41:28,718 INFO [make_plots] finished rendering nemo nowcast-green research 06nov26 figures 2026-11-06 14:41:28,774 INFO [make_plots] nemo research plots for 2026-11-06 nowcast-green completed 2026-11-06 14:41:28,878 INFO [manager] checklist updated with [plots] items from make_plots worker 2026-11-06 20:25:27,137 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/00/ to /results/forcing/atmospheric/continental2.5/GRIB/20261107/00/ 2026-11-06 20:25:27,139 INFO [collect_weather] 2026-11-07 2.5km weather forecast 00 collection complete 2026-11-06 20:25:27,173 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker 2026-11-06 20:25:27,848 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['06', '2.5km'], host='localhost') 2026-11-06 20:25:27,859 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['06', '--fcst-date', '2026-11-07'], host='localhost') 2026-11-06 20:25:29,150 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20261107/00/ 2026-11-06 20:25:29,151 INFO [crop_gribs] 2026-11-07 00 GRIBs cropping complete 2026-11-06 20:25:29,158 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker 2026-11-06 20:25:30,208 INFO [collect_weather] running in process 842479 2026-11-06 20:25:30,209 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2026-11-06 20:25:30,210 INFO [collect_weather] publishing log messages to tcp://*:5562 2026-11-06 20:25:30,211 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555 2026-11-06 20:25:30,244 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/06/ 2026-11-06 20:25:31,859 INFO [crop_gribs] running in process 842480 2026-11-06 20:25:31,860 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2026-11-06 20:25:31,861 INFO [crop_gribs] publishing log messages to tcp://*:5563 2026-11-06 20:25:31,862 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555 2026-11-06 20:25:31,863 INFO [crop_gribs] cropping 2026-11-07 ECCC HRDPS 2.5km continental 06Z GRIB files to SalishSeaCast subdomain 2026-11-06 20:25:31,874 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20261107/06/ 2026-11-07 02:25:41,946 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/06/ to /results/forcing/atmospheric/continental2.5/GRIB/20261107/06/ 2026-11-07 02:25:41,948 INFO [collect_weather] 2026-11-07 2.5km weather forecast 06 collection complete 2026-11-07 02:25:41,951 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker 2026-11-07 02:25:42,518 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Capilano', '--data-date', '2026-11-06'], host='localhost') 2026-11-07 02:25:42,529 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'ChilliwackVedder', '--data-date', '2026-11-06'], host='localhost') 2026-11-07 02:25:42,535 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'ClowhomClowhomLake', '--data-date', '2026-11-06'], host='localhost') 2026-11-07 02:25:42,546 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Englishman', '--data-date', '2026-11-06'], host='localhost') 2026-11-07 02:25:42,553 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Fraser', '--data-date', '2026-11-06'], host='localhost') 2026-11-07 02:25:42,558 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'HomathkoMouth', '--data-date', '2026-11-06'], host='localhost') 2026-11-07 02:25:42,562 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'NicomeklLangley', '--data-date', '2026-11-06'], host='localhost') 2026-11-07 02:25:42,567 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'RobertsCreek', '--data-date', '2026-11-06'], host='localhost') 2026-11-07 02:25:42,578 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SalmonSayward', '--data-date', '2026-11-06'], host='localhost') 2026-11-07 02:25:42,584 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SanJuanPortRenfrew', '--data-date', '2026-11-06'], host='localhost') 2026-11-07 02:25:42,592 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SquamishBrackendale', '--data-date', '2026-11-06'], host='localhost') 2026-11-07 02:25:42,597 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaScotty', '--data-date', '2026-11-06'], host='localhost') 2026-11-07 02:25:42,603 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaBypass', '--data-date', '2026-11-06'], host='localhost') 2026-11-07 02:25:42,610 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaDiversion', '--data-date', '2026-11-06'], host='localhost') 2026-11-07 02:25:42,624 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ctd', args=['SCVIP'], host='localhost') 2026-11-07 02:25:42,629 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ctd', args=['SEVIP'], host='localhost') 2026-11-07 02:25:42,639 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ferry', args=['TWDP'], host='localhost') 2026-11-07 02:25:42,647 INFO [manager] launching NextWorker(module='nowcast.workers.get_vfpa_hadcp', args=['--data-date', '2026-11-06'], host='localhost') 2026-11-07 02:25:42,662 INFO [manager] launching NextWorker(module='nowcast.workers.collect_NeahBay_ssh', args=['00'], host='localhost') 2026-11-07 02:25:42,667 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['12', '2.5km'], host='localhost') 2026-11-07 02:25:42,680 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['12'], host='localhost') 2026-11-07 02:25:42,895 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20261107/06/ 2026-11-07 02:25:42,896 INFO [crop_gribs] 2026-11-07 06 GRIBs cropping complete 2026-11-07 02:25:42,899 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker 2026-11-07 02:25:43,838 INFO [manager] launching NextWorker(module='nowcast.workers.grib_to_netcdf', args=['forecast2'], host='localhost') 2026-11-07 02:25:46,052 INFO [collect_NeahBay_ssh] running in process 853499 2026-11-07 02:25:46,053 INFO [collect_NeahBay_ssh] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2026-11-07 02:25:46,054 INFO [collect_NeahBay_ssh] publishing log messages to tcp://*:5562 2026-11-07 02:25:46,055 INFO [collect_NeahBay_ssh] connected to skookum.eos.ubc.ca port 5555 2026-11-07 02:25:46,058 INFO [collect_NeahBay_ssh] collecting Neah Bay ssh 00Z obs/forecast for 2026-11-07 2026-11-07 02:25:46,069 INFO [collect_weather] running in process 853500 2026-11-07 02:25:46,070 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2026-11-07 02:25:46,072 INFO [collect_weather] publishing log messages to tcp://*:5563 2026-11-07 02:25:46,073 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555 2026-11-07 02:25:46,104 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/12/ 2026-11-07 02:25:47,047 INFO [collect_river_data] running in process 853481 2026-11-07 02:25:47,048 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2026-11-07 02:25:47,049 INFO [collect_river_data] publishing log messages to tcp://*:5564 2026-11-07 02:25:47,050 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2026-11-07 02:25:47,051 INFO [collect_river_data] Collecting ECCC ChilliwackVedder river data for 2026-11-06 2026-11-07 02:25:47,053 INFO [collect_river_data] running in process 853485 2026-11-07 02:25:47,055 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2026-11-07 02:25:47,056 INFO [collect_river_data] publishing log messages to tcp://*:5565 2026-11-07 02:25:47,057 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2026-11-07 02:25:47,058 INFO [collect_river_data] Collecting ECCC HomathkoMouth river data for 2026-11-06 2026-11-07 02:25:47,118 INFO [collect_river_data] running in process 853491 2026-11-07 02:25:47,119 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2026-11-07 02:25:47,119 INFO [collect_river_data] publishing log messages to tcp://*:5566 2026-11-07 02:25:47,120 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2026-11-07 02:25:47,121 INFO [collect_river_data] Collecting ECCC SquamishBrackendale river data for 2026-11-06 2026-11-07 02:25:47,150 INFO [collect_river_data] running in process 853487 2026-11-07 02:25:47,151 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2026-11-07 02:25:47,153 INFO [collect_river_data] publishing log messages to tcp://*:5567 2026-11-07 02:25:47,153 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2026-11-07 02:25:47,154 INFO [collect_river_data] Collecting ECCC RobertsCreek river data for 2026-11-06 2026-11-07 02:25:47,163 INFO [collect_river_data] running in process 853482 2026-11-07 02:25:47,164 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2026-11-07 02:25:47,165 INFO [collect_river_data] publishing log messages to tcp://*:5568 2026-11-07 02:25:47,166 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2026-11-07 02:25:47,167 INFO [collect_river_data] Collecting ECCC ClowhomClowhomLake river data for 2026-11-06 2026-11-07 02:25:47,172 INFO [collect_river_data] Appended ECCC SquamishBrackendale river average discharge for 2026-11-06 to: /results/forcing/rivers/observations/Squamish_Brackendale_flow 2026-11-07 02:25:47,172 INFO [collect_river_data] ECCC SquamishBrackendale river data collection for 2026-11-06 completed 2026-11-07 02:25:47,175 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2026-11-07 02:25:47,187 INFO [collect_river_data] Appended ECCC ChilliwackVedder river average discharge for 2026-11-06 to: /results/forcing/rivers/observations/Chilliwack_Vedder_flow 2026-11-07 02:25:47,188 INFO [collect_river_data] ECCC ChilliwackVedder river data collection for 2026-11-06 completed 2026-11-07 02:25:47,190 INFO [collect_river_data] running in process 853479 2026-11-07 02:25:47,191 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2026-11-07 02:25:47,191 INFO [collect_river_data] publishing log messages to tcp://*:5569 2026-11-07 02:25:47,192 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2026-11-07 02:25:47,193 INFO [collect_river_data] Collecting ECCC Capilano river data for 2026-11-06 2026-11-07 02:25:47,198 INFO [collect_river_data] running in process 853492 2026-11-07 02:25:47,199 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2026-11-07 02:25:47,199 INFO [collect_river_data] publishing log messages to tcp://*:5570 2026-11-07 02:25:47,200 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2026-11-07 02:25:47,202 INFO [collect_river_data] Collecting ECCC TheodosiaScotty river data for 2026-11-06 2026-11-07 02:25:47,203 INFO [collect_river_data] Appended ECCC ClowhomClowhomLake river average discharge for 2026-11-06 to: /results/forcing/rivers/observations/Clowhom_ClowhomLake_flow 2026-11-07 02:25:47,203 INFO [collect_river_data] ECCC ClowhomClowhomLake river data collection for 2026-11-06 completed 2026-11-07 02:25:47,205 INFO [collect_river_data] Appended ECCC HomathkoMouth river average discharge for 2026-11-06 to: /results/forcing/rivers/observations/Homathko_Mouth_flow 2026-11-07 02:25:47,206 INFO [collect_river_data] ECCC HomathkoMouth river data collection for 2026-11-06 completed 2026-11-07 02:25:47,218 INFO [collect_river_data] running in process 853488 2026-11-07 02:25:47,219 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2026-11-07 02:25:47,220 INFO [collect_river_data] publishing log messages to tcp://*:5571 2026-11-07 02:25:47,221 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2026-11-07 02:25:47,222 INFO [collect_river_data] Collecting ECCC SalmonSayward river data for 2026-11-06 2026-11-07 02:25:47,226 INFO [collect_river_data] running in process 853484 2026-11-07 02:25:47,227 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2026-11-07 02:25:47,228 INFO [collect_river_data] publishing log messages to tcp://*:5572 2026-11-07 02:25:47,228 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2026-11-07 02:25:47,229 INFO [collect_river_data] Collecting ECCC Fraser river data for 2026-11-06 2026-11-07 02:25:47,230 INFO [collect_river_data] running in process 853483 2026-11-07 02:25:47,235 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2026-11-07 02:25:47,235 INFO [collect_river_data] publishing log messages to tcp://*:5573 2026-11-07 02:25:47,236 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2026-11-07 02:25:47,237 INFO [collect_river_data] Collecting ECCC Englishman river data for 2026-11-06 2026-11-07 02:25:47,239 INFO [collect_river_data] running in process 853490 2026-11-07 02:25:47,240 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2026-11-07 02:25:47,241 INFO [collect_river_data] Appended ECCC RobertsCreek river average discharge for 2026-11-06 to: /results/forcing/rivers/observations/RobertsCreek_flow 2026-11-07 02:25:47,241 INFO [collect_river_data] publishing log messages to tcp://*:5574 2026-11-07 02:25:47,244 INFO [collect_river_data] ECCC RobertsCreek river data collection for 2026-11-06 completed 2026-11-07 02:25:47,245 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2026-11-07 02:25:47,248 INFO [collect_river_data] Collecting ECCC SanJuanPortRenfrew river data for 2026-11-06 2026-11-07 02:25:47,249 INFO [collect_river_data] running in process 853486 2026-11-07 02:25:47,250 INFO [collect_river_data] running in process 853494 2026-11-07 02:25:47,251 INFO [collect_river_data] running in process 853493 2026-11-07 02:25:47,252 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2026-11-07 02:25:47,253 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2026-11-07 02:25:47,254 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2026-11-07 02:25:47,255 INFO [collect_river_data] Appended ECCC TheodosiaScotty river average discharge for 2026-11-06 to: /results/forcing/rivers/observations/Theodosia_Scotty_flow 2026-11-07 02:25:47,256 INFO [collect_river_data] publishing log messages to tcp://*:5575 2026-11-07 02:25:47,257 INFO [collect_river_data] publishing log messages to tcp://*:5576 2026-11-07 02:25:47,262 INFO [collect_river_data] publishing log messages to tcp://*:5577 2026-11-07 02:25:47,262 INFO [collect_river_data] ECCC TheodosiaScotty river data collection for 2026-11-06 completed 2026-11-07 02:25:47,264 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2026-11-07 02:25:47,265 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2026-11-07 02:25:47,265 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2026-11-07 02:25:47,267 INFO [collect_river_data] Collecting ECCC NicomeklLangley river data for 2026-11-06 2026-11-07 02:25:47,268 INFO [collect_river_data] Collecting ECCC TheodosiaDiversion river data for 2026-11-06 2026-11-07 02:25:47,269 INFO [collect_river_data] Collecting ECCC TheodosiaBypass river data for 2026-11-06 2026-11-07 02:25:47,274 INFO [collect_river_data] Appended ECCC SalmonSayward river average discharge for 2026-11-06 to: /results/forcing/rivers/observations/Salmon_Sayward_flow 2026-11-07 02:25:47,274 INFO [collect_river_data] ECCC SalmonSayward river data collection for 2026-11-06 completed 2026-11-07 02:25:47,288 INFO [collect_river_data] Appended ECCC Englishman river average discharge for 2026-11-06 to: /data/dlatorne/SOG-projects/SOG-forcing/ECget/Englishman_flow 2026-11-07 02:25:47,291 INFO [collect_river_data] ECCC Englishman river data collection for 2026-11-06 completed 2026-11-07 02:25:47,294 INFO [collect_river_data] Appended ECCC Fraser river average discharge for 2026-11-06 to: /data/dlatorne/SOG-projects/SOG-forcing/ECget/Fraser_flow 2026-11-07 02:25:47,295 INFO [collect_river_data] Appended ECCC Capilano river average discharge for 2026-11-06 to: /opp/observations/rivers/Capilano/Caplilano_08GA010_day_avg_flow 2026-11-07 02:25:47,296 INFO [collect_river_data] ECCC Fraser river data collection for 2026-11-06 completed 2026-11-07 02:25:47,296 INFO [collect_river_data] ECCC Capilano river data collection for 2026-11-06 completed 2026-11-07 02:25:47,300 INFO [collect_river_data] Appended ECCC SanJuanPortRenfrew river average discharge for 2026-11-06 to: /results/forcing/rivers/observations/SanJuan_PortRenfrew_flow 2026-11-07 02:25:47,301 INFO [collect_river_data] ECCC SanJuanPortRenfrew river data collection for 2026-11-06 completed 2026-11-07 02:25:47,310 INFO [collect_river_data] Appended ECCC NicomeklLangley river average discharge for 2026-11-06 to: /results/forcing/rivers/observations/Nicomekl_Langley_flow 2026-11-07 02:25:47,311 INFO [collect_river_data] ECCC NicomeklLangley river data collection for 2026-11-06 completed 2026-11-07 02:25:47,339 INFO [collect_river_data] Appended ECCC TheodosiaBypass river average discharge for 2026-11-06 to: /results/forcing/rivers/observations/Theodosia_Bypass_flow 2026-11-07 02:25:47,340 INFO [collect_river_data] ECCC TheodosiaBypass river data collection for 2026-11-06 completed 2026-11-07 02:25:47,343 INFO [collect_river_data] Appended ECCC TheodosiaDiversion river average discharge for 2026-11-06 to: /results/forcing/rivers/observations/Theodosia_Diversion_flow 2026-11-07 02:25:47,344 INFO [collect_river_data] ECCC TheodosiaDiversion river data collection for 2026-11-06 completed 2026-11-07 02:25:47,347 INFO [collect_NeahBay_ssh] 2026-11-07T00:00:00-07:00 Neah Bay ssh 00Z obs/forecast data collection complete 2026-11-07 02:25:47,598 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2026-11-07 02:25:47,600 INFO [grib_to_netcdf] running in process 853714 2026-11-07 02:25:47,602 INFO [grib_to_netcdf] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2026-11-07 02:25:47,603 INFO [grib_to_netcdf] publishing log messages to tcp://*:5578 2026-11-07 02:25:47,605 INFO [grib_to_netcdf] connected to skookum.eos.ubc.ca port 5555 2026-11-07 02:25:47,606 INFO [grib_to_netcdf] creating NEMO-atmos forcing files for 2026-11-07 forecast2 run 2026-11-07 02:25:47,621 INFO [get_vfpa_hadcp] running in process 853498 2026-11-07 02:25:47,622 INFO [get_vfpa_hadcp] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2026-11-07 02:25:47,623 INFO [get_vfpa_hadcp] publishing log messages to tcp://*:5579 2026-11-07 02:25:47,624 INFO [get_vfpa_hadcp] connected to skookum.eos.ubc.ca port 5555 2026-11-07 02:25:47,625 INFO [get_vfpa_hadcp] processing VFPA HADCP data from 2nd Narrows Rail Bridge for 2026-11-06 2026-11-07 02:25:47,850 INFO [crop_gribs] running in process 853501 2026-11-07 02:25:47,851 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2026-11-07 02:25:47,852 INFO [crop_gribs] publishing log messages to tcp://*:5566 2026-11-07 02:25:47,853 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555 2026-11-07 02:25:47,854 INFO [crop_gribs] cropping 2026-11-07 ECCC HRDPS 2.5km continental 12Z GRIB files to SalishSeaCast subdomain 2026-11-07 02:25:47,866 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20261107/12/ 2026-11-07 02:25:47,973 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2026-11-07 02:25:48,328 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2026-11-07 02:25:48,733 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2026-11-07 02:25:48,984 INFO [get_onc_ferry] running in process 853497 2026-11-07 02:25:48,985 INFO [get_onc_ferry] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2026-11-07 02:25:48,986 INFO [get_onc_ferry] publishing log messages to tcp://*:5564 2026-11-07 02:25:48,987 INFO [get_onc_ferry] connected to skookum.eos.ubc.ca port 5555 2026-11-07 02:25:48,988 INFO [get_onc_ferry] requesting ONC TWDP.N1 NAV data for 2026-11-06 2026-11-07 02:25:48,996 INFO [get_onc_ctd] running in process 853495 2026-11-07 02:25:48,998 INFO [get_onc_ctd] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2026-11-07 02:25:48,999 INFO [get_onc_ctd] publishing log messages to tcp://*:5565 2026-11-07 02:25:48,999 INFO [get_onc_ctd] connected to skookum.eos.ubc.ca port 5555 2026-11-07 02:25:49,000 INFO [get_onc_ctd] requesting ONC SCVIP CTD T&S data for 2026-11-06 2026-11-07 02:25:49,013 INFO [get_onc_ctd] running in process 853496 2026-11-07 02:25:49,016 INFO [get_onc_ctd] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2026-11-07 02:25:49,017 INFO [get_onc_ctd] publishing log messages to tcp://*:5568 2026-11-07 02:25:49,018 INFO [get_onc_ctd] connected to skookum.eos.ubc.ca port 5555 2026-11-07 02:25:49,018 INFO [get_onc_ctd] requesting ONC SEVIP CTD T&S data for 2026-11-06 2026-11-07 02:25:49,056 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2026-11-07 02:25:49,409 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2026-11-07 02:25:49,731 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2026-11-07 02:25:50,050 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2026-11-07 02:25:50,055 INFO [get_vfpa_hadcp] added VFPA HADCP data from 2nd Narrows Rail Bridge for 2026-11-06 to /opp/observations/AISDATA/netcdf/VFPA_2ND_NARROWS_HADCP_2s_202611.nc 2026-11-07 02:25:50,055 INFO [get_vfpa_hadcp] VFPA HADCP observations added to 2026-11 netcdf file 2026-11-07 02:25:50,375 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2026-11-07 02:25:50,681 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2026-11-07 02:25:50,984 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2026-11-07 02:25:51,326 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2026-11-07 02:25:51,666 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2026-11-07 02:25:52,022 INFO [manager] checklist updated with [Neah Bay ssh data] items from collect_NeahBay_ssh worker 2026-11-07 02:25:52,333 INFO [manager] launching NextWorker(module='nowcast.workers.make_ssh_files', args=['forecast2', '--run-date', '2026-11-07'], host='localhost') 2026-11-07 02:25:52,340 INFO [manager] checklist updated with [VFPA HADCP data] items from get_vfpa_hadcp worker 2026-11-07 02:25:52,715 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['VFPA-HADCP'], host='localhost') 2026-11-07 02:25:54,044 INFO [ping_erddap] running in process 853849 2026-11-07 02:25:54,044 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2026-11-07 02:25:54,045 INFO [ping_erddap] publishing log messages to tcp://*:5562 2026-11-07 02:25:54,046 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555 2026-11-07 02:25:54,047 INFO [ping_erddap] VFPA-HADCP ERDDAP dataset flag file(s) created 2026-11-07 02:25:54,052 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker 2026-11-07 02:25:56,283 INFO [make_ssh_files] running in process 853836 2026-11-07 02:25:56,285 INFO [make_ssh_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2026-11-07 02:25:56,286 INFO [make_ssh_files] publishing log messages to tcp://*:5562 2026-11-07 02:25:56,286 INFO [make_ssh_files] connected to skookum.eos.ubc.ca port 5555 2026-11-07 02:25:56,287 INFO [make_ssh_files] building forecast2 sea surface height boundary conditions file(s) from 2026-11-07 NOAA Neah Bay 00Z observation and forecast values 2026-11-07 02:25:58,720 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2026m11d02.nc 2026-11-07 02:25:58,855 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2026m11d08.nc 2026-11-07 02:25:58,942 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2026m11d03.nc 2026-11-07 02:25:59,173 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2026m11d04.nc 2026-11-07 02:25:59,377 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2026m11d05.nc 2026-11-07 02:25:59,581 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2026m11d06.nc 2026-11-07 02:25:59,873 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2026m11d07.nc 2026-11-07 02:26:00,045 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2026m11d08.nc 2026-11-07 02:26:00,289 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2026m11d09.nc 2026-11-07 02:26:00,466 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2026m11d10.nc 2026-11-07 02:26:00,747 INFO [make_ssh_files] sea surface height boundary file for 2026-11-07 forecast2 run created 2026-11-07 02:26:00,751 INFO [manager] checklist updated with [sea surface height forcing] items from make_ssh_files worker 2026-11-07 02:26:01,294 INFO [manager] launching NextWorker(module='nowcast.workers.make_runoff_file', args=['v202108'], host='localhost') 2026-11-07 02:26:01,886 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2026m11d09.nc 2026-11-07 02:26:01,895 INFO [grib_to_netcdf] 2026-11-07 NEMO-atmos forcing file for forecast2 created 2026-11-07 02:26:01,897 INFO [manager] checklist updated with [weather forcing] items from grib_to_netcdf worker 2026-11-07 02:26:03,439 INFO [make_runoff_file] running in process 853886 2026-11-07 02:26:03,441 INFO [make_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2026-11-07 02:26:03,442 INFO [make_runoff_file] publishing log messages to tcp://*:5562 2026-11-07 02:26:03,443 INFO [make_runoff_file] connected to skookum.eos.ubc.ca port 5555 2026-11-07 02:26:03,445 INFO [make_runoff_file] calculating NEMO runoff forcing for v202108 bathymetry for 2026-11-06 2026-11-07 02:26:04,650 ERROR [make_runoff_file] no 2026-11-06 discharge obs for SkagitMountVernon: patching 2026-11-07 02:26:05,057 ERROR [make_runoff_file] no 2026-11-06 discharge obs for SnohomishMonroe: patching 2026-11-07 02:26:05,297 ERROR [make_runoff_file] no 2026-11-06 discharge obs for NisquallyMcKenna: patching 2026-11-07 02:26:05,411 ERROR [make_runoff_file] no 2026-11-06 discharge obs for GreenwaterGreenwater: patching 2026-11-07 02:26:06,304 INFO [get_onc_ctd] 2026-11-06 ONC SCVIP CTD T&S file created 2026-11-07 02:26:06,308 INFO [manager] checklist updated with [ONC CTD data] items from get_onc_ctd worker 2026-11-07 02:26:06,650 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['SCVIP-CTD'], host='localhost') 2026-11-07 02:26:06,988 INFO [make_runoff_file] stored NEMO runoff forcing for v202108 bathymetry for 2026-11-06: /results/forcing/rivers/R202108Dailies_y2026m11d06.nc 2026-11-07 02:26:06,988 INFO [make_runoff_file] 2026-11-06 runoff file creation completed 2026-11-07 02:26:06,990 INFO [manager] checklist updated with [rivers forcing] items from make_runoff_file worker 2026-11-07 02:26:07,302 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'forecast2'], host='localhost') 2026-11-07 02:26:07,304 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.nibi', 'forecast2'], host='localhost') 2026-11-07 02:26:07,308 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'forecast2'], host='localhost') 2026-11-07 02:26:07,900 INFO [ping_erddap] running in process 853908 2026-11-07 02:26:07,901 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2026-11-07 02:26:07,902 INFO [ping_erddap] publishing log messages to tcp://*:5562 2026-11-07 02:26:07,903 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555 2026-11-07 02:26:07,903 INFO [ping_erddap] SCVIP-CTD ERDDAP dataset flag file(s) created 2026-11-07 02:26:07,905 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker 2026-11-07 02:26:08,126 INFO [get_onc_ferry] requesting ONC TWDP TSG data for 2026-11-06 2026-11-07 02:26:08,817 INFO [upload_forcing] running in process 853910 2026-11-07 02:26:08,818 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2026-11-07 02:26:08,819 INFO [upload_forcing] publishing log messages to tcp://*:5562 2026-11-07 02:26:08,820 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2026-11-07 02:26:08,842 INFO [upload_forcing] running in process 853911 2026-11-07 02:26:08,843 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2026-11-07 02:26:08,844 INFO [upload_forcing] publishing log messages to tcp://*:5565 2026-11-07 02:26:08,845 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2026-11-07 02:26:08,845 INFO [upload_forcing] running in process 853912 2026-11-07 02:26:08,846 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2026-11-07 02:26:08,847 INFO [upload_forcing] publishing log messages to tcp://*:5567 2026-11-07 02:26:08,848 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2026-11-07 02:26:09,278 INFO [get_onc_ctd] 2026-11-06 ONC SEVIP CTD T&S file created 2026-11-07 02:26:09,281 INFO [manager] checklist updated with [ONC CTD data] items from get_onc_ctd worker 2026-11-07 02:26:09,791 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['SEVIP-CTD'], host='localhost') 2026-11-07 02:26:10,340 INFO [upload_forcing] LiveOcean boundary conditions file not found; created symlink to /results/forcing/LiveOcean/boundary_conditions/LiveOcean_v201905_y2026m11d06.nc 2026-11-07 02:26:10,399 INFO [upload_forcing] forecast2 2026-11-07 forcing files upload to optimum-hindcast completed 2026-11-07 02:26:10,403 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2026-11-07 02:26:10,960 INFO [ping_erddap] running in process 853941 2026-11-07 02:26:10,961 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2026-11-07 02:26:10,962 INFO [ping_erddap] publishing log messages to tcp://*:5568 2026-11-07 02:26:10,963 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555 2026-11-07 02:26:10,965 INFO [ping_erddap] SEVIP-CTD ERDDAP dataset flag file(s) created 2026-11-07 02:26:10,966 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker 2026-11-07 02:26:11,206 INFO [upload_forcing] forecast2 2026-11-07 forcing files upload to arbutus.cloud-nowcast completed 2026-11-07 02:26:11,275 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2026-11-07 02:26:11,579 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2026-11-07'], host='localhost') 2026-11-07 02:26:12,763 INFO [make_forcing_links] running in process 853947 2026-11-07 02:26:12,764 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2026-11-07 02:26:12,765 INFO [make_forcing_links] publishing log messages to tcp://*:5562 2026-11-07 02:26:12,766 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555 2026-11-07 02:26:13,376 INFO [make_forcing_links] forecast2 2026-11-07 forcing file links on arbutus.cloud-nowcast created 2026-11-07 02:26:13,380 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker 2026-11-07 02:26:13,898 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2026-11-07'], host='arbutus.cloud-nowcast') 2026-11-07 02:26:16,015 INFO [upload_forcing] forecast2 2026-11-07 forcing files upload to robot.nibi completed 2026-11-07 02:26:16,019 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2026-11-07 02:26:16,130 INFO [run_NEMO] running in process 231108 2026-11-07 02:26:16,132 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2026-11-07 02:26:16,133 INFO [run_NEMO] publishing log messages to tcp://*:5556 2026-11-07 02:26:22,812 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555 2026-11-07 02:26:24,839 INFO [run_NEMO] forecast2: launching /nemoShare/MEOPAR/nowcast-sys/runs/06nov26forecast2_2026-11-07T022622.880049-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast 2026-11-07 02:26:24,879 INFO [run_NEMO] forecast2 NEMO run for 2026-11-07 on arbutus.cloud-nowcast started 2026-11-07 02:26:25,044 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker 2026-11-07 02:26:25,362 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'forecast2'], host='arbutus.cloud-nowcast') 2026-11-07 02:26:26,214 INFO [watch_NEMO] running in process 231251 2026-11-07 02:26:26,215 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2026-11-07 02:26:26,216 INFO [watch_NEMO] publishing log messages to tcp://*:5557 2026-11-07 02:26:26,217 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555 2026-11-07 02:26:26,309 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: time.step not found; continuing to watch... 2026-11-07 02:26:29,982 INFO [get_onc_ferry] requesting ONC TWDP OXYSENSOR data for 2026-11-06 2026-11-07 02:26:51,704 INFO [get_onc_ferry] requesting ONC TWDP TURBCHLFL data for 2026-11-06 2026-11-07 02:27:13,819 INFO [get_onc_ferry] requesting ONC TWDP CO2SENSOR data for 2026-11-06 2026-11-07 02:27:14,088 WARNING [get_onc_ferry] No ONC TWDP CO2SENSOR partial_pressure,co2 data for 2026-11-06; substituting empty dataset 2026-11-07 02:27:14,094 WARNING [get_onc_ferry] No ONC TWDP CO2SENSOR partial_pressure data for 2026-11-06; substituting empty dataset 2026-11-07 02:27:14,096 WARNING [get_onc_ferry] No ONC TWDP CO2SENSOR co2 data for 2026-11-06; substituting empty dataset 2026-11-07 02:27:14,097 INFO [get_onc_ferry] requesting ONC TWDP TEMPHUMID data for 2026-11-06 2026-11-07 02:27:14,323 WARNING [get_onc_ferry] No ONC TWDP TEMPHUMID air_temperature,rel_humidity data for 2026-11-06; substituting empty dataset 2026-11-07 02:27:14,329 WARNING [get_onc_ferry] No ONC TWDP TEMPHUMID air_temperature data for 2026-11-06; substituting empty dataset 2026-11-07 02:27:14,330 WARNING [get_onc_ferry] No ONC TWDP TEMPHUMID rel_humidity data for 2026-11-06; substituting empty dataset 2026-11-07 02:27:14,331 INFO [get_onc_ferry] requesting ONC TWDP BARPRESS data for 2026-11-06 2026-11-07 02:27:14,599 WARNING [get_onc_ferry] No ONC TWDP BARPRESS barometric_pressure data for 2026-11-06; substituting empty dataset 2026-11-07 02:27:14,603 WARNING [get_onc_ferry] No ONC TWDP BARPRESS barometric_pressure data for 2026-11-06; substituting empty dataset 2026-11-07 02:27:14,604 INFO [get_onc_ferry] requesting ONC TWDP PYRANOMETER data for 2026-11-06 2026-11-07 02:27:14,862 WARNING [get_onc_ferry] No ONC TWDP PYRANOMETER solar_radiation data for 2026-11-06; substituting empty dataset 2026-11-07 02:27:14,866 WARNING [get_onc_ferry] No ONC TWDP PYRANOMETER solar_radiation data for 2026-11-06; substituting empty dataset 2026-11-07 02:27:14,867 INFO [get_onc_ferry] requesting ONC TWDP PYRGEOMETER data for 2026-11-06 2026-11-07 02:27:15,069 WARNING [get_onc_ferry] No ONC TWDP PYRGEOMETER downward_radiation data for 2026-11-06; substituting empty dataset 2026-11-07 02:27:15,073 WARNING [get_onc_ferry] No ONC TWDP PYRGEOMETER downward_radiation data for 2026-11-06; substituting empty dataset 2026-11-07 02:27:15,229 WARNING [get_onc_ferry] ONC TWDP CO2SENSOR partial_pressure data for 2026-11-06 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs 2026-11-07 02:27:15,244 WARNING [get_onc_ferry] ONC TWDP CO2SENSOR co2 data for 2026-11-06 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs 2026-11-07 02:27:15,260 WARNING [get_onc_ferry] ONC TWDP TEMPHUMID air_temperature data for 2026-11-06 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs 2026-11-07 02:27:15,275 WARNING [get_onc_ferry] ONC TWDP TEMPHUMID rel_humidity data for 2026-11-06 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs 2026-11-07 02:27:15,289 WARNING [get_onc_ferry] ONC TWDP BARPRESS barometric_pressure data for 2026-11-06 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs 2026-11-07 02:27:15,304 WARNING [get_onc_ferry] ONC TWDP PYRANOMETER solar_radiation data for 2026-11-06 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs 2026-11-07 02:27:15,319 WARNING [get_onc_ferry] ONC TWDP PYRGEOMETER downward_radiation data for 2026-11-06 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs 2026-11-07 02:27:28,006 INFO [get_onc_ferry] 2026-11-06 ONC TWDP ferry data file created 2026-11-07 02:27:28,171 INFO [manager] checklist updated with [ONC ferry data] items from get_onc_ferry worker 2026-11-07 02:27:28,331 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['TWDP-ferry'], host='localhost') 2026-11-07 02:27:29,584 INFO [ping_erddap] running in process 853991 2026-11-07 02:27:29,585 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2026-11-07 02:27:29,586 INFO [ping_erddap] publishing log messages to tcp://*:5562 2026-11-07 02:27:29,587 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555 2026-11-07 02:27:29,589 INFO [ping_erddap] TWDP-ferry ERDDAP dataset flag file(s) created 2026-11-07 02:27:29,592 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker 2026-11-07 02:28:26,316 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 19613175 = 2026-11-08 04:09:20 UTC, 13.9% complete 2026-11-07 02:30:26,318 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 19613627 = 2026-11-08 09:10:40 UTC, 30.6% complete 2026-11-07 02:32:26,319 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 19614074 = 2026-11-08 14:08:40 UTC, 47.2% complete 2026-11-07 02:34:26,320 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 19614524 = 2026-11-08 19:08:40 UTC, 63.8% complete 2026-11-07 02:36:26,322 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 19614979 = 2026-11-09 00:12:00 UTC, 80.7% complete 2026-11-07 02:38:26,324 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 19615437 = 2026-11-09 05:17:20 UTC, 97.7% complete 2026-11-07 02:40:26,329 INFO [watch_NEMO] forecast2 NEMO run on arbutus.cloud-nowcast completed 2026-11-07 02:40:26,334 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker 2026-11-07 02:40:27,065 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_wind_file', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2026-11-07'], host='arbutus.cloud-nowcast') 2026-11-07 02:40:27,079 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_current_file', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2026-11-07'], host='arbutus.cloud-nowcast') 2026-11-07 02:40:27,084 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2026-11-06'], host='localhost') 2026-11-07 02:40:28,386 INFO [download_results] running in process 854301 2026-11-07 02:40:28,387 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2026-11-07 02:40:28,388 INFO [download_results] publishing log messages to tcp://*:5562 2026-11-07 02:40:28,389 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2026-11-07 02:40:28,390 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/forecast2/06nov26 to /results/SalishSea/forecast2.202111 2026-11-07 02:40:28,421 INFO [make_ww3_wind_file] running in process 231530 2026-11-07 02:40:28,422 INFO [make_ww3_wind_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2026-11-07 02:40:28,423 INFO [make_ww3_wind_file] publishing log messages to tcp://*:5570 2026-11-07 02:40:28,425 INFO [make_ww3_wind_file] connected to skookum.eos.ubc.ca port 5555 2026-11-07 02:40:28,426 INFO [make_ww3_wind_file] Creating wwatch3 wind forcing file for 2026-11-07 forecast2 run 2026-11-07 02:40:28,441 INFO [make_ww3_current_file] running in process 231531 2026-11-07 02:40:28,442 INFO [make_ww3_current_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2026-11-07 02:40:28,444 INFO [make_ww3_current_file] publishing log messages to tcp://*:5571 2026-11-07 02:40:28,446 INFO [make_ww3_current_file] connected to skookum.eos.ubc.ca port 5555 2026-11-07 02:40:28,447 INFO [make_ww3_current_file] Creating wwatch3 currents forcing file for 2026-11-07 forecast2 run 2026-11-07 02:40:38,717 INFO [make_ww3_wind_file] wwatch3 wind forcing file created on arbutus.cloud-nowcast for 2026-11-07 forecast2 run 2026-11-07 02:40:38,721 INFO [manager] checklist updated with [WW3 wind forcing] items from make_ww3_wind_file worker 2026-11-07 02:41:09,909 INFO [download_results] forecast2 2026-11-06 results files from arbutus.cloud-nowcast downloaded 2026-11-07 02:41:09,914 INFO [manager] checklist updated with [results files] items from download_results worker 2026-11-07 02:41:10,739 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['forecast2', '--run-date', '2026-11-06'], host='localhost') 2026-11-07 02:41:12,965 INFO [make_CHS_currents_file] running in process 854319 2026-11-07 02:41:12,967 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2026-11-07 02:41:12,968 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5562 2026-11-07 02:41:12,969 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555 2026-11-07 02:41:50,845 INFO [make_CHS_currents_file] Made CHS currents file for 2026-11-06 for forecast2 2026-11-07 02:41:50,849 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker 2026-11-07 02:41:51,184 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['nemo', 'forecast2', '--run-date', '2026-11-06'], host='localhost') 2026-11-07 02:41:52,480 INFO [update_forecast_datasets] running in process 854355 2026-11-07 02:41:52,481 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2026-11-07 02:41:52,482 INFO [update_forecast_datasets] publishing log messages to tcp://*:5562 2026-11-07 02:41:52,483 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555 2026-11-07 02:41:52,484 INFO [update_forecast_datasets] updating nemo forecast directory for forecast2/06nov26 run 2026-11-07 02:43:50,868 INFO [make_ww3_current_file] wwatch3 currents forcing file created on arbutus.cloud-nowcast for 2026-11-07 forecast2 run 2026-11-07 02:43:50,873 INFO [manager] checklist updated with [WW3 currents forcing] items from make_ww3_current_file worker 2026-11-07 02:43:51,206 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2026-11-07'], host='arbutus.cloud-nowcast') 2026-11-07 02:43:52,778 INFO [run_ww3] running in process 231918 2026-11-07 02:43:52,783 INFO [run_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2026-11-07 02:43:52,784 INFO [run_ww3] publishing log messages to tcp://*:5572 2026-11-07 02:43:52,785 INFO [run_ww3] connected to skookum.eos.ubc.ca port 5555 2026-11-07 02:43:52,789 INFO [run_ww3] Created run directory /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast2_2026-11-07T024352.614789-0700 2026-11-07 02:43:52,789 INFO [run_ww3] forecast2: launching /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast2_2026-11-07T024352.614789-0700/SoGWW3.sh on arbutus.cloud-nowcast 2026-11-07 02:43:52,791 INFO [run_ww3] forecast2 WaveWatch3 run for 2026-11-07 on arbutus.cloud-nowcast started 2026-11-07 02:43:52,792 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker 2026-11-07 02:43:53,000 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'forecast2'], host='arbutus.cloud-nowcast') 2026-11-07 02:43:53,859 INFO [watch_ww3] running in process 231935 2026-11-07 02:43:53,860 INFO [watch_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2026-11-07 02:43:53,861 INFO [watch_ww3] publishing log messages to tcp://*:5573 2026-11-07 02:43:53,863 INFO [watch_ww3] connected to skookum.eos.ubc.ca port 5555 2026-11-07 02:43:53,921 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch... 2026-11-07 02:45:53,921 INFO [watch_ww3] forecast2 WWATCH3 run on arbutus.cloud-nowcast completed 2026-11-07 02:45:53,926 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker 2026-11-07 02:45:54,467 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2026-11-07'], host='localhost') 2026-11-07 02:45:55,678 INFO [download_wwatch3_results] running in process 854476 2026-11-07 02:45:55,679 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2026-11-07 02:45:55,680 INFO [download_wwatch3_results] publishing log messages to tcp://*:5564 2026-11-07 02:45:55,681 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555 2026-11-07 02:46:03,792 INFO [download_wwatch3_results] forecast2 2026-11-07 results files from arbutus.cloud-nowcast downloaded 2026-11-07 02:46:03,795 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker 2026-11-07 02:46:04,130 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['wwatch3', 'forecast2', '--run-date', '2026-11-07'], host='localhost') 2026-11-07 02:46:05,334 INFO [update_forecast_datasets] running in process 854494 2026-11-07 02:46:05,335 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2026-11-07 02:46:05,336 INFO [update_forecast_datasets] publishing log messages to tcp://*:5564 2026-11-07 02:46:05,337 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555 2026-11-07 02:46:05,338 INFO [update_forecast_datasets] updating wwatch3 most_recent_forecast directory from forecast2/07nov26 run 2026-11-07 02:46:05,340 INFO [update_forecast_datasets] updated wwatch3 most_recent_forecast directory from forecast2/07nov26 run 2026-11-07 02:46:05,342 INFO [update_forecast_datasets] updating wwatch3 forecast directory for forecast2/07nov26 run 2026-11-07 02:46:05,620 INFO [update_forecast_datasets] updated wwatch3 forecast directory for forecast2/07nov26 run: /results/SalishSea/rolling-forecasts/wwatch3 2026-11-07 02:46:05,622 INFO [update_forecast_datasets] wwatch3 2026-11-07 forecast2 rolling forecast datasets updated 2026-11-07 02:46:05,625 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker 2026-11-07 02:46:05,958 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['wwatch3-forecast'], host='localhost') 2026-11-07 02:46:07,112 INFO [ping_erddap] running in process 854499 2026-11-07 02:46:07,113 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2026-11-07 02:46:07,114 INFO [ping_erddap] publishing log messages to tcp://*:5564 2026-11-07 02:46:07,115 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555 2026-11-07 02:46:07,117 INFO [ping_erddap] wwatch3-forecast ERDDAP dataset flag file(s) created 2026-11-07 02:46:07,119 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker 2026-11-07 02:46:07,439 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['wwatch3', 'forecast', 'publish', '--run-date', '2026-11-06'], host='localhost') 2026-11-07 02:46:11,572 INFO [make_plots] running in process 854504 2026-11-07 02:46:11,574 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2026-11-07 02:46:11,575 INFO [make_plots] publishing log messages to tcp://*:5564 2026-11-07 02:46:11,576 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555 2026-11-07 02:46:11,577 INFO [make_plots] preparing render list for 2026-11-06 WaveWatch3 forecast publish figures 2026-11-07 02:46:11,578 INFO [make_plots] added Halibut Bank figure to 2026-11-06 WaveWatch3 forecast publish render list 2026-11-07 02:46:11,579 INFO [make_plots] added Sentry Shoal figure to 2026-11-06 WaveWatch3 forecast publish render list 2026-11-07 02:46:11,580 INFO [make_plots] starting to render wwatch3 forecast publish 06nov26 figures 2026-11-07 02:46:13,866 CRITICAL [make_plots] unhandled exception: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/.pixi/envs/default/lib/python3.14/site-packages/xarray/backends/file_manager.py", line 211, in _acquire_with_cache_info file = self._cache[self._key] ~~~~~~~~~~~^^^^^^^^^^^ File "/SalishSeaCast/SalishSeaNowcast/.pixi/envs/default/lib/python3.14/site-packages/xarray/backends/lru_cache.py", line 56, in __getitem__ value = self._cache[key] ~~~~~~~~~~~^^^^^ KeyError: [, ('https://salishsea.eos.ubc.ca/erddap/griddap/ubcSSf2DWaveFields30mV17-02',), 'r', (('clobber', True), ('diskless', False), ('format', 'NETCDF4'), ('persist', False)), '3aad99f5-8d03-4a08-ac46-527dd9839d15'] During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/.pixi/envs/default/lib/python3.14/site-packages/nemo_nowcast/worker.py", line 388, in _do_work checklist = self.worker_func( self._parsed_args, self.config, self.tell_manager ) File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 233, in make_plots checklist = _render_figures( config, model, run_type, plot_type, dmy, fig_functions, test_figure_id ) File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 767, in _render_figures fig = _calc_figure(fig_func, args, kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 834, in _calc_figure fig = fig_func(*args, **kwargs) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/wwatch3/wave_height_period.py", line 68, in make_figure plot_data = _prep_plot_data(buoy, wwatch3_dataset_url) File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/wwatch3/wave_height_period.py", line 78, in _prep_plot_data wwatch3_fields = xarray.open_dataset(wwatch3_dataset_url) File "/SalishSeaCast/SalishSeaNowcast/.pixi/envs/default/lib/python3.14/site-packages/xarray/backends/api.py", line 760, in open_dataset backend_ds = backend.open_dataset( filename_or_obj, ...<2 lines>... **kwargs, ) File "/SalishSeaCast/SalishSeaNowcast/.pixi/envs/default/lib/python3.14/site-packages/xarray/backends/netCDF4_.py", line 682, in open_dataset store = NetCDF4DataStore.open( filename_or_obj, ...<8 lines>... autoclose=autoclose, ) File "/SalishSeaCast/SalishSeaNowcast/.pixi/envs/default/lib/python3.14/site-packages/xarray/backends/netCDF4_.py", line 468, in open return cls(manager, group=group, mode=mode, lock=lock, autoclose=autoclose) File "/SalishSeaCast/SalishSeaNowcast/.pixi/envs/default/lib/python3.14/site-packages/xarray/backends/netCDF4_.py", line 398, in __init__ self.format = self.ds.data_model ^^^^^^^ File "/SalishSeaCast/SalishSeaNowcast/.pixi/envs/default/lib/python3.14/site-packages/xarray/backends/netCDF4_.py", line 477, in ds return self._acquire() ~~~~~~~~~~~~~^^ File "/SalishSeaCast/SalishSeaNowcast/.pixi/envs/default/lib/python3.14/site-packages/xarray/backends/netCDF4_.py", line 471, in _acquire with self._manager.acquire_context(needs_lock) as root: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^ File "/SalishSeaCast/SalishSeaNowcast/.pixi/envs/default/lib/python3.14/contextlib.py", line 141, in __enter__ return next(self.gen) File "/SalishSeaCast/SalishSeaNowcast/.pixi/envs/default/lib/python3.14/site-packages/xarray/backends/file_manager.py", line 199, in acquire_context file, cached = self._acquire_with_cache_info(needs_lock) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^ File "/SalishSeaCast/SalishSeaNowcast/.pixi/envs/default/lib/python3.14/site-packages/xarray/backends/file_manager.py", line 217, in _acquire_with_cache_info file = self._opener(*self._args, **kwargs) File "src/netCDF4/_netCDF4.pyx", line 2522, in netCDF4._netCDF4.Dataset.__init__ File "src/netCDF4/_netCDF4.pyx", line 2159, in netCDF4._netCDF4._ensure_nc_success OSError: [Errno -70] NetCDF: DAP server error: 'https://salishsea.eos.ubc.ca/erddap/griddap/ubcSSf2DWaveFields30mV17-02' 2026-11-07 02:46:24,666 INFO [update_forecast_datasets] updated nemo forecast directory for forecast2/06nov26 run: /results/SalishSea/rolling-forecasts/nemo 2026-11-07 02:46:24,667 INFO [update_forecast_datasets] nemo 2026-11-06 forecast2 rolling forecast datasets updated 2026-11-07 02:46:24,669 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker 2026-11-07 02:46:25,011 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nemo-forecast'], host='localhost') 2026-11-07 02:46:25,018 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'forecast2', 'publish', '--run-date', '2026-11-06'], host='localhost') 2026-11-07 02:46:25,023 INFO [manager] launching NextWorker(module='nowcast.workers.make_surface_current_tiles', args=['forecast2', '--run-date', '2026-11-06'], host='localhost') 2026-11-07 02:46:26,223 INFO [ping_erddap] running in process 854555 2026-11-07 02:46:26,224 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2026-11-07 02:46:26,225 INFO [ping_erddap] publishing log messages to tcp://*:5562 2026-11-07 02:46:26,226 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555 2026-11-07 02:46:26,235 INFO [ping_erddap] nemo-forecast ERDDAP dataset flag file(s) created 2026-11-07 02:46:26,241 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker 2026-11-07 02:46:27,698 INFO [make_surface_current_tiles] running in process 854558 2026-11-07 02:46:27,701 INFO [make_surface_current_tiles] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2026-11-07 02:46:27,703 INFO [make_surface_current_tiles] publishing log messages to tcp://*:5562 2026-11-07 02:46:27,704 INFO [make_surface_current_tiles] connected to skookum.eos.ubc.ca port 5555 2026-11-07 02:46:27,705 INFO [make_surface_current_tiles] starting rendering of tiles for 2026-11-06 forecast2 in 6 processes into /results/nowcast-sys/figures/surface_currents/forecast2/06nov26 2026-11-07 02:46:28,244 INFO [make_plots] running in process 854557 2026-11-07 02:46:28,245 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2026-11-07 02:46:28,245 INFO [make_plots] publishing log messages to tcp://*:5564 2026-11-07 02:46:28,246 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555 2026-11-07 02:46:28,761 INFO [make_plots] preparing render list for 2026-11-06 NEMO forecast2 publish figures 2026-11-07 02:46:28,848 INFO [make_plots] added Website_thumbnail figure to 2026-11-06 NEMO forecast2 publish render list 2026-11-07 02:46:28,849 INFO [make_plots] added Threshold_website figure to 2026-11-06 NEMO forecast2 publish render list 2026-11-07 02:46:28,850 INFO [make_plots] added PA_tidal_predictions figure to 2026-11-06 NEMO forecast2 publish render list 2026-11-07 02:46:28,851 INFO [make_plots] added Boundary Bay figure to 2026-11-06 NEMO forecast2 publish render list 2026-11-07 02:46:28,852 INFO [make_plots] added Campbell River figure to 2026-11-06 NEMO forecast2 publish render list 2026-11-07 02:46:28,853 INFO [make_plots] added Cherry Point figure to 2026-11-06 NEMO forecast2 publish render list 2026-11-07 02:46:28,854 INFO [make_plots] added Friday Harbor figure to 2026-11-06 NEMO forecast2 publish render list 2026-11-07 02:46:28,855 INFO [make_plots] added Halfmoon Bay figure to 2026-11-06 NEMO forecast2 publish render list 2026-11-07 02:46:28,856 INFO [make_plots] added Nanaimo figure to 2026-11-06 NEMO forecast2 publish render list 2026-11-07 02:46:28,856 INFO [make_plots] added Neah Bay figure to 2026-11-06 NEMO forecast2 publish render list 2026-11-07 02:46:28,857 INFO [make_plots] added New Westminster figure to 2026-11-06 NEMO forecast2 publish render list 2026-11-07 02:46:28,858 INFO [make_plots] added Patricia Bay figure to 2026-11-06 NEMO forecast2 publish render list 2026-11-07 02:46:28,858 INFO [make_plots] added Point Atkinson figure to 2026-11-06 NEMO forecast2 publish render list 2026-11-07 02:46:28,859 INFO [make_plots] added Port Renfrew figure to 2026-11-06 NEMO forecast2 publish render list 2026-11-07 02:46:28,860 INFO [make_plots] added Sand Heads figure to 2026-11-06 NEMO forecast2 publish render list 2026-11-07 02:46:28,861 INFO [make_plots] added Sandy Cove figure to 2026-11-06 NEMO forecast2 publish render list 2026-11-07 02:46:28,862 INFO [make_plots] added Squamish figure to 2026-11-06 NEMO forecast2 publish render list 2026-11-07 02:46:28,863 INFO [make_plots] added Victoria figure to 2026-11-06 NEMO forecast2 publish render list 2026-11-07 02:46:28,864 INFO [make_plots] added Woodwards Landing figure to 2026-11-06 NEMO forecast2 publish render list 2026-11-07 02:46:28,864 INFO [make_plots] starting to render nemo forecast2 publish 06nov26 figures 2026-11-07 02:53:51,740 WARNING [make_plots] invalid station number for Boundary Bay station: None 2026-11-07 02:53:51,741 WARNING [make_plots] can't resolve a valid CHS station code for Boundary Bay 2026-11-07 02:54:18,789 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb996/data for station 08074 Campbell River from 2026-11-02 00:00:00Z to 2026-11-09 05:55:00Z 2026-11-07 02:55:27,761 WARNING [make_plots] invalid station number for Halfmoon Bay station: None 2026-11-07 02:55:27,762 WARNING [make_plots] can't resolve a valid CHS station code for Halfmoon Bay 2026-11-07 02:55:52,568 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb96d/data for station 07917 Nanaimo from 2026-11-02 00:00:00Z to 2026-11-09 05:55:00Z 2026-11-07 02:56:35,530 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e13d0f4a073c4bbf93/data for station 07654 New Westminster from 2026-11-02 00:00:00Z to 2026-11-09 05:55:00Z 2026-11-07 02:56:58,948 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd2d/data for station 07277 Patricia Bay from 2026-11-02 00:00:00Z to 2026-11-09 05:55:00Z 2026-11-07 02:57:25,470 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94c/data for station 07795 Point Atkinson from 2026-11-02 00:00:00Z to 2026-11-09 05:55:00Z 2026-11-07 02:57:50,381 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e23d0f4a073c4bc060/data for station 08525 Port Renfrew from 2026-11-02 00:00:00Z to 2026-11-09 05:55:00Z 2026-11-07 02:58:12,324 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb93c/data for station 07594 Sand Heads from 2026-11-02 00:00:00Z to 2026-11-09 05:55:00Z 2026-11-07 02:58:34,308 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc43b/data for station 07786 Sandy Cove from 2026-11-02 00:00:00Z to 2026-11-09 05:55:00Z 2026-11-07 02:58:58,243 WARNING [make_plots] invalid station number for Squamish station: None 2026-11-07 02:58:58,244 WARNING [make_plots] can't resolve a valid CHS station code for Squamish 2026-11-07 02:59:18,092 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd1e/data for station 07120 Victoria from 2026-11-02 00:00:00Z to 2026-11-09 05:55:00Z 2026-11-07 02:59:26,919 INFO [make_surface_current_tiles] starting PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast2/06nov26 2026-11-07 02:59:34,362 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc42d/data for station 07610 Woodwards Landing from 2026-11-02 00:00:00Z to 2026-11-09 05:55:00Z 2026-11-07 02:59:34,853 INFO [make_plots] no obs water level data available from 5cebf1e43d0f4a073c4bc42d during 2026-11-02 00:00:00Z to 2026-11-09 05:55:00Z 2026-11-07 02:59:51,724 INFO [make_plots] finished rendering nemo forecast2 publish 06nov26 figures 2026-11-07 02:59:51,726 INFO [make_plots] nemo publish plots for 2026-11-06 forecast2 completed 2026-11-07 02:59:51,735 INFO [manager] checklist updated with [plots] items from make_plots worker 2026-11-07 02:59:52,061 INFO [manager] launching NextWorker(module='nowcast.workers.make_feeds', args=['forecast2', '--run-date', '2026-11-06'], host='localhost') 2026-11-07 02:59:55,620 INFO [make_feeds] running in process 856966 2026-11-07 02:59:55,621 INFO [make_feeds] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2026-11-07 02:59:55,623 INFO [make_feeds] publishing log messages to tcp://*:5564 2026-11-07 02:59:55,623 INFO [make_feeds] connected to skookum.eos.ubc.ca port 5555 2026-11-07 03:00:10,898 INFO [make_feeds] ATOM feeds for 2026-11-06 forecast2 run completed 2026-11-07 03:00:10,902 INFO [manager] checklist updated with [storm surge alert feeds] items from make_feeds worker 2026-11-07 03:00:11,227 INFO [manager] launching NextWorker(module='nemo_nowcast.workers.clear_checklist', args=[], host='localhost') 2026-11-07 03:00:12,284 INFO [clear_checklist] running in process 856989 2026-11-07 03:00:12,285 INFO [clear_checklist] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2026-11-07 03:00:12,286 INFO [clear_checklist] publishing log messages to tcp://*:5564 2026-11-07 03:00:12,287 INFO [clear_checklist] connected to skookum.eos.ubc.ca port 5555 2026-11-07 03:00:12,288 INFO [clear_checklist] requesting that manager clear system state checklist 2026-11-07 03:00:12,290 INFO [manager] writing checklist to log file 2026-11-07 03:00:12,338 INFO [manager] checklist cleared 2026-11-07 03:00:12,341 INFO [clear_checklist] nowcast system checklist cleared 2026-11-07 03:00:12,344 INFO [manager] launching NextWorker(module='nemo_nowcast.workers.rotate_logs', args=[], host='localhost') 2026-11-07 03:00:13,541 INFO [rotate_logs] running in process 856995 2026-11-07 03:00:13,542 INFO [rotate_logs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2026-11-07 03:00:13,543 INFO [rotate_logs] publishing log messages to tcp://*:5564 2026-11-07 03:00:13,544 INFO [rotate_logs] connected to skookum.eos.ubc.ca port 5555 2026-11-07 03:00:13,545 INFO [rotate_logs] rotating log files