2026-09-08 04:24:55,261 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/nowcast.log
2026-09-08 04:24:55,270 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/nowcast.debug.log
2026-09-08 04:24:55,275 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/checklist.log
2026-09-08 04:24:55,276 INFO [rotate_logs] log files rotated
2026-09-08 04:24:55,281 INFO [manager] checklist updated with [log rotation] items from rotate_logs worker
2026-09-08 04:26:46,190 INFO [make_surface_current_tiles] finished PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast2/07sep26
2026-09-08 04:26:46,207 INFO [make_surface_current_tiles] finished rendering of tiles for 2026-09-07 forecast2 into /results/nowcast-sys/figures/surface_currents/forecast2/07sep26
2026-09-08 04:26:46,208 INFO [make_surface_current_tiles] surface current tile figures for 2026-09-07 forecast2 completed
2026-09-08 04:26:46,487 INFO [manager] checklist updated with [surface current tiles] items from make_surface_current_tiles worker
2026-09-08 09:11:14,626 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/12/ to /results/forcing/atmospheric/continental2.5/GRIB/20260908/12/
2026-09-08 09:11:14,628 INFO [collect_weather] 2026-09-08 2.5km weather forecast 12 collection complete
2026-09-08 09:11:14,630 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2026-09-08 09:11:15,001 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'SkagitMountVernon', '--data-date', '2026-09-07'], host='localhost')
2026-09-08 09:11:15,008 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'SnohomishMonroe', '--data-date', '2026-09-07'], host='localhost')
2026-09-08 09:11:15,013 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'NisquallyMcKenna', '--data-date', '2026-09-07'], host='localhost')
2026-09-08 09:11:15,019 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'GreenwaterGreenwater', '--data-date', '2026-09-07'], host='localhost')
2026-09-08 09:11:15,024 INFO [manager] launching NextWorker(module='nowcast.workers.make_turbidity_file', args=[], host='localhost')
2026-09-08 09:11:15,033 INFO [manager] launching NextWorker(module='nowcast.workers.collect_NeahBay_ssh', args=['06'], host='localhost')
2026-09-08 09:11:15,037 INFO [manager] launching NextWorker(module='nowcast.workers.download_live_ocean', args=[], host='localhost')
2026-09-08 09:11:15,046 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['18', '2.5km'], host='localhost')
2026-09-08 09:11:15,055 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['18'], host='localhost')
2026-09-08 09:11:16,890 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20260908/12/
2026-09-08 09:11:16,891 INFO [crop_gribs] 2026-09-08 12 GRIBs cropping complete
2026-09-08 09:11:16,897 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2026-09-08 09:11:17,233 INFO [collect_weather] running in process 3146767
2026-09-08 09:11:17,234 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-08 09:11:17,235 INFO [collect_weather] publishing log messages to tcp://*:5562
2026-09-08 09:11:17,236 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2026-09-08 09:11:17,245 INFO [collect_NeahBay_ssh] running in process 3146765
2026-09-08 09:11:17,246 INFO [collect_NeahBay_ssh] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-08 09:11:17,247 INFO [collect_NeahBay_ssh] publishing log messages to tcp://*:5563
2026-09-08 09:11:17,248 INFO [collect_NeahBay_ssh] connected to skookum.eos.ubc.ca port 5555
2026-09-08 09:11:17,249 INFO [collect_NeahBay_ssh] collecting Neah Bay ssh 06Z obs/forecast for 2026-09-08
2026-09-08 09:11:17,252 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/18/
2026-09-08 09:11:17,284 INFO [manager] launching NextWorker(module='nowcast.workers.grib_to_netcdf', args=['nowcast+'], host='localhost')
2026-09-08 09:11:17,406 INFO [download_live_ocean] running in process 3146766
2026-09-08 09:11:17,407 INFO [download_live_ocean] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-08 09:11:17,408 INFO [download_live_ocean] publishing log messages to tcp://*:5564
2026-09-08 09:11:17,408 INFO [download_live_ocean] connected to skookum.eos.ubc.ca port 5555
2026-09-08 09:11:17,409 INFO [download_live_ocean] downloading Salish Sea western boundary day-averaged LiveOcean file for 2026-09-08
2026-09-08 09:11:17,833 INFO [collect_river_data] running in process 3146759
2026-09-08 09:11:17,834 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-08 09:11:17,835 INFO [collect_river_data] publishing log messages to tcp://*:5565
2026-09-08 09:11:17,836 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-09-08 09:11:17,836 INFO [collect_river_data] Collecting USGS SkagitMountVernon river data for 2026-09-07
2026-09-08 09:11:17,900 INFO [collect_river_data] running in process 3146763
2026-09-08 09:11:17,901 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-08 09:11:17,902 INFO [collect_river_data] publishing log messages to tcp://*:5566
2026-09-08 09:11:17,903 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-09-08 09:11:17,904 INFO [collect_river_data] Collecting USGS GreenwaterGreenwater river data for 2026-09-07
2026-09-08 09:11:17,924 INFO [collect_river_data] running in process 3146761
2026-09-08 09:11:17,925 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-08 09:11:17,926 INFO [collect_river_data] publishing log messages to tcp://*:5567
2026-09-08 09:11:17,927 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-09-08 09:11:17,928 INFO [collect_river_data] Collecting USGS SnohomishMonroe river data for 2026-09-07
2026-09-08 09:11:17,961 INFO [collect_river_data] running in process 3146762
2026-09-08 09:11:17,962 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-08 09:11:17,963 INFO [collect_river_data] publishing log messages to tcp://*:5568
2026-09-08 09:11:17,964 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-09-08 09:11:17,965 INFO [collect_river_data] Collecting USGS NisquallyMcKenna river data for 2026-09-07
2026-09-08 09:11:18,056 INFO [make_turbidity_file] running in process 3146764
2026-09-08 09:11:18,057 INFO [make_turbidity_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-08 09:11:18,058 INFO [make_turbidity_file] publishing log messages to tcp://*:5569
2026-09-08 09:11:18,059 INFO [make_turbidity_file] connected to skookum.eos.ubc.ca port 5555
2026-09-08 09:11:18,060 INFO [make_turbidity_file] Creating Fraser River turbidity forcing file for 2026-09-08
2026-09-08 09:11:18,351 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12200500&startDT=2026-09-07&endDT=2026-09-07 "HTTP/1.1 200 "
2026-09-08 09:11:18,520 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12089500&startDT=2026-09-07&endDT=2026-09-07 "HTTP/1.1 200 "
2026-09-08 09:11:18,526 INFO [crop_gribs] running in process 3146768
2026-09-08 09:11:18,528 INFO [collect_river_data] Appended USGS SkagitMountVernon river average discharge for 2026-09-07 to: /results/forcing/rivers/observations/Skagit_MountVernon_flow
2026-09-08 09:11:18,528 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-09-08 09:11:18,529 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12097500&startDT=2026-09-07&endDT=2026-09-07 "HTTP/1.1 200 "
2026-09-08 09:11:18,530 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-08 09:11:18,531 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12150800&startDT=2026-09-07&endDT=2026-09-07 "HTTP/1.1 200 "
2026-09-08 09:11:18,531 INFO [collect_river_data] USGS SkagitMountVernon river data collection for 2026-09-07 completed
2026-09-08 09:11:18,533 INFO [crop_gribs] publishing log messages to tcp://*:5570
2026-09-08 09:11:18,535 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2026-09-08 09:11:18,537 INFO [crop_gribs] cropping 2026-09-08 ECCC HRDPS 2.5km continental 18Z GRIB files to SalishSeaCast subdomain
2026-09-08 09:11:18,540 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20260908/18/
2026-09-08 09:11:18,687 INFO [collect_NeahBay_ssh] 2026-09-08T00:00:00-07:00 Neah Bay ssh 06Z obs/forecast data collection complete
2026-09-08 09:11:18,689 INFO [manager] checklist updated with [Neah Bay ssh data] items from collect_NeahBay_ssh worker
2026-09-08 09:11:18,834 INFO [manager] launching NextWorker(module='nowcast.workers.make_ssh_files', args=['nowcast', '--run-date', '2026-09-08'], host='localhost')
2026-09-08 09:11:19,424 INFO [collect_river_data] Appended USGS GreenwaterGreenwater river average discharge for 2026-09-07 to: /results/forcing/rivers/observations/Greenwater_Greenwater_flow
2026-09-08 09:11:19,427 INFO [collect_river_data] USGS GreenwaterGreenwater river data collection for 2026-09-07 completed
2026-09-08 09:11:19,429 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-09-08 09:11:19,455 INFO [collect_river_data] Appended USGS SnohomishMonroe river average discharge for 2026-09-07 to: /results/forcing/rivers/observations/Snohomish_Monroe_flow
2026-09-08 09:11:19,456 INFO [collect_river_data] USGS SnohomishMonroe river data collection for 2026-09-07 completed
2026-09-08 09:11:19,566 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-09-08 09:11:19,599 INFO [grib_to_netcdf] running in process 3146864
2026-09-08 09:11:19,600 INFO [grib_to_netcdf] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-08 09:11:19,602 INFO [grib_to_netcdf] publishing log messages to tcp://*:5563
2026-09-08 09:11:19,602 INFO [grib_to_netcdf] connected to skookum.eos.ubc.ca port 5555
2026-09-08 09:11:19,603 INFO [grib_to_netcdf] creating NEMO-atmos forcing files for 2026-09-08 nowcast and forecast runs
2026-09-08 09:11:19,633 INFO [collect_river_data] Appended USGS NisquallyMcKenna river average discharge for 2026-09-07 to: /results/forcing/rivers/observations/Nisqually_McKenna_flow
2026-09-08 09:11:19,634 INFO [collect_river_data] USGS NisquallyMcKenna river data collection for 2026-09-07 completed
2026-09-08 09:11:19,709 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-09-08 09:11:22,203 INFO [make_ssh_files] running in process 3146896
2026-09-08 09:11:22,204 INFO [make_ssh_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-08 09:11:22,205 INFO [make_ssh_files] publishing log messages to tcp://*:5565
2026-09-08 09:11:22,206 INFO [make_ssh_files] connected to skookum.eos.ubc.ca port 5555
2026-09-08 09:11:22,208 INFO [make_ssh_files] building nowcast sea surface height boundary conditions file(s) from 2026-09-08 NOAA Neah Bay 06Z observation and forecast values
2026-09-08 09:11:24,589 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2026m09d04.nc
2026-09-08 09:11:24,770 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2026m09d05.nc
2026-09-08 09:11:24,955 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2026m09d06.nc
2026-09-08 09:11:25,143 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2026m09d07.nc
2026-09-08 09:11:28,424 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2026m09d08.nc
2026-09-08 09:11:28,430 INFO [make_turbidity_file] 2026-09-08 Fraser River turbidity file creation complete
2026-09-08 09:11:28,432 INFO [manager] checklist updated with [Fraser River turbidity] items from make_turbidity_file worker
2026-09-08 09:11:28,648 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2026m09d09.nc
2026-09-08 09:11:29,246 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2026m09d10.nc
2026-09-08 09:11:29,786 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2026m09d11.nc
2026-09-08 09:11:29,929 INFO [make_ssh_files] sea surface height boundary file for 2026-09-08 nowcast run created
2026-09-08 09:11:29,932 INFO [manager] checklist updated with [sea surface height forcing] items from make_ssh_files worker
2026-09-08 09:11:30,069 INFO [manager] launching NextWorker(module='nowcast.workers.make_runoff_file', args=['v202108'], host='localhost')
2026-09-08 09:11:32,207 INFO [make_runoff_file] running in process 3146939
2026-09-08 09:11:32,208 INFO [make_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-08 09:11:32,210 INFO [make_runoff_file] publishing log messages to tcp://*:5565
2026-09-08 09:11:32,211 INFO [make_runoff_file] connected to skookum.eos.ubc.ca port 5555
2026-09-08 09:11:32,212 INFO [make_runoff_file] calculating NEMO runoff forcing for v202108 bathymetry for 2026-09-07
2026-09-08 09:11:42,448 INFO [make_runoff_file] stored NEMO runoff forcing for v202108 bathymetry for 2026-09-07: /results/forcing/rivers/R202108Dailies_y2026m09d07.nc
2026-09-08 09:11:42,449 INFO [make_runoff_file] 2026-09-07 runoff file creation completed
2026-09-08 09:11:42,451 INFO [manager] checklist updated with [rivers forcing] items from make_runoff_file worker
2026-09-08 09:12:38,401 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/hrdps_y2026m09d08.nc
2026-09-08 09:12:47,664 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2026m09d09.nc
2026-09-08 09:12:54,680 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2026m09d10.nc
2026-09-08 09:12:54,694 INFO [grib_to_netcdf] 2026-09-08 NEMO-atmos forcing file for nowcast+ created
2026-09-08 09:12:54,698 INFO [manager] checklist updated with [weather forcing] items from grib_to_netcdf worker
2026-09-08 12:11:22,364 ERROR [download_live_ocean] giving up after 37 attempts: 403 Client Error: Forbidden for url: https://s3.kopah.uw.edu/liveocean-share/f2026.09.08/ubc_done.txt for https://s3.kopah.uw.edu/liveocean-share/f2026.09.08/ubc_done.txt
2026-09-08 12:11:22,372 CRITICAL [download_live_ocean] 2026-09-08 Live Ocean file for Salish Sea western boundary download failed
2026-09-08 12:53:35,083 INFO [upload_forcing] running in process 3156493
2026-09-08 12:53:35,084 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-08 12:53:35,085 INFO [upload_forcing] publishing log messages to tcp://*:5563
2026-09-08 12:53:35,087 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-09-08 12:53:37,673 INFO [upload_forcing] nowcast+ 2026-09-08 forcing files upload to arbutus.cloud-nowcast completed
2026-09-08 12:53:37,677 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-09-08 12:53:37,822 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'nowcast+', '--run-date', '2026-09-08'], host='localhost')
2026-09-08 12:53:39,261 INFO [make_forcing_links] running in process 3156552
2026-09-08 12:53:39,262 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-08 12:53:39,264 INFO [make_forcing_links] publishing log messages to tcp://*:5563
2026-09-08 12:53:39,265 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2026-09-08 12:53:39,954 INFO [make_forcing_links] nowcast+ 2026-09-08 forcing file links on arbutus.cloud-nowcast created
2026-09-08 12:53:39,958 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2026-09-08 12:53:40,079 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2026-09-08'], host='arbutus.cloud-nowcast')
2026-09-08 12:53:41,429 INFO [run_NEMO] running in process 868958
2026-09-08 12:53:41,430 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-09-08 12:53:41,432 INFO [run_NEMO] publishing log messages to tcp://*:5556
2026-09-08 12:53:41,433 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2026-09-08 12:53:44,812 WARNING [run_NEMO] revision and status requested for non-existent repo: /nemoShare/MEOPAR/nowcast-sys/moad_tools
2026-09-08 12:53:44,813 WARNING [run_NEMO] revision and status requested for non-existent repo: /nemoShare/MEOPAR/nowcast-sys/NEMO-Cmd
2026-09-08 12:53:44,814 WARNING [run_NEMO] revision and status requested for non-existent repo: /nemoShare/MEOPAR/nowcast-sys/NEMO_Nowcast
2026-09-08 12:53:44,820 WARNING [run_NEMO] There are uncommitted changes in /nemoShare/MEOPAR/nowcast-sys/SalishSeaCmd
2026-09-08 12:53:44,829 WARNING [run_NEMO] revision and status requested for non-existent repo: /nemoShare/MEOPAR/nowcast-sys/tools
2026-09-08 12:53:44,834 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2026-09-08 12:53:44,840 INFO [run_NEMO] nowcast: launching /nemoShare/MEOPAR/nowcast-sys/runs/08sep26nowcast_2026-09-08T125341.522161-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2026-09-08 12:53:44,842 INFO [run_NEMO] nowcast NEMO run for 2026-09-08 on arbutus.cloud-nowcast started
2026-09-08 12:53:44,962 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'nowcast'], host='arbutus.cloud-nowcast')
2026-09-08 12:53:45,865 INFO [watch_NEMO] running in process 869112
2026-09-08 12:53:45,866 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-09-08 12:53:45,867 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2026-09-08 12:53:45,869 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2026-09-08 12:53:45,966 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2026-09-08 12:54:14,201 INFO [upload_forcing] running in process 3156716
2026-09-08 12:54:14,202 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-08 12:54:14,203 INFO [upload_forcing] publishing log messages to tcp://*:5563
2026-09-08 12:54:14,204 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-09-08 12:54:15,798 INFO [upload_forcing] nowcast+ 2026-09-08 forcing files upload to optimum-hindcast completed
2026-09-08 12:54:15,801 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-09-08 12:54:30,381 INFO [upload_forcing] running in process 3156874
2026-09-08 12:54:30,382 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-08 12:54:30,384 INFO [upload_forcing] publishing log messages to tcp://*:5563
2026-09-08 12:54:30,385 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-09-08 12:54:39,013 INFO [upload_forcing] nowcast+ 2026-09-08 forcing files upload to robot.nibi completed
2026-09-08 12:54:39,018 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-09-08 12:58:45,971 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 19482092 = 2026-09-08 11:40:40 UTC, 48.7% complete
2026-09-08 13:03:45,975 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 19483200 = 2026-09-08 23:59:20 UTC, 100.0% complete
2026-09-08 13:08:45,989 INFO [watch_NEMO] nowcast NEMO run on arbutus.cloud-nowcast completed
2026-09-08 13:08:45,993 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2026-09-08 13:08:46,304 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'ssh', '--run-date', '2026-09-08'], host='localhost')
2026-09-08 13:08:46,311 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2026-09-08'], host='localhost')
2026-09-08 13:08:47,841 INFO [download_results] running in process 3157363
2026-09-08 13:08:47,842 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-08 13:08:47,844 INFO [download_results] publishing log messages to tcp://*:5563
2026-09-08 13:08:47,845 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2026-09-08 13:08:47,846 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/nowcast/08sep26 to /results/SalishSea/nowcast-blue.202111
2026-09-08 13:08:47,940 INFO [make_forcing_links] running in process 3157362
2026-09-08 13:08:47,942 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-08 13:08:47,943 INFO [make_forcing_links] publishing log messages to tcp://*:5564
2026-09-08 13:08:47,944 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2026-09-08 13:08:48,888 INFO [make_forcing_links] ssh 2026-09-08 forcing file links on arbutus.cloud-nowcast created
2026-09-08 13:08:48,891 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2026-09-08 13:08:49,042 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2026-09-08'], host='arbutus.cloud-nowcast')
2026-09-08 13:08:50,051 INFO [run_NEMO] running in process 869333
2026-09-08 13:08:50,052 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-09-08 13:08:50,054 INFO [run_NEMO] publishing log messages to tcp://*:5556
2026-09-08 13:08:50,056 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2026-09-08 13:08:50,644 WARNING [run_NEMO] revision and status requested for non-existent repo: /nemoShare/MEOPAR/nowcast-sys/moad_tools
2026-09-08 13:08:50,646 WARNING [run_NEMO] revision and status requested for non-existent repo: /nemoShare/MEOPAR/nowcast-sys/NEMO-Cmd
2026-09-08 13:08:50,648 WARNING [run_NEMO] revision and status requested for non-existent repo: /nemoShare/MEOPAR/nowcast-sys/NEMO_Nowcast
2026-09-08 13:08:51,016 WARNING [run_NEMO] There are uncommitted changes in /nemoShare/MEOPAR/nowcast-sys/SalishSeaCmd
2026-09-08 13:08:51,696 WARNING [run_NEMO] revision and status requested for non-existent repo: /nemoShare/MEOPAR/nowcast-sys/tools
2026-09-08 13:08:52,151 INFO [run_NEMO] forecast: launching /nemoShare/MEOPAR/nowcast-sys/runs/08sep26forecast_2026-09-08T130850.129985-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2026-09-08 13:08:52,198 INFO [run_NEMO] forecast NEMO run for 2026-09-08 on arbutus.cloud-nowcast started
2026-09-08 13:08:52,203 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2026-09-08 13:08:52,361 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'forecast'], host='arbutus.cloud-nowcast')
2026-09-08 13:08:53,286 INFO [watch_NEMO] running in process 869487
2026-09-08 13:08:53,287 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-09-08 13:08:53,288 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2026-09-08 13:08:53,290 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2026-09-08 13:08:53,390 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2026-09-08 13:09:42,437 INFO [download_results] nowcast 2026-09-08 results files from arbutus.cloud-nowcast downloaded
2026-09-08 13:09:42,442 INFO [manager] checklist updated with [results files] items from download_results worker
2026-09-08 13:09:42,586 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast', 'research', '--run-date', '2026-09-08'], host='localhost')
2026-09-08 13:09:42,592 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast', 'comparison', '--run-date', '2026-09-07'], host='localhost')
2026-09-08 13:09:42,597 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['nowcast', '--run-date', '2026-09-08'], host='localhost')
2026-09-08 13:09:45,822 INFO [make_CHS_currents_file] running in process 3157396
2026-09-08 13:09:45,824 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-08 13:09:45,826 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5563
2026-09-08 13:09:45,829 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555
2026-09-08 13:09:47,970 INFO [make_plots] running in process 3157393
2026-09-08 13:09:47,971 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-08 13:09:47,972 INFO [make_plots] publishing log messages to tcp://*:5564
2026-09-08 13:09:47,973 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2026-09-08 13:09:47,981 INFO [make_plots] running in process 3157395
2026-09-08 13:09:48,007 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-08 13:09:48,016 INFO [make_plots] publishing log messages to tcp://*:5565
2026-09-08 13:09:48,022 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2026-09-08 13:09:48,464 INFO [make_plots] preparing render list for 2026-09-08 NEMO nowcast-blue research figures
2026-09-08 13:09:48,466 INFO [make_plots] preparing render list for 2026-09-07 NEMO nowcast-blue comparison figures
2026-09-08 13:09:48,586 INFO [make_plots] added SH_wind figure to 2026-09-07 NEMO nowcast-blue comparison render list
2026-09-08 13:09:48,587 INFO [make_plots] added Compare_VENUS_East figure to 2026-09-07 NEMO nowcast-blue comparison render list
2026-09-08 13:09:48,588 INFO [make_plots] added Compare_VENUS_Central figure to 2026-09-07 NEMO nowcast-blue comparison render list
2026-09-08 13:09:48,589 INFO [make_plots] starting to render nemo nowcast comparison 07sep26 figures
2026-09-08 13:09:49,828 INFO [make_plots] added salinity_thalweg_and_surface figures to 2026-09-08 NEMO nowcast-blue research render list
2026-09-08 13:09:50,776 INFO [make_plots] added temperature_thalweg_and_surface figures to 2026-09-08 NEMO nowcast-blue research render list
2026-09-08 13:09:50,777 INFO [make_plots] added Currents_sections_and_surface figure to 2026-09-08 NEMO nowcast-blue research render list
2026-09-08 13:09:50,778 INFO [make_plots] added Currents_at_VENUS_Central figure to 2026-09-08 NEMO nowcast-blue research render list
2026-09-08 13:09:50,779 INFO [make_plots] added Currents_at_VENUS_East figure to 2026-09-08 NEMO nowcast-blue research render list
2026-09-08 13:09:50,780 INFO [make_plots] starting to render nemo nowcast research 08sep26 figures
2026-09-08 13:09:51,639 WARNING [make_plots] findfont: Failed to find font weight medium, now using 400.
2026-09-08 13:09:51,643 WARNING [make_plots] findfont: Failed to find font weight medium, now using 400.
2026-09-08 13:10:18,599 WARNING [make_plots] findfont: Failed to find font weight medium, now using 400.
2026-09-08 13:10:18,600 WARNING [make_plots] findfont: Failed to find font weight medium, now using 400.
2026-09-08 13:10:18,803 WARNING [make_plots] findfont: Failed to find font weight medium, now using 400.
2026-09-08 13:10:18,805 WARNING [make_plots] findfont: Failed to find font weight medium, now using 400.
2026-09-08 13:10:19,968 INFO [make_CHS_currents_file] Made CHS currents file for 2026-09-08 for nowcast
2026-09-08 13:10:19,971 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker
2026-09-08 13:10:51,259 INFO [make_plots] finished rendering nemo nowcast research 08sep26 figures
2026-09-08 13:10:51,262 INFO [make_plots] nemo research plots for 2026-09-08 nowcast completed
2026-09-08 13:10:51,280 INFO [manager] checklist updated with [plots] items from make_plots worker
2026-09-08 13:11:17,708 INFO [make_plots] finished rendering nemo nowcast comparison 07sep26 figures
2026-09-08 13:11:17,711 INFO [make_plots] nemo comparison plots for 2026-09-07 nowcast completed
2026-09-08 13:11:17,716 INFO [manager] checklist updated with [plots] items from make_plots worker
2026-09-08 13:13:53,396 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 19484294 = 2026-09-09 12:08:40 UTC, 33.7% complete
2026-09-08 13:18:53,401 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 19485492 = 2026-09-10 01:27:20 UTC, 70.7% complete
2026-09-08 13:23:53,409 INFO [watch_NEMO] forecast NEMO run on arbutus.cloud-nowcast completed
2026-09-08 13:23:53,414 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2026-09-08 13:23:53,729 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_wind_file', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2026-09-08'], host='arbutus.cloud-nowcast')
2026-09-08 13:23:53,743 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_current_file', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2026-09-08'], host='arbutus.cloud-nowcast')
2026-09-08 13:23:53,748 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'turbidity', '--run-date', '2026-09-08'], host='localhost')
2026-09-08 13:23:53,753 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.nibi', 'turbidity', '--run-date', '2026-09-08'], host='localhost')
2026-09-08 13:23:53,758 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'turbidity', '--run-date', '2026-09-08'], host='localhost')
2026-09-08 13:23:53,763 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2026-09-08'], host='localhost')
2026-09-08 13:23:54,973 INFO [make_ww3_wind_file] running in process 870143
2026-09-08 13:23:54,974 INFO [make_ww3_wind_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-09-08 13:23:54,975 INFO [make_ww3_wind_file] publishing log messages to tcp://*:5570
2026-09-08 13:23:54,976 INFO [make_ww3_wind_file] connected to skookum.eos.ubc.ca port 5555
2026-09-08 13:23:54,977 INFO [make_ww3_wind_file] Creating wwatch3 wind forcing file for 2026-09-08 forecast run
2026-09-08 13:23:54,981 INFO [make_ww3_current_file] running in process 870144
2026-09-08 13:23:54,982 INFO [make_ww3_current_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-09-08 13:23:54,983 INFO [make_ww3_current_file] publishing log messages to tcp://*:5571
2026-09-08 13:23:54,984 INFO [make_ww3_current_file] connected to skookum.eos.ubc.ca port 5555
2026-09-08 13:23:54,985 INFO [make_ww3_current_file] Creating wwatch3 currents forcing file for 2026-09-08 forecast run
2026-09-08 13:23:55,260 INFO [download_results] running in process 3157745
2026-09-08 13:23:55,262 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-08 13:23:55,263 INFO [download_results] publishing log messages to tcp://*:5563
2026-09-08 13:23:55,264 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2026-09-08 13:23:55,265 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/forecast/08sep26 to /results/SalishSea/forecast.202111
2026-09-08 13:23:55,380 INFO [upload_forcing] running in process 3157743
2026-09-08 13:23:55,381 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-08 13:23:55,382 INFO [upload_forcing] publishing log messages to tcp://*:5564
2026-09-08 13:23:55,383 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-09-08 13:23:55,404 INFO [upload_forcing] running in process 3157742
2026-09-08 13:23:55,406 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-08 13:23:55,407 INFO [upload_forcing] publishing log messages to tcp://*:5565
2026-09-08 13:23:55,408 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-09-08 13:23:55,416 INFO [upload_forcing] running in process 3157744
2026-09-08 13:23:55,417 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-08 13:23:55,418 INFO [upload_forcing] publishing log messages to tcp://*:5566
2026-09-08 13:23:55,419 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-09-08 13:23:56,122 INFO [upload_forcing] turbidity 2026-09-08 forcing files upload to optimum-hindcast completed
2026-09-08 13:23:56,126 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-09-08 13:23:56,339 INFO [upload_forcing] turbidity 2026-09-08 forcing files upload to arbutus.cloud-nowcast completed
2026-09-08 13:23:56,343 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-09-08 13:23:56,508 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2026-09-08'], host='localhost')
2026-09-08 13:23:58,246 INFO [make_forcing_links] running in process 3157770
2026-09-08 13:23:58,247 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-08 13:23:58,248 INFO [make_forcing_links] publishing log messages to tcp://*:5565
2026-09-08 13:23:58,250 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2026-09-08 13:23:58,754 INFO [make_forcing_links] nowcast-green 2026-09-08 forcing file links on arbutus.cloud-nowcast created
2026-09-08 13:23:58,757 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2026-09-08 13:23:58,911 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2026-09-08'], host='arbutus.cloud-nowcast')
2026-09-08 13:23:59,388 INFO [upload_forcing] turbidity 2026-09-08 forcing files upload to robot.nibi completed
2026-09-08 13:23:59,392 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-09-08 13:23:59,893 INFO [run_NEMO] running in process 870613
2026-09-08 13:23:59,894 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-09-08 13:23:59,896 INFO [run_NEMO] publishing log messages to tcp://*:5556
2026-09-08 13:23:59,897 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2026-09-08 13:24:00,487 WARNING [run_NEMO] revision and status requested for non-existent repo: /nemoShare/MEOPAR/nowcast-sys/moad_tools
2026-09-08 13:24:00,488 WARNING [run_NEMO] revision and status requested for non-existent repo: /nemoShare/MEOPAR/nowcast-sys/NEMO-Cmd
2026-09-08 13:24:00,489 WARNING [run_NEMO] revision and status requested for non-existent repo: /nemoShare/MEOPAR/nowcast-sys/NEMO_Nowcast
2026-09-08 13:24:00,981 WARNING [run_NEMO] There are uncommitted changes in /nemoShare/MEOPAR/nowcast-sys/SalishSeaCmd
2026-09-08 13:24:01,815 WARNING [run_NEMO] revision and status requested for non-existent repo: /nemoShare/MEOPAR/nowcast-sys/tools
2026-09-08 13:24:02,369 INFO [run_NEMO] nowcast-green: launching /nemoShare/MEOPAR/nowcast-sys/runs/08sep26nowcast-green_2026-09-08T132359.935139-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2026-09-08 13:24:02,418 INFO [run_NEMO] nowcast-green NEMO run for 2026-09-08 on arbutus.cloud-nowcast started
2026-09-08 13:24:02,423 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2026-09-08 13:24:02,581 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'nowcast-green'], host='arbutus.cloud-nowcast')
2026-09-08 13:24:03,308 INFO [make_ww3_wind_file] wwatch3 wind forcing file created on arbutus.cloud-nowcast for 2026-09-08 forecast run
2026-09-08 13:24:03,311 INFO [manager] checklist updated with [WW3 wind forcing] items from make_ww3_wind_file worker
2026-09-08 13:24:03,483 INFO [watch_NEMO] running in process 870781
2026-09-08 13:24:03,484 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-09-08 13:24:03,486 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2026-09-08 13:24:03,488 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2026-09-08 13:24:03,593 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2026-09-08 13:25:02,799 INFO [download_results] forecast 2026-09-08 results files from arbutus.cloud-nowcast downloaded
2026-09-08 13:25:02,803 INFO [manager] checklist updated with [results files] items from download_results worker
2026-09-08 13:25:02,958 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['forecast', '--run-date', '2026-09-08'], host='localhost')
2026-09-08 13:25:05,442 INFO [make_CHS_currents_file] running in process 3157804
2026-09-08 13:25:05,444 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-08 13:25:05,445 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5563
2026-09-08 13:25:05,446 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555
2026-09-08 13:25:52,721 INFO [make_CHS_currents_file] Made CHS currents file for 2026-09-08 for forecast
2026-09-08 13:25:52,724 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker
2026-09-08 13:25:52,878 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['nemo', 'forecast', '--run-date', '2026-09-08'], host='localhost')
2026-09-08 13:25:54,383 INFO [update_forecast_datasets] running in process 3157836
2026-09-08 13:25:54,384 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-08 13:25:54,385 INFO [update_forecast_datasets] publishing log messages to tcp://*:5563
2026-09-08 13:25:54,387 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2026-09-08 13:25:54,388 INFO [update_forecast_datasets] updating nemo forecast directory for forecast/08sep26 run
2026-09-08 13:25:54,819 INFO [update_forecast_datasets] updated nemo forecast directory for forecast/08sep26 run: /results/SalishSea/rolling-forecasts/nemo
2026-09-08 13:25:54,821 INFO [update_forecast_datasets] nemo 2026-09-08 forecast rolling forecast datasets updated
2026-09-08 13:25:54,824 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2026-09-08 13:25:54,979 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nemo-forecast'], host='localhost')
2026-09-08 13:25:54,984 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'forecast', 'publish', '--run-date', '2026-09-08'], host='localhost')
2026-09-08 13:25:54,989 INFO [manager] launching NextWorker(module='nowcast.workers.make_surface_current_tiles', args=['forecast', '--run-date', '2026-09-08'], host='localhost')
2026-09-08 13:25:56,415 INFO [ping_erddap] running in process 3157842
2026-09-08 13:25:56,416 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-08 13:25:56,417 INFO [ping_erddap] publishing log messages to tcp://*:5563
2026-09-08 13:25:56,419 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2026-09-08 13:25:56,438 INFO [ping_erddap] nemo-forecast ERDDAP dataset flag file(s) created
2026-09-08 13:25:56,442 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2026-09-08 13:25:57,944 INFO [make_surface_current_tiles] running in process 3157845
2026-09-08 13:25:57,946 INFO [make_surface_current_tiles] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-08 13:25:57,948 INFO [make_surface_current_tiles] publishing log messages to tcp://*:5563
2026-09-08 13:25:57,948 INFO [make_surface_current_tiles] connected to skookum.eos.ubc.ca port 5555
2026-09-08 13:25:57,977 INFO [make_surface_current_tiles] starting rendering of tiles for 2026-09-08 forecast in 6 processes into /results/nowcast-sys/figures/surface_currents/forecast/08sep26
2026-09-08 13:25:58,870 INFO [make_plots] running in process 3157844
2026-09-08 13:25:58,871 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-08 13:25:58,872 INFO [make_plots] publishing log messages to tcp://*:5564
2026-09-08 13:25:58,872 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2026-09-08 13:25:59,270 INFO [make_plots] preparing render list for 2026-09-08 NEMO forecast publish figures
2026-09-08 13:25:59,357 INFO [make_plots] added Website_thumbnail figure to 2026-09-08 NEMO forecast publish render list
2026-09-08 13:25:59,358 INFO [make_plots] added Threshold_website figure to 2026-09-08 NEMO forecast publish render list
2026-09-08 13:25:59,359 INFO [make_plots] added PA_tidal_predictions figure to 2026-09-08 NEMO forecast publish render list
2026-09-08 13:25:59,360 INFO [make_plots] added Boundary Bay figure to 2026-09-08 NEMO forecast publish render list
2026-09-08 13:25:59,361 INFO [make_plots] added Campbell River figure to 2026-09-08 NEMO forecast publish render list
2026-09-08 13:25:59,362 INFO [make_plots] added Cherry Point figure to 2026-09-08 NEMO forecast publish render list
2026-09-08 13:25:59,363 INFO [make_plots] added Friday Harbor figure to 2026-09-08 NEMO forecast publish render list
2026-09-08 13:25:59,363 INFO [make_plots] added Halfmoon Bay figure to 2026-09-08 NEMO forecast publish render list
2026-09-08 13:25:59,364 INFO [make_plots] added Nanaimo figure to 2026-09-08 NEMO forecast publish render list
2026-09-08 13:25:59,365 INFO [make_plots] added Neah Bay figure to 2026-09-08 NEMO forecast publish render list
2026-09-08 13:25:59,366 INFO [make_plots] added New Westminster figure to 2026-09-08 NEMO forecast publish render list
2026-09-08 13:25:59,367 INFO [make_plots] added Patricia Bay figure to 2026-09-08 NEMO forecast publish render list
2026-09-08 13:25:59,367 INFO [make_plots] added Point Atkinson figure to 2026-09-08 NEMO forecast publish render list
2026-09-08 13:25:59,368 INFO [make_plots] added Port Renfrew figure to 2026-09-08 NEMO forecast publish render list
2026-09-08 13:25:59,369 INFO [make_plots] added Sand Heads figure to 2026-09-08 NEMO forecast publish render list
2026-09-08 13:25:59,370 INFO [make_plots] added Sandy Cove figure to 2026-09-08 NEMO forecast publish render list
2026-09-08 13:25:59,371 INFO [make_plots] added Squamish figure to 2026-09-08 NEMO forecast publish render list
2026-09-08 13:25:59,371 INFO [make_plots] added Victoria figure to 2026-09-08 NEMO forecast publish render list
2026-09-08 13:25:59,372 INFO [make_plots] added Woodwards Landing figure to 2026-09-08 NEMO forecast publish render list
2026-09-08 13:25:59,373 INFO [make_plots] starting to render nemo forecast publish 08sep26 figures
2026-09-08 13:26:28,548 INFO [make_ww3_current_file] wwatch3 currents forcing file created on arbutus.cloud-nowcast for 2026-09-08 forecast run
2026-09-08 13:26:28,552 INFO [manager] checklist updated with [WW3 currents forcing] items from make_ww3_current_file worker
2026-09-08 13:26:28,848 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2026-09-08'], host='arbutus.cloud-nowcast')
2026-09-08 13:26:29,674 INFO [run_ww3] running in process 870975
2026-09-08 13:26:29,675 INFO [run_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-09-08 13:26:29,676 INFO [run_ww3] publishing log messages to tcp://*:5572
2026-09-08 13:26:29,677 INFO [run_ww3] connected to skookum.eos.ubc.ca port 5555
2026-09-08 13:26:29,680 INFO [run_ww3] Created run directory /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/nowcast_2026-09-08T132629.668683-0700
2026-09-08 13:26:29,682 INFO [run_ww3] nowcast: launching /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/nowcast_2026-09-08T132629.668683-0700/SoGWW3.sh on arbutus.cloud-nowcast
2026-09-08 13:26:29,728 INFO [run_ww3] nowcast WaveWatch3 run for 2026-09-08 on arbutus.cloud-nowcast started
2026-09-08 13:26:29,732 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker
2026-09-08 13:26:30,026 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'nowcast'], host='arbutus.cloud-nowcast')
2026-09-08 13:26:30,849 INFO [watch_ww3] running in process 870994
2026-09-08 13:26:30,850 INFO [watch_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-09-08 13:26:30,851 INFO [watch_ww3] publishing log messages to tcp://*:5573
2026-09-08 13:26:30,852 INFO [watch_ww3] connected to skookum.eos.ubc.ca port 5555
2026-09-08 13:26:30,913 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2026-09-08 13:28:33,407 WARNING [make_plots] findfont: Failed to find font weight medium, now using 400.
2026-09-08 13:28:33,409 WARNING [make_plots] findfont: Failed to find font weight medium, now using 400.
2026-09-08 13:28:33,413 WARNING [make_plots] findfont: Failed to find font weight medium, now using 400.
2026-09-08 13:28:33,415 WARNING [make_plots] findfont: Failed to find font weight medium, now using 400.
2026-09-08 13:28:33,748 WARNING [make_plots] findfont: Failed to find font weight medium, now using 400.
2026-09-08 13:28:33,752 WARNING [make_plots] findfont: Failed to find font weight medium, now using 400.
2026-09-08 13:28:33,756 WARNING [make_plots] findfont: Failed to find font weight medium, now using 400.
2026-09-08 13:28:33,760 WARNING [make_plots] findfont: Failed to find font weight medium, now using 400.
2026-09-08 13:28:33,774 WARNING [make_plots] findfont: Failed to find font weight medium, now using 400.
2026-09-08 13:28:33,778 WARNING [make_plots] findfont: Failed to find font weight medium, now using 400.
2026-09-08 13:29:03,598 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19481480 = 2026-09-08 04:52:40 UTC, 20.3% complete
2026-09-08 13:31:30,917 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 405, 93.8% complete
2026-09-08 13:33:47,146 WARNING [make_plots] findfont: Failed to find font weight medium, now using 400.
2026-09-08 13:33:47,148 WARNING [make_plots] findfont: Failed to find font weight medium, now using 400.
2026-09-08 13:33:48,298 WARNING [make_plots] invalid station number for Boundary Bay station: None
2026-09-08 13:33:48,299 WARNING [make_plots] can't resolve a valid CHS station code for Boundary Bay
2026-09-08 13:34:03,602 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19481970 = 2026-09-08 10:19:20 UTC, 43.0% complete
2026-09-08 13:34:06,080 WARNING [make_plots] findfont: Failed to find font weight medium, now using 400.
2026-09-08 13:34:06,082 WARNING [make_plots] findfont: Failed to find font weight medium, now using 400.
2026-09-08 13:34:09,998 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb996/data for station 08074 Campbell River from 2026-09-03 00:00:00Z to 2026-09-10 11:55:00Z
2026-09-08 13:35:18,010 WARNING [make_plots] invalid station number for Halfmoon Bay station: None
2026-09-08 13:35:18,017 WARNING [make_plots] can't resolve a valid CHS station code for Halfmoon Bay
2026-09-08 13:35:42,201 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb96d/data for station 07917 Nanaimo from 2026-09-03 00:00:00Z to 2026-09-10 11:55:00Z
2026-09-08 13:36:30,783 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e13d0f4a073c4bbf93/data for station 07654 New Westminster from 2026-09-03 00:00:00Z to 2026-09-10 11:55:00Z
2026-09-08 13:36:30,919 INFO [watch_ww3] nowcast WWATCH3 run on arbutus.cloud-nowcast completed
2026-09-08 13:36:30,924 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2026-09-08 13:36:31,229 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2026-09-08'], host='localhost')
2026-09-08 13:36:31,236 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2026-09-08'], host='arbutus.cloud-nowcast')
2026-09-08 13:36:32,072 INFO [run_ww3] running in process 871544
2026-09-08 13:36:32,074 INFO [run_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-09-08 13:36:32,076 INFO [run_ww3] publishing log messages to tcp://*:5572
2026-09-08 13:36:32,078 INFO [run_ww3] connected to skookum.eos.ubc.ca port 5555
2026-09-08 13:36:32,085 INFO [run_ww3] Created run directory /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast_2026-09-08T133632.063982-0700
2026-09-08 13:36:32,087 INFO [run_ww3] forecast: launching /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast_2026-09-08T133632.063982-0700/SoGWW3.sh on arbutus.cloud-nowcast
2026-09-08 13:36:32,124 INFO [run_ww3] forecast WaveWatch3 run for 2026-09-08 on arbutus.cloud-nowcast started
2026-09-08 13:36:32,128 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker
2026-09-08 13:36:32,295 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'forecast'], host='arbutus.cloud-nowcast')
2026-09-08 13:36:33,115 INFO [watch_ww3] running in process 871563
2026-09-08 13:36:33,118 INFO [watch_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-09-08 13:36:33,120 INFO [watch_ww3] publishing log messages to tcp://*:5573
2026-09-08 13:36:33,122 INFO [watch_ww3] connected to skookum.eos.ubc.ca port 5555
2026-09-08 13:36:33,178 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2026-09-08 13:36:33,193 INFO [download_wwatch3_results] running in process 3160033
2026-09-08 13:36:33,194 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-08 13:36:33,196 INFO [download_wwatch3_results] publishing log messages to tcp://*:5565
2026-09-08 13:36:33,197 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2026-09-08 13:36:45,713 INFO [make_surface_current_tiles] starting PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast/08sep26
2026-09-08 13:36:46,072 INFO [download_wwatch3_results] nowcast 2026-09-08 results files from arbutus.cloud-nowcast downloaded
2026-09-08 13:36:46,075 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2026-09-08 13:36:51,335 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd2d/data for station 07277 Patricia Bay from 2026-09-03 00:00:00Z to 2026-09-10 11:55:00Z
2026-09-08 13:37:08,230 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94c/data for station 07795 Point Atkinson from 2026-09-03 00:00:00Z to 2026-09-10 11:55:00Z
2026-09-08 13:37:26,174 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e23d0f4a073c4bc060/data for station 08525 Port Renfrew from 2026-09-03 00:00:00Z to 2026-09-10 11:55:00Z
2026-09-08 13:37:42,182 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb93c/data for station 07594 Sand Heads from 2026-09-03 00:00:00Z to 2026-09-10 11:55:00Z
2026-09-08 13:37:59,748 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc43b/data for station 07786 Sandy Cove from 2026-09-03 00:00:00Z to 2026-09-10 11:55:00Z
2026-09-08 13:38:17,139 WARNING [make_plots] invalid station number for Squamish station: None
2026-09-08 13:38:17,140 WARNING [make_plots] can't resolve a valid CHS station code for Squamish
2026-09-08 13:38:33,870 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd1e/data for station 07120 Victoria from 2026-09-03 00:00:00Z to 2026-09-10 11:55:00Z
2026-09-08 13:38:49,988 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc42d/data for station 07610 Woodwards Landing from 2026-09-03 00:00:00Z to 2026-09-10 11:55:00Z
2026-09-08 13:38:50,474 INFO [make_plots] no obs water level data available from 5cebf1e43d0f4a073c4bc42d during 2026-09-03 00:00:00Z to 2026-09-10 11:55:00Z
2026-09-08 13:39:03,601 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19482446 = 2026-09-08 15:36:40 UTC, 65.1% complete
2026-09-08 13:39:06,892 INFO [make_plots] finished rendering nemo forecast publish 08sep26 figures
2026-09-08 13:39:06,895 INFO [make_plots] nemo publish plots for 2026-09-08 forecast completed
2026-09-08 13:39:06,901 INFO [manager] checklist updated with [plots] items from make_plots worker
2026-09-08 13:39:07,056 INFO [manager] launching NextWorker(module='nowcast.workers.make_feeds', args=['forecast', '--run-date', '2026-09-08'], host='localhost')
2026-09-08 13:39:10,590 INFO [make_feeds] running in process 3160106
2026-09-08 13:39:10,591 INFO [make_feeds] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-08 13:39:10,592 INFO [make_feeds] publishing log messages to tcp://*:5564
2026-09-08 13:39:10,593 INFO [make_feeds] connected to skookum.eos.ubc.ca port 5555
2026-09-08 13:39:25,921 INFO [make_feeds] ATOM feeds for 2026-09-08 forecast run completed
2026-09-08 13:39:25,924 INFO [manager] checklist updated with [storm surge alert feeds] items from make_feeds worker
2026-09-08 13:41:33,179 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 444, 68.5% complete
2026-09-08 13:41:45,862 INFO [make_surface_current_tiles] finished PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast/08sep26
2026-09-08 13:41:45,880 INFO [make_surface_current_tiles] finished rendering of tiles for 2026-09-08 forecast into /results/nowcast-sys/figures/surface_currents/forecast/08sep26
2026-09-08 13:41:45,881 INFO [make_surface_current_tiles] surface current tile figures for 2026-09-08 forecast completed
2026-09-08 13:41:46,186 INFO [manager] checklist updated with [surface current tiles] items from make_surface_current_tiles worker
2026-09-08 13:44:03,601 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19482915 = 2026-09-08 20:49:20 UTC, 86.8% complete
2026-09-08 13:46:33,177 INFO [watch_ww3] forecast WWATCH3 run on arbutus.cloud-nowcast completed
2026-09-08 13:46:33,182 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2026-09-08 13:46:33,628 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2026-09-08'], host='localhost')
2026-09-08 13:46:35,155 INFO [download_wwatch3_results] running in process 3160279
2026-09-08 13:46:35,156 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-08 13:46:35,157 INFO [download_wwatch3_results] publishing log messages to tcp://*:5563
2026-09-08 13:46:35,158 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2026-09-08 13:46:48,434 INFO [download_wwatch3_results] forecast 2026-09-08 results files from arbutus.cloud-nowcast downloaded
2026-09-08 13:46:48,438 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2026-09-08 13:46:48,736 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['wwatch3', 'forecast', '--run-date', '2026-09-08'], host='localhost')
2026-09-08 13:46:50,246 INFO [update_forecast_datasets] running in process 3160289
2026-09-08 13:46:50,247 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-08 13:46:50,248 INFO [update_forecast_datasets] publishing log messages to tcp://*:5563
2026-09-08 13:46:50,249 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2026-09-08 13:46:50,250 INFO [update_forecast_datasets] updating wwatch3 most_recent_forecast directory from forecast/08sep26 run
2026-09-08 13:46:50,255 INFO [update_forecast_datasets] updated wwatch3 most_recent_forecast directory from forecast/08sep26 run
2026-09-08 13:46:50,256 INFO [update_forecast_datasets] updating wwatch3 forecast directory for forecast/08sep26 run
2026-09-08 13:46:50,312 INFO [update_forecast_datasets] updated wwatch3 forecast directory for forecast/08sep26 run: /results/SalishSea/rolling-forecasts/wwatch3
2026-09-08 13:46:50,313 INFO [update_forecast_datasets] wwatch3 2026-09-08 forecast rolling forecast datasets updated
2026-09-08 13:46:50,317 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2026-09-08 13:46:50,625 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['wwatch3-forecast'], host='localhost')
2026-09-08 13:46:52,116 INFO [ping_erddap] running in process 3160295
2026-09-08 13:46:52,117 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-08 13:46:52,118 INFO [ping_erddap] publishing log messages to tcp://*:5563
2026-09-08 13:46:52,119 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2026-09-08 13:46:52,121 INFO [ping_erddap] wwatch3-forecast ERDDAP dataset flag file(s) created
2026-09-08 13:46:52,122 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2026-09-08 13:46:52,410 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['wwatch3', 'forecast', 'publish', '--run-date', '2026-09-08'], host='localhost')
2026-09-08 13:46:56,186 INFO [make_plots] running in process 3160300
2026-09-08 13:46:56,187 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-08 13:46:56,188 INFO [make_plots] publishing log messages to tcp://*:5563
2026-09-08 13:46:56,190 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2026-09-08 13:46:56,191 INFO [make_plots] preparing render list for 2026-09-08 WaveWatch3 forecast publish figures
2026-09-08 13:46:56,192 INFO [make_plots] added Halibut Bank figure to 2026-09-08 WaveWatch3 forecast publish render list
2026-09-08 13:46:56,193 INFO [make_plots] added Sentry Shoal figure to 2026-09-08 WaveWatch3 forecast publish render list
2026-09-08 13:46:56,194 INFO [make_plots] starting to render wwatch3 forecast publish 08sep26 figures
2026-09-08 13:46:56,635 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46146 Halibut Bank
2026-09-08 13:47:02,335 WARNING [make_plots] findfont: Failed to find font weight medium, now using 400.
2026-09-08 13:47:02,336 WARNING [make_plots] findfont: Failed to find font weight medium, now using 400.
2026-09-08 13:47:02,361 WARNING [make_plots] findfont: Failed to find font weight medium, now using 400.
2026-09-08 13:47:02,363 WARNING [make_plots] findfont: Failed to find font weight medium, now using 400.
2026-09-08 13:47:02,869 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46131 Sentry Shoal
2026-09-08 13:47:07,747 INFO [make_plots] finished rendering wwatch3 forecast publish 08sep26 figures
2026-09-08 13:47:07,748 INFO [make_plots] wwatch3 publish plots for 2026-09-08 forecast completed
2026-09-08 13:47:07,752 INFO [manager] checklist updated with [plots] items from make_plots worker
2026-09-08 13:49:03,609 INFO [watch_NEMO] nowcast-green NEMO run on arbutus.cloud-nowcast completed
2026-09-08 13:49:03,614 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2026-09-08 13:49:04,048 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2026-09-08'], host='localhost')
2026-09-08 13:49:05,765 INFO [download_results] running in process 3160373
2026-09-08 13:49:05,766 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-08 13:49:05,767 INFO [download_results] publishing log messages to tcp://*:5563
2026-09-08 13:49:05,768 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2026-09-08 13:49:05,769 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/nowcast-green/08sep26 to /results2/SalishSea/nowcast-green.202111
2026-09-08 13:51:32,714 INFO [download_results] nowcast-green 2026-09-08 results files from arbutus.cloud-nowcast downloaded
2026-09-08 13:51:32,720 INFO [manager] checklist updated with [results files] items from download_results worker
2026-09-08 13:51:33,020 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast-green', 'research', '--run-date', '2026-09-08'], host='localhost')
2026-09-08 13:51:33,024 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nowcast-green'], host='localhost')
2026-09-08 13:51:33,029 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'biology', '--run-date', '2026-09-08'], host='localhost')
2026-09-08 13:51:33,035 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'physics', '--run-date', '2026-09-08'], host='localhost')
2026-09-08 13:51:33,046 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'chemistry', '--run-date', '2026-09-08'], host='localhost')
2026-09-08 13:51:34,714 INFO [ping_erddap] running in process 3160432
2026-09-08 13:51:34,716 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-08 13:51:34,717 INFO [ping_erddap] publishing log messages to tcp://*:5563
2026-09-08 13:51:34,719 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2026-09-08 13:51:34,724 INFO [ping_erddap] nowcast-green ERDDAP dataset flag file(s) created
2026-09-08 13:51:34,725 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2026-09-08 13:51:35,901 INFO [make_averaged_dataset] running in process 3160433
2026-09-08 13:51:35,902 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-08 13:51:35,903 INFO [make_averaged_dataset] publishing log messages to tcp://*:5563
2026-09-08 13:51:35,904 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555
2026-09-08 13:51:35,905 INFO [make_averaged_dataset] creating day-averaged dataset for 08-Sep-2026 biology
2026-09-08 13:51:35,908 INFO [make_averaged_dataset] running in process 3160434
2026-09-08 13:51:35,909 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-08 13:51:35,910 INFO [make_averaged_dataset] publishing log messages to tcp://*:5564
2026-09-08 13:51:35,910 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555
2026-09-08 13:51:35,911 INFO [make_averaged_dataset] creating day-averaged dataset for 08-Sep-2026 physics
2026-09-08 13:51:35,922 INFO [make_averaged_dataset] running in process 3160435
2026-09-08 13:51:35,923 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-08 13:51:35,924 INFO [make_averaged_dataset] publishing log messages to tcp://*:5565
2026-09-08 13:51:35,925 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555
2026-09-08 13:51:35,926 INFO [make_averaged_dataset] creating day-averaged dataset for 08-Sep-2026 chemistry
2026-09-08 13:51:37,175 INFO [make_plots] running in process 3160430
2026-09-08 13:51:37,176 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-08 13:51:37,177 INFO [make_plots] publishing log messages to tcp://*:5566
2026-09-08 13:51:37,177 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2026-09-08 13:51:37,583 INFO [make_plots] preparing render list for 2026-09-08 NEMO nowcast-green research figures
2026-09-08 13:51:39,473 INFO [make_plots] added temperature_thalweg_and_surface figures to 2026-09-08 NEMO nowcast-green research render list
2026-09-08 13:51:40,262 INFO [make_plots] added nitrate_thalweg_and_surface figures to 2026-09-08 NEMO nowcast-green research render list
2026-09-08 13:51:41,040 INFO [make_plots] added ammonium_thalweg_and_surface figures to 2026-09-08 NEMO nowcast-green research render list
2026-09-08 13:51:41,801 INFO [make_plots] added silicon_thalweg_and_surface figures to 2026-09-08 NEMO nowcast-green research render list
2026-09-08 13:51:42,568 INFO [make_plots] added dissolved_organic_nitrogen_thalweg_and_surface figures to 2026-09-08 NEMO nowcast-green research render list
2026-09-08 13:51:43,340 INFO [make_plots] added particulate_organic_nitrogen_thalweg_and_surface figures to 2026-09-08 NEMO nowcast-green research render list
2026-09-08 13:51:44,115 INFO [make_plots] added biogenic_silicon_thalweg_and_surface figures to 2026-09-08 NEMO nowcast-green research render list
2026-09-08 13:51:44,997 INFO [make_plots] added diatoms_thalweg_and_surface figures to 2026-09-08 NEMO nowcast-green research render list
2026-09-08 13:51:45,837 INFO [make_plots] added flagellates_thalweg_and_surface figures to 2026-09-08 NEMO nowcast-green research render list
2026-09-08 13:51:46,666 INFO [make_plots] added microzooplankton_thalweg_and_surface figures to 2026-09-08 NEMO nowcast-green research render list
2026-09-08 13:51:47,491 INFO [make_plots] added mesozooplankton_thalweg_and_surface figures to 2026-09-08 NEMO nowcast-green research render list
2026-09-08 13:51:48,231 INFO [make_plots] added turbidity_thalweg_and_surface figures to 2026-09-08 NEMO nowcast-green research render list
2026-09-08 13:51:49,331 INFO [make_plots] added temperature_salinity_timeseries figure to 2026-09-08 NEMO nowcast-green research render list
2026-09-08 13:51:49,331 INFO [make_plots] added nitrate_diatoms_timeseries figure to 2026-09-08 NEMO nowcast-green research render list
2026-09-08 13:51:49,332 INFO [make_plots] added diatoms_flagellates_timeseries figure to 2026-09-08 NEMO nowcast-green research render list
2026-09-08 13:51:49,334 INFO [make_plots] added z1_z2_zooplankton_timeseries figure to 2026-09-08 NEMO nowcast-green research render list
2026-09-08 13:51:49,335 INFO [make_plots] starting to render nemo nowcast-green research 08sep26 figures
2026-09-08 13:51:50,006 WARNING [make_plots] findfont: Failed to find font weight medium, now using 400.
2026-09-08 13:51:50,008 WARNING [make_plots] findfont: Failed to find font weight medium, now using 400.
2026-09-08 13:52:18,140 INFO [make_averaged_dataset] day-averaged dataset for 08-Sep-2026 physics created
2026-09-08 13:52:18,144 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker
2026-09-08 13:52:39,769 INFO [make_averaged_dataset] day-averaged dataset for 08-Sep-2026 chemistry created
2026-09-08 13:52:39,772 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker
2026-09-08 13:52:47,128 INFO [make_averaged_dataset] day-averaged dataset for 08-Sep-2026 biology created
2026-09-08 13:52:47,131 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker
2026-09-08 14:10:34,656 WARNING [make_plots] findfont: Failed to find font weight medium, now using 400.
2026-09-08 14:10:34,658 WARNING [make_plots] findfont: Failed to find font weight medium, now using 400.
2026-09-08 14:25:18,543 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/18/ to /results/forcing/atmospheric/continental2.5/GRIB/20260908/18/
2026-09-08 14:25:18,544 INFO [collect_weather] 2026-09-08 2.5km weather forecast 18 collection complete
2026-09-08 14:25:18,547 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2026-09-08 14:25:19,089 INFO [manager] launching NextWorker(module='nowcast.workers.download_weather', args=['00', '1km'], host='localhost')
2026-09-08 14:25:19,096 INFO [manager] launching NextWorker(module='nowcast.workers.download_weather', args=['12', '1km'], host='localhost')
2026-09-08 14:25:19,100 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['00', '2.5km'], host='localhost')
2026-09-08 14:25:19,104 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['00', '--fcst-date', '2026-09-09'], host='localhost')
2026-09-08 14:25:19,186 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20260908/18/
2026-09-08 14:25:19,187 INFO [crop_gribs] 2026-09-08 18 GRIBs cropping complete
2026-09-08 14:25:19,190 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2026-09-08 14:25:20,848 INFO [download_weather] running in process 3161514
2026-09-08 14:25:20,849 INFO [download_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-08 14:25:20,850 INFO [download_weather] publishing log messages to tcp://*:5562
2026-09-08 14:25:20,851 INFO [download_weather] connected to skookum.eos.ubc.ca port 5555
2026-09-08 14:25:20,852 INFO [download_weather] downloading 12 1 km forecast GRIB2 files for 20260908
2026-09-08 14:25:20,857 INFO [collect_weather] running in process 3161515
2026-09-08 14:25:20,858 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-08 14:25:20,859 INFO [collect_weather] publishing log messages to tcp://*:5563
2026-09-08 14:25:20,860 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2026-09-08 14:25:20,881 INFO [download_weather] running in process 3161512
2026-09-08 14:25:20,882 INFO [download_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-08 14:25:20,883 INFO [download_weather] publishing log messages to tcp://*:5564
2026-09-08 14:25:20,884 INFO [download_weather] connected to skookum.eos.ubc.ca port 5555
2026-09-08 14:25:20,885 INFO [download_weather] downloading 00 1 km forecast GRIB2 files for 20260908
2026-09-08 14:25:21,004 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/00/
2026-09-08 14:25:21,742 INFO [crop_gribs] running in process 3161516
2026-09-08 14:25:21,745 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-08 14:25:21,747 INFO [crop_gribs] publishing log messages to tcp://*:5565
2026-09-08 14:25:21,748 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2026-09-08 14:25:21,749 INFO [crop_gribs] cropping 2026-09-09 ECCC HRDPS 2.5km continental 00Z GRIB files to SalishSeaCast subdomain
2026-09-08 14:25:21,757 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20260909/00/
2026-09-08 14:28:51,934 INFO [download_weather] 2026-09-08 1km weather forecast 00 downloads complete
2026-09-08 14:28:51,938 INFO [manager] checklist updated with [weather forecast] items from download_weather worker
2026-09-08 14:28:56,992 INFO [download_weather] 2026-09-08 1km weather forecast 12 downloads complete
2026-09-08 14:28:56,995 INFO [manager] checklist updated with [weather forecast] items from download_weather worker
2026-09-08 14:48:33,922 INFO [make_plots] finished rendering nemo nowcast-green research 08sep26 figures
2026-09-08 14:48:33,933 INFO [make_plots] nemo research plots for 2026-09-08 nowcast-green completed
2026-09-08 14:48:33,998 INFO [manager] checklist updated with [plots] items from make_plots worker
2026-09-08 20:26:59,706 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/00/ to /results/forcing/atmospheric/continental2.5/GRIB/20260909/00/
2026-09-08 20:26:59,707 INFO [collect_weather] 2026-09-09 2.5km weather forecast 00 collection complete
2026-09-08 20:26:59,712 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2026-09-08 20:27:00,208 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['06', '2.5km'], host='localhost')
2026-09-08 20:27:00,215 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['06', '--fcst-date', '2026-09-09'], host='localhost')
2026-09-08 20:27:01,146 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20260909/00/
2026-09-08 20:27:01,147 INFO [crop_gribs] 2026-09-09 00 GRIBs cropping complete
2026-09-08 20:27:01,149 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2026-09-08 20:27:03,312 INFO [collect_weather] running in process 3169403
2026-09-08 20:27:03,313 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-08 20:27:03,315 INFO [collect_weather] publishing log messages to tcp://*:5562
2026-09-08 20:27:03,317 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2026-09-08 20:27:03,331 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/06/
2026-09-08 20:27:04,447 INFO [crop_gribs] running in process 3169405
2026-09-08 20:27:04,451 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-08 20:27:04,452 INFO [crop_gribs] publishing log messages to tcp://*:5563
2026-09-08 20:27:04,453 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2026-09-08 20:27:04,454 INFO [crop_gribs] cropping 2026-09-09 ECCC HRDPS 2.5km continental 06Z GRIB files to SalishSeaCast subdomain
2026-09-08 20:27:04,460 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20260909/06/
2026-09-09 02:26:51,038 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/06/ to /results/forcing/atmospheric/continental2.5/GRIB/20260909/06/
2026-09-09 02:26:51,039 INFO [collect_weather] 2026-09-09 2.5km weather forecast 06 collection complete
2026-09-09 02:26:51,043 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2026-09-09 02:26:51,604 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Capilano', '--data-date', '2026-09-08'], host='localhost')
2026-09-09 02:26:51,611 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'ChilliwackVedder', '--data-date', '2026-09-08'], host='localhost')
2026-09-09 02:26:51,616 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'ClowhomClowhomLake', '--data-date', '2026-09-08'], host='localhost')
2026-09-09 02:26:51,624 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Englishman', '--data-date', '2026-09-08'], host='localhost')
2026-09-09 02:26:51,630 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Fraser', '--data-date', '2026-09-08'], host='localhost')
2026-09-09 02:26:51,637 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'HomathkoMouth', '--data-date', '2026-09-08'], host='localhost')
2026-09-09 02:26:51,641 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'NicomeklLangley', '--data-date', '2026-09-08'], host='localhost')
2026-09-09 02:26:51,652 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'RobertsCreek', '--data-date', '2026-09-08'], host='localhost')
2026-09-09 02:26:51,657 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SalmonSayward', '--data-date', '2026-09-08'], host='localhost')
2026-09-09 02:26:51,661 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SanJuanPortRenfrew', '--data-date', '2026-09-08'], host='localhost')
2026-09-09 02:26:51,665 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SquamishBrackendale', '--data-date', '2026-09-08'], host='localhost')
2026-09-09 02:26:51,679 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaScotty', '--data-date', '2026-09-08'], host='localhost')
2026-09-09 02:26:51,689 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaBypass', '--data-date', '2026-09-08'], host='localhost')
2026-09-09 02:26:51,700 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaDiversion', '--data-date', '2026-09-08'], host='localhost')
2026-09-09 02:26:51,708 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ctd', args=['SCVIP'], host='localhost')
2026-09-09 02:26:51,734 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ctd', args=['SEVIP'], host='localhost')
2026-09-09 02:26:51,747 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ferry', args=['TWDP'], host='localhost')
2026-09-09 02:26:51,752 INFO [manager] launching NextWorker(module='nowcast.workers.get_vfpa_hadcp', args=['--data-date', '2026-09-08'], host='localhost')
2026-09-09 02:26:51,765 INFO [manager] launching NextWorker(module='nowcast.workers.collect_NeahBay_ssh', args=['00'], host='localhost')
2026-09-09 02:26:51,794 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['12', '2.5km'], host='localhost')
2026-09-09 02:26:51,824 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['12'], host='localhost')
2026-09-09 02:26:56,114 INFO [collect_NeahBay_ssh] running in process 3177086
2026-09-09 02:26:56,116 INFO [collect_NeahBay_ssh] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-09 02:26:56,117 INFO [collect_NeahBay_ssh] publishing log messages to tcp://*:5562
2026-09-09 02:26:56,118 INFO [collect_NeahBay_ssh] connected to skookum.eos.ubc.ca port 5555
2026-09-09 02:26:56,119 INFO [collect_NeahBay_ssh] collecting Neah Bay ssh 00Z obs/forecast for 2026-09-09
2026-09-09 02:26:56,173 INFO [collect_weather] running in process 3177087
2026-09-09 02:26:56,174 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-09 02:26:56,175 INFO [collect_weather] publishing log messages to tcp://*:5564
2026-09-09 02:26:56,176 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2026-09-09 02:26:56,950 INFO [collect_river_data] running in process 3177078
2026-09-09 02:26:56,952 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-09 02:26:56,955 INFO [collect_river_data] publishing log messages to tcp://*:5565
2026-09-09 02:26:56,958 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-09-09 02:26:56,962 INFO [collect_river_data] Collecting ECCC TheodosiaScotty river data for 2026-09-08
2026-09-09 02:26:56,966 INFO [collect_river_data] running in process 3177067
2026-09-09 02:26:56,969 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-09 02:26:56,971 INFO [collect_river_data] publishing log messages to tcp://*:5566
2026-09-09 02:26:56,974 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-09-09 02:26:56,976 INFO [collect_river_data] Collecting ECCC ChilliwackVedder river data for 2026-09-08
2026-09-09 02:26:56,977 INFO [collect_river_data] running in process 3177072
2026-09-09 02:26:56,979 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-09 02:26:56,981 INFO [collect_river_data] publishing log messages to tcp://*:5567
2026-09-09 02:26:56,982 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-09-09 02:26:56,983 INFO [collect_river_data] Collecting ECCC NicomeklLangley river data for 2026-09-08
2026-09-09 02:26:57,010 INFO [collect_river_data] running in process 3177071
2026-09-09 02:26:57,012 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-09 02:26:57,013 INFO [collect_river_data] publishing log messages to tcp://*:5568
2026-09-09 02:26:57,016 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-09-09 02:26:57,018 INFO [collect_river_data] Collecting ECCC HomathkoMouth river data for 2026-09-08
2026-09-09 02:26:57,067 INFO [collect_river_data] running in process 3177080
2026-09-09 02:26:57,068 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-09 02:26:57,069 INFO [collect_river_data] publishing log messages to tcp://*:5569
2026-09-09 02:26:57,070 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-09-09 02:26:57,071 INFO [collect_river_data] Collecting ECCC TheodosiaDiversion river data for 2026-09-08
2026-09-09 02:26:57,080 INFO [collect_river_data] running in process 3177077
2026-09-09 02:26:57,081 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-09 02:26:57,081 INFO [collect_river_data] publishing log messages to tcp://*:5570
2026-09-09 02:26:57,083 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-09-09 02:26:57,083 INFO [collect_river_data] Collecting ECCC SquamishBrackendale river data for 2026-09-08
2026-09-09 02:26:57,089 INFO [collect_river_data] running in process 3177070
2026-09-09 02:26:57,090 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-09 02:26:57,091 INFO [collect_river_data] publishing log messages to tcp://*:5571
2026-09-09 02:26:57,092 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-09-09 02:26:57,093 INFO [collect_river_data] Collecting ECCC Fraser river data for 2026-09-08
2026-09-09 02:26:57,098 INFO [collect_river_data] running in process 3177073
2026-09-09 02:26:57,102 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-09 02:26:57,104 INFO [collect_river_data] publishing log messages to tcp://*:5572
2026-09-09 02:26:57,107 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-09-09 02:26:57,108 INFO [collect_river_data] Collecting ECCC RobertsCreek river data for 2026-09-08
2026-09-09 02:26:57,109 INFO [collect_river_data] running in process 3177075
2026-09-09 02:26:57,110 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-09 02:26:57,112 INFO [collect_river_data] publishing log messages to tcp://*:5573
2026-09-09 02:26:57,113 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-09-09 02:26:57,113 INFO [collect_river_data] Collecting ECCC SalmonSayward river data for 2026-09-08
2026-09-09 02:26:57,118 INFO [collect_river_data] Appended ECCC Fraser river average discharge for 2026-09-08 to: /data/dlatorne/SOG-projects/SOG-forcing/ECget/Fraser_flow
2026-09-09 02:26:57,120 INFO [collect_river_data] ECCC Fraser river data collection for 2026-09-08 completed
2026-09-09 02:26:57,123 INFO [collect_river_data] running in process 3177076
2026-09-09 02:26:57,124 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-09 02:26:57,125 INFO [collect_river_data] publishing log messages to tcp://*:5574
2026-09-09 02:26:57,126 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-09-09 02:26:57,131 INFO [collect_river_data] Collecting ECCC SanJuanPortRenfrew river data for 2026-09-08
2026-09-09 02:26:57,132 INFO [collect_river_data] running in process 3177079
2026-09-09 02:26:57,133 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-09 02:26:57,134 INFO [collect_river_data] publishing log messages to tcp://*:5575
2026-09-09 02:26:57,135 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-09-09 02:26:57,137 INFO [collect_river_data] Collecting ECCC TheodosiaBypass river data for 2026-09-08
2026-09-09 02:26:57,141 INFO [collect_river_data] running in process 3177068
2026-09-09 02:26:57,142 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-09 02:26:57,143 INFO [collect_river_data] publishing log messages to tcp://*:5576
2026-09-09 02:26:57,146 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-09-09 02:26:57,147 INFO [collect_river_data] Collecting ECCC ClowhomClowhomLake river data for 2026-09-08
2026-09-09 02:26:57,148 INFO [collect_river_data] running in process 3177065
2026-09-09 02:26:57,149 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-09 02:26:57,150 INFO [collect_river_data] publishing log messages to tcp://*:5577
2026-09-09 02:26:57,151 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-09-09 02:26:57,152 INFO [collect_river_data] Collecting ECCC Capilano river data for 2026-09-08
2026-09-09 02:26:57,176 INFO [collect_river_data] running in process 3177069
2026-09-09 02:26:57,177 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-09 02:26:57,178 INFO [collect_river_data] publishing log messages to tcp://*:5578
2026-09-09 02:26:57,178 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-09-09 02:26:57,179 INFO [collect_river_data] Collecting ECCC Englishman river data for 2026-09-08
2026-09-09 02:26:57,208 INFO [collect_river_data] Appended ECCC Englishman river average discharge for 2026-09-08 to: /data/dlatorne/SOG-projects/SOG-forcing/ECget/Englishman_flow
2026-09-09 02:26:57,209 INFO [collect_river_data] ECCC Englishman river data collection for 2026-09-08 completed
2026-09-09 02:26:57,239 INFO [collect_river_data] Appended ECCC Capilano river average discharge for 2026-09-08 to: /opp/observations/rivers/Capilano/Caplilano_08GA010_day_avg_flow
2026-09-09 02:26:57,240 INFO [collect_river_data] ECCC Capilano river data collection for 2026-09-08 completed
2026-09-09 02:26:57,267 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-09-09 02:26:57,494 INFO [get_vfpa_hadcp] running in process 3177085
2026-09-09 02:26:57,495 INFO [get_vfpa_hadcp] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-09 02:26:57,495 INFO [get_vfpa_hadcp] publishing log messages to tcp://*:5579
2026-09-09 02:26:57,496 INFO [get_vfpa_hadcp] connected to skookum.eos.ubc.ca port 5555
2026-09-09 02:26:57,497 INFO [get_vfpa_hadcp] processing VFPA HADCP data from 2nd Narrows Rail Bridge for 2026-09-08
2026-09-09 02:26:57,517 WARNING [get_vfpa_hadcp] file not found: /opp/observations/AISDATA/20260908T000000Z-20260908T010000Z.csv
2026-09-09 02:26:57,529 CRITICAL [get_vfpa_hadcp] unhandled exception:
Traceback (most recent call last):
  File "/SalishSeaCast/SalishSeaNowcast/.pixi/envs/default/lib/python3.14/site-packages/nemo_nowcast/worker.py", line 388, in _do_work
    checklist = self.worker_func(
        self._parsed_args, self.config, self.tell_manager
    )
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/get_vfpa_hadcp.py", line 105, in get_vfpa_hadcp
    ds = _make_hour_dataset(csv_dir, data_date, place)
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/get_vfpa_hadcp.py", line 170, in _make_hour_dataset
    ds = _csv_to_dataset(csv_dir / csv_filename, place)
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/get_vfpa_hadcp.py", line 246, in _csv_to_dataset
    df = pandas.read_csv(csv_file, skiprows=3)
  File "/SalishSeaCast/SalishSeaNowcast/.pixi/envs/default/lib/python3.14/site-packages/pandas/io/parsers/readers.py", line 1026, in read_csv
    return _read(filepath_or_buffer, kwds)
  File "/SalishSeaCast/SalishSeaNowcast/.pixi/envs/default/lib/python3.14/site-packages/pandas/io/parsers/readers.py", line 620, in _read
    parser = TextFileReader(filepath_or_buffer, **kwds)
  File "/SalishSeaCast/SalishSeaNowcast/.pixi/envs/default/lib/python3.14/site-packages/pandas/io/parsers/readers.py", line 1620, in __init__
    self._engine = self._make_engine(f, self.engine)
                   ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^
  File "/SalishSeaCast/SalishSeaNowcast/.pixi/envs/default/lib/python3.14/site-packages/pandas/io/parsers/readers.py", line 1880, in _make_engine
    self.handles = get_handle(
                   ~~~~~~~~~~^
        f,
        ^^
    ...<6 lines>...
        storage_options=self.options.get("storage_options", None),
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/SalishSeaCast/SalishSeaNowcast/.pixi/envs/default/lib/python3.14/site-packages/pandas/io/common.py", line 873, in get_handle
    handle = open(
        handle,
    ...<3 lines>...
        newline="",
    )
FileNotFoundError: [Errno 2] No such file or directory: '/opp/observations/AISDATA/20260908T000000Z-20260908T010000Z.csv'
2026-09-09 02:26:57,809 INFO [crop_gribs] running in process 3177088
2026-09-09 02:26:57,813 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-09 02:26:57,814 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-09-09 02:26:57,815 INFO [crop_gribs] publishing log messages to tcp://*:5590
2026-09-09 02:26:57,815 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2026-09-09 02:26:57,816 INFO [crop_gribs] cropping 2026-09-09 ECCC HRDPS 2.5km continental 12Z GRIB files to SalishSeaCast subdomain
2026-09-09 02:26:57,817 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20260909/12/
2026-09-09 02:26:58,012 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-09-09 02:26:58,099 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/12/
2026-09-09 02:26:58,391 INFO [get_onc_ferry] running in process 3177084
2026-09-09 02:26:58,392 INFO [get_onc_ferry] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-09 02:26:58,393 INFO [get_onc_ferry] publishing log messages to tcp://*:5571
2026-09-09 02:26:58,394 INFO [get_onc_ferry] connected to skookum.eos.ubc.ca port 5555
2026-09-09 02:26:58,396 INFO [get_onc_ferry] requesting ONC TWDP.N1 NAV data for 2026-09-08
2026-09-09 02:26:58,431 INFO [get_onc_ctd] running in process 3177081
2026-09-09 02:26:58,433 INFO [get_onc_ctd] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-09 02:26:58,434 INFO [get_onc_ctd] publishing log messages to tcp://*:5577
2026-09-09 02:26:58,435 INFO [get_onc_ctd] connected to skookum.eos.ubc.ca port 5555
2026-09-09 02:26:58,437 INFO [get_onc_ctd] requesting ONC SCVIP CTD T&S data for 2026-09-08
2026-09-09 02:26:58,496 INFO [get_onc_ctd] running in process 3177083
2026-09-09 02:26:58,497 INFO [get_onc_ctd] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-09 02:26:58,498 INFO [get_onc_ctd] publishing log messages to tcp://*:5591
2026-09-09 02:26:58,499 INFO [get_onc_ctd] connected to skookum.eos.ubc.ca port 5555
2026-09-09 02:26:58,500 INFO [get_onc_ctd] requesting ONC SEVIP CTD T&S data for 2026-09-08
2026-09-09 02:26:58,559 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20260909/06/
2026-09-09 02:26:58,560 INFO [crop_gribs] 2026-09-09 06 GRIBs cropping complete
2026-09-09 02:26:58,563 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2026-09-09 02:26:59,069 INFO [manager] launching NextWorker(module='nowcast.workers.grib_to_netcdf', args=['forecast2'], host='localhost')
2026-09-09 02:26:59,388 INFO [collect_river_data] Appended ECCC SanJuanPortRenfrew river average discharge for 2026-09-08 to: /results/forcing/rivers/observations/SanJuan_PortRenfrew_flow
2026-09-09 02:26:59,389 INFO [collect_river_data] ECCC SanJuanPortRenfrew river data collection for 2026-09-08 completed
2026-09-09 02:26:59,391 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-09-09 02:26:59,393 INFO [collect_river_data] Appended ECCC ChilliwackVedder river average discharge for 2026-09-08 to: /results/forcing/rivers/observations/Chilliwack_Vedder_flow
2026-09-09 02:26:59,394 INFO [collect_river_data] ECCC ChilliwackVedder river data collection for 2026-09-08 completed
2026-09-09 02:26:59,402 INFO [collect_river_data] Appended ECCC ClowhomClowhomLake river average discharge for 2026-09-08 to: /results/forcing/rivers/observations/Clowhom_ClowhomLake_flow
2026-09-09 02:26:59,403 INFO [collect_river_data] ECCC ClowhomClowhomLake river data collection for 2026-09-08 completed
2026-09-09 02:26:59,431 INFO [collect_river_data] Appended ECCC NicomeklLangley river average discharge for 2026-09-08 to: /results/forcing/rivers/observations/Nicomekl_Langley_flow
2026-09-09 02:26:59,432 INFO [collect_river_data] ECCC NicomeklLangley river data collection for 2026-09-08 completed
2026-09-09 02:26:59,435 INFO [collect_river_data] Appended ECCC TheodosiaBypass river average discharge for 2026-09-08 to: /results/forcing/rivers/observations/Theodosia_Bypass_flow
2026-09-09 02:26:59,436 INFO [collect_river_data] ECCC TheodosiaBypass river data collection for 2026-09-08 completed
2026-09-09 02:26:59,441 INFO [collect_river_data] Appended ECCC HomathkoMouth river average discharge for 2026-09-08 to: /results/forcing/rivers/observations/Homathko_Mouth_flow
2026-09-09 02:26:59,442 INFO [collect_river_data] ECCC HomathkoMouth river data collection for 2026-09-08 completed
2026-09-09 02:26:59,447 INFO [collect_river_data] Appended ECCC SalmonSayward river average discharge for 2026-09-08 to: /results/forcing/rivers/observations/Salmon_Sayward_flow
2026-09-09 02:26:59,448 INFO [collect_river_data] ECCC SalmonSayward river data collection for 2026-09-08 completed
2026-09-09 02:26:59,487 INFO [collect_river_data] Appended ECCC SquamishBrackendale river average discharge for 2026-09-08 to: /results/forcing/rivers/observations/Squamish_Brackendale_flow
2026-09-09 02:26:59,488 INFO [collect_river_data] ECCC SquamishBrackendale river data collection for 2026-09-08 completed
2026-09-09 02:26:59,494 INFO [collect_river_data] Appended ECCC TheodosiaScotty river average discharge for 2026-09-08 to: /results/forcing/rivers/observations/Theodosia_Scotty_flow
2026-09-09 02:26:59,495 INFO [collect_river_data] ECCC TheodosiaScotty river data collection for 2026-09-08 completed
2026-09-09 02:26:59,603 INFO [collect_river_data] Appended ECCC RobertsCreek river average discharge for 2026-09-08 to: /results/forcing/rivers/observations/RobertsCreek_flow
2026-09-09 02:26:59,604 INFO [collect_river_data] ECCC RobertsCreek river data collection for 2026-09-08 completed
2026-09-09 02:26:59,634 INFO [collect_NeahBay_ssh] 2026-09-09T00:00:00-07:00 Neah Bay ssh 00Z obs/forecast data collection complete
2026-09-09 02:26:59,708 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-09-09 02:27:00,006 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-09-09 02:27:00,267 INFO [collect_river_data] Appended ECCC TheodosiaDiversion river average discharge for 2026-09-08 to: /results/forcing/rivers/observations/Theodosia_Diversion_flow
2026-09-09 02:27:00,268 INFO [collect_river_data] ECCC TheodosiaDiversion river data collection for 2026-09-08 completed
2026-09-09 02:27:00,344 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-09-09 02:27:00,704 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-09-09 02:27:01,037 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-09-09 02:27:01,346 INFO [grib_to_netcdf] running in process 3177405
2026-09-09 02:27:01,355 INFO [grib_to_netcdf] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-09 02:27:01,365 INFO [grib_to_netcdf] publishing log messages to tcp://*:5563
2026-09-09 02:27:01,376 INFO [grib_to_netcdf] connected to skookum.eos.ubc.ca port 5555
2026-09-09 02:27:01,378 INFO [grib_to_netcdf] creating NEMO-atmos forcing files for 2026-09-09 forecast2 run
2026-09-09 02:27:01,398 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-09-09 02:27:01,740 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-09-09 02:27:02,075 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-09-09 02:27:02,407 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-09-09 02:27:02,705 INFO [manager] checklist updated with [Neah Bay ssh data] items from collect_NeahBay_ssh worker
2026-09-09 02:27:03,003 INFO [manager] launching NextWorker(module='nowcast.workers.make_ssh_files', args=['forecast2', '--run-date', '2026-09-09'], host='localhost')
2026-09-09 02:27:03,009 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-09-09 02:27:07,144 INFO [make_ssh_files] running in process 3177433
2026-09-09 02:27:07,145 INFO [make_ssh_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-09 02:27:07,146 INFO [make_ssh_files] publishing log messages to tcp://*:5562
2026-09-09 02:27:07,147 INFO [make_ssh_files] connected to skookum.eos.ubc.ca port 5555
2026-09-09 02:27:07,148 INFO [make_ssh_files] building forecast2 sea surface height boundary conditions file(s) from 2026-09-09 NOAA Neah Bay 00Z observation and forecast values
2026-09-09 02:27:13,315 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2026m09d04.nc
2026-09-09 02:27:14,730 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2026m09d05.nc
2026-09-09 02:27:15,780 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2026m09d06.nc
2026-09-09 02:27:16,406 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2026m09d07.nc
2026-09-09 02:27:16,619 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2026m09d08.nc
2026-09-09 02:27:17,674 INFO [get_onc_ferry] requesting ONC TWDP TSG data for 2026-09-08
2026-09-09 02:27:17,848 INFO [get_onc_ctd] 2026-09-08 ONC SCVIP CTD T&S file created
2026-09-09 02:27:17,851 INFO [manager] checklist updated with [ONC CTD data] items from get_onc_ctd worker
2026-09-09 02:27:18,245 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['SCVIP-CTD'], host='localhost')
2026-09-09 02:27:18,709 INFO [get_onc_ctd] 2026-09-08 ONC SEVIP CTD T&S file created
2026-09-09 02:27:18,711 INFO [manager] checklist updated with [ONC CTD data] items from get_onc_ctd worker
2026-09-09 02:27:18,777 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2026m09d09.nc
2026-09-09 02:27:19,133 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['SEVIP-CTD'], host='localhost')
2026-09-09 02:27:19,752 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2026m09d10.nc
2026-09-09 02:27:19,759 INFO [ping_erddap] running in process 3177479
2026-09-09 02:27:19,759 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-09 02:27:19,760 INFO [ping_erddap] publishing log messages to tcp://*:5565
2026-09-09 02:27:19,761 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2026-09-09 02:27:19,763 INFO [ping_erddap] SCVIP-CTD ERDDAP dataset flag file(s) created
2026-09-09 02:27:19,765 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2026-09-09 02:27:20,629 INFO [ping_erddap] running in process 3177484
2026-09-09 02:27:20,630 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-09 02:27:20,631 INFO [ping_erddap] publishing log messages to tcp://*:5565
2026-09-09 02:27:20,632 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2026-09-09 02:27:20,634 INFO [ping_erddap] SEVIP-CTD ERDDAP dataset flag file(s) created
2026-09-09 02:27:20,636 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2026-09-09 02:27:20,642 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2026m09d11.nc
2026-09-09 02:27:20,670 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2026m09d10.nc
2026-09-09 02:27:20,870 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2026m09d12.nc
2026-09-09 02:27:21,092 INFO [make_ssh_files] sea surface height boundary file for 2026-09-09 forecast2 run created
2026-09-09 02:27:21,096 INFO [manager] checklist updated with [sea surface height forcing] items from make_ssh_files worker
2026-09-09 02:27:21,468 INFO [manager] launching NextWorker(module='nowcast.workers.make_runoff_file', args=['v202108'], host='localhost')
2026-09-09 02:27:23,852 INFO [make_runoff_file] running in process 3177499
2026-09-09 02:27:23,854 INFO [make_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-09 02:27:23,856 INFO [make_runoff_file] publishing log messages to tcp://*:5562
2026-09-09 02:27:23,857 INFO [make_runoff_file] connected to skookum.eos.ubc.ca port 5555
2026-09-09 02:27:23,857 INFO [make_runoff_file] calculating NEMO runoff forcing for v202108 bathymetry for 2026-09-08
2026-09-09 02:27:25,764 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2026m09d11.nc
2026-09-09 02:27:25,783 INFO [grib_to_netcdf] 2026-09-09 NEMO-atmos forcing file for forecast2 created
2026-09-09 02:27:25,797 INFO [manager] checklist updated with [weather forcing] items from grib_to_netcdf worker
2026-09-09 02:27:30,954 ERROR [make_runoff_file] no 2026-09-08 discharge obs for SkagitMountVernon: patching
2026-09-09 02:27:32,908 ERROR [make_runoff_file] no 2026-09-08 discharge obs for SnohomishMonroe: patching
2026-09-09 02:27:35,558 ERROR [make_runoff_file] no 2026-09-08 discharge obs for NisquallyMcKenna: patching
2026-09-09 02:27:37,086 ERROR [make_runoff_file] no 2026-09-08 discharge obs for GreenwaterGreenwater: patching
2026-09-09 02:27:40,913 INFO [make_runoff_file] stored NEMO runoff forcing for v202108 bathymetry for 2026-09-08: /results/forcing/rivers/R202108Dailies_y2026m09d08.nc
2026-09-09 02:27:40,915 INFO [make_runoff_file] 2026-09-08 runoff file creation completed
2026-09-09 02:27:40,918 INFO [manager] checklist updated with [rivers forcing] items from make_runoff_file worker
2026-09-09 02:27:41,259 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'forecast2'], host='localhost')
2026-09-09 02:27:41,264 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.nibi', 'forecast2'], host='localhost')
2026-09-09 02:27:41,269 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'forecast2'], host='localhost')
2026-09-09 02:27:41,349 INFO [get_onc_ferry] requesting ONC TWDP OXYSENSOR data for 2026-09-08
2026-09-09 02:27:43,152 INFO [upload_forcing] running in process 3177528
2026-09-09 02:27:43,152 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-09 02:27:43,154 INFO [upload_forcing] publishing log messages to tcp://*:5562
2026-09-09 02:27:43,155 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-09-09 02:27:43,156 INFO [upload_forcing] running in process 3177530
2026-09-09 02:27:43,157 INFO [upload_forcing] running in process 3177529
2026-09-09 02:27:43,159 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-09 02:27:43,160 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-09 02:27:43,161 INFO [upload_forcing] publishing log messages to tcp://*:5563
2026-09-09 02:27:43,163 INFO [upload_forcing] publishing log messages to tcp://*:5565
2026-09-09 02:27:43,165 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-09-09 02:27:43,165 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-09-09 02:27:44,241 CRITICAL [upload_forcing] unhandled exception:
Traceback (most recent call last):
  File "/SalishSeaCast/SalishSeaNowcast/.pixi/envs/default/lib/python3.14/site-packages/paramiko/sftp_client.py", line 130, in __init__
    server_version = self._send_version()
  File "/SalishSeaCast/SalishSeaNowcast/.pixi/envs/default/lib/python3.14/site-packages/paramiko/sftp.py", line 145, in _send_version
    t, data = self._read_packet()
              ~~~~~~~~~~~~~~~~~^^
  File "/SalishSeaCast/SalishSeaNowcast/.pixi/envs/default/lib/python3.14/site-packages/paramiko/sftp.py", line 212, in _read_packet
    x = self._read_all(4)
  File "/SalishSeaCast/SalishSeaNowcast/.pixi/envs/default/lib/python3.14/site-packages/paramiko/sftp.py", line 199, in _read_all
    raise EOFError()
EOFError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/SalishSeaCast/SalishSeaNowcast/.pixi/envs/default/lib/python3.14/site-packages/nemo_nowcast/worker.py", line 388, in _do_work
    checklist = self.worker_func(
        self._parsed_args, self.config, self.tell_manager
    )
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/upload_forcing.py", line 95, in upload_forcing
    ssh_client, sftp_client = ssh_sftp.sftp(host_name, ssh_key)
                              ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/ssh_sftp.py", line 146, in sftp
    sftp_client = ssh_client.open_sftp()
  File "/SalishSeaCast/SalishSeaNowcast/.pixi/envs/default/lib/python3.14/site-packages/paramiko/client.py", line 606, in open_sftp
    return self._transport.open_sftp_client()
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/SalishSeaCast/SalishSeaNowcast/.pixi/envs/default/lib/python3.14/site-packages/paramiko/transport.py", line 1210, in open_sftp_client
    return SFTPClient.from_transport(self)
           ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^
  File "/SalishSeaCast/SalishSeaNowcast/.pixi/envs/default/lib/python3.14/site-packages/paramiko/sftp_client.py", line 170, in from_transport
    return cls(chan)
  File "/SalishSeaCast/SalishSeaNowcast/.pixi/envs/default/lib/python3.14/site-packages/paramiko/sftp_client.py", line 132, in __init__
    raise SSHException("EOF during negotiation")
paramiko.ssh_exception.SSHException: EOF during negotiation
2026-09-09 02:27:48,225 INFO [upload_forcing] LiveOcean boundary conditions file not found; created symlink to /results/forcing/LiveOcean/boundary_conditions/LiveOcean_v201905_y2026m09d08.nc
2026-09-09 02:27:48,226 INFO [upload_forcing] LiveOcean boundary conditions file not found; created symlink to /results/forcing/LiveOcean/boundary_conditions/LiveOcean_v201905_y2026m09d08.nc
2026-09-09 02:27:48,316 INFO [upload_forcing] forecast2 2026-09-09 forcing files upload to optimum-hindcast completed
2026-09-09 02:27:48,318 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-09-09 02:27:48,329 INFO [upload_forcing] forecast2 2026-09-09 forcing files upload to arbutus.cloud-nowcast completed
2026-09-09 02:27:48,644 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-09-09 02:27:48,945 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2026-09-09'], host='localhost')
2026-09-09 02:27:50,452 INFO [make_forcing_links] running in process 3177552
2026-09-09 02:27:50,454 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-09 02:27:50,455 INFO [make_forcing_links] publishing log messages to tcp://*:5562
2026-09-09 02:27:50,456 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2026-09-09 02:27:51,380 INFO [make_forcing_links] forecast2 2026-09-09 forcing file links on arbutus.cloud-nowcast created
2026-09-09 02:27:51,383 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2026-09-09 02:27:51,712 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2026-09-09'], host='arbutus.cloud-nowcast')
2026-09-09 02:27:53,056 INFO [run_NEMO] running in process 874887
2026-09-09 02:27:53,059 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-09-09 02:27:53,060 INFO [run_NEMO] publishing log messages to tcp://*:5556
2026-09-09 02:27:53,061 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2026-09-09 02:28:03,692 INFO [get_onc_ferry] requesting ONC TWDP TURBCHLFL data for 2026-09-08
2026-09-09 02:28:08,719 WARNING [run_NEMO] revision and status requested for non-existent repo: /nemoShare/MEOPAR/nowcast-sys/moad_tools
2026-09-09 02:28:08,720 WARNING [run_NEMO] revision and status requested for non-existent repo: /nemoShare/MEOPAR/nowcast-sys/NEMO-Cmd
2026-09-09 02:28:08,721 WARNING [run_NEMO] revision and status requested for non-existent repo: /nemoShare/MEOPAR/nowcast-sys/NEMO_Nowcast
2026-09-09 02:28:09,100 WARNING [run_NEMO] There are uncommitted changes in /nemoShare/MEOPAR/nowcast-sys/SalishSeaCmd
2026-09-09 02:28:09,788 WARNING [run_NEMO] revision and status requested for non-existent repo: /nemoShare/MEOPAR/nowcast-sys/tools
2026-09-09 02:28:10,238 INFO [run_NEMO] forecast2: launching /nemoShare/MEOPAR/nowcast-sys/runs/08sep26forecast2_2026-09-09T022808.203668-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2026-09-09 02:28:10,283 INFO [run_NEMO] forecast2 NEMO run for 2026-09-09 on arbutus.cloud-nowcast started
2026-09-09 02:28:10,287 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2026-09-09 02:28:10,602 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'forecast2'], host='arbutus.cloud-nowcast')
2026-09-09 02:28:11,503 INFO [watch_NEMO] running in process 875045
2026-09-09 02:28:11,505 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-09-09 02:28:11,506 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2026-09-09 02:28:11,507 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2026-09-09 02:28:11,608 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2026-09-09 02:28:26,123 INFO [get_onc_ferry] requesting ONC TWDP CO2SENSOR data for 2026-09-08
2026-09-09 02:28:26,298 WARNING [get_onc_ferry] No ONC TWDP CO2SENSOR partial_pressure,co2 data for 2026-09-08; substituting empty dataset
2026-09-09 02:28:26,305 WARNING [get_onc_ferry] No ONC TWDP CO2SENSOR partial_pressure data for 2026-09-08; substituting empty dataset
2026-09-09 02:28:26,307 WARNING [get_onc_ferry] No ONC TWDP CO2SENSOR co2 data for 2026-09-08; substituting empty dataset
2026-09-09 02:28:26,308 INFO [get_onc_ferry] requesting ONC TWDP TEMPHUMID data for 2026-09-08
2026-09-09 02:28:26,503 WARNING [get_onc_ferry] No ONC TWDP TEMPHUMID air_temperature,rel_humidity data for 2026-09-08; substituting empty dataset
2026-09-09 02:28:26,509 WARNING [get_onc_ferry] No ONC TWDP TEMPHUMID air_temperature data for 2026-09-08; substituting empty dataset
2026-09-09 02:28:26,511 WARNING [get_onc_ferry] No ONC TWDP TEMPHUMID rel_humidity data for 2026-09-08; substituting empty dataset
2026-09-09 02:28:26,512 INFO [get_onc_ferry] requesting ONC TWDP BARPRESS data for 2026-09-08
2026-09-09 02:28:26,697 WARNING [get_onc_ferry] No ONC TWDP BARPRESS barometric_pressure data for 2026-09-08; substituting empty dataset
2026-09-09 02:28:26,702 WARNING [get_onc_ferry] No ONC TWDP BARPRESS barometric_pressure data for 2026-09-08; substituting empty dataset
2026-09-09 02:28:26,703 INFO [get_onc_ferry] requesting ONC TWDP PYRANOMETER data for 2026-09-08
2026-09-09 02:28:26,836 WARNING [get_onc_ferry] No ONC TWDP PYRANOMETER solar_radiation data for 2026-09-08; substituting empty dataset
2026-09-09 02:28:26,840 WARNING [get_onc_ferry] No ONC TWDP PYRANOMETER solar_radiation data for 2026-09-08; substituting empty dataset
2026-09-09 02:28:26,841 INFO [get_onc_ferry] requesting ONC TWDP PYRGEOMETER data for 2026-09-08
2026-09-09 02:28:26,977 WARNING [get_onc_ferry] No ONC TWDP PYRGEOMETER downward_radiation data for 2026-09-08; substituting empty dataset
2026-09-09 02:28:26,983 WARNING [get_onc_ferry] No ONC TWDP PYRGEOMETER downward_radiation data for 2026-09-08; substituting empty dataset
2026-09-09 02:28:27,127 WARNING [get_onc_ferry] ONC TWDP CO2SENSOR partial_pressure data for 2026-09-08 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2026-09-09 02:28:27,142 WARNING [get_onc_ferry] ONC TWDP CO2SENSOR co2 data for 2026-09-08 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2026-09-09 02:28:27,158 WARNING [get_onc_ferry] ONC TWDP TEMPHUMID air_temperature data for 2026-09-08 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2026-09-09 02:28:27,172 WARNING [get_onc_ferry] ONC TWDP TEMPHUMID rel_humidity data for 2026-09-08 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2026-09-09 02:28:27,187 WARNING [get_onc_ferry] ONC TWDP BARPRESS barometric_pressure data for 2026-09-08 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2026-09-09 02:28:27,201 WARNING [get_onc_ferry] ONC TWDP PYRANOMETER solar_radiation data for 2026-09-08 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2026-09-09 02:28:27,216 WARNING [get_onc_ferry] ONC TWDP PYRGEOMETER downward_radiation data for 2026-09-08 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2026-09-09 02:28:33,413 INFO [get_onc_ferry] 2026-09-08 ONC TWDP ferry data file created
2026-09-09 02:28:33,415 INFO [manager] checklist updated with [ONC ferry data] items from get_onc_ferry worker
2026-09-09 02:28:33,687 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['TWDP-ferry'], host='localhost')
2026-09-09 02:28:35,147 INFO [ping_erddap] running in process 3177581
2026-09-09 02:28:35,148 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-09 02:28:35,149 INFO [ping_erddap] publishing log messages to tcp://*:5562
2026-09-09 02:28:35,151 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2026-09-09 02:28:35,152 INFO [ping_erddap] TWDP-ferry ERDDAP dataset flag file(s) created
2026-09-09 02:28:35,154 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2026-09-09 02:33:11,609 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 19486423 = 2026-09-10 11:48:00 UTC, 39.3% complete
2026-09-09 02:38:11,612 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 19487612 = 2026-09-11 01:00:40 UTC, 83.4% complete
2026-09-09 02:43:11,618 INFO [watch_NEMO] forecast2 NEMO run on arbutus.cloud-nowcast completed
2026-09-09 02:43:11,623 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2026-09-09 02:43:12,195 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_wind_file', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2026-09-09'], host='arbutus.cloud-nowcast')
2026-09-09 02:43:12,208 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_current_file', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2026-09-09'], host='arbutus.cloud-nowcast')
2026-09-09 02:43:12,213 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2026-09-08'], host='localhost')
2026-09-09 02:43:13,555 INFO [make_ww3_current_file] running in process 875297
2026-09-09 02:43:13,557 INFO [make_ww3_current_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-09-09 02:43:13,558 INFO [make_ww3_current_file] publishing log messages to tcp://*:5571
2026-09-09 02:43:13,559 INFO [make_ww3_current_file] connected to skookum.eos.ubc.ca port 5555
2026-09-09 02:43:13,560 INFO [make_ww3_current_file] Creating wwatch3 currents forcing file for 2026-09-09 forecast2 run
2026-09-09 02:43:13,576 INFO [make_ww3_wind_file] running in process 875296
2026-09-09 02:43:13,577 INFO [make_ww3_wind_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-09-09 02:43:13,578 INFO [make_ww3_wind_file] publishing log messages to tcp://*:5570
2026-09-09 02:43:13,579 INFO [make_ww3_wind_file] connected to skookum.eos.ubc.ca port 5555
2026-09-09 02:43:13,579 INFO [make_ww3_wind_file] Creating wwatch3 wind forcing file for 2026-09-09 forecast2 run
2026-09-09 02:43:13,753 INFO [download_results] running in process 3177897
2026-09-09 02:43:13,754 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-09 02:43:13,755 INFO [download_results] publishing log messages to tcp://*:5562
2026-09-09 02:43:13,756 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2026-09-09 02:43:13,757 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/forecast2/08sep26 to /results/SalishSea/forecast2.202111
2026-09-09 02:43:21,862 INFO [make_ww3_wind_file] wwatch3 wind forcing file created on arbutus.cloud-nowcast for 2026-09-09 forecast2 run
2026-09-09 02:43:21,868 INFO [manager] checklist updated with [WW3 wind forcing] items from make_ww3_wind_file worker
2026-09-09 02:44:24,567 INFO [download_results] forecast2 2026-09-08 results files from arbutus.cloud-nowcast downloaded
2026-09-09 02:44:24,571 INFO [manager] checklist updated with [results files] items from download_results worker
2026-09-09 02:44:24,933 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['forecast2', '--run-date', '2026-09-08'], host='localhost')
2026-09-09 02:44:27,343 INFO [make_CHS_currents_file] running in process 3177927
2026-09-09 02:44:27,345 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-09 02:44:27,346 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5562
2026-09-09 02:44:27,347 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555
2026-09-09 02:45:07,418 INFO [make_CHS_currents_file] Made CHS currents file for 2026-09-08 for forecast2
2026-09-09 02:45:07,421 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker
2026-09-09 02:45:07,750 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['nemo', 'forecast2', '--run-date', '2026-09-08'], host='localhost')
2026-09-09 02:45:09,114 INFO [update_forecast_datasets] running in process 3177961
2026-09-09 02:45:09,115 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-09 02:45:09,116 INFO [update_forecast_datasets] publishing log messages to tcp://*:5562
2026-09-09 02:45:09,117 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2026-09-09 02:45:09,119 INFO [update_forecast_datasets] updating nemo forecast directory for forecast2/08sep26 run
2026-09-09 02:46:34,459 INFO [make_ww3_current_file] wwatch3 currents forcing file created on arbutus.cloud-nowcast for 2026-09-09 forecast2 run
2026-09-09 02:46:34,464 INFO [manager] checklist updated with [WW3 currents forcing] items from make_ww3_current_file worker
2026-09-09 02:46:34,800 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2026-09-09'], host='arbutus.cloud-nowcast')
2026-09-09 02:46:35,696 INFO [run_ww3] running in process 875707
2026-09-09 02:46:35,697 INFO [run_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-09-09 02:46:35,698 INFO [run_ww3] publishing log messages to tcp://*:5572
2026-09-09 02:46:35,700 INFO [run_ww3] connected to skookum.eos.ubc.ca port 5555
2026-09-09 02:46:35,703 INFO [run_ww3] Created run directory /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast2_2026-09-09T024635.692782-0700
2026-09-09 02:46:35,705 INFO [run_ww3] forecast2: launching /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast2_2026-09-09T024635.692782-0700/SoGWW3.sh on arbutus.cloud-nowcast
2026-09-09 02:46:35,753 INFO [run_ww3] forecast2 WaveWatch3 run for 2026-09-09 on arbutus.cloud-nowcast started
2026-09-09 02:46:35,757 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker
2026-09-09 02:46:36,087 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'forecast2'], host='arbutus.cloud-nowcast')
2026-09-09 02:46:36,964 INFO [watch_ww3] running in process 875724
2026-09-09 02:46:36,966 INFO [watch_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-09-09 02:46:36,967 INFO [watch_ww3] publishing log messages to tcp://*:5573
2026-09-09 02:46:36,968 INFO [watch_ww3] connected to skookum.eos.ubc.ca port 5555
2026-09-09 02:46:37,034 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2026-09-09 02:51:37,038 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 423, 78.3% complete
2026-09-09 02:56:37,038 INFO [watch_ww3] forecast2 WWATCH3 run on arbutus.cloud-nowcast completed
2026-09-09 02:56:37,043 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2026-09-09 02:56:37,629 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2026-09-09'], host='localhost')
2026-09-09 02:56:39,322 INFO [download_wwatch3_results] running in process 3178256
2026-09-09 02:56:39,324 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-09 02:56:39,325 INFO [download_wwatch3_results] publishing log messages to tcp://*:5563
2026-09-09 02:56:39,326 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2026-09-09 02:56:57,470 INFO [download_wwatch3_results] forecast2 2026-09-09 results files from arbutus.cloud-nowcast downloaded
2026-09-09 02:56:57,474 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2026-09-09 02:56:57,751 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['wwatch3', 'forecast2', '--run-date', '2026-09-09'], host='localhost')
2026-09-09 02:56:59,227 INFO [update_forecast_datasets] running in process 3178268
2026-09-09 02:56:59,229 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-09 02:56:59,230 INFO [update_forecast_datasets] publishing log messages to tcp://*:5563
2026-09-09 02:56:59,231 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2026-09-09 02:56:59,232 INFO [update_forecast_datasets] updating wwatch3 most_recent_forecast directory from forecast2/09sep26 run
2026-09-09 02:56:59,235 INFO [update_forecast_datasets] updated wwatch3 most_recent_forecast directory from forecast2/09sep26 run
2026-09-09 02:56:59,236 INFO [update_forecast_datasets] updating wwatch3 forecast directory for forecast2/09sep26 run
2026-09-09 02:57:29,572 INFO [update_forecast_datasets] updated wwatch3 forecast directory for forecast2/09sep26 run: /results/SalishSea/rolling-forecasts/wwatch3
2026-09-09 02:57:29,573 INFO [update_forecast_datasets] wwatch3 2026-09-09 forecast2 rolling forecast datasets updated
2026-09-09 02:57:29,578 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2026-09-09 02:57:29,916 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['wwatch3-forecast'], host='localhost')
2026-09-09 02:57:31,238 INFO [ping_erddap] running in process 3178294
2026-09-09 02:57:31,240 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-09 02:57:31,241 INFO [ping_erddap] publishing log messages to tcp://*:5563
2026-09-09 02:57:31,242 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2026-09-09 02:57:31,244 INFO [ping_erddap] wwatch3-forecast ERDDAP dataset flag file(s) created
2026-09-09 02:57:31,246 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2026-09-09 02:57:31,585 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['wwatch3', 'forecast', 'publish', '--run-date', '2026-09-08'], host='localhost')
2026-09-09 02:57:36,230 INFO [make_plots] running in process 3178301
2026-09-09 02:57:36,231 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-09 02:57:36,233 INFO [make_plots] publishing log messages to tcp://*:5563
2026-09-09 02:57:36,234 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2026-09-09 02:57:36,235 INFO [make_plots] preparing render list for 2026-09-08 WaveWatch3 forecast publish figures
2026-09-09 02:57:36,236 INFO [make_plots] added Halibut Bank figure to 2026-09-08 WaveWatch3 forecast publish render list
2026-09-09 02:57:36,237 INFO [make_plots] added Sentry Shoal figure to 2026-09-08 WaveWatch3 forecast publish render list
2026-09-09 02:57:36,238 INFO [make_plots] starting to render wwatch3 forecast publish 08sep26 figures
2026-09-09 02:57:36,718 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46146 Halibut Bank
2026-09-09 02:57:41,755 WARNING [make_plots] findfont: Failed to find font weight medium, now using 400.
2026-09-09 02:57:41,758 WARNING [make_plots] findfont: Failed to find font weight medium, now using 400.
2026-09-09 02:57:41,778 WARNING [make_plots] findfont: Failed to find font weight medium, now using 400.
2026-09-09 02:57:41,780 WARNING [make_plots] findfont: Failed to find font weight medium, now using 400.
2026-09-09 02:57:46,036 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46131 Sentry Shoal
2026-09-09 02:57:53,771 INFO [make_plots] finished rendering wwatch3 forecast publish 08sep26 figures
2026-09-09 02:57:53,772 INFO [make_plots] wwatch3 publish plots for 2026-09-08 forecast completed
2026-09-09 02:57:53,776 INFO [manager] checklist updated with [plots] items from make_plots worker
2026-09-09 02:58:20,355 INFO [update_forecast_datasets] updated nemo forecast directory for forecast2/08sep26 run: /results/SalishSea/rolling-forecasts/nemo
2026-09-09 02:58:20,355 INFO [update_forecast_datasets] nemo 2026-09-08 forecast2 rolling forecast datasets updated
2026-09-09 02:58:20,359 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2026-09-09 02:58:20,702 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nemo-forecast'], host='localhost')
2026-09-09 02:58:20,707 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'forecast2', 'publish', '--run-date', '2026-09-08'], host='localhost')
2026-09-09 02:58:20,712 INFO [manager] launching NextWorker(module='nowcast.workers.make_surface_current_tiles', args=['forecast2', '--run-date', '2026-09-08'], host='localhost')
2026-09-09 02:58:22,557 INFO [ping_erddap] running in process 3178360
2026-09-09 02:58:22,558 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-09 02:58:22,559 INFO [ping_erddap] publishing log messages to tcp://*:5562
2026-09-09 02:58:22,560 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2026-09-09 02:58:22,574 INFO [ping_erddap] nemo-forecast ERDDAP dataset flag file(s) created
2026-09-09 02:58:22,593 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2026-09-09 02:58:24,158 INFO [make_surface_current_tiles] running in process 3178363
2026-09-09 02:58:24,159 INFO [make_surface_current_tiles] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-09 02:58:24,160 INFO [make_surface_current_tiles] publishing log messages to tcp://*:5562
2026-09-09 02:58:24,161 INFO [make_surface_current_tiles] connected to skookum.eos.ubc.ca port 5555
2026-09-09 02:58:24,979 INFO [make_plots] running in process 3178362
2026-09-09 02:58:24,981 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-09 02:58:24,983 INFO [make_plots] publishing log messages to tcp://*:5563
2026-09-09 02:58:24,984 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2026-09-09 02:58:25,106 INFO [make_surface_current_tiles] starting rendering of tiles for 2026-09-08 forecast2 in 6 processes into /results/nowcast-sys/figures/surface_currents/forecast2/08sep26
2026-09-09 02:58:25,452 INFO [make_plots] preparing render list for 2026-09-08 NEMO forecast2 publish figures
2026-09-09 02:58:27,532 INFO [make_plots] added Website_thumbnail figure to 2026-09-08 NEMO forecast2 publish render list
2026-09-09 02:58:27,533 INFO [make_plots] added Threshold_website figure to 2026-09-08 NEMO forecast2 publish render list
2026-09-09 02:58:27,534 INFO [make_plots] added PA_tidal_predictions figure to 2026-09-08 NEMO forecast2 publish render list
2026-09-09 02:58:27,534 INFO [make_plots] added Boundary Bay figure to 2026-09-08 NEMO forecast2 publish render list
2026-09-09 02:58:27,535 INFO [make_plots] added Campbell River figure to 2026-09-08 NEMO forecast2 publish render list
2026-09-09 02:58:27,536 INFO [make_plots] added Cherry Point figure to 2026-09-08 NEMO forecast2 publish render list
2026-09-09 02:58:27,536 INFO [make_plots] added Friday Harbor figure to 2026-09-08 NEMO forecast2 publish render list
2026-09-09 02:58:27,537 INFO [make_plots] added Halfmoon Bay figure to 2026-09-08 NEMO forecast2 publish render list
2026-09-09 02:58:27,538 INFO [make_plots] added Nanaimo figure to 2026-09-08 NEMO forecast2 publish render list
2026-09-09 02:58:27,538 INFO [make_plots] added Neah Bay figure to 2026-09-08 NEMO forecast2 publish render list
2026-09-09 02:58:27,539 INFO [make_plots] added New Westminster figure to 2026-09-08 NEMO forecast2 publish render list
2026-09-09 02:58:27,540 INFO [make_plots] added Patricia Bay figure to 2026-09-08 NEMO forecast2 publish render list
2026-09-09 02:58:27,541 INFO [make_plots] added Point Atkinson figure to 2026-09-08 NEMO forecast2 publish render list
2026-09-09 02:58:27,541 INFO [make_plots] added Port Renfrew figure to 2026-09-08 NEMO forecast2 publish render list
2026-09-09 02:58:27,542 INFO [make_plots] added Sand Heads figure to 2026-09-08 NEMO forecast2 publish render list
2026-09-09 02:58:27,543 INFO [make_plots] added Sandy Cove figure to 2026-09-08 NEMO forecast2 publish render list
2026-09-09 02:58:27,544 INFO [make_plots] added Squamish figure to 2026-09-08 NEMO forecast2 publish render list
2026-09-09 02:58:27,545 INFO [make_plots] added Victoria figure to 2026-09-08 NEMO forecast2 publish render list
2026-09-09 02:58:27,545 INFO [make_plots] added Woodwards Landing figure to 2026-09-08 NEMO forecast2 publish render list
2026-09-09 02:58:27,546 INFO [make_plots] starting to render nemo forecast2 publish 08sep26 figures
2026-09-09 03:01:02,050 WARNING [make_plots] findfont: Failed to find font weight medium, now using 400.
2026-09-09 03:01:02,052 WARNING [make_plots] findfont: Failed to find font weight medium, now using 400.
2026-09-09 03:01:02,056 WARNING [make_plots] findfont: Failed to find font weight medium, now using 400.
2026-09-09 03:01:02,058 WARNING [make_plots] findfont: Failed to find font weight medium, now using 400.
2026-09-09 03:01:02,372 WARNING [make_plots] findfont: Failed to find font weight medium, now using 400.
2026-09-09 03:01:02,374 WARNING [make_plots] findfont: Failed to find font weight medium, now using 400.
2026-09-09 03:01:02,377 WARNING [make_plots] findfont: Failed to find font weight medium, now using 400.
2026-09-09 03:01:02,379 WARNING [make_plots] findfont: Failed to find font weight medium, now using 400.
2026-09-09 03:01:02,389 WARNING [make_plots] findfont: Failed to find font weight medium, now using 400.
2026-09-09 03:01:02,391 WARNING [make_plots] findfont: Failed to find font weight medium, now using 400.
2026-09-09 03:05:51,408 WARNING [make_plots] findfont: Failed to find font weight medium, now using 400.
2026-09-09 03:05:51,409 WARNING [make_plots] findfont: Failed to find font weight medium, now using 400.
2026-09-09 03:05:53,139 WARNING [make_plots] invalid station number for Boundary Bay station: None
2026-09-09 03:05:53,140 WARNING [make_plots] can't resolve a valid CHS station code for Boundary Bay
2026-09-09 03:06:10,150 WARNING [make_plots] findfont: Failed to find font weight medium, now using 400.
2026-09-09 03:06:10,152 WARNING [make_plots] findfont: Failed to find font weight medium, now using 400.
2026-09-09 03:06:17,736 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb996/data for station 08074 Campbell River from 2026-09-04 00:00:00Z to 2026-09-11 05:55:00Z
2026-09-09 03:07:24,924 WARNING [make_plots] invalid station number for Halfmoon Bay station: None
2026-09-09 03:07:24,925 WARNING [make_plots] can't resolve a valid CHS station code for Halfmoon Bay
2026-09-09 03:07:45,316 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb96d/data for station 07917 Nanaimo from 2026-09-04 00:00:00Z to 2026-09-11 05:55:00Z
2026-09-09 03:08:24,944 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e13d0f4a073c4bbf93/data for station 07654 New Westminster from 2026-09-04 00:00:00Z to 2026-09-11 05:55:00Z
2026-09-09 03:08:48,658 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd2d/data for station 07277 Patricia Bay from 2026-09-04 00:00:00Z to 2026-09-11 05:55:00Z
2026-09-09 03:09:10,651 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94c/data for station 07795 Point Atkinson from 2026-09-04 00:00:00Z to 2026-09-11 05:55:00Z
2026-09-09 03:09:33,215 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e23d0f4a073c4bc060/data for station 08525 Port Renfrew from 2026-09-04 00:00:00Z to 2026-09-11 05:55:00Z
2026-09-09 03:09:52,123 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb93c/data for station 07594 Sand Heads from 2026-09-04 00:00:00Z to 2026-09-11 05:55:00Z
2026-09-09 03:10:17,262 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc43b/data for station 07786 Sandy Cove from 2026-09-04 00:00:00Z to 2026-09-11 05:55:00Z
2026-09-09 03:10:40,380 WARNING [make_plots] invalid station number for Squamish station: None
2026-09-09 03:10:40,380 WARNING [make_plots] can't resolve a valid CHS station code for Squamish
2026-09-09 03:11:02,430 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd1e/data for station 07120 Victoria from 2026-09-04 00:00:00Z to 2026-09-11 05:55:00Z
2026-09-09 03:11:22,829 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc42d/data for station 07610 Woodwards Landing from 2026-09-04 00:00:00Z to 2026-09-11 05:55:00Z
2026-09-09 03:11:23,303 INFO [make_plots] no obs water level data available from 5cebf1e43d0f4a073c4bc42d during 2026-09-04 00:00:00Z to 2026-09-11 05:55:00Z
2026-09-09 03:11:44,793 INFO [make_plots] finished rendering nemo forecast2 publish 08sep26 figures
2026-09-09 03:11:44,796 INFO [make_plots] nemo publish plots for 2026-09-08 forecast2 completed
2026-09-09 03:11:44,802 INFO [manager] checklist updated with [plots] items from make_plots worker
2026-09-09 03:11:45,136 INFO [manager] launching NextWorker(module='nowcast.workers.make_feeds', args=['forecast2', '--run-date', '2026-09-08'], host='localhost')
2026-09-09 03:11:49,023 INFO [make_feeds] running in process 3180062
2026-09-09 03:11:49,024 INFO [make_feeds] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-09 03:11:49,025 INFO [make_feeds] publishing log messages to tcp://*:5563
2026-09-09 03:11:49,027 INFO [make_feeds] connected to skookum.eos.ubc.ca port 5555
2026-09-09 03:12:07,389 INFO [make_feeds] ATOM feeds for 2026-09-08 forecast2 run completed
2026-09-09 03:12:07,393 INFO [manager] checklist updated with [storm surge alert feeds] items from make_feeds worker
2026-09-09 03:12:07,750 INFO [manager] launching NextWorker(module='nemo_nowcast.workers.clear_checklist', args=[], host='localhost')
2026-09-09 03:12:09,508 INFO [clear_checklist] running in process 3180122
2026-09-09 03:12:09,509 INFO [clear_checklist] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-09 03:12:09,511 INFO [clear_checklist] publishing log messages to tcp://*:5563
2026-09-09 03:12:09,512 INFO [clear_checklist] connected to skookum.eos.ubc.ca port 5555
2026-09-09 03:12:09,513 INFO [clear_checklist] requesting that manager clear system state checklist
2026-09-09 03:12:09,514 INFO [manager] writing checklist to log file
2026-09-09 03:12:09,561 INFO [manager] checklist cleared
2026-09-09 03:12:09,563 INFO [clear_checklist] nowcast system checklist cleared
2026-09-09 03:12:09,565 INFO [manager] launching NextWorker(module='nemo_nowcast.workers.rotate_logs', args=[], host='localhost')
2026-09-09 03:12:11,179 INFO [rotate_logs] running in process 3180128
2026-09-09 03:12:11,180 INFO [rotate_logs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-09 03:12:11,181 INFO [rotate_logs] publishing log messages to tcp://*:5563
2026-09-09 03:12:11,182 INFO [rotate_logs] connected to skookum.eos.ubc.ca port 5555
2026-09-09 03:12:11,183 INFO [rotate_logs] rotating log files