2026-07-12 03:48:27,997 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/nowcast.log
2026-07-12 03:48:28,006 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/nowcast.debug.log
2026-07-12 03:48:28,013 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/checklist.log
2026-07-12 03:48:28,013 INFO [rotate_logs] log files rotated
2026-07-12 03:48:28,181 INFO [manager] checklist updated with [log rotation] items from rotate_logs worker
2026-07-12 03:50:20,718 INFO [make_surface_current_tiles] finished PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast2/11jul26
2026-07-12 03:50:20,741 INFO [make_surface_current_tiles] finished rendering of tiles for 2026-07-11 forecast2 into /results/nowcast-sys/figures/surface_currents/forecast2/11jul26
2026-07-12 03:50:20,743 INFO [make_surface_current_tiles] surface current tile figures for 2026-07-11 forecast2 completed
2026-07-12 03:50:21,009 INFO [manager] checklist updated with [surface current tiles] items from make_surface_current_tiles worker
2026-07-12 03:52:25,982 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 405, 75.0% complete
2026-07-12 03:57:25,993 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 504, 93.3% complete
2026-07-12 04:02:25,988 INFO [watch_ww3] forecast2 WWATCH3 run on arbutus.cloud-nowcast completed
2026-07-12 04:02:25,989 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2026-07-12 04:02:26,344 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2026-07-12'], host='localhost')
2026-07-12 04:02:27,595 INFO [download_wwatch3_results] running in process 253768
2026-07-12 04:02:27,596 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-07-12 04:02:27,597 INFO [download_wwatch3_results] publishing log messages to tcp://*:5562
2026-07-12 04:02:27,598 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2026-07-12 04:03:06,022 INFO [download_wwatch3_results] forecast2 2026-07-12 results files from arbutus.cloud-nowcast downloaded
2026-07-12 04:03:06,025 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2026-07-12 04:03:06,179 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['wwatch3', 'forecast2', '--run-date', '2026-07-12'], host='localhost')
2026-07-12 04:03:07,400 INFO [update_forecast_datasets] running in process 253793
2026-07-12 04:03:07,401 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-07-12 04:03:07,402 INFO [update_forecast_datasets] publishing log messages to tcp://*:5562
2026-07-12 04:03:07,403 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2026-07-12 04:03:07,404 INFO [update_forecast_datasets] updating wwatch3 most_recent_forecast directory from forecast2/12jul26 run
2026-07-12 04:03:07,428 INFO [update_forecast_datasets] updated wwatch3 most_recent_forecast directory from forecast2/12jul26 run
2026-07-12 04:03:07,429 INFO [update_forecast_datasets] updating wwatch3 forecast directory for forecast2/12jul26 run
2026-07-12 04:03:34,008 INFO [update_forecast_datasets] updated wwatch3 forecast directory for forecast2/12jul26 run: /results/SalishSea/rolling-forecasts/wwatch3
2026-07-12 04:03:34,008 INFO [update_forecast_datasets] wwatch3 2026-07-12 forecast2 rolling forecast datasets updated
2026-07-12 04:03:34,012 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2026-07-12 04:03:34,151 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['wwatch3-forecast'], host='localhost')
2026-07-12 04:03:35,290 INFO [ping_erddap] running in process 253816
2026-07-12 04:03:35,291 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-07-12 04:03:35,292 INFO [ping_erddap] publishing log messages to tcp://*:5562
2026-07-12 04:03:35,293 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2026-07-12 04:03:35,295 INFO [ping_erddap] wwatch3-forecast ERDDAP dataset flag file(s) created
2026-07-12 04:03:35,297 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2026-07-12 04:03:35,614 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['wwatch3', 'forecast2', 'publish', '--run-date', '2026-07-12'], host='localhost')
2026-07-12 04:03:38,659 INFO [make_plots] running in process 253821
2026-07-12 04:03:38,660 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-07-12 04:03:38,661 INFO [make_plots] publishing log messages to tcp://*:5562
2026-07-12 04:03:38,662 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2026-07-12 04:03:38,662 INFO [make_plots] preparing render list for 2026-07-12 WaveWatch3 forecast2 publish figures
2026-07-12 04:03:38,663 INFO [make_plots] added Halibut Bank figure to 2026-07-12 WaveWatch3 forecast2 publish render list
2026-07-12 04:03:38,664 INFO [make_plots] added Sentry Shoal figure to 2026-07-12 WaveWatch3 forecast2 publish render list
2026-07-12 04:03:38,664 INFO [make_plots] starting to render wwatch3 forecast2 publish 12jul26 figures
2026-07-12 04:03:39,338 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46146 Halibut Bank
2026-07-12 04:03:45,224 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46131 Sentry Shoal
2026-07-12 04:03:49,510 INFO [make_plots] finished rendering wwatch3 forecast2 publish 12jul26 figures
2026-07-12 04:03:49,511 INFO [make_plots] wwatch3 publish plots for 2026-07-12 forecast2 completed
2026-07-12 04:03:49,515 INFO [manager] checklist updated with [plots] items from make_plots worker
2026-07-12 08:50:49,171 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/12/ to /results/forcing/atmospheric/continental2.5/GRIB/20260712/12/
2026-07-12 08:50:49,173 INFO [collect_weather] 2026-07-12 2.5km weather forecast 12 collection complete
2026-07-12 08:50:49,179 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2026-07-12 08:50:49,629 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'SkagitMountVernon', '--data-date', '2026-07-11'], host='localhost')
2026-07-12 08:50:49,638 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'SnohomishMonroe', '--data-date', '2026-07-11'], host='localhost')
2026-07-12 08:50:49,641 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'NisquallyMcKenna', '--data-date', '2026-07-11'], host='localhost')
2026-07-12 08:50:49,654 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'GreenwaterGreenwater', '--data-date', '2026-07-11'], host='localhost')
2026-07-12 08:50:49,661 INFO [manager] launching NextWorker(module='nowcast.workers.make_turbidity_file', args=[], host='localhost')
2026-07-12 08:50:49,669 INFO [manager] launching NextWorker(module='nowcast.workers.collect_NeahBay_ssh', args=['06'], host='localhost')
2026-07-12 08:50:49,674 INFO [manager] launching NextWorker(module='nowcast.workers.download_live_ocean', args=[], host='localhost')
2026-07-12 08:50:49,678 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['18', '2.5km'], host='localhost')
2026-07-12 08:50:49,685 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['18'], host='localhost')
2026-07-12 08:50:49,814 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20260712/12/
2026-07-12 08:50:49,816 INFO [crop_gribs] 2026-07-12 12 GRIBs cropping complete
2026-07-12 08:50:49,821 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2026-07-12 08:50:50,203 INFO [manager] launching NextWorker(module='nowcast.workers.grib_to_netcdf', args=['nowcast+'], host='localhost')
2026-07-12 08:50:52,425 INFO [download_live_ocean] running in process 259996
2026-07-12 08:50:52,426 INFO [download_live_ocean] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-07-12 08:50:52,427 INFO [download_live_ocean] publishing log messages to tcp://*:5562
2026-07-12 08:50:52,428 INFO [download_live_ocean] connected to skookum.eos.ubc.ca port 5555
2026-07-12 08:50:52,430 INFO [download_live_ocean] downloading Salish Sea western boundary day-averaged LiveOcean file for 2026-07-12
2026-07-12 08:50:52,432 INFO [collect_weather] running in process 259997
2026-07-12 08:50:52,433 INFO [collect_NeahBay_ssh] running in process 259995
2026-07-12 08:50:52,434 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-07-12 08:50:52,435 INFO [collect_NeahBay_ssh] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-07-12 08:50:52,436 INFO [collect_weather] publishing log messages to tcp://*:5563
2026-07-12 08:50:52,437 INFO [collect_NeahBay_ssh] publishing log messages to tcp://*:5564
2026-07-12 08:50:52,438 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2026-07-12 08:50:52,439 INFO [collect_NeahBay_ssh] connected to skookum.eos.ubc.ca port 5555
2026-07-12 08:50:52,440 INFO [collect_NeahBay_ssh] collecting Neah Bay ssh 06Z obs/forecast for 2026-07-12
2026-07-12 08:50:52,463 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/18/
2026-07-12 08:50:53,016 INFO [download_live_ocean] downloaded 10211947 bytes from https://s3.kopah.uw.edu/liveocean-share/f2026.07.12/ubc.nc to /results/forcing/LiveOcean/downloaded/20260712/low_passed_UBC.nc
2026-07-12 08:50:53,018 INFO [download_live_ocean] Deflating in up to 1 concurrent sub-processes
2026-07-12 08:50:53,379 INFO [collect_river_data] running in process 259992
2026-07-12 08:50:53,383 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-07-12 08:50:53,384 INFO [collect_river_data] publishing log messages to tcp://*:5565
2026-07-12 08:50:53,385 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-07-12 08:50:53,386 INFO [collect_river_data] Collecting USGS GreenwaterGreenwater river data for 2026-07-11
2026-07-12 08:50:53,387 INFO [collect_river_data] running in process 259990
2026-07-12 08:50:53,388 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-07-12 08:50:53,389 INFO [collect_river_data] publishing log messages to tcp://*:5566
2026-07-12 08:50:53,390 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-07-12 08:50:53,391 INFO [collect_river_data] Collecting USGS NisquallyMcKenna river data for 2026-07-11
2026-07-12 08:50:53,392 INFO [collect_river_data] running in process 259989
2026-07-12 08:50:53,393 INFO [collect_river_data] running in process 259987
2026-07-12 08:50:53,394 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-07-12 08:50:53,395 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-07-12 08:50:53,396 INFO [collect_river_data] publishing log messages to tcp://*:5567
2026-07-12 08:50:53,397 INFO [collect_river_data] publishing log messages to tcp://*:5568
2026-07-12 08:50:53,398 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-07-12 08:50:53,399 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-07-12 08:50:53,400 INFO [collect_river_data] Collecting USGS SnohomishMonroe river data for 2026-07-11
2026-07-12 08:50:53,401 INFO [collect_river_data] Collecting USGS SkagitMountVernon river data for 2026-07-11
2026-07-12 08:50:53,561 INFO [make_turbidity_file] running in process 259994
2026-07-12 08:50:53,562 INFO [make_turbidity_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-07-12 08:50:53,563 INFO [make_turbidity_file] publishing log messages to tcp://*:5569
2026-07-12 08:50:53,564 INFO [make_turbidity_file] connected to skookum.eos.ubc.ca port 5555
2026-07-12 08:50:53,565 INFO [make_turbidity_file] Creating Fraser River turbidity forcing file for 2026-07-12
2026-07-12 08:50:53,580 INFO [grib_to_netcdf] running in process 260001
2026-07-12 08:50:53,581 INFO [grib_to_netcdf] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-07-12 08:50:53,582 INFO [grib_to_netcdf] publishing log messages to tcp://*:5570
2026-07-12 08:50:53,583 INFO [grib_to_netcdf] connected to skookum.eos.ubc.ca port 5555
2026-07-12 08:50:53,584 INFO [grib_to_netcdf] creating NEMO-atmos forcing files for 2026-07-12 nowcast and forecast runs
2026-07-12 08:50:54,015 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12150800&startDT=2026-07-11&endDT=2026-07-11 "HTTP/1.1 200 "
2026-07-12 08:50:54,023 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12200500&startDT=2026-07-11&endDT=2026-07-11 "HTTP/1.1 200 "
2026-07-12 08:50:54,037 INFO [collect_NeahBay_ssh] 2026-07-12T00:00:00-07:00 Neah Bay ssh 06Z obs/forecast data collection complete
2026-07-12 08:50:54,040 INFO [manager] checklist updated with [Neah Bay ssh data] items from collect_NeahBay_ssh worker
2026-07-12 08:50:54,042 INFO [crop_gribs] running in process 259998
2026-07-12 08:50:54,043 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-07-12 08:50:54,044 INFO [crop_gribs] publishing log messages to tcp://*:5571
2026-07-12 08:50:54,045 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2026-07-12 08:50:54,046 INFO [crop_gribs] cropping 2026-07-12 ECCC HRDPS 2.5km continental 18Z GRIB files to SalishSeaCast subdomain
2026-07-12 08:50:54,060 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20260712/18/
2026-07-12 08:50:54,138 INFO [download_live_ocean] netCDF4 deflated /results/forcing/LiveOcean/downloaded/20260712/low_passed_UBC.nc
2026-07-12 08:50:54,139 INFO [download_live_ocean] 2026-07-12 Live Ocean file for Salish Sea western boundary downloaded
2026-07-12 08:50:54,202 INFO [collect_river_data] Appended USGS SnohomishMonroe river average discharge for 2026-07-11 to: /results/forcing/rivers/observations/Snohomish_Monroe_flow
2026-07-12 08:50:54,221 INFO [collect_river_data] USGS SnohomishMonroe river data collection for 2026-07-11 completed
2026-07-12 08:50:54,236 INFO [collect_river_data] Appended USGS SkagitMountVernon river average discharge for 2026-07-11 to: /results/forcing/rivers/observations/Skagit_MountVernon_flow
2026-07-12 08:50:54,237 INFO [collect_river_data] USGS SkagitMountVernon river data collection for 2026-07-11 completed
2026-07-12 08:50:54,239 INFO [manager] launching NextWorker(module='nowcast.workers.make_ssh_files', args=['nowcast', '--run-date', '2026-07-12'], host='localhost')
2026-07-12 08:50:54,246 INFO [manager] checklist updated with [Live Ocean products] items from download_live_ocean worker
2026-07-12 08:50:54,687 INFO [manager] launching NextWorker(module='nowcast.workers.make_live_ocean_files', args=['--run-date', '2026-07-12'], host='localhost')
2026-07-12 08:50:54,762 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-07-12 08:50:54,979 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-07-12 08:50:56,374 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12089500&startDT=2026-07-11&endDT=2026-07-11 "HTTP/1.1 200 "
2026-07-12 08:50:56,378 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12097500&startDT=2026-07-11&endDT=2026-07-11 "HTTP/1.1 200 "
2026-07-12 08:50:56,395 INFO [collect_river_data] Appended USGS NisquallyMcKenna river average discharge for 2026-07-11 to: /results/forcing/rivers/observations/Nisqually_McKenna_flow
2026-07-12 08:50:56,395 INFO [collect_river_data] USGS NisquallyMcKenna river data collection for 2026-07-11 completed
2026-07-12 08:50:56,398 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-07-12 08:50:56,399 INFO [collect_river_data] Appended USGS GreenwaterGreenwater river average discharge for 2026-07-11 to: /results/forcing/rivers/observations/Greenwater_Greenwater_flow
2026-07-12 08:50:56,400 INFO [collect_river_data] USGS GreenwaterGreenwater river data collection for 2026-07-11 completed
2026-07-12 08:50:56,581 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-07-12 08:50:56,977 INFO [make_turbidity_file] 2026-07-12 Fraser River turbidity file creation complete
2026-07-12 08:50:56,979 INFO [manager] checklist updated with [Fraser River turbidity] items from make_turbidity_file worker
2026-07-12 08:50:59,168 INFO [make_live_ocean_files] running in process 260152
2026-07-12 08:50:59,169 INFO [make_live_ocean_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-07-12 08:50:59,171 INFO [make_live_ocean_files] publishing log messages to tcp://*:5562
2026-07-12 08:50:59,172 INFO [make_live_ocean_files] connected to skookum.eos.ubc.ca port 5555
2026-07-12 08:50:59,173 INFO [make_live_ocean_files] Creating T&S western boundary conditions file from 2026-07-12 Live Ocean run
2026-07-12 08:50:59,220 INFO [make_ssh_files] running in process 260136
2026-07-12 08:50:59,221 INFO [make_ssh_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-07-12 08:50:59,222 INFO [make_ssh_files] publishing log messages to tcp://*:5564
2026-07-12 08:50:59,224 INFO [make_ssh_files] connected to skookum.eos.ubc.ca port 5555
2026-07-12 08:50:59,225 INFO [make_ssh_files] building nowcast sea surface height boundary conditions file(s) from 2026-07-12 NOAA Neah Bay 06Z observation and forecast values
2026-07-12 08:51:01,386 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2026m07d08.nc
2026-07-12 08:51:01,579 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2026m07d09.nc
2026-07-12 08:51:01,780 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2026m07d10.nc
2026-07-12 08:51:01,987 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2026m07d11.nc
2026-07-12 08:51:02,250 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2026m07d12.nc
2026-07-12 08:51:02,479 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2026m07d13.nc
2026-07-12 08:51:02,700 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2026m07d14.nc
2026-07-12 08:51:02,906 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2026m07d15.nc
2026-07-12 08:51:03,106 INFO [make_ssh_files] sea surface height boundary file for 2026-07-12 nowcast run created
2026-07-12 08:51:03,109 INFO [manager] checklist updated with [sea surface height forcing] items from make_ssh_files worker
2026-07-12 08:51:03,258 INFO [manager] launching NextWorker(module='nowcast.workers.make_runoff_file', args=['v202108'], host='localhost')
2026-07-12 08:51:05,981 INFO [make_runoff_file] running in process 260222
2026-07-12 08:51:05,982 INFO [make_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-07-12 08:51:05,983 INFO [make_runoff_file] publishing log messages to tcp://*:5564
2026-07-12 08:51:05,985 INFO [make_runoff_file] connected to skookum.eos.ubc.ca port 5555
2026-07-12 08:51:05,986 INFO [make_runoff_file] calculating NEMO runoff forcing for v202108 bathymetry for 2026-07-11
2026-07-12 08:51:07,928 ERROR [make_runoff_file] no 2026-07-11 discharge obs for Theodosia: patching
2026-07-12 08:51:09,244 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/hrdps_y2026m07d12.nc
2026-07-12 08:51:10,285 INFO [make_runoff_file] stored NEMO runoff forcing for v202108 bathymetry for 2026-07-11: /results/forcing/rivers/R202108Dailies_y2026m07d11.nc
2026-07-12 08:51:10,286 INFO [make_runoff_file] 2026-07-11 runoff file creation completed
2026-07-12 08:51:10,287 INFO [manager] checklist updated with [rivers forcing] items from make_runoff_file worker
2026-07-12 08:51:20,243 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2026m07d13.nc
2026-07-12 08:51:25,600 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2026m07d14.nc
2026-07-12 08:51:25,615 INFO [grib_to_netcdf] 2026-07-12 NEMO-atmos forcing file for nowcast+ created
2026-07-12 08:51:25,618 INFO [manager] checklist updated with [weather forcing] items from grib_to_netcdf worker
2026-07-12 08:51:43,335 INFO [make_live_ocean_files] Stored T&S western boundary conditions file: /results/forcing/LiveOcean/boundary_conditions/LiveOcean_v201905_y2026m07d12.nc
2026-07-12 08:51:43,336 INFO [make_live_ocean_files] 2026-07-12 Live Ocean western boundary conditions files created
2026-07-12 08:51:43,339 INFO [manager] checklist updated with [Live Ocean boundary conditions] items from make_live_ocean_files worker
2026-07-12 08:51:43,522 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['weather'], host='localhost')
2026-07-12 08:51:43,527 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'nowcast+'], host='localhost')
2026-07-12 08:51:43,532 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'nowcast+'], host='localhost')
2026-07-12 08:51:43,539 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.nibi', 'nowcast+'], host='localhost')
2026-07-12 08:51:43,545 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'nowcast+'], host='localhost')
2026-07-12 08:51:45,051 INFO [ping_erddap] running in process 260283
2026-07-12 08:51:45,053 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-07-12 08:51:45,053 INFO [ping_erddap] publishing log messages to tcp://*:5562
2026-07-12 08:51:45,054 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2026-07-12 08:51:45,057 INFO [ping_erddap] weather ERDDAP dataset flag file(s) created
2026-07-12 08:51:45,059 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2026-07-12 08:51:45,430 INFO [upload_forcing] running in process 260285
2026-07-12 08:51:45,431 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-07-12 08:51:45,432 INFO [upload_forcing] publishing log messages to tcp://*:5564
2026-07-12 08:51:45,433 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-07-12 08:51:45,470 INFO [upload_forcing] running in process 260288
2026-07-12 08:51:45,471 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-07-12 08:51:45,472 INFO [upload_forcing] publishing log messages to tcp://*:5565
2026-07-12 08:51:45,473 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-07-12 08:51:45,474 INFO [upload_forcing] running in process 260287
2026-07-12 08:51:45,476 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-07-12 08:51:45,477 INFO [upload_forcing] publishing log messages to tcp://*:5566
2026-07-12 08:51:45,478 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-07-12 08:51:45,487 INFO [upload_forcing] running in process 260286
2026-07-12 08:51:45,488 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-07-12 08:51:45,490 INFO [upload_forcing] publishing log messages to tcp://*:5562
2026-07-12 08:51:45,491 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-07-12 08:51:47,092 INFO [upload_forcing] nowcast+ 2026-07-12 forcing files upload to optimum-hindcast completed
2026-07-12 08:51:47,128 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-07-12 08:51:51,304 INFO [upload_forcing] nowcast+ 2026-07-12 forcing files upload to arbutus.cloud-nowcast completed
2026-07-12 08:51:51,309 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-07-12 08:51:51,499 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'nowcast+', '--run-date', '2026-07-12'], host='localhost')
2026-07-12 08:51:52,712 INFO [upload_forcing] nowcast+ 2026-07-12 forcing files upload to orcinus-nowcast-agrif completed
2026-07-12 08:51:52,716 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-07-12 08:51:53,333 INFO [make_forcing_links] running in process 260318
2026-07-12 08:51:53,334 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-07-12 08:51:53,335 INFO [make_forcing_links] publishing log messages to tcp://*:5562
2026-07-12 08:51:53,336 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2026-07-12 08:51:53,500 INFO [upload_forcing] nowcast+ 2026-07-12 forcing files upload to robot.nibi completed
2026-07-12 08:51:53,505 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-07-12 08:51:55,107 INFO [make_forcing_links] nowcast+ 2026-07-12 forcing file links on arbutus.cloud-nowcast created
2026-07-12 08:51:55,142 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2026-07-12 08:51:55,269 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2026-07-12'], host='arbutus.cloud-nowcast')
2026-07-12 08:51:57,617 INFO [run_NEMO] running in process 20454
2026-07-12 08:51:57,623 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-07-12 08:51:57,624 INFO [run_NEMO] publishing log messages to tcp://*:5556
2026-07-12 08:51:57,677 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2026-07-12 08:51:58,433 WARNING [run_NEMO] There are uncommitted changes in /nemoShare/MEOPAR/nowcast-sys/SS-run-sets
2026-07-12 08:51:58,797 INFO [run_NEMO] nowcast: launching /nemoShare/MEOPAR/nowcast-sys/runs/12jul26nowcast_2026-07-12T085157.674400-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2026-07-12 08:51:58,814 INFO [run_NEMO] nowcast NEMO run for 2026-07-12 on arbutus.cloud-nowcast started
2026-07-12 08:51:58,841 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2026-07-12 08:51:59,002 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'nowcast'], host='arbutus.cloud-nowcast')
2026-07-12 08:52:01,934 INFO [watch_NEMO] running in process 20730
2026-07-12 08:52:01,935 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-07-12 08:52:01,936 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2026-07-12 08:52:01,937 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2026-07-12 08:52:02,032 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2026-07-12 08:57:02,100 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 19356236 = 2026-07-12 05:16:40 UTC, 22.0% complete
2026-07-12 09:02:02,021 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 19356802 = 2026-07-12 11:34:00 UTC, 48.2% complete
2026-07-12 09:07:02,022 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 19357368 = 2026-07-12 17:51:20 UTC, 74.4% complete
2026-07-12 09:12:02,032 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 19357920 = 2026-07-12 23:59:20 UTC, 100.0% complete
2026-07-12 09:17:02,026 INFO [watch_NEMO] nowcast NEMO run on arbutus.cloud-nowcast completed
2026-07-12 09:17:02,030 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2026-07-12 09:17:02,516 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'ssh', '--run-date', '2026-07-12'], host='localhost')
2026-07-12 09:17:02,523 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2026-07-12'], host='localhost')
2026-07-12 09:17:04,365 INFO [download_results] running in process 260819
2026-07-12 09:17:04,366 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-07-12 09:17:04,367 INFO [download_results] publishing log messages to tcp://*:5562
2026-07-12 09:17:04,369 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2026-07-12 09:17:04,370 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/nowcast/12jul26 to /results/SalishSea/nowcast-blue.202111
2026-07-12 09:17:04,500 INFO [make_forcing_links] running in process 260818
2026-07-12 09:17:04,501 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-07-12 09:17:04,502 INFO [make_forcing_links] publishing log messages to tcp://*:5564
2026-07-12 09:17:04,503 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2026-07-12 09:17:05,593 INFO [make_forcing_links] ssh 2026-07-12 forcing file links on arbutus.cloud-nowcast created
2026-07-12 09:17:05,601 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2026-07-12 09:17:05,797 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2026-07-12'], host='arbutus.cloud-nowcast')
2026-07-12 09:17:08,094 INFO [run_NEMO] running in process 21180
2026-07-12 09:17:08,099 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-07-12 09:17:08,105 INFO [run_NEMO] publishing log messages to tcp://*:5556
2026-07-12 09:17:08,112 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2026-07-12 09:17:08,867 WARNING [run_NEMO] There are uncommitted changes in /nemoShare/MEOPAR/nowcast-sys/SS-run-sets
2026-07-12 09:17:09,197 INFO [run_NEMO] forecast: launching /nemoShare/MEOPAR/nowcast-sys/runs/12jul26forecast_2026-07-12T091708.165398-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2026-07-12 09:17:09,211 INFO [run_NEMO] forecast NEMO run for 2026-07-12 on arbutus.cloud-nowcast started
2026-07-12 09:17:09,221 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2026-07-12 09:17:09,405 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'forecast'], host='arbutus.cloud-nowcast')
2026-07-12 09:17:11,608 INFO [watch_NEMO] running in process 21458
2026-07-12 09:17:11,617 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-07-12 09:17:11,638 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2026-07-12 09:17:11,658 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2026-07-12 09:17:11,722 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2026-07-12 09:17:56,602 INFO [download_results] nowcast 2026-07-12 results files from arbutus.cloud-nowcast downloaded
2026-07-12 09:17:56,607 INFO [manager] checklist updated with [results files] items from download_results worker
2026-07-12 09:17:56,763 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast', 'research', '--run-date', '2026-07-12'], host='localhost')
2026-07-12 09:17:56,768 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast', 'comparison', '--run-date', '2026-07-11'], host='localhost')
2026-07-12 09:17:56,774 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['nowcast', '--run-date', '2026-07-12'], host='localhost')
2026-07-12 09:18:00,259 INFO [make_CHS_currents_file] running in process 260853
2026-07-12 09:18:00,261 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-07-12 09:18:00,261 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5562
2026-07-12 09:18:00,263 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555
2026-07-12 09:18:01,513 INFO [make_plots] running in process 260850
2026-07-12 09:18:01,514 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-07-12 09:18:01,515 INFO [make_plots] publishing log messages to tcp://*:5564
2026-07-12 09:18:01,516 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2026-07-12 09:18:01,522 INFO [make_plots] running in process 260852
2026-07-12 09:18:01,524 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-07-12 09:18:01,525 INFO [make_plots] publishing log messages to tcp://*:5565
2026-07-12 09:18:01,526 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2026-07-12 09:18:02,071 INFO [make_plots] preparing render list for 2026-07-11 NEMO nowcast-blue comparison figures
2026-07-12 09:18:02,072 INFO [make_plots] preparing render list for 2026-07-12 NEMO nowcast-blue research figures
2026-07-12 09:18:02,191 INFO [make_plots] added SH_wind figure to 2026-07-11 NEMO nowcast-blue comparison render list
2026-07-12 09:18:02,215 INFO [make_plots] added Compare_VENUS_East figure to 2026-07-11 NEMO nowcast-blue comparison render list
2026-07-12 09:18:02,233 INFO [make_plots] added Compare_VENUS_Central figure to 2026-07-11 NEMO nowcast-blue comparison render list
2026-07-12 09:18:02,239 INFO [make_plots] starting to render nemo nowcast comparison 11jul26 figures
2026-07-12 09:18:03,594 INFO [make_plots] added salinity_thalweg_and_surface figures to 2026-07-12 NEMO nowcast-blue research render list
2026-07-12 09:18:04,676 INFO [make_plots] added temperature_thalweg_and_surface figures to 2026-07-12 NEMO nowcast-blue research render list
2026-07-12 09:18:04,677 INFO [make_plots] added Currents_sections_and_surface figure to 2026-07-12 NEMO nowcast-blue research render list
2026-07-12 09:18:04,678 INFO [make_plots] added Currents_at_VENUS_Central figure to 2026-07-12 NEMO nowcast-blue research render list
2026-07-12 09:18:04,679 INFO [make_plots] added Currents_at_VENUS_East figure to 2026-07-12 NEMO nowcast-blue research render list
2026-07-12 09:18:04,680 INFO [make_plots] starting to render nemo nowcast research 12jul26 figures
2026-07-12 09:18:37,341 INFO [make_CHS_currents_file] Made CHS currents file for 2026-07-12 for nowcast
2026-07-12 09:18:37,344 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker
2026-07-12 09:19:14,045 INFO [make_plots] finished rendering nemo nowcast research 12jul26 figures
2026-07-12 09:19:14,048 INFO [make_plots] nemo research plots for 2026-07-12 nowcast completed
2026-07-12 09:19:14,062 INFO [manager] checklist updated with [plots] items from make_plots worker
2026-07-12 09:19:36,811 INFO [make_plots] finished rendering nemo nowcast comparison 11jul26 figures
2026-07-12 09:19:36,829 INFO [make_plots] nemo comparison plots for 2026-07-11 nowcast completed
2026-07-12 09:19:36,861 INFO [manager] checklist updated with [plots] items from make_plots worker
2026-07-12 09:22:11,720 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 19358399 = 2026-07-13 05:18:40 UTC, 14.8% complete
2026-07-12 09:27:11,738 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 19358917 = 2026-07-13 11:04:00 UTC, 30.8% complete
2026-07-12 09:32:11,721 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 19359449 = 2026-07-13 16:58:40 UTC, 47.2% complete
2026-07-12 09:37:11,725 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 19360004 = 2026-07-13 23:08:40 UTC, 64.3% complete
2026-07-12 09:42:11,728 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 19360522 = 2026-07-14 04:54:00 UTC, 80.3% complete
2026-07-12 09:47:11,729 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 19361084 = 2026-07-14 11:08:40 UTC, 97.7% complete
2026-07-12 09:52:11,773 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2026-07-12 09:52:11,788 INFO [watch_NEMO] forecast NEMO run on arbutus.cloud-nowcast completed
2026-07-12 09:52:12,157 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_wind_file', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2026-07-12'], host='arbutus.cloud-nowcast')
2026-07-12 09:52:12,166 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_current_file', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2026-07-12'], host='arbutus.cloud-nowcast')
2026-07-12 09:52:12,170 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'turbidity', '--run-date', '2026-07-12'], host='localhost')
2026-07-12 09:52:12,177 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'turbidity', '--run-date', '2026-07-12'], host='localhost')
2026-07-12 09:52:12,181 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.nibi', 'turbidity', '--run-date', '2026-07-12'], host='localhost')
2026-07-12 09:52:12,186 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'turbidity', '--run-date', '2026-07-12'], host='localhost')
2026-07-12 09:52:12,190 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2026-07-12'], host='localhost')
2026-07-12 09:52:14,025 INFO [download_results] running in process 261661
2026-07-12 09:52:14,026 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-07-12 09:52:14,027 INFO [download_results] publishing log messages to tcp://*:5562
2026-07-12 09:52:14,028 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2026-07-12 09:52:14,029 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/forecast/12jul26 to /results/SalishSea/forecast.202111
2026-07-12 09:52:14,177 INFO [upload_forcing] running in process 261658
2026-07-12 09:52:14,178 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-07-12 09:52:14,180 INFO [upload_forcing] publishing log messages to tcp://*:5564
2026-07-12 09:52:14,181 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-07-12 09:52:14,196 INFO [upload_forcing] running in process 261656
2026-07-12 09:52:14,197 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-07-12 09:52:14,198 INFO [upload_forcing] publishing log messages to tcp://*:5565
2026-07-12 09:52:14,200 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-07-12 09:52:14,242 INFO [upload_forcing] running in process 261657
2026-07-12 09:52:14,243 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-07-12 09:52:14,244 INFO [upload_forcing] publishing log messages to tcp://*:5566
2026-07-12 09:52:14,245 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-07-12 09:52:14,246 INFO [upload_forcing] running in process 261659
2026-07-12 09:52:14,248 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-07-12 09:52:14,249 INFO [upload_forcing] publishing log messages to tcp://*:5567
2026-07-12 09:52:14,250 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-07-12 09:52:15,007 INFO [make_ww3_current_file] running in process 22016
2026-07-12 09:52:15,008 INFO [make_ww3_current_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-07-12 09:52:15,009 INFO [make_ww3_current_file] publishing log messages to tcp://*:5571
2026-07-12 09:52:15,018 INFO [make_ww3_current_file] connected to skookum.eos.ubc.ca port 5555
2026-07-12 09:52:15,019 INFO [make_ww3_current_file] Creating wwatch3 currents forcing file for 2026-07-12 forecast run
2026-07-12 09:52:15,159 INFO [upload_forcing] turbidity 2026-07-12 forcing files upload to optimum-hindcast completed
2026-07-12 09:52:15,162 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-07-12 09:52:17,563 INFO [upload_forcing] turbidity 2026-07-12 forcing files upload to robot.nibi completed
2026-07-12 09:52:17,599 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-07-12 09:52:19,708 INFO [upload_forcing] turbidity 2026-07-12 forcing files upload to orcinus-nowcast-agrif completed
2026-07-12 09:52:19,712 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-07-12 09:52:19,999 INFO [upload_forcing] turbidity 2026-07-12 forcing files upload to arbutus.cloud-nowcast completed
2026-07-12 09:52:20,003 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-07-12 09:52:20,170 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2026-07-12'], host='localhost')
2026-07-12 09:52:22,279 INFO [make_forcing_links] running in process 261698
2026-07-12 09:52:22,282 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-07-12 09:52:22,283 INFO [make_forcing_links] publishing log messages to tcp://*:5564
2026-07-12 09:52:22,285 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2026-07-12 09:52:26,908 INFO [make_forcing_links] nowcast-green 2026-07-12 forcing file links on arbutus.cloud-nowcast created
2026-07-12 09:52:26,913 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2026-07-12 09:52:27,098 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2026-07-12'], host='arbutus.cloud-nowcast')
2026-07-12 09:52:28,432 INFO [make_ww3_wind_file] wwatch3 wind forcing file created on arbutus.cloud-nowcast for 2026-07-12 forecast run
2026-07-12 09:52:28,467 INFO [manager] checklist updated with [WW3 wind forcing] items from make_ww3_wind_file worker
2026-07-12 09:52:29,353 INFO [run_NEMO] running in process 22651
2026-07-12 09:52:29,368 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-07-12 09:52:29,371 INFO [run_NEMO] publishing log messages to tcp://*:5556
2026-07-12 09:52:29,374 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2026-07-12 09:52:30,237 WARNING [run_NEMO] There are uncommitted changes in /nemoShare/MEOPAR/nowcast-sys/SS-run-sets
2026-07-12 09:52:30,624 INFO [run_NEMO] nowcast-green: launching /nemoShare/MEOPAR/nowcast-sys/runs/12jul26nowcast-green_2026-07-12T095229.389093-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2026-07-12 09:52:30,641 INFO [run_NEMO] nowcast-green NEMO run for 2026-07-12 on arbutus.cloud-nowcast started
2026-07-12 09:52:30,643 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2026-07-12 09:52:30,814 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'nowcast-green'], host='arbutus.cloud-nowcast')
2026-07-12 09:52:32,997 INFO [watch_NEMO] running in process 22931
2026-07-12 09:52:32,999 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-07-12 09:52:33,022 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2026-07-12 09:52:33,042 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2026-07-12 09:52:33,102 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2026-07-12 09:53:47,908 INFO [download_results] forecast 2026-07-12 results files from arbutus.cloud-nowcast downloaded
2026-07-12 09:53:47,911 INFO [manager] checklist updated with [results files] items from download_results worker
2026-07-12 09:53:48,113 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['forecast', '--run-date', '2026-07-12'], host='localhost')
2026-07-12 09:53:51,047 INFO [make_CHS_currents_file] running in process 261736
2026-07-12 09:53:51,048 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-07-12 09:53:51,049 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5562
2026-07-12 09:53:51,050 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555
2026-07-12 09:54:40,154 INFO [make_CHS_currents_file] Made CHS currents file for 2026-07-12 for forecast
2026-07-12 09:54:40,158 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker
2026-07-12 09:54:40,321 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['nemo', 'forecast', '--run-date', '2026-07-12'], host='localhost')
2026-07-12 09:54:41,768 INFO [update_forecast_datasets] running in process 261776
2026-07-12 09:54:41,769 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-07-12 09:54:41,771 INFO [update_forecast_datasets] publishing log messages to tcp://*:5562
2026-07-12 09:54:41,772 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2026-07-12 09:54:41,773 INFO [update_forecast_datasets] updating nemo forecast directory for forecast/12jul26 run
2026-07-12 09:54:42,207 INFO [update_forecast_datasets] updated nemo forecast directory for forecast/12jul26 run: /results/SalishSea/rolling-forecasts/nemo
2026-07-12 09:54:42,208 INFO [update_forecast_datasets] nemo 2026-07-12 forecast rolling forecast datasets updated
2026-07-12 09:54:42,211 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2026-07-12 09:54:42,377 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nemo-forecast'], host='localhost')
2026-07-12 09:54:42,381 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'forecast', 'publish', '--run-date', '2026-07-12'], host='localhost')
2026-07-12 09:54:42,386 INFO [manager] launching NextWorker(module='nowcast.workers.make_surface_current_tiles', args=['forecast', '--run-date', '2026-07-12'], host='localhost')
2026-07-12 09:54:43,865 INFO [ping_erddap] running in process 261781
2026-07-12 09:54:43,866 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-07-12 09:54:43,867 INFO [ping_erddap] publishing log messages to tcp://*:5562
2026-07-12 09:54:43,867 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2026-07-12 09:54:43,876 INFO [ping_erddap] nemo-forecast ERDDAP dataset flag file(s) created
2026-07-12 09:54:43,878 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2026-07-12 09:54:45,901 INFO [make_surface_current_tiles] running in process 261784
2026-07-12 09:54:45,905 INFO [make_surface_current_tiles] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-07-12 09:54:45,906 INFO [make_surface_current_tiles] publishing log messages to tcp://*:5562
2026-07-12 09:54:45,908 INFO [make_surface_current_tiles] connected to skookum.eos.ubc.ca port 5555
2026-07-12 09:54:45,927 INFO [make_surface_current_tiles] starting rendering of tiles for 2026-07-12 forecast in 6 processes into /results/nowcast-sys/figures/surface_currents/forecast/12jul26
2026-07-12 09:54:48,300 INFO [make_plots] running in process 261783
2026-07-12 09:54:48,301 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-07-12 09:54:48,302 INFO [make_plots] publishing log messages to tcp://*:5564
2026-07-12 09:54:48,303 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2026-07-12 09:54:48,802 INFO [make_plots] preparing render list for 2026-07-12 NEMO forecast publish figures
2026-07-12 09:54:49,284 INFO [make_plots] added Website_thumbnail figure to 2026-07-12 NEMO forecast publish render list
2026-07-12 09:54:49,319 INFO [make_plots] added Threshold_website figure to 2026-07-12 NEMO forecast publish render list
2026-07-12 09:54:49,321 INFO [make_plots] added PA_tidal_predictions figure to 2026-07-12 NEMO forecast publish render list
2026-07-12 09:54:49,322 INFO [make_plots] added Boundary Bay figure to 2026-07-12 NEMO forecast publish render list
2026-07-12 09:54:49,322 INFO [make_plots] added Campbell River figure to 2026-07-12 NEMO forecast publish render list
2026-07-12 09:54:49,323 INFO [make_plots] added Cherry Point figure to 2026-07-12 NEMO forecast publish render list
2026-07-12 09:54:49,324 INFO [make_plots] added Friday Harbor figure to 2026-07-12 NEMO forecast publish render list
2026-07-12 09:54:49,325 INFO [make_plots] added Halfmoon Bay figure to 2026-07-12 NEMO forecast publish render list
2026-07-12 09:54:49,326 INFO [make_plots] added Nanaimo figure to 2026-07-12 NEMO forecast publish render list
2026-07-12 09:54:49,327 INFO [make_plots] added Neah Bay figure to 2026-07-12 NEMO forecast publish render list
2026-07-12 09:54:49,327 INFO [make_plots] added New Westminster figure to 2026-07-12 NEMO forecast publish render list
2026-07-12 09:54:49,328 INFO [make_plots] added Patricia Bay figure to 2026-07-12 NEMO forecast publish render list
2026-07-12 09:54:49,328 INFO [make_plots] added Point Atkinson figure to 2026-07-12 NEMO forecast publish render list
2026-07-12 09:54:49,329 INFO [make_plots] added Port Renfrew figure to 2026-07-12 NEMO forecast publish render list
2026-07-12 09:54:49,329 INFO [make_plots] added Sand Heads figure to 2026-07-12 NEMO forecast publish render list
2026-07-12 09:54:49,330 INFO [make_plots] added Sandy Cove figure to 2026-07-12 NEMO forecast publish render list
2026-07-12 09:54:49,330 INFO [make_plots] added Squamish figure to 2026-07-12 NEMO forecast publish render list
2026-07-12 09:54:49,331 INFO [make_plots] added Victoria figure to 2026-07-12 NEMO forecast publish render list
2026-07-12 09:54:49,332 INFO [make_plots] added Woodwards Landing figure to 2026-07-12 NEMO forecast publish render list
2026-07-12 09:54:49,332 INFO [make_plots] starting to render nemo forecast publish 12jul26 figures
2026-07-12 09:55:02,656 INFO [make_ww3_current_file] wwatch3 currents forcing file created on arbutus.cloud-nowcast for 2026-07-12 forecast run
2026-07-12 09:55:02,685 INFO [manager] checklist updated with [WW3 currents forcing] items from make_ww3_current_file worker
2026-07-12 09:55:02,852 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2026-07-12'], host='arbutus.cloud-nowcast')
2026-07-12 09:55:05,236 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker
2026-07-12 09:55:05,453 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'nowcast'], host='arbutus.cloud-nowcast')
2026-07-12 09:55:07,621 INFO [watch_ww3] running in process 23758
2026-07-12 09:55:07,622 INFO [watch_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-07-12 09:55:07,623 INFO [watch_ww3] publishing log messages to tcp://*:5573
2026-07-12 09:55:07,624 INFO [watch_ww3] connected to skookum.eos.ubc.ca port 5555
2026-07-12 09:55:07,827 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2026-07-12 09:57:33,102 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19355933 = 2026-07-12 01:54:40 UTC, 8.0% complete
2026-07-12 10:00:07,905 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 93, 21.5% complete
2026-07-12 10:02:33,103 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19356123 = 2026-07-12 04:01:20 UTC, 16.8% complete
2026-07-12 10:03:53,567 WARNING [make_plots] invalid station number for Boundary Bay station: None
2026-07-12 10:03:53,568 WARNING [make_plots] can't resolve a valid CHS station code for Boundary Bay
2026-07-12 10:04:22,750 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb996/data for station 08074 Campbell River from 2026-07-07 00:00:00Z to 2026-07-14 11:55:00Z
2026-07-12 10:05:07,815 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 198, 45.8% complete
2026-07-12 10:05:44,799 WARNING [make_plots] invalid station number for Halfmoon Bay station: None
2026-07-12 10:05:44,808 WARNING [make_plots] can't resolve a valid CHS station code for Halfmoon Bay
2026-07-12 10:05:51,606 INFO [make_surface_current_tiles] starting PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast/12jul26
2026-07-12 10:06:03,417 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb96d/data for station 07917 Nanaimo from 2026-07-07 00:00:00Z to 2026-07-14 11:55:00Z
2026-07-12 10:06:39,735 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e13d0f4a073c4bbf93/data for station 07654 New Westminster from 2026-07-07 00:00:00Z to 2026-07-14 11:55:00Z
2026-07-12 10:06:59,293 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd2d/data for station 07277 Patricia Bay from 2026-07-07 00:00:00Z to 2026-07-14 11:55:00Z
2026-07-12 10:07:18,740 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94c/data for station 07795 Point Atkinson from 2026-07-07 00:00:00Z to 2026-07-14 11:55:00Z
2026-07-12 10:07:33,125 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19356314 = 2026-07-12 06:08:40 UTC, 25.6% complete
2026-07-12 10:07:37,128 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e23d0f4a073c4bc060/data for station 08525 Port Renfrew from 2026-07-07 00:00:00Z to 2026-07-14 11:55:00Z
2026-07-12 10:07:54,632 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb93c/data for station 07594 Sand Heads from 2026-07-07 00:00:00Z to 2026-07-14 11:55:00Z
2026-07-12 10:08:13,599 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc43b/data for station 07786 Sandy Cove from 2026-07-07 00:00:00Z to 2026-07-14 11:55:00Z
2026-07-12 10:08:33,508 WARNING [make_plots] invalid station number for Squamish station: None
2026-07-12 10:08:33,517 WARNING [make_plots] can't resolve a valid CHS station code for Squamish
2026-07-12 10:08:52,236 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd1e/data for station 07120 Victoria from 2026-07-07 00:00:00Z to 2026-07-14 11:55:00Z
2026-07-12 10:09:09,987 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc42d/data for station 07610 Woodwards Landing from 2026-07-07 00:00:00Z to 2026-07-14 11:55:00Z
2026-07-12 10:09:10,371 INFO [make_plots] no obs water level data available from 5cebf1e43d0f4a073c4bc42d during 2026-07-07 00:00:00Z to 2026-07-14 11:55:00Z
2026-07-12 10:09:28,913 INFO [make_plots] finished rendering nemo forecast publish 12jul26 figures
2026-07-12 10:09:28,916 INFO [make_plots] nemo publish plots for 2026-07-12 forecast completed
2026-07-12 10:09:28,923 INFO [manager] checklist updated with [plots] items from make_plots worker
2026-07-12 10:09:29,116 INFO [manager] launching NextWorker(module='nowcast.workers.make_feeds', args=['forecast', '--run-date', '2026-07-12'], host='localhost')
2026-07-12 10:09:34,251 INFO [make_feeds] running in process 266649
2026-07-12 10:09:34,252 INFO [make_feeds] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-07-12 10:09:34,253 INFO [make_feeds] publishing log messages to tcp://*:5564
2026-07-12 10:09:34,254 INFO [make_feeds] connected to skookum.eos.ubc.ca port 5555
2026-07-12 10:09:50,171 INFO [make_feeds] ATOM feeds for 2026-07-12 forecast run completed
2026-07-12 10:09:50,191 INFO [manager] checklist updated with [storm surge alert feeds] items from make_feeds worker
2026-07-12 10:10:07,846 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 297, 68.8% complete
2026-07-12 10:11:08,438 INFO [make_surface_current_tiles] finished PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast/12jul26
2026-07-12 10:11:08,462 INFO [make_surface_current_tiles] finished rendering of tiles for 2026-07-12 forecast into /results/nowcast-sys/figures/surface_currents/forecast/12jul26
2026-07-12 10:11:08,463 INFO [make_surface_current_tiles] surface current tile figures for 2026-07-12 forecast completed
2026-07-12 10:11:08,749 INFO [manager] checklist updated with [surface current tiles] items from make_surface_current_tiles worker
2026-07-12 10:12:33,149 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19356505 = 2026-07-12 08:16:00 UTC, 34.5% complete
2026-07-12 10:15:07,864 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 399, 92.4% complete
2026-07-12 10:17:33,165 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19356698 = 2026-07-12 10:24:40 UTC, 43.4% complete
2026-07-12 10:20:07,878 INFO [watch_ww3] nowcast WWATCH3 run on arbutus.cloud-nowcast completed
2026-07-12 10:20:07,881 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2026-07-12 10:20:08,343 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2026-07-12'], host='localhost')
2026-07-12 10:20:08,349 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2026-07-12'], host='arbutus.cloud-nowcast')
2026-07-12 10:20:09,807 INFO [download_wwatch3_results] running in process 266876
2026-07-12 10:20:09,808 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-07-12 10:20:09,809 INFO [download_wwatch3_results] publishing log messages to tcp://*:5562
2026-07-12 10:20:09,810 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2026-07-12 10:20:10,620 INFO [run_ww3] running in process 24017
2026-07-12 10:20:10,621 INFO [run_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-07-12 10:20:10,623 INFO [run_ww3] publishing log messages to tcp://*:5572
2026-07-12 10:20:10,624 INFO [run_ww3] connected to skookum.eos.ubc.ca port 5555
2026-07-12 10:20:10,628 INFO [run_ww3] Created run directory /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast_2026-07-12T102010.553693-0700
2026-07-12 10:20:10,629 INFO [run_ww3] forecast: launching /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast_2026-07-12T102010.553693-0700/SoGWW3.sh on arbutus.cloud-nowcast
2026-07-12 10:20:10,651 INFO [run_ww3] forecast WaveWatch3 run for 2026-07-12 on arbutus.cloud-nowcast started
2026-07-12 10:20:10,686 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker
2026-07-12 10:20:10,986 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'forecast'], host='arbutus.cloud-nowcast')
2026-07-12 10:20:13,050 INFO [watch_ww3] running in process 24172
2026-07-12 10:20:13,074 INFO [watch_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-07-12 10:20:13,093 INFO [watch_ww3] publishing log messages to tcp://*:5573
2026-07-12 10:20:13,106 INFO [watch_ww3] connected to skookum.eos.ubc.ca port 5555
2026-07-12 10:20:13,146 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2026-07-12 10:20:48,989 INFO [download_wwatch3_results] nowcast 2026-07-12 results files from arbutus.cloud-nowcast downloaded
2026-07-12 10:20:48,993 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2026-07-12 10:22:33,176 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19356888 = 2026-07-12 12:31:20 UTC, 52.2% complete
2026-07-12 10:25:13,154 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 93, 14.4% complete
2026-07-12 10:27:33,328 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19357079 = 2026-07-12 14:38:40 UTC, 61.0% complete
2026-07-12 10:30:13,164 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 192, 29.6% complete
2026-07-12 10:32:33,192 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19357270 = 2026-07-12 16:46:00 UTC, 69.9% complete
2026-07-12 10:35:13,175 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 291, 44.9% complete
2026-07-12 10:37:33,198 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19357461 = 2026-07-12 18:53:20 UTC, 78.7% complete
2026-07-12 10:40:13,174 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 396, 61.1% complete
2026-07-12 10:42:33,206 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19357652 = 2026-07-12 21:00:40 UTC, 87.6% complete
2026-07-12 10:45:13,343 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 498, 76.9% complete
2026-07-12 10:47:33,211 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19357844 = 2026-07-12 23:08:40 UTC, 96.5% complete
2026-07-12 10:50:13,166 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 597, 92.1% complete
2026-07-12 10:52:33,153 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19357920 = 2026-07-12 23:59:20 UTC, 100.0% complete
2026-07-12 10:55:13,145 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2026-07-12 10:55:13,199 INFO [watch_ww3] forecast WWATCH3 run on arbutus.cloud-nowcast completed
2026-07-12 10:55:13,699 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2026-07-12'], host='localhost')
2026-07-12 10:55:15,965 INFO [download_wwatch3_results] running in process 267605
2026-07-12 10:55:15,966 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-07-12 10:55:15,967 INFO [download_wwatch3_results] publishing log messages to tcp://*:5562
2026-07-12 10:55:15,968 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2026-07-12 10:55:56,596 INFO [download_wwatch3_results] forecast 2026-07-12 results files from arbutus.cloud-nowcast downloaded
2026-07-12 10:55:56,599 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2026-07-12 10:55:56,948 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['wwatch3', 'forecast', '--run-date', '2026-07-12'], host='localhost')
2026-07-12 10:55:58,456 INFO [update_forecast_datasets] running in process 267633
2026-07-12 10:55:58,458 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-07-12 10:55:58,459 INFO [update_forecast_datasets] publishing log messages to tcp://*:5562
2026-07-12 10:55:58,460 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2026-07-12 10:55:58,461 INFO [update_forecast_datasets] updating wwatch3 most_recent_forecast directory from forecast/12jul26 run
2026-07-12 10:55:58,465 INFO [update_forecast_datasets] updated wwatch3 most_recent_forecast directory from forecast/12jul26 run
2026-07-12 10:55:58,466 INFO [update_forecast_datasets] updating wwatch3 forecast directory for forecast/12jul26 run
2026-07-12 10:55:58,573 INFO [update_forecast_datasets] updated wwatch3 forecast directory for forecast/12jul26 run: /results/SalishSea/rolling-forecasts/wwatch3
2026-07-12 10:55:58,599 INFO [update_forecast_datasets] wwatch3 2026-07-12 forecast rolling forecast datasets updated
2026-07-12 10:55:58,634 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2026-07-12 10:55:58,915 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['wwatch3-forecast'], host='localhost')
2026-07-12 10:56:00,455 INFO [ping_erddap] running in process 267639
2026-07-12 10:56:00,456 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-07-12 10:56:00,457 INFO [ping_erddap] publishing log messages to tcp://*:5562
2026-07-12 10:56:00,459 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2026-07-12 10:56:00,460 INFO [ping_erddap] wwatch3-forecast ERDDAP dataset flag file(s) created
2026-07-12 10:56:00,462 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2026-07-12 10:56:00,773 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['wwatch3', 'forecast', 'publish', '--run-date', '2026-07-12'], host='localhost')
2026-07-12 10:56:08,391 INFO [make_plots] running in process 267644
2026-07-12 10:56:08,392 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-07-12 10:56:08,393 INFO [make_plots] publishing log messages to tcp://*:5562
2026-07-12 10:56:08,394 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2026-07-12 10:56:08,395 INFO [make_plots] preparing render list for 2026-07-12 WaveWatch3 forecast publish figures
2026-07-12 10:56:08,397 INFO [make_plots] added Halibut Bank figure to 2026-07-12 WaveWatch3 forecast publish render list
2026-07-12 10:56:08,398 INFO [make_plots] added Sentry Shoal figure to 2026-07-12 WaveWatch3 forecast publish render list
2026-07-12 10:56:08,398 INFO [make_plots] starting to render wwatch3 forecast publish 12jul26 figures
2026-07-12 10:56:09,318 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46146 Halibut Bank
2026-07-12 10:56:17,843 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46131 Sentry Shoal
2026-07-12 10:56:23,018 INFO [make_plots] finished rendering wwatch3 forecast publish 12jul26 figures
2026-07-12 10:56:23,019 INFO [make_plots] wwatch3 publish plots for 2026-07-12 forecast completed
2026-07-12 10:56:23,023 INFO [manager] checklist updated with [plots] items from make_plots worker
2026-07-12 10:57:33,120 INFO [watch_NEMO] nowcast-green NEMO run on arbutus.cloud-nowcast completed
2026-07-12 10:57:33,145 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2026-07-12 10:57:33,656 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2026-07-12'], host='localhost')
2026-07-12 10:57:35,154 INFO [download_results] running in process 267701
2026-07-12 10:57:35,155 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-07-12 10:57:35,156 INFO [download_results] publishing log messages to tcp://*:5562
2026-07-12 10:57:35,157 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2026-07-12 10:57:35,158 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/nowcast-green/12jul26 to /results2/SalishSea/nowcast-green.202111
2026-07-12 11:01:04,438 INFO [download_results] nowcast-green 2026-07-12 results files from arbutus.cloud-nowcast downloaded
2026-07-12 11:01:04,444 INFO [manager] checklist updated with [results files] items from download_results worker
2026-07-12 11:01:04,762 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast-green', 'research', '--run-date', '2026-07-12'], host='localhost')
2026-07-12 11:01:04,769 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nowcast-green'], host='localhost')
2026-07-12 11:01:04,773 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'biology', '--run-date', '2026-07-12'], host='localhost')
2026-07-12 11:01:04,778 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'physics', '--run-date', '2026-07-12'], host='localhost')
2026-07-12 11:01:04,785 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'chemistry', '--run-date', '2026-07-12'], host='localhost')
2026-07-12 11:01:06,600 INFO [ping_erddap] running in process 267781
2026-07-12 11:01:06,613 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-07-12 11:01:06,615 INFO [ping_erddap] publishing log messages to tcp://*:5562
2026-07-12 11:01:06,615 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2026-07-12 11:01:06,622 INFO [ping_erddap] nowcast-green ERDDAP dataset flag file(s) created
2026-07-12 11:01:06,623 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2026-07-12 11:01:09,000 INFO [make_averaged_dataset] running in process 267782
2026-07-12 11:01:09,003 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-07-12 11:01:09,005 INFO [make_averaged_dataset] publishing log messages to tcp://*:5562
2026-07-12 11:01:09,007 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555
2026-07-12 11:01:09,008 INFO [make_averaged_dataset] creating day-averaged dataset for 12-Jul-2026 biology
2026-07-12 11:01:09,009 INFO [make_averaged_dataset] running in process 267784
2026-07-12 11:01:09,011 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-07-12 11:01:09,012 INFO [make_averaged_dataset] publishing log messages to tcp://*:5564
2026-07-12 11:01:09,013 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555
2026-07-12 11:01:09,014 INFO [make_averaged_dataset] creating day-averaged dataset for 12-Jul-2026 chemistry
2026-07-12 11:01:09,019 INFO [make_averaged_dataset] running in process 267783
2026-07-12 11:01:09,021 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-07-12 11:01:09,022 INFO [make_averaged_dataset] publishing log messages to tcp://*:5565
2026-07-12 11:01:09,023 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555
2026-07-12 11:01:09,024 INFO [make_averaged_dataset] creating day-averaged dataset for 12-Jul-2026 physics
2026-07-12 11:01:11,041 INFO [make_plots] running in process 267779
2026-07-12 11:01:11,042 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-07-12 11:01:11,043 INFO [make_plots] publishing log messages to tcp://*:5566
2026-07-12 11:01:11,044 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2026-07-12 11:01:11,672 INFO [make_plots] preparing render list for 2026-07-12 NEMO nowcast-green research figures
2026-07-12 11:01:13,803 INFO [make_plots] added temperature_thalweg_and_surface figures to 2026-07-12 NEMO nowcast-green research render list
2026-07-12 11:01:14,608 INFO [make_plots] added nitrate_thalweg_and_surface figures to 2026-07-12 NEMO nowcast-green research render list
2026-07-12 11:01:15,411 INFO [make_plots] added ammonium_thalweg_and_surface figures to 2026-07-12 NEMO nowcast-green research render list
2026-07-12 11:01:16,201 INFO [make_plots] added silicon_thalweg_and_surface figures to 2026-07-12 NEMO nowcast-green research render list
2026-07-12 11:01:16,966 INFO [make_plots] added dissolved_organic_nitrogen_thalweg_and_surface figures to 2026-07-12 NEMO nowcast-green research render list
2026-07-12 11:01:17,737 INFO [make_plots] added particulate_organic_nitrogen_thalweg_and_surface figures to 2026-07-12 NEMO nowcast-green research render list
2026-07-12 11:01:18,504 INFO [make_plots] added biogenic_silicon_thalweg_and_surface figures to 2026-07-12 NEMO nowcast-green research render list
2026-07-12 11:01:19,400 INFO [make_plots] added diatoms_thalweg_and_surface figures to 2026-07-12 NEMO nowcast-green research render list
2026-07-12 11:01:20,276 INFO [make_plots] added flagellates_thalweg_and_surface figures to 2026-07-12 NEMO nowcast-green research render list
2026-07-12 11:01:21,140 INFO [make_plots] added microzooplankton_thalweg_and_surface figures to 2026-07-12 NEMO nowcast-green research render list
2026-07-12 11:01:22,033 INFO [make_plots] added mesozooplankton_thalweg_and_surface figures to 2026-07-12 NEMO nowcast-green research render list
2026-07-12 11:01:22,824 INFO [make_plots] added turbidity_thalweg_and_surface figures to 2026-07-12 NEMO nowcast-green research render list
2026-07-12 11:01:24,079 INFO [make_plots] added temperature_salinity_timeseries figure to 2026-07-12 NEMO nowcast-green research render list
2026-07-12 11:01:24,099 INFO [make_plots] added nitrate_diatoms_timeseries figure to 2026-07-12 NEMO nowcast-green research render list
2026-07-12 11:01:24,117 INFO [make_plots] added diatoms_flagellates_timeseries figure to 2026-07-12 NEMO nowcast-green research render list
2026-07-12 11:01:24,135 INFO [make_plots] added z1_z2_zooplankton_timeseries figure to 2026-07-12 NEMO nowcast-green research render list
2026-07-12 11:01:24,139 INFO [make_plots] starting to render nemo nowcast-green research 12jul26 figures
2026-07-12 11:02:07,977 INFO [make_averaged_dataset] day-averaged dataset for 12-Jul-2026 physics created
2026-07-12 11:02:07,980 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker
2026-07-12 11:02:26,390 INFO [make_averaged_dataset] day-averaged dataset for 12-Jul-2026 chemistry created
2026-07-12 11:02:26,393 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker
2026-07-12 11:02:37,089 INFO [make_averaged_dataset] day-averaged dataset for 12-Jul-2026 biology created
2026-07-12 11:02:37,092 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker
2026-07-12 12:06:40,681 INFO [make_plots] finished rendering nemo nowcast-green research 12jul26 figures
2026-07-12 12:06:40,682 INFO [make_plots] nemo research plots for 2026-07-12 nowcast-green completed
2026-07-12 12:06:40,744 INFO [manager] checklist updated with [plots] items from make_plots worker
2026-07-12 14:56:24,074 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/18/ to /results/forcing/atmospheric/continental2.5/GRIB/20260712/18/
2026-07-12 14:56:24,075 INFO [collect_weather] 2026-07-12 2.5km weather forecast 18 collection complete
2026-07-12 14:56:24,078 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2026-07-12 14:56:24,514 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20260712/18/
2026-07-12 14:56:24,516 INFO [crop_gribs] 2026-07-12 18 GRIBs cropping complete
2026-07-12 14:56:24,638 INFO [manager] launching NextWorker(module='nowcast.workers.download_weather', args=['00', '1km'], host='localhost')
2026-07-12 14:56:24,647 INFO [manager] launching NextWorker(module='nowcast.workers.download_weather', args=['12', '1km'], host='localhost')
2026-07-12 14:56:24,653 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['00', '2.5km'], host='localhost')
2026-07-12 14:56:24,659 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['00', '--fcst-date', '2026-07-13'], host='localhost')
2026-07-12 14:56:24,665 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2026-07-12 14:56:26,821 INFO [download_weather] running in process 273093
2026-07-12 14:56:26,823 INFO [download_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-07-12 14:56:26,824 INFO [download_weather] publishing log messages to tcp://*:5562
2026-07-12 14:56:26,826 INFO [download_weather] connected to skookum.eos.ubc.ca port 5555
2026-07-12 14:56:26,827 INFO [download_weather] downloading 12 1 km forecast GRIB2 files for 20260712
2026-07-12 14:56:26,829 INFO [download_weather] running in process 273091
2026-07-12 14:56:26,829 INFO [collect_weather] running in process 273094
2026-07-12 14:56:26,830 INFO [download_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-07-12 14:56:26,831 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-07-12 14:56:26,832 INFO [download_weather] publishing log messages to tcp://*:5563
2026-07-12 14:56:26,833 INFO [collect_weather] publishing log messages to tcp://*:5564
2026-07-12 14:56:26,834 INFO [download_weather] connected to skookum.eos.ubc.ca port 5555
2026-07-12 14:56:26,834 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2026-07-12 14:56:26,835 INFO [download_weather] downloading 00 1 km forecast GRIB2 files for 20260712
2026-07-12 14:56:26,932 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/00/
2026-07-12 14:56:28,431 INFO [crop_gribs] running in process 273095
2026-07-12 14:56:28,433 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-07-12 14:56:28,434 INFO [crop_gribs] publishing log messages to tcp://*:5565
2026-07-12 14:56:28,435 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2026-07-12 14:56:28,436 INFO [crop_gribs] cropping 2026-07-13 ECCC HRDPS 2.5km continental 00Z GRIB files to SalishSeaCast subdomain
2026-07-12 14:56:28,449 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20260713/00/
2026-07-12 14:59:58,784 INFO [download_weather] 2026-07-12 1km weather forecast 12 downloads complete
2026-07-12 14:59:58,787 INFO [manager] checklist updated with [weather forecast] items from download_weather worker
2026-07-12 15:00:01,036 INFO [download_weather] 2026-07-12 1km weather forecast 00 downloads complete
2026-07-12 15:00:01,039 INFO [manager] checklist updated with [weather forecast] items from download_weather worker
2026-07-12 20:51:38,542 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/00/ to /results/forcing/atmospheric/continental2.5/GRIB/20260713/00/
2026-07-12 20:51:38,544 INFO [collect_weather] 2026-07-13 2.5km weather forecast 00 collection complete
2026-07-12 20:51:38,547 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2026-07-12 20:51:39,111 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['06', '2.5km'], host='localhost')
2026-07-12 20:51:39,119 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['06', '--fcst-date', '2026-07-13'], host='localhost')
2026-07-12 20:51:39,882 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20260713/00/
2026-07-12 20:51:39,883 INFO [crop_gribs] 2026-07-13 00 GRIBs cropping complete
2026-07-12 20:51:39,885 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2026-07-12 20:51:41,016 INFO [collect_weather] running in process 280446
2026-07-12 20:51:41,017 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-07-12 20:51:41,018 INFO [collect_weather] publishing log messages to tcp://*:5562
2026-07-12 20:51:41,019 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2026-07-12 20:51:41,037 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/06/
2026-07-12 20:51:42,637 INFO [crop_gribs] running in process 280448
2026-07-12 20:51:42,639 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-07-12 20:51:42,640 INFO [crop_gribs] publishing log messages to tcp://*:5563
2026-07-12 20:51:42,641 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2026-07-12 20:51:42,643 INFO [crop_gribs] cropping 2026-07-13 ECCC HRDPS 2.5km continental 06Z GRIB files to SalishSeaCast subdomain
2026-07-12 20:51:42,656 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20260713/06/
2026-07-13 02:53:50,678 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/06/ to /results/forcing/atmospheric/continental2.5/GRIB/20260713/06/
2026-07-13 02:53:50,680 INFO [collect_weather] 2026-07-13 2.5km weather forecast 06 collection complete
2026-07-13 02:53:50,682 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2026-07-13 02:53:51,252 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Capilano', '--data-date', '2026-07-12'], host='localhost')
2026-07-13 02:53:51,259 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'ChilliwackVedder', '--data-date', '2026-07-12'], host='localhost')
2026-07-13 02:53:51,264 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'ClowhomClowhomLake', '--data-date', '2026-07-12'], host='localhost')
2026-07-13 02:53:51,269 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Englishman', '--data-date', '2026-07-12'], host='localhost')
2026-07-13 02:53:51,273 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Fraser', '--data-date', '2026-07-12'], host='localhost')
2026-07-13 02:53:51,274 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20260713/06/
2026-07-13 02:53:51,275 INFO [crop_gribs] 2026-07-13 06 GRIBs cropping complete
2026-07-13 02:53:51,277 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'HomathkoMouth', '--data-date', '2026-07-12'], host='localhost')
2026-07-13 02:53:51,281 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'NicomeklLangley', '--data-date', '2026-07-12'], host='localhost')
2026-07-13 02:53:51,287 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'RobertsCreek', '--data-date', '2026-07-12'], host='localhost')
2026-07-13 02:53:51,294 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SalmonSayward', '--data-date', '2026-07-12'], host='localhost')
2026-07-13 02:53:51,302 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SanJuanPortRenfrew', '--data-date', '2026-07-12'], host='localhost')
2026-07-13 02:53:51,317 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SquamishBrackendale', '--data-date', '2026-07-12'], host='localhost')
2026-07-13 02:53:51,329 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaScotty', '--data-date', '2026-07-12'], host='localhost')
2026-07-13 02:53:51,343 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaBypass', '--data-date', '2026-07-12'], host='localhost')
2026-07-13 02:53:51,356 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaDiversion', '--data-date', '2026-07-12'], host='localhost')
2026-07-13 02:53:51,370 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ctd', args=['SCVIP'], host='localhost')
2026-07-13 02:53:51,382 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ctd', args=['SEVIP'], host='localhost')
2026-07-13 02:53:51,386 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ferry', args=['TWDP'], host='localhost')
2026-07-13 02:53:51,395 INFO [manager] launching NextWorker(module='nowcast.workers.get_vfpa_hadcp', args=['--data-date', '2026-07-12'], host='localhost')
2026-07-13 02:53:51,407 INFO [manager] launching NextWorker(module='nowcast.workers.collect_NeahBay_ssh', args=['00'], host='localhost')
2026-07-13 02:53:51,414 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['12', '2.5km'], host='localhost')
2026-07-13 02:53:51,424 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['12'], host='localhost')
2026-07-13 02:53:51,437 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2026-07-13 02:53:52,604 INFO [manager] launching NextWorker(module='nowcast.workers.grib_to_netcdf', args=['forecast2'], host='localhost')
2026-07-13 02:53:54,492 INFO [collect_NeahBay_ssh] running in process 288101
2026-07-13 02:53:54,496 INFO [collect_NeahBay_ssh] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-07-13 02:53:54,497 INFO [collect_NeahBay_ssh] publishing log messages to tcp://*:5562
2026-07-13 02:53:54,498 INFO [collect_NeahBay_ssh] connected to skookum.eos.ubc.ca port 5555
2026-07-13 02:53:54,499 INFO [collect_NeahBay_ssh] collecting Neah Bay ssh 00Z obs/forecast for 2026-07-13
2026-07-13 02:53:54,521 INFO [collect_weather] running in process 288102
2026-07-13 02:53:54,530 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-07-13 02:53:54,531 INFO [collect_weather] publishing log messages to tcp://*:5563
2026-07-13 02:53:54,532 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2026-07-13 02:53:54,547 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/12/
2026-07-13 02:53:55,542 INFO [collect_river_data] running in process 288088
2026-07-13 02:53:55,543 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-07-13 02:53:55,546 INFO [collect_river_data] publishing log messages to tcp://*:5564
2026-07-13 02:53:55,547 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-07-13 02:53:55,548 INFO [collect_river_data] Collecting ECCC NicomeklLangley river data for 2026-07-12
2026-07-13 02:53:55,604 INFO [collect_river_data] running in process 288084
2026-07-13 02:53:55,606 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-07-13 02:53:55,607 INFO [collect_river_data] publishing log messages to tcp://*:5565
2026-07-13 02:53:55,609 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-07-13 02:53:55,610 INFO [collect_river_data] Collecting ECCC ClowhomClowhomLake river data for 2026-07-12
2026-07-13 02:53:55,626 INFO [collect_river_data] running in process 288085
2026-07-13 02:53:55,627 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-07-13 02:53:55,627 INFO [collect_river_data] publishing log messages to tcp://*:5566
2026-07-13 02:53:55,630 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-07-13 02:53:55,631 INFO [collect_river_data] Collecting ECCC Englishman river data for 2026-07-12
2026-07-13 02:53:55,638 INFO [collect_river_data] running in process 288086
2026-07-13 02:53:55,639 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-07-13 02:53:55,640 INFO [collect_river_data] publishing log messages to tcp://*:5567
2026-07-13 02:53:55,640 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-07-13 02:53:55,641 INFO [collect_river_data] Collecting ECCC Fraser river data for 2026-07-12
2026-07-13 02:53:55,655 INFO [collect_river_data] Appended ECCC Englishman river average discharge for 2026-07-12 to: /data/dlatorne/SOG-projects/SOG-forcing/ECget/Englishman_flow
2026-07-13 02:53:55,656 INFO [collect_river_data] ECCC Englishman river data collection for 2026-07-12 completed
2026-07-13 02:53:55,658 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-07-13 02:53:55,668 INFO [collect_river_data] running in process 288081
2026-07-13 02:53:55,669 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-07-13 02:53:55,671 INFO [collect_river_data] publishing log messages to tcp://*:5568
2026-07-13 02:53:55,672 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-07-13 02:53:55,673 INFO [collect_river_data] Collecting ECCC Capilano river data for 2026-07-12
2026-07-13 02:53:55,675 INFO [collect_river_data] running in process 288083
2026-07-13 02:53:55,676 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-07-13 02:53:55,677 INFO [collect_river_data] Appended ECCC Fraser river average discharge for 2026-07-12 to: /data/dlatorne/SOG-projects/SOG-forcing/ECget/Fraser_flow
2026-07-13 02:53:55,678 INFO [collect_river_data] publishing log messages to tcp://*:5569
2026-07-13 02:53:55,679 INFO [collect_river_data] ECCC Fraser river data collection for 2026-07-12 completed
2026-07-13 02:53:55,679 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-07-13 02:53:55,680 INFO [collect_river_data] Collecting ECCC ChilliwackVedder river data for 2026-07-12
2026-07-13 02:53:55,731 INFO [collect_river_data] running in process 288092
2026-07-13 02:53:55,733 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-07-13 02:53:55,734 INFO [collect_river_data] publishing log messages to tcp://*:5570
2026-07-13 02:53:55,736 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-07-13 02:53:55,737 INFO [collect_river_data] Collecting ECCC SquamishBrackendale river data for 2026-07-12
2026-07-13 02:53:55,740 INFO [collect_river_data] Appended ECCC Capilano river average discharge for 2026-07-12 to: /opp/observations/rivers/Capilano/Caplilano_08GA010_day_avg_flow
2026-07-13 02:53:55,741 INFO [collect_river_data] ECCC Capilano river data collection for 2026-07-12 completed
2026-07-13 02:53:55,744 INFO [collect_river_data] Appended ECCC ClowhomClowhomLake river average discharge for 2026-07-12 to: /results/forcing/rivers/observations/Clowhom_ClowhomLake_flow
2026-07-13 02:53:55,746 INFO [collect_river_data] ECCC ClowhomClowhomLake river data collection for 2026-07-12 completed
2026-07-13 02:53:55,838 INFO [collect_river_data] Appended ECCC ChilliwackVedder river average discharge for 2026-07-12 to: /results/forcing/rivers/observations/Chilliwack_Vedder_flow
2026-07-13 02:53:55,844 INFO [collect_river_data] Appended ECCC NicomeklLangley river average discharge for 2026-07-12 to: /results/forcing/rivers/observations/Nicomekl_Langley_flow
2026-07-13 02:53:55,848 INFO [collect_river_data] ECCC ChilliwackVedder river data collection for 2026-07-12 completed
2026-07-13 02:53:55,849 INFO [collect_river_data] ECCC NicomeklLangley river data collection for 2026-07-12 completed
2026-07-13 02:53:55,852 INFO [collect_river_data] running in process 288090
2026-07-13 02:53:55,854 INFO [collect_river_data] running in process 288087
2026-07-13 02:53:55,854 INFO [collect_river_data] running in process 288095
2026-07-13 02:53:55,857 INFO [collect_river_data] running in process 288089
2026-07-13 02:53:55,859 INFO [collect_river_data] running in process 288091
2026-07-13 02:53:55,864 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-07-13 02:53:55,865 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-07-13 02:53:55,866 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-07-13 02:53:55,867 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-07-13 02:53:55,869 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-07-13 02:53:55,870 INFO [collect_river_data] publishing log messages to tcp://*:5571
2026-07-13 02:53:55,871 INFO [collect_river_data] Appended ECCC SquamishBrackendale river average discharge for 2026-07-12 to: /results/forcing/rivers/observations/Squamish_Brackendale_flow
2026-07-13 02:53:55,872 INFO [collect_river_data] publishing log messages to tcp://*:5572
2026-07-13 02:53:55,873 INFO [collect_river_data] publishing log messages to tcp://*:5573
2026-07-13 02:53:55,874 INFO [collect_river_data] publishing log messages to tcp://*:5575
2026-07-13 02:53:55,875 INFO [collect_river_data] publishing log messages to tcp://*:5574
2026-07-13 02:53:55,877 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-07-13 02:53:55,878 INFO [collect_river_data] ECCC SquamishBrackendale river data collection for 2026-07-12 completed
2026-07-13 02:53:55,879 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-07-13 02:53:55,880 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-07-13 02:53:55,881 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-07-13 02:53:55,882 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-07-13 02:53:55,882 INFO [collect_river_data] Collecting ECCC SalmonSayward river data for 2026-07-12
2026-07-13 02:53:55,883 INFO [collect_river_data] Collecting ECCC HomathkoMouth river data for 2026-07-12
2026-07-13 02:53:55,884 INFO [collect_river_data] Collecting ECCC TheodosiaDiversion river data for 2026-07-12
2026-07-13 02:53:55,885 INFO [collect_river_data] Collecting ECCC SanJuanPortRenfrew river data for 2026-07-12
2026-07-13 02:53:55,886 INFO [collect_river_data] Collecting ECCC RobertsCreek river data for 2026-07-12
2026-07-13 02:53:55,888 CRITICAL [collect_river_data] unhandled exception:
Traceback (most recent call last):
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/pandas/core/indexes/datetimes.py", line 610, in get_loc
    return self._partial_date_slice(reso, parsed)
           ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/pandas/core/indexes/datetimelike.py", line 333, in _partial_date_slice
    raise KeyError
KeyError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/SalishSeaCast/NEMO_Nowcast/nemo_nowcast/worker.py", line 387, in _do_work
    checklist = self.worker_func(
        self._parsed_args, self.config, self.tell_manager
    )
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/collect_river_data.py", line 109, in collect_river_data
    day_avg_discharge = day_avg_discharge_funcs[data_src](river_name, data_date, config)
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/collect_river_data.py", line 142, in _calc_eccc_day_avg_discharge
    day_avg_discharge = df.loc[f"{data_date.format('YYYY-MM-DD')}"].mean()[
                        ~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/pandas/core/indexing.py", line 1192, in __getitem__
    return self._getitem_axis(maybe_callable, axis=axis)
           ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/pandas/core/indexing.py", line 1432, in _getitem_axis
    return self._get_label(key, axis=axis)
           ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/pandas/core/indexing.py", line 1382, in _get_label
    return self.obj.xs(label, axis=axis)
           ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/pandas/core/generic.py", line 4323, in xs
    loc = index.get_loc(key)
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/pandas/core/indexes/datetimes.py", line 612, in get_loc
    raise KeyError(key) from err
KeyError: '2026-07-12'
2026-07-13 02:53:56,146 INFO [collect_river_data] Appended ECCC SalmonSayward river average discharge for 2026-07-12 to: /results/forcing/rivers/observations/Salmon_Sayward_flow
2026-07-13 02:53:56,147 INFO [collect_river_data] Appended ECCC RobertsCreek river average discharge for 2026-07-12 to: /results/forcing/rivers/observations/RobertsCreek_flow
2026-07-13 02:53:56,149 INFO [collect_river_data] Appended ECCC HomathkoMouth river average discharge for 2026-07-12 to: /results/forcing/rivers/observations/Homathko_Mouth_flow
2026-07-13 02:53:56,150 INFO [collect_river_data] Appended ECCC SanJuanPortRenfrew river average discharge for 2026-07-12 to: /results/forcing/rivers/observations/SanJuan_PortRenfrew_flow
2026-07-13 02:53:56,152 INFO [collect_river_data] ECCC SalmonSayward river data collection for 2026-07-12 completed
2026-07-13 02:53:56,153 INFO [collect_river_data] ECCC RobertsCreek river data collection for 2026-07-12 completed
2026-07-13 02:53:56,154 INFO [collect_river_data] ECCC HomathkoMouth river data collection for 2026-07-12 completed
2026-07-13 02:53:56,155 INFO [collect_river_data] ECCC SanJuanPortRenfrew river data collection for 2026-07-12 completed
2026-07-13 02:53:56,158 INFO [collect_river_data] running in process 288093
2026-07-13 02:53:56,158 INFO [collect_river_data] running in process 288094
2026-07-13 02:53:56,160 INFO [get_vfpa_hadcp] running in process 288100
2026-07-13 02:53:56,160 INFO [get_vfpa_hadcp] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-07-13 02:53:56,162 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-07-13 02:53:56,163 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-07-13 02:53:56,163 INFO [get_vfpa_hadcp] publishing log messages to tcp://*:5578
2026-07-13 02:53:56,164 INFO [collect_river_data] publishing log messages to tcp://*:5576
2026-07-13 02:53:56,166 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-07-13 02:53:56,167 INFO [collect_river_data] publishing log messages to tcp://*:5577
2026-07-13 02:53:56,168 INFO [get_vfpa_hadcp] connected to skookum.eos.ubc.ca port 5555
2026-07-13 02:53:56,170 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-07-13 02:53:56,171 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-07-13 02:53:56,171 INFO [get_vfpa_hadcp] processing VFPA HADCP data from 2nd Narrows Rail Bridge for 2026-07-12
2026-07-13 02:53:56,172 INFO [collect_river_data] Collecting ECCC TheodosiaScotty river data for 2026-07-12
2026-07-13 02:53:56,174 INFO [collect_river_data] Collecting ECCC TheodosiaBypass river data for 2026-07-12
2026-07-13 02:53:56,174 WARNING [get_vfpa_hadcp] file not found: /opp/observations/AISDATA/20260712T000000Z-20260712T010000Z.csv
2026-07-13 02:53:56,182 INFO [collect_river_data] Appended ECCC TheodosiaBypass river average discharge for 2026-07-12 to: /results/forcing/rivers/observations/Theodosia_Bypass_flow
2026-07-13 02:53:56,184 INFO [collect_river_data] Appended ECCC TheodosiaScotty river average discharge for 2026-07-12 to: /results/forcing/rivers/observations/Theodosia_Scotty_flow
2026-07-13 02:53:56,186 INFO [collect_river_data] ECCC TheodosiaBypass river data collection for 2026-07-12 completed
2026-07-13 02:53:56,190 INFO [collect_river_data] ECCC TheodosiaScotty river data collection for 2026-07-12 completed
2026-07-13 02:53:56,194 CRITICAL [get_vfpa_hadcp] unhandled exception:
Traceback (most recent call last):
  File "/SalishSeaCast/NEMO_Nowcast/nemo_nowcast/worker.py", line 387, in _do_work
    checklist = self.worker_func(
        self._parsed_args, self.config, self.tell_manager
    )
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/get_vfpa_hadcp.py", line 105, in get_vfpa_hadcp
    ds = _make_hour_dataset(csv_dir, data_date, place)
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/get_vfpa_hadcp.py", line 170, in _make_hour_dataset
    ds = _csv_to_dataset(csv_dir / csv_filename, place)
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/get_vfpa_hadcp.py", line 246, in _csv_to_dataset
    df = pandas.read_csv(csv_file, skiprows=3)
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/pandas/io/parsers/readers.py", line 1026, in read_csv
    return _read(filepath_or_buffer, kwds)
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/pandas/io/parsers/readers.py", line 620, in _read
    parser = TextFileReader(filepath_or_buffer, **kwds)
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/pandas/io/parsers/readers.py", line 1620, in __init__
    self._engine = self._make_engine(f, self.engine)
                   ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/pandas/io/parsers/readers.py", line 1880, in _make_engine
    self.handles = get_handle(
                   ~~~~~~~~~~^
        f,
        ^^
    ...<6 lines>...
        storage_options=self.options.get("storage_options", None),
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/pandas/io/common.py", line 873, in get_handle
    handle = open(
        handle,
    ...<3 lines>...
        newline="",
    )
FileNotFoundError: [Errno 2] No such file or directory: '/opp/observations/AISDATA/20260712T000000Z-20260712T010000Z.csv'
2026-07-13 02:53:56,235 INFO [crop_gribs] running in process 288103
2026-07-13 02:53:56,236 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-07-13 02:53:56,238 INFO [crop_gribs] publishing log messages to tcp://*:5579
2026-07-13 02:53:56,238 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2026-07-13 02:53:56,239 INFO [crop_gribs] cropping 2026-07-13 ECCC HRDPS 2.5km continental 12Z GRIB files to SalishSeaCast subdomain
2026-07-13 02:53:56,256 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20260713/12/
2026-07-13 02:53:56,430 INFO [collect_NeahBay_ssh] 2026-07-13T00:00:00-07:00 Neah Bay ssh 00Z obs/forecast data collection complete
2026-07-13 02:53:56,470 INFO [grib_to_netcdf] running in process 288272
2026-07-13 02:53:56,471 INFO [grib_to_netcdf] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-07-13 02:53:56,472 INFO [grib_to_netcdf] publishing log messages to tcp://*:5566
2026-07-13 02:53:56,473 INFO [grib_to_netcdf] connected to skookum.eos.ubc.ca port 5555
2026-07-13 02:53:56,473 INFO [grib_to_netcdf] creating NEMO-atmos forcing files for 2026-07-13 forecast2 run
2026-07-13 02:53:56,516 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-07-13 02:53:56,889 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-07-13 02:53:57,224 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-07-13 02:53:57,240 INFO [get_onc_ferry] running in process 288099
2026-07-13 02:53:57,241 INFO [get_onc_ferry] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-07-13 02:53:57,242 INFO [get_onc_ferry] publishing log messages to tcp://*:5567
2026-07-13 02:53:57,244 INFO [get_onc_ferry] connected to skookum.eos.ubc.ca port 5555
2026-07-13 02:53:57,245 INFO [get_onc_ferry] requesting ONC TWDP.N1 NAV data for 2026-07-12
2026-07-13 02:53:57,283 INFO [get_onc_ctd] running in process 288098
2026-07-13 02:53:57,284 INFO [get_onc_ctd] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-07-13 02:53:57,285 INFO [get_onc_ctd] publishing log messages to tcp://*:5568
2026-07-13 02:53:57,286 INFO [get_onc_ctd] connected to skookum.eos.ubc.ca port 5555
2026-07-13 02:53:57,287 INFO [get_onc_ctd] requesting ONC SEVIP CTD T&S data for 2026-07-12
2026-07-13 02:53:57,311 INFO [get_onc_ctd] running in process 288097
2026-07-13 02:53:57,312 INFO [get_onc_ctd] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-07-13 02:53:57,313 INFO [get_onc_ctd] publishing log messages to tcp://*:5590
2026-07-13 02:53:57,314 INFO [get_onc_ctd] connected to skookum.eos.ubc.ca port 5555
2026-07-13 02:53:57,315 INFO [get_onc_ctd] requesting ONC SCVIP CTD T&S data for 2026-07-12
2026-07-13 02:53:57,564 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-07-13 02:53:57,891 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-07-13 02:53:58,227 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-07-13 02:53:58,541 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-07-13 02:53:58,848 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-07-13 02:53:59,195 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-07-13 02:53:59,490 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-07-13 02:53:59,798 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-07-13 02:54:00,101 INFO [manager] checklist updated with [Neah Bay ssh data] items from collect_NeahBay_ssh worker
2026-07-13 02:54:00,403 INFO [manager] launching NextWorker(module='nowcast.workers.make_ssh_files', args=['forecast2', '--run-date', '2026-07-13'], host='localhost')
2026-07-13 02:54:04,035 INFO [make_ssh_files] running in process 288452
2026-07-13 02:54:04,036 INFO [make_ssh_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-07-13 02:54:04,037 INFO [make_ssh_files] publishing log messages to tcp://*:5562
2026-07-13 02:54:04,038 INFO [make_ssh_files] connected to skookum.eos.ubc.ca port 5555
2026-07-13 02:54:04,039 INFO [make_ssh_files] building forecast2 sea surface height boundary conditions file(s) from 2026-07-13 NOAA Neah Bay 00Z observation and forecast values
2026-07-13 02:54:06,309 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2026m07d08.nc
2026-07-13 02:54:06,521 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2026m07d09.nc
2026-07-13 02:54:06,724 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2026m07d10.nc
2026-07-13 02:54:06,906 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2026m07d11.nc
2026-07-13 02:54:07,117 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2026m07d12.nc
2026-07-13 02:54:07,394 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2026m07d13.nc
2026-07-13 02:54:07,673 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2026m07d14.nc
2026-07-13 02:54:07,935 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2026m07d15.nc
2026-07-13 02:54:07,943 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2026m07d14.nc
2026-07-13 02:54:08,114 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2026m07d16.nc
2026-07-13 02:54:08,298 INFO [make_ssh_files] sea surface height boundary file for 2026-07-13 forecast2 run created
2026-07-13 02:54:08,303 INFO [manager] checklist updated with [sea surface height forcing] items from make_ssh_files worker
2026-07-13 02:54:08,666 INFO [manager] launching NextWorker(module='nowcast.workers.make_runoff_file', args=['v202108'], host='localhost')
2026-07-13 02:54:10,835 INFO [make_runoff_file] running in process 288512
2026-07-13 02:54:10,836 INFO [make_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-07-13 02:54:10,837 INFO [make_runoff_file] publishing log messages to tcp://*:5562
2026-07-13 02:54:10,837 INFO [make_runoff_file] connected to skookum.eos.ubc.ca port 5555
2026-07-13 02:54:10,840 INFO [make_runoff_file] calculating NEMO runoff forcing for v202108 bathymetry for 2026-07-12
2026-07-13 02:54:11,246 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2026m07d15.nc
2026-07-13 02:54:11,253 INFO [grib_to_netcdf] 2026-07-13 NEMO-atmos forcing file for forecast2 created
2026-07-13 02:54:11,256 INFO [manager] checklist updated with [weather forcing] items from grib_to_netcdf worker
2026-07-13 02:54:11,902 ERROR [make_runoff_file] no 2026-07-12 discharge obs for SkagitMountVernon: patching
2026-07-13 02:54:12,316 ERROR [make_runoff_file] no 2026-07-12 discharge obs for SnohomishMonroe: patching
2026-07-13 02:54:12,521 ERROR [make_runoff_file] no 2026-07-12 discharge obs for NisquallyMcKenna: patching
2026-07-13 02:54:12,626 ERROR [make_runoff_file] no 2026-07-12 discharge obs for GreenwaterGreenwater: patching
2026-07-13 02:54:12,948 ERROR [make_runoff_file] no 2026-07-12 discharge obs for Theodosia: patching
2026-07-13 02:54:14,202 INFO [make_runoff_file] stored NEMO runoff forcing for v202108 bathymetry for 2026-07-12: /results/forcing/rivers/R202108Dailies_y2026m07d12.nc
2026-07-13 02:54:14,202 INFO [make_runoff_file] 2026-07-12 runoff file creation completed
2026-07-13 02:54:14,204 INFO [manager] checklist updated with [rivers forcing] items from make_runoff_file worker
2026-07-13 02:54:14,540 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'forecast2'], host='localhost')
2026-07-13 02:54:14,546 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'forecast2'], host='localhost')
2026-07-13 02:54:14,550 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.nibi', 'forecast2'], host='localhost')
2026-07-13 02:54:14,557 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'forecast2'], host='localhost')
2026-07-13 02:54:16,075 INFO [upload_forcing] running in process 288537
2026-07-13 02:54:16,076 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-07-13 02:54:16,077 INFO [upload_forcing] publishing log messages to tcp://*:5562
2026-07-13 02:54:16,078 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-07-13 02:54:16,087 INFO [upload_forcing] running in process 288539
2026-07-13 02:54:16,088 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-07-13 02:54:16,090 INFO [upload_forcing] publishing log messages to tcp://*:5564
2026-07-13 02:54:16,091 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-07-13 02:54:16,126 INFO [upload_forcing] running in process 288536
2026-07-13 02:54:16,127 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-07-13 02:54:16,128 INFO [upload_forcing] publishing log messages to tcp://*:5565
2026-07-13 02:54:16,129 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-07-13 02:54:16,139 INFO [upload_forcing] running in process 288538
2026-07-13 02:54:16,140 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-07-13 02:54:16,141 INFO [upload_forcing] publishing log messages to tcp://*:5566
2026-07-13 02:54:16,142 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-07-13 02:54:16,886 INFO [get_onc_ctd] 2026-07-12 ONC SCVIP CTD T&S file created
2026-07-13 02:54:16,889 INFO [manager] checklist updated with [ONC CTD data] items from get_onc_ctd worker
2026-07-13 02:54:17,255 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['SCVIP-CTD'], host='localhost')
2026-07-13 02:54:17,358 INFO [upload_forcing] LiveOcean boundary conditions file not found; created symlink to /results/forcing/LiveOcean/boundary_conditions/LiveOcean_v201905_y2026m07d12.nc
2026-07-13 02:54:17,415 INFO [upload_forcing] forecast2 2026-07-13 forcing files upload to optimum-hindcast completed
2026-07-13 02:54:17,419 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-07-13 02:54:18,146 INFO [get_onc_ctd] 2026-07-12 ONC SEVIP CTD T&S file created
2026-07-13 02:54:18,149 INFO [manager] checklist updated with [ONC CTD data] items from get_onc_ctd worker
2026-07-13 02:54:18,350 INFO [ping_erddap] running in process 288563
2026-07-13 02:54:18,351 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-07-13 02:54:18,352 INFO [ping_erddap] publishing log messages to tcp://*:5564
2026-07-13 02:54:18,353 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2026-07-13 02:54:18,355 INFO [ping_erddap] SCVIP-CTD ERDDAP dataset flag file(s) created
2026-07-13 02:54:18,442 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['SEVIP-CTD'], host='localhost')
2026-07-13 02:54:18,448 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2026-07-13 02:54:19,603 INFO [ping_erddap] running in process 288570
2026-07-13 02:54:19,605 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-07-13 02:54:19,606 INFO [ping_erddap] publishing log messages to tcp://*:5564
2026-07-13 02:54:19,607 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2026-07-13 02:54:19,608 INFO [ping_erddap] SEVIP-CTD ERDDAP dataset flag file(s) created
2026-07-13 02:54:19,610 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2026-07-13 02:54:20,196 INFO [get_onc_ferry] requesting ONC TWDP TSG data for 2026-07-12
2026-07-13 02:54:22,078 INFO [upload_forcing] forecast2 2026-07-13 forcing files upload to robot.nibi completed
2026-07-13 02:54:22,083 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-07-13 02:54:22,188 INFO [upload_forcing] forecast2 2026-07-13 forcing files upload to orcinus-nowcast-agrif completed
2026-07-13 02:54:22,403 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-07-13 02:54:42,460 INFO [upload_forcing] forecast2 2026-07-13 forcing files upload to arbutus.cloud-nowcast completed
2026-07-13 02:54:42,464 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-07-13 02:54:42,795 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2026-07-13'], host='localhost')
2026-07-13 02:54:44,058 INFO [make_forcing_links] running in process 288595
2026-07-13 02:54:44,060 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-07-13 02:54:44,061 INFO [make_forcing_links] publishing log messages to tcp://*:5562
2026-07-13 02:54:44,062 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2026-07-13 02:54:44,656 INFO [get_onc_ferry] requesting ONC TWDP OXYSENSOR data for 2026-07-12
2026-07-13 02:54:49,953 INFO [make_forcing_links] forecast2 2026-07-13 forcing file links on arbutus.cloud-nowcast created
2026-07-13 02:54:49,957 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2026-07-13 02:54:50,305 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2026-07-13'], host='arbutus.cloud-nowcast')
2026-07-13 02:55:00,333 WARNING [run_NEMO] There are uncommitted changes in /nemoShare/MEOPAR/nowcast-sys/SS-run-sets
2026-07-13 02:55:00,723 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2026-07-13 02:55:00,771 INFO [run_NEMO] forecast2: launching /nemoShare/MEOPAR/nowcast-sys/runs/12jul26forecast2_2026-07-13T025459.541090-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2026-07-13 02:55:00,782 INFO [run_NEMO] forecast2 NEMO run for 2026-07-13 on arbutus.cloud-nowcast started
2026-07-13 02:55:01,036 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'forecast2'], host='arbutus.cloud-nowcast')
2026-07-13 02:55:04,408 INFO [watch_NEMO] running in process 31596
2026-07-13 02:55:04,409 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-07-13 02:55:04,410 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2026-07-13 02:55:04,411 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2026-07-13 02:55:04,499 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2026-07-13 02:55:12,326 INFO [get_onc_ferry] requesting ONC TWDP TURBCHLFL data for 2026-07-12
2026-07-13 02:55:42,618 INFO [get_onc_ferry] requesting ONC TWDP CO2SENSOR data for 2026-07-12
2026-07-13 02:55:42,712 WARNING [get_onc_ferry] No ONC TWDP CO2SENSOR partial_pressure,co2 data for 2026-07-12; substituting empty dataset
2026-07-13 02:55:42,719 WARNING [get_onc_ferry] No ONC TWDP CO2SENSOR partial_pressure data for 2026-07-12; substituting empty dataset
2026-07-13 02:55:42,720 WARNING [get_onc_ferry] No ONC TWDP CO2SENSOR co2 data for 2026-07-12; substituting empty dataset
2026-07-13 02:55:42,721 INFO [get_onc_ferry] requesting ONC TWDP TEMPHUMID data for 2026-07-12
2026-07-13 02:55:42,781 WARNING [get_onc_ferry] No ONC TWDP TEMPHUMID air_temperature,rel_humidity data for 2026-07-12; substituting empty dataset
2026-07-13 02:55:42,786 WARNING [get_onc_ferry] No ONC TWDP TEMPHUMID air_temperature data for 2026-07-12; substituting empty dataset
2026-07-13 02:55:42,788 WARNING [get_onc_ferry] No ONC TWDP TEMPHUMID rel_humidity data for 2026-07-12; substituting empty dataset
2026-07-13 02:55:42,789 INFO [get_onc_ferry] requesting ONC TWDP BARPRESS data for 2026-07-12
2026-07-13 02:55:42,851 WARNING [get_onc_ferry] No ONC TWDP BARPRESS barometric_pressure data for 2026-07-12; substituting empty dataset
2026-07-13 02:55:42,854 WARNING [get_onc_ferry] No ONC TWDP BARPRESS barometric_pressure data for 2026-07-12; substituting empty dataset
2026-07-13 02:55:42,855 INFO [get_onc_ferry] requesting ONC TWDP PYRANOMETER data for 2026-07-12
2026-07-13 02:55:43,079 WARNING [get_onc_ferry] No ONC TWDP PYRANOMETER solar_radiation data for 2026-07-12; substituting empty dataset
2026-07-13 02:55:43,082 WARNING [get_onc_ferry] No ONC TWDP PYRANOMETER solar_radiation data for 2026-07-12; substituting empty dataset
2026-07-13 02:55:43,083 INFO [get_onc_ferry] requesting ONC TWDP PYRGEOMETER data for 2026-07-12
2026-07-13 02:55:43,139 WARNING [get_onc_ferry] No ONC TWDP PYRGEOMETER downward_radiation data for 2026-07-12; substituting empty dataset
2026-07-13 02:55:43,142 WARNING [get_onc_ferry] No ONC TWDP PYRGEOMETER downward_radiation data for 2026-07-12; substituting empty dataset
2026-07-13 02:55:43,338 WARNING [get_onc_ferry] ONC TWDP CO2SENSOR partial_pressure data for 2026-07-12 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2026-07-13 02:55:43,355 WARNING [get_onc_ferry] ONC TWDP CO2SENSOR co2 data for 2026-07-12 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2026-07-13 02:55:43,372 WARNING [get_onc_ferry] ONC TWDP TEMPHUMID air_temperature data for 2026-07-12 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2026-07-13 02:55:43,390 WARNING [get_onc_ferry] ONC TWDP TEMPHUMID rel_humidity data for 2026-07-12 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2026-07-13 02:55:43,407 WARNING [get_onc_ferry] ONC TWDP BARPRESS barometric_pressure data for 2026-07-12 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2026-07-13 02:55:43,426 WARNING [get_onc_ferry] ONC TWDP PYRANOMETER solar_radiation data for 2026-07-12 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2026-07-13 02:55:43,443 WARNING [get_onc_ferry] ONC TWDP PYRGEOMETER downward_radiation data for 2026-07-12 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2026-07-13 02:55:58,264 INFO [get_onc_ferry] 2026-07-12 ONC TWDP ferry data file created
2026-07-13 02:55:58,267 INFO [manager] checklist updated with [ONC ferry data] items from get_onc_ferry worker
2026-07-13 02:55:58,609 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['TWDP-ferry'], host='localhost')
2026-07-13 02:55:59,762 INFO [ping_erddap] running in process 288634
2026-07-13 02:55:59,763 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-07-13 02:55:59,765 INFO [ping_erddap] publishing log messages to tcp://*:5562
2026-07-13 02:55:59,766 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2026-07-13 02:55:59,768 INFO [ping_erddap] TWDP-ferry ERDDAP dataset flag file(s) created
2026-07-13 02:55:59,770 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2026-07-13 03:00:04,479 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 19360574 = 2026-07-14 05:28:40 UTC, 18.3% complete
2026-07-13 03:05:04,451 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 19361084 = 2026-07-14 11:08:40 UTC, 37.2% complete
2026-07-13 03:10:04,547 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 19361625 = 2026-07-14 17:09:20 UTC, 57.2% complete
2026-07-13 03:15:04,524 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 19362168 = 2026-07-14 23:11:20 UTC, 77.3% complete
2026-07-13 03:20:04,504 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 19362699 = 2026-07-15 05:05:20 UTC, 97.0% complete
2026-07-13 03:25:04,429 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2026-07-13 03:25:04,468 INFO [watch_NEMO] forecast2 NEMO run on arbutus.cloud-nowcast completed
2026-07-13 03:25:05,106 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_wind_file', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2026-07-13'], host='arbutus.cloud-nowcast')
2026-07-13 03:25:05,114 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_current_file', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2026-07-13'], host='arbutus.cloud-nowcast')
2026-07-13 03:25:05,118 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2026-07-12'], host='localhost')
2026-07-13 03:25:06,443 INFO [download_results] running in process 289282
2026-07-13 03:25:06,444 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-07-13 03:25:06,445 INFO [download_results] publishing log messages to tcp://*:5562
2026-07-13 03:25:06,446 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2026-07-13 03:25:06,448 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/forecast2/12jul26 to /results/SalishSea/forecast2.202111
2026-07-13 03:25:08,357 INFO [make_ww3_wind_file] running in process 31932
2026-07-13 03:25:08,358 INFO [make_ww3_wind_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-07-13 03:25:08,359 INFO [make_ww3_wind_file] publishing log messages to tcp://*:5570
2026-07-13 03:25:08,361 INFO [make_ww3_wind_file] connected to skookum.eos.ubc.ca port 5555
2026-07-13 03:25:08,362 INFO [make_ww3_wind_file] Creating wwatch3 wind forcing file for 2026-07-13 forecast2 run
2026-07-13 03:25:17,132 INFO [make_ww3_wind_file] wwatch3 wind forcing file created on arbutus.cloud-nowcast for 2026-07-13 forecast2 run
2026-07-13 03:25:17,136 INFO [manager] checklist updated with [WW3 wind forcing] items from make_ww3_wind_file worker
2026-07-13 03:27:01,042 INFO [download_results] forecast2 2026-07-12 results files from arbutus.cloud-nowcast downloaded
2026-07-13 03:27:01,046 INFO [manager] checklist updated with [results files] items from download_results worker
2026-07-13 03:27:01,378 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['forecast2', '--run-date', '2026-07-12'], host='localhost')
2026-07-13 03:27:03,747 INFO [make_CHS_currents_file] running in process 289325
2026-07-13 03:27:03,748 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-07-13 03:27:03,749 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5562
2026-07-13 03:27:03,750 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555
2026-07-13 03:27:41,416 INFO [make_CHS_currents_file] Made CHS currents file for 2026-07-12 for forecast2
2026-07-13 03:27:41,419 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker
2026-07-13 03:27:41,693 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['nemo', 'forecast2', '--run-date', '2026-07-12'], host='localhost')
2026-07-13 03:27:42,938 INFO [update_forecast_datasets] running in process 289366
2026-07-13 03:27:42,940 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-07-13 03:27:42,941 INFO [update_forecast_datasets] publishing log messages to tcp://*:5562
2026-07-13 03:27:42,942 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2026-07-13 03:27:42,943 INFO [update_forecast_datasets] updating nemo forecast directory for forecast2/12jul26 run
2026-07-13 03:28:51,391 INFO [manager] checklist updated with [WW3 currents forcing] items from make_ww3_current_file worker
2026-07-13 03:28:51,526 INFO [make_ww3_current_file] wwatch3 currents forcing file created on arbutus.cloud-nowcast for 2026-07-13 forecast2 run
2026-07-13 03:28:51,701 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2026-07-13'], host='arbutus.cloud-nowcast')
2026-07-13 03:28:55,884 INFO [run_ww3] running in process 32476
2026-07-13 03:28:55,894 INFO [run_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-07-13 03:28:55,895 INFO [run_ww3] publishing log messages to tcp://*:5572
2026-07-13 03:28:55,896 INFO [run_ww3] connected to skookum.eos.ubc.ca port 5555
2026-07-13 03:28:55,898 INFO [run_ww3] Created run directory /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast2_2026-07-13T032855.883875-0700
2026-07-13 03:28:55,905 INFO [run_ww3] forecast2: launching /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast2_2026-07-13T032855.883875-0700/SoGWW3.sh on arbutus.cloud-nowcast
2026-07-13 03:28:55,915 INFO [run_ww3] forecast2 WaveWatch3 run for 2026-07-13 on arbutus.cloud-nowcast started
2026-07-13 03:28:55,955 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker
2026-07-13 03:28:56,281 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'forecast2'], host='arbutus.cloud-nowcast')
2026-07-13 03:28:59,934 INFO [watch_ww3] running in process 32563
2026-07-13 03:28:59,936 INFO [watch_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-07-13 03:28:59,937 INFO [watch_ww3] publishing log messages to tcp://*:5573
2026-07-13 03:28:59,938 INFO [watch_ww3] connected to skookum.eos.ubc.ca port 5555
2026-07-13 03:28:59,983 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2026-07-13 03:31:59,618 INFO [update_forecast_datasets] updated nemo forecast directory for forecast2/12jul26 run: /results/SalishSea/rolling-forecasts/nemo
2026-07-13 03:31:59,619 INFO [update_forecast_datasets] nemo 2026-07-12 forecast2 rolling forecast datasets updated
2026-07-13 03:31:59,622 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2026-07-13 03:31:59,958 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nemo-forecast'], host='localhost')
2026-07-13 03:31:59,964 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'forecast2', 'publish', '--run-date', '2026-07-12'], host='localhost')
2026-07-13 03:31:59,969 INFO [manager] launching NextWorker(module='nowcast.workers.make_surface_current_tiles', args=['forecast2', '--run-date', '2026-07-12'], host='localhost')
2026-07-13 03:32:01,356 INFO [ping_erddap] running in process 289509
2026-07-13 03:32:01,358 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-07-13 03:32:01,359 INFO [ping_erddap] publishing log messages to tcp://*:5562
2026-07-13 03:32:01,360 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2026-07-13 03:32:01,372 INFO [ping_erddap] nemo-forecast ERDDAP dataset flag file(s) created
2026-07-13 03:32:01,373 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2026-07-13 03:32:02,919 INFO [make_surface_current_tiles] running in process 289512
2026-07-13 03:32:02,920 INFO [make_surface_current_tiles] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-07-13 03:32:02,921 INFO [make_surface_current_tiles] publishing log messages to tcp://*:5562
2026-07-13 03:32:02,922 INFO [make_surface_current_tiles] connected to skookum.eos.ubc.ca port 5555
2026-07-13 03:32:02,923 INFO [make_surface_current_tiles] starting rendering of tiles for 2026-07-12 forecast2 in 6 processes into /results/nowcast-sys/figures/surface_currents/forecast2/12jul26
2026-07-13 03:32:04,116 INFO [make_plots] running in process 289511
2026-07-13 03:32:04,118 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-07-13 03:32:04,118 INFO [make_plots] publishing log messages to tcp://*:5564
2026-07-13 03:32:04,119 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2026-07-13 03:32:04,644 INFO [make_plots] preparing render list for 2026-07-12 NEMO forecast2 publish figures
2026-07-13 03:32:04,968 INFO [make_plots] added Website_thumbnail figure to 2026-07-12 NEMO forecast2 publish render list
2026-07-13 03:32:04,969 INFO [make_plots] added Threshold_website figure to 2026-07-12 NEMO forecast2 publish render list
2026-07-13 03:32:04,969 INFO [make_plots] added PA_tidal_predictions figure to 2026-07-12 NEMO forecast2 publish render list
2026-07-13 03:32:04,970 INFO [make_plots] added Boundary Bay figure to 2026-07-12 NEMO forecast2 publish render list
2026-07-13 03:32:04,971 INFO [make_plots] added Campbell River figure to 2026-07-12 NEMO forecast2 publish render list
2026-07-13 03:32:04,972 INFO [make_plots] added Cherry Point figure to 2026-07-12 NEMO forecast2 publish render list
2026-07-13 03:32:04,973 INFO [make_plots] added Friday Harbor figure to 2026-07-12 NEMO forecast2 publish render list
2026-07-13 03:32:04,973 INFO [make_plots] added Halfmoon Bay figure to 2026-07-12 NEMO forecast2 publish render list
2026-07-13 03:32:04,974 INFO [make_plots] added Nanaimo figure to 2026-07-12 NEMO forecast2 publish render list
2026-07-13 03:32:04,975 INFO [make_plots] added Neah Bay figure to 2026-07-12 NEMO forecast2 publish render list
2026-07-13 03:32:04,975 INFO [make_plots] added New Westminster figure to 2026-07-12 NEMO forecast2 publish render list
2026-07-13 03:32:04,976 INFO [make_plots] added Patricia Bay figure to 2026-07-12 NEMO forecast2 publish render list
2026-07-13 03:32:04,976 INFO [make_plots] added Point Atkinson figure to 2026-07-12 NEMO forecast2 publish render list
2026-07-13 03:32:04,977 INFO [make_plots] added Port Renfrew figure to 2026-07-12 NEMO forecast2 publish render list
2026-07-13 03:32:04,978 INFO [make_plots] added Sand Heads figure to 2026-07-12 NEMO forecast2 publish render list
2026-07-13 03:32:04,979 INFO [make_plots] added Sandy Cove figure to 2026-07-12 NEMO forecast2 publish render list
2026-07-13 03:32:04,980 INFO [make_plots] added Squamish figure to 2026-07-12 NEMO forecast2 publish render list
2026-07-13 03:32:04,980 INFO [make_plots] added Victoria figure to 2026-07-12 NEMO forecast2 publish render list
2026-07-13 03:32:04,981 INFO [make_plots] added Woodwards Landing figure to 2026-07-12 NEMO forecast2 publish render list
2026-07-13 03:32:04,982 INFO [make_plots] starting to render nemo forecast2 publish 12jul26 figures
2026-07-13 03:33:59,966 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 93, 17.2% complete
2026-07-13 03:38:59,956 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 192, 35.6% complete
2026-07-13 03:39:34,130 WARNING [make_plots] invalid station number for Boundary Bay station: None
2026-07-13 03:39:34,131 WARNING [make_plots] can't resolve a valid CHS station code for Boundary Bay
2026-07-13 03:39:57,536 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb996/data for station 08074 Campbell River from 2026-07-08 00:00:00Z to 2026-07-15 05:55:00Z
2026-07-13 03:41:09,941 WARNING [make_plots] invalid station number for Halfmoon Bay station: None
2026-07-13 03:41:09,942 WARNING [make_plots] can't resolve a valid CHS station code for Halfmoon Bay
2026-07-13 03:41:34,613 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb96d/data for station 07917 Nanaimo from 2026-07-08 00:00:00Z to 2026-07-15 05:55:00Z
2026-07-13 03:42:22,579 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e13d0f4a073c4bbf93/data for station 07654 New Westminster from 2026-07-08 00:00:00Z to 2026-07-15 05:55:00Z
2026-07-13 03:42:46,176 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd2d/data for station 07277 Patricia Bay from 2026-07-08 00:00:00Z to 2026-07-15 05:55:00Z
2026-07-13 03:43:07,082 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94c/data for station 07795 Point Atkinson from 2026-07-08 00:00:00Z to 2026-07-15 05:55:00Z
2026-07-13 03:43:18,011 INFO [make_surface_current_tiles] starting PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast2/12jul26
2026-07-13 03:43:24,854 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e23d0f4a073c4bc060/data for station 08525 Port Renfrew from 2026-07-08 00:00:00Z to 2026-07-15 05:55:00Z
2026-07-13 03:43:41,078 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb93c/data for station 07594 Sand Heads from 2026-07-08 00:00:00Z to 2026-07-15 05:55:00Z
2026-07-13 03:43:59,478 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc43b/data for station 07786 Sandy Cove from 2026-07-08 00:00:00Z to 2026-07-15 05:55:00Z
2026-07-13 03:43:59,949 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 297, 55.0% complete
2026-07-13 03:44:17,140 WARNING [make_plots] invalid station number for Squamish station: None
2026-07-13 03:44:17,141 WARNING [make_plots] can't resolve a valid CHS station code for Squamish
2026-07-13 03:44:34,808 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd1e/data for station 07120 Victoria from 2026-07-08 00:00:00Z to 2026-07-15 05:55:00Z
2026-07-13 03:44:50,422 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc42d/data for station 07610 Woodwards Landing from 2026-07-08 00:00:00Z to 2026-07-15 05:55:00Z
2026-07-13 03:44:50,805 INFO [make_plots] no obs water level data available from 5cebf1e43d0f4a073c4bc42d during 2026-07-08 00:00:00Z to 2026-07-15 05:55:00Z
2026-07-13 03:45:07,753 INFO [make_plots] finished rendering nemo forecast2 publish 12jul26 figures
2026-07-13 03:45:07,755 INFO [make_plots] nemo publish plots for 2026-07-12 forecast2 completed
2026-07-13 03:45:07,763 INFO [manager] checklist updated with [plots] items from make_plots worker
2026-07-13 03:45:08,081 INFO [manager] launching NextWorker(module='nowcast.workers.make_feeds', args=['forecast2', '--run-date', '2026-07-12'], host='localhost')
2026-07-13 03:45:11,155 INFO [make_feeds] running in process 294664
2026-07-13 03:45:11,156 INFO [make_feeds] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-07-13 03:45:11,157 INFO [make_feeds] publishing log messages to tcp://*:5564
2026-07-13 03:45:11,158 INFO [make_feeds] connected to skookum.eos.ubc.ca port 5555
2026-07-13 03:45:26,159 INFO [make_feeds] ATOM feeds for 2026-07-12 forecast2 run completed
2026-07-13 03:45:26,162 INFO [manager] checklist updated with [storm surge alert feeds] items from make_feeds worker
2026-07-13 03:45:26,486 INFO [manager] launching NextWorker(module='nemo_nowcast.workers.clear_checklist', args=[], host='localhost')
2026-07-13 03:45:27,554 INFO [clear_checklist] running in process 294689
2026-07-13 03:45:27,555 INFO [clear_checklist] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-07-13 03:45:27,556 INFO [clear_checklist] publishing log messages to tcp://*:5564
2026-07-13 03:45:27,557 INFO [clear_checklist] connected to skookum.eos.ubc.ca port 5555
2026-07-13 03:45:27,558 INFO [clear_checklist] requesting that manager clear system state checklist
2026-07-13 03:45:27,561 INFO [manager] writing checklist to log file
2026-07-13 03:45:27,602 INFO [manager] checklist cleared
2026-07-13 03:45:27,604 INFO [clear_checklist] nowcast system checklist cleared
2026-07-13 03:45:27,607 INFO [manager] launching NextWorker(module='nemo_nowcast.workers.rotate_logs', args=[], host='localhost')
2026-07-13 03:45:28,561 INFO [rotate_logs] running in process 294697
2026-07-13 03:45:28,562 INFO [rotate_logs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-07-13 03:45:28,563 INFO [rotate_logs] publishing log messages to tcp://*:5564
2026-07-13 03:45:28,564 INFO [rotate_logs] connected to skookum.eos.ubc.ca port 5555
2026-07-13 03:45:28,565 INFO [rotate_logs] rotating log files