2026-06-12 03:26:46,659 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/nowcast.log
2026-06-12 03:26:46,668 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/nowcast.debug.log
2026-06-12 03:26:46,674 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/checklist.log
2026-06-12 03:26:46,675 INFO [rotate_logs] log files rotated
2026-06-12 03:26:46,683 INFO [manager] checklist updated with [log rotation] items from rotate_logs worker
2026-06-12 03:29:02,893 INFO [make_surface_current_tiles] finished PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast2/11jun26
2026-06-12 03:29:02,916 INFO [make_surface_current_tiles] finished rendering of tiles for 2026-06-11 forecast2 into /results/nowcast-sys/figures/surface_currents/forecast2/11jun26
2026-06-12 03:29:02,917 INFO [make_surface_current_tiles] surface current tile figures for 2026-06-11 forecast2 completed
2026-06-12 03:29:03,385 INFO [manager] checklist updated with [surface current tiles] items from make_surface_current_tiles worker
2026-06-12 03:30:56,760 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 408, 75.6% complete
2026-06-12 03:35:56,761 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 513, 95.0% complete
2026-06-12 03:40:56,762 INFO [watch_ww3] forecast2 WWATCH3 run on arbutus.cloud-nowcast completed
2026-06-12 03:40:56,765 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2026-06-12 03:40:57,134 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2026-06-12'], host='localhost')
2026-06-12 03:40:58,445 INFO [download_wwatch3_results] running in process 66163
2026-06-12 03:40:58,446 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-06-12 03:40:58,447 INFO [download_wwatch3_results] publishing log messages to tcp://*:5562
2026-06-12 03:40:58,449 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2026-06-12 03:41:35,475 INFO [download_wwatch3_results] forecast2 2026-06-12 results files from arbutus.cloud-nowcast downloaded
2026-06-12 03:41:35,478 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2026-06-12 03:41:35,628 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['wwatch3', 'forecast2', '--run-date', '2026-06-12'], host='localhost')
2026-06-12 03:41:36,664 INFO [update_forecast_datasets] running in process 66183
2026-06-12 03:41:36,665 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-06-12 03:41:36,666 INFO [update_forecast_datasets] publishing log messages to tcp://*:5562
2026-06-12 03:41:36,667 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2026-06-12 03:41:36,669 INFO [update_forecast_datasets] updating wwatch3 most_recent_forecast directory from forecast2/12jun26 run
2026-06-12 03:41:36,742 INFO [update_forecast_datasets] updated wwatch3 most_recent_forecast directory from forecast2/12jun26 run
2026-06-12 03:41:36,743 INFO [update_forecast_datasets] updating wwatch3 forecast directory for forecast2/12jun26 run
2026-06-12 03:42:00,159 INFO [update_forecast_datasets] updated wwatch3 forecast directory for forecast2/12jun26 run: /results/SalishSea/rolling-forecasts/wwatch3
2026-06-12 03:42:00,160 INFO [update_forecast_datasets] wwatch3 2026-06-12 forecast2 rolling forecast datasets updated
2026-06-12 03:42:00,163 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2026-06-12 03:42:00,314 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['wwatch3-forecast'], host='localhost')
2026-06-12 03:42:01,499 INFO [ping_erddap] running in process 66199
2026-06-12 03:42:01,500 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-06-12 03:42:01,502 INFO [ping_erddap] publishing log messages to tcp://*:5562
2026-06-12 03:42:01,503 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2026-06-12 03:42:01,505 INFO [ping_erddap] wwatch3-forecast ERDDAP dataset flag file(s) created
2026-06-12 03:42:01,508 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2026-06-12 03:42:01,653 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['wwatch3', 'forecast2', 'publish', '--run-date', '2026-06-12'], host='localhost')
2026-06-12 03:42:04,683 INFO [make_plots] running in process 66204
2026-06-12 03:42:04,685 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-06-12 03:42:04,686 INFO [make_plots] publishing log messages to tcp://*:5562
2026-06-12 03:42:04,687 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2026-06-12 03:42:04,688 INFO [make_plots] preparing render list for 2026-06-12 WaveWatch3 forecast2 publish figures
2026-06-12 03:42:04,689 INFO [make_plots] added Halibut Bank figure to 2026-06-12 WaveWatch3 forecast2 publish render list
2026-06-12 03:42:04,690 INFO [make_plots] added Sentry Shoal figure to 2026-06-12 WaveWatch3 forecast2 publish render list
2026-06-12 03:42:04,691 INFO [make_plots] starting to render wwatch3 forecast2 publish 12jun26 figures
2026-06-12 03:42:05,329 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46146 Halibut Bank
2026-06-12 03:42:11,254 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46131 Sentry Shoal
2026-06-12 03:42:15,246 INFO [make_plots] finished rendering wwatch3 forecast2 publish 12jun26 figures
2026-06-12 03:42:15,247 INFO [make_plots] wwatch3 publish plots for 2026-06-12 forecast2 completed
2026-06-12 03:42:15,251 INFO [manager] checklist updated with [plots] items from make_plots worker
2026-06-12 08:40:29,339 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/12/ to /results/forcing/atmospheric/continental2.5/GRIB/20260612/12/
2026-06-12 08:40:29,341 INFO [collect_weather] 2026-06-12 2.5km weather forecast 12 collection complete
2026-06-12 08:40:29,346 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2026-06-12 08:40:29,671 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'SkagitMountVernon', '--data-date', '2026-06-11'], host='localhost')
2026-06-12 08:40:29,678 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'SnohomishMonroe', '--data-date', '2026-06-11'], host='localhost')
2026-06-12 08:40:29,684 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'NisquallyMcKenna', '--data-date', '2026-06-11'], host='localhost')
2026-06-12 08:40:29,688 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'GreenwaterGreenwater', '--data-date', '2026-06-11'], host='localhost')
2026-06-12 08:40:29,694 INFO [manager] launching NextWorker(module='nowcast.workers.make_turbidity_file', args=[], host='localhost')
2026-06-12 08:40:29,700 INFO [manager] launching NextWorker(module='nowcast.workers.collect_NeahBay_ssh', args=['06'], host='localhost')
2026-06-12 08:40:29,711 INFO [manager] launching NextWorker(module='nowcast.workers.download_live_ocean', args=[], host='localhost')
2026-06-12 08:40:29,716 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['18', '2.5km'], host='localhost')
2026-06-12 08:40:29,738 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['18'], host='localhost')
2026-06-12 08:40:30,933 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20260612/12/
2026-06-12 08:40:30,934 INFO [crop_gribs] 2026-06-12 12 GRIBs cropping complete
2026-06-12 08:40:30,936 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2026-06-12 08:40:31,365 INFO [collect_weather] running in process 72980
2026-06-12 08:40:31,366 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-06-12 08:40:31,366 INFO [collect_weather] publishing log messages to tcp://*:5562
2026-06-12 08:40:31,367 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2026-06-12 08:40:31,372 INFO [collect_NeahBay_ssh] running in process 72978
2026-06-12 08:40:31,373 INFO [collect_NeahBay_ssh] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-06-12 08:40:31,374 INFO [collect_NeahBay_ssh] publishing log messages to tcp://*:5563
2026-06-12 08:40:31,375 INFO [collect_NeahBay_ssh] connected to skookum.eos.ubc.ca port 5555
2026-06-12 08:40:31,375 INFO [collect_NeahBay_ssh] collecting Neah Bay ssh 06Z obs/forecast for 2026-06-12
2026-06-12 08:40:31,382 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/18/
2026-06-12 08:40:31,408 INFO [manager] launching NextWorker(module='nowcast.workers.grib_to_netcdf', args=['nowcast+'], host='localhost')
2026-06-12 08:40:31,551 INFO [download_live_ocean] running in process 72979
2026-06-12 08:40:31,552 INFO [download_live_ocean] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-06-12 08:40:31,553 INFO [download_live_ocean] publishing log messages to tcp://*:5564
2026-06-12 08:40:31,554 INFO [download_live_ocean] connected to skookum.eos.ubc.ca port 5555
2026-06-12 08:40:31,555 INFO [download_live_ocean] downloading Salish Sea western boundary day-averaged LiveOcean file for 2026-06-12
2026-06-12 08:40:31,966 INFO [collect_river_data] running in process 72976
2026-06-12 08:40:31,968 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-06-12 08:40:31,969 INFO [collect_river_data] publishing log messages to tcp://*:5565
2026-06-12 08:40:31,970 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-06-12 08:40:31,971 INFO [collect_river_data] Collecting USGS GreenwaterGreenwater river data for 2026-06-11
2026-06-12 08:40:31,982 INFO [collect_river_data] running in process 72972
2026-06-12 08:40:31,988 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-06-12 08:40:31,989 INFO [collect_river_data] publishing log messages to tcp://*:5566
2026-06-12 08:40:31,990 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-06-12 08:40:31,990 INFO [collect_river_data] Collecting USGS SkagitMountVernon river data for 2026-06-11
2026-06-12 08:40:32,020 INFO [collect_river_data] running in process 72974
2026-06-12 08:40:32,025 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-06-12 08:40:32,026 INFO [collect_river_data] publishing log messages to tcp://*:5567
2026-06-12 08:40:32,027 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-06-12 08:40:32,029 INFO [collect_river_data] Collecting USGS SnohomishMonroe river data for 2026-06-11
2026-06-12 08:40:32,058 INFO [collect_river_data] running in process 72975
2026-06-12 08:40:32,060 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-06-12 08:40:32,061 INFO [collect_river_data] publishing log messages to tcp://*:5568
2026-06-12 08:40:32,062 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-06-12 08:40:32,062 INFO [collect_river_data] Collecting USGS NisquallyMcKenna river data for 2026-06-11
2026-06-12 08:40:32,068 INFO [make_turbidity_file] running in process 72977
2026-06-12 08:40:32,069 INFO [make_turbidity_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-06-12 08:40:32,070 INFO [make_turbidity_file] publishing log messages to tcp://*:5569
2026-06-12 08:40:32,070 INFO [make_turbidity_file] connected to skookum.eos.ubc.ca port 5555
2026-06-12 08:40:32,071 INFO [make_turbidity_file] Creating Fraser River turbidity forcing file for 2026-06-12
2026-06-12 08:40:32,386 INFO [crop_gribs] running in process 72982
2026-06-12 08:40:32,387 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-06-12 08:40:32,388 INFO [crop_gribs] publishing log messages to tcp://*:5570
2026-06-12 08:40:32,389 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2026-06-12 08:40:32,389 INFO [crop_gribs] cropping 2026-06-12 ECCC HRDPS 2.5km continental 18Z GRIB files to SalishSeaCast subdomain
2026-06-12 08:40:32,403 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20260612/18/
2026-06-12 08:40:32,542 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12150800&startDT=2026-06-11&endDT=2026-06-11 "HTTP/1.1 200 "
2026-06-12 08:40:32,548 INFO [collect_river_data] Appended USGS SnohomishMonroe river average discharge for 2026-06-11 to: /results/forcing/rivers/observations/Snohomish_Monroe_flow
2026-06-12 08:40:32,549 INFO [collect_river_data] USGS SnohomishMonroe river data collection for 2026-06-11 completed
2026-06-12 08:40:32,552 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-06-12 08:40:32,553 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12097500&startDT=2026-06-11&endDT=2026-06-11 "HTTP/1.1 200 "
2026-06-12 08:40:32,554 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12200500&startDT=2026-06-11&endDT=2026-06-11 "HTTP/1.1 200 "
2026-06-12 08:40:32,561 INFO [collect_river_data] Appended USGS SkagitMountVernon river average discharge for 2026-06-11 to: /results/forcing/rivers/observations/Skagit_MountVernon_flow
2026-06-12 08:40:32,562 INFO [collect_river_data] Appended USGS GreenwaterGreenwater river average discharge for 2026-06-11 to: /results/forcing/rivers/observations/Greenwater_Greenwater_flow
2026-06-12 08:40:32,563 INFO [collect_river_data] USGS SkagitMountVernon river data collection for 2026-06-11 completed
2026-06-12 08:40:32,564 INFO [collect_river_data] USGS GreenwaterGreenwater river data collection for 2026-06-11 completed
2026-06-12 08:40:32,567 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12089500&startDT=2026-06-11&endDT=2026-06-11 "HTTP/1.1 200 "
2026-06-12 08:40:32,572 INFO [collect_river_data] Appended USGS NisquallyMcKenna river average discharge for 2026-06-11 to: /results/forcing/rivers/observations/Nisqually_McKenna_flow
2026-06-12 08:40:32,573 INFO [collect_river_data] USGS NisquallyMcKenna river data collection for 2026-06-11 completed
2026-06-12 08:40:32,698 INFO [collect_NeahBay_ssh] 2026-06-12T00:00:00-07:00 Neah Bay ssh 06Z obs/forecast data collection complete
2026-06-12 08:40:32,705 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-06-12 08:40:32,859 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-06-12 08:40:33,007 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-06-12 08:40:33,155 INFO [manager] checklist updated with [Neah Bay ssh data] items from collect_NeahBay_ssh worker
2026-06-12 08:40:33,286 INFO [manager] launching NextWorker(module='nowcast.workers.make_ssh_files', args=['nowcast', '--run-date', '2026-06-12'], host='localhost')
2026-06-12 08:40:33,336 INFO [grib_to_netcdf] running in process 73069
2026-06-12 08:40:33,337 INFO [grib_to_netcdf] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-06-12 08:40:33,338 INFO [grib_to_netcdf] publishing log messages to tcp://*:5565
2026-06-12 08:40:33,339 INFO [grib_to_netcdf] connected to skookum.eos.ubc.ca port 5555
2026-06-12 08:40:33,340 INFO [grib_to_netcdf] creating NEMO-atmos forcing files for 2026-06-12 nowcast and forecast runs
2026-06-12 08:40:34,546 INFO [make_turbidity_file] 2026-06-12 Fraser River turbidity file creation complete
2026-06-12 08:40:34,548 INFO [manager] checklist updated with [Fraser River turbidity] items from make_turbidity_file worker
2026-06-12 08:40:35,943 INFO [make_ssh_files] running in process 73114
2026-06-12 08:40:35,945 INFO [make_ssh_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-06-12 08:40:35,946 INFO [make_ssh_files] publishing log messages to tcp://*:5563
2026-06-12 08:40:35,947 INFO [make_ssh_files] connected to skookum.eos.ubc.ca port 5555
2026-06-12 08:40:35,948 INFO [make_ssh_files] building nowcast sea surface height boundary conditions file(s) from 2026-06-12 NOAA Neah Bay 06Z observation and forecast values
2026-06-12 08:40:37,548 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2026m06d08.nc
2026-06-12 08:40:37,725 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2026m06d09.nc
2026-06-12 08:40:37,897 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2026m06d10.nc
2026-06-12 08:40:38,068 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2026m06d11.nc
2026-06-12 08:40:38,269 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2026m06d12.nc
2026-06-12 08:40:38,460 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2026m06d13.nc
2026-06-12 08:40:38,658 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2026m06d14.nc
2026-06-12 08:40:38,860 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2026m06d15.nc
2026-06-12 08:40:38,984 INFO [make_ssh_files] sea surface height boundary file for 2026-06-12 nowcast run created
2026-06-12 08:40:38,987 INFO [manager] checklist updated with [sea surface height forcing] items from make_ssh_files worker
2026-06-12 08:40:39,136 INFO [manager] launching NextWorker(module='nowcast.workers.make_runoff_file', args=['v202108'], host='localhost')
2026-06-12 08:40:41,000 INFO [make_runoff_file] running in process 73167
2026-06-12 08:40:41,001 INFO [make_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-06-12 08:40:41,003 INFO [make_runoff_file] publishing log messages to tcp://*:5563
2026-06-12 08:40:41,004 INFO [make_runoff_file] connected to skookum.eos.ubc.ca port 5555
2026-06-12 08:40:41,005 INFO [make_runoff_file] calculating NEMO runoff forcing for v202108 bathymetry for 2026-06-11
2026-06-12 08:40:42,390 ERROR [make_runoff_file] no 2026-06-11 discharge obs for Theodosia: patching
2026-06-12 08:40:44,012 INFO [make_runoff_file] stored NEMO runoff forcing for v202108 bathymetry for 2026-06-11: /results/forcing/rivers/R202108Dailies_y2026m06d11.nc
2026-06-12 08:40:44,013 INFO [make_runoff_file] 2026-06-11 runoff file creation completed
2026-06-12 08:40:44,015 INFO [manager] checklist updated with [rivers forcing] items from make_runoff_file worker
2026-06-12 08:40:44,879 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/hrdps_y2026m06d12.nc
2026-06-12 08:40:53,536 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2026m06d13.nc
2026-06-12 08:40:58,484 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2026m06d14.nc
2026-06-12 08:40:58,499 INFO [grib_to_netcdf] 2026-06-12 NEMO-atmos forcing file for nowcast+ created
2026-06-12 08:40:58,502 INFO [manager] checklist updated with [weather forcing] items from grib_to_netcdf worker
2026-06-12 11:40:35,434 ERROR [download_live_ocean] giving up after 37 attempts: 403 Client Error: Forbidden for url: https://s3.kopah.uw.edu/liveocean-share/f2026.06.12/ubc_done.txt for https://s3.kopah.uw.edu/liveocean-share/f2026.06.12/ubc_done.txt
2026-06-12 11:40:35,826 CRITICAL [download_live_ocean] 2026-06-12 Live Ocean file for Salish Sea western boundary download failed
2026-06-12 11:43:26,188 INFO [download_live_ocean] running in process 78098
2026-06-12 11:43:26,189 INFO [download_live_ocean] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-06-12 11:43:26,191 INFO [download_live_ocean] publishing log messages to tcp://*:5563
2026-06-12 11:43:26,194 INFO [download_live_ocean] connected to skookum.eos.ubc.ca port 5555
2026-06-12 11:43:26,194 INFO [download_live_ocean] downloading Salish Sea western boundary day-averaged LiveOcean file for 2026-06-12
2026-06-12 12:28:27,579 INFO [download_live_ocean] downloaded 10220256 bytes from https://s3.kopah.uw.edu/liveocean-share/f2026.06.12/ubc.nc to /results/forcing/LiveOcean/downloaded/20260612/low_passed_UBC.nc
2026-06-12 12:28:27,580 INFO [download_live_ocean] Deflating in up to 1 concurrent sub-processes
2026-06-12 12:28:28,609 INFO [download_live_ocean] netCDF4 deflated /results/forcing/LiveOcean/downloaded/20260612/low_passed_UBC.nc
2026-06-12 12:28:28,610 INFO [download_live_ocean] 2026-06-12 Live Ocean file for Salish Sea western boundary downloaded
2026-06-12 12:28:28,613 INFO [manager] checklist updated with [Live Ocean products] items from download_live_ocean worker
2026-06-12 12:28:28,901 INFO [manager] launching NextWorker(module='nowcast.workers.make_live_ocean_files', args=['--run-date', '2026-06-12'], host='localhost')
2026-06-12 12:28:31,771 INFO [make_live_ocean_files] running in process 79076
2026-06-12 12:28:31,772 INFO [make_live_ocean_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-06-12 12:28:31,774 INFO [make_live_ocean_files] publishing log messages to tcp://*:5563
2026-06-12 12:28:31,775 INFO [make_live_ocean_files] connected to skookum.eos.ubc.ca port 5555
2026-06-12 12:28:31,776 INFO [make_live_ocean_files] Creating T&S western boundary conditions file from 2026-06-12 Live Ocean run
2026-06-12 12:29:11,692 INFO [make_live_ocean_files] Stored T&S western boundary conditions file: /results/forcing/LiveOcean/boundary_conditions/LiveOcean_v201905_y2026m06d12.nc
2026-06-12 12:29:11,693 INFO [make_live_ocean_files] 2026-06-12 Live Ocean western boundary conditions files created
2026-06-12 12:29:11,696 INFO [manager] checklist updated with [Live Ocean boundary conditions] items from make_live_ocean_files worker
2026-06-12 12:29:11,847 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['weather'], host='localhost')
2026-06-12 12:29:11,851 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'nowcast+'], host='localhost')
2026-06-12 12:29:11,854 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'nowcast+'], host='localhost')
2026-06-12 12:29:11,858 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.nibi', 'nowcast+'], host='localhost')
2026-06-12 12:29:11,863 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'nowcast+'], host='localhost')
2026-06-12 12:29:13,154 INFO [ping_erddap] running in process 79118
2026-06-12 12:29:13,156 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-06-12 12:29:13,157 INFO [ping_erddap] publishing log messages to tcp://*:5563
2026-06-12 12:29:13,159 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2026-06-12 12:29:13,161 INFO [ping_erddap] weather ERDDAP dataset flag file(s) created
2026-06-12 12:29:13,163 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2026-06-12 12:29:13,194 INFO [upload_forcing] running in process 79123
2026-06-12 12:29:13,196 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-06-12 12:29:13,197 INFO [upload_forcing] publishing log messages to tcp://*:5564
2026-06-12 12:29:13,198 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-06-12 12:29:13,232 INFO [upload_forcing] running in process 79121
2026-06-12 12:29:13,233 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-06-12 12:29:13,234 INFO [upload_forcing] publishing log messages to tcp://*:5565
2026-06-12 12:29:13,236 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-06-12 12:29:13,237 INFO [upload_forcing] running in process 79122
2026-06-12 12:29:13,238 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-06-12 12:29:13,239 INFO [upload_forcing] publishing log messages to tcp://*:5566
2026-06-12 12:29:13,240 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-06-12 12:29:13,269 INFO [upload_forcing] running in process 79120
2026-06-12 12:29:13,270 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-06-12 12:29:13,271 INFO [upload_forcing] publishing log messages to tcp://*:5567
2026-06-12 12:29:13,272 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-06-12 12:29:14,866 INFO [upload_forcing] nowcast+ 2026-06-12 forcing files upload to optimum-hindcast completed
2026-06-12 12:29:14,868 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-06-12 12:29:16,601 INFO [upload_forcing] nowcast+ 2026-06-12 forcing files upload to arbutus.cloud-nowcast completed
2026-06-12 12:29:16,605 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-06-12 12:29:16,761 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'nowcast+', '--run-date', '2026-06-12'], host='localhost')
2026-06-12 12:29:18,218 INFO [make_forcing_links] running in process 79155
2026-06-12 12:29:18,219 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-06-12 12:29:18,221 INFO [make_forcing_links] publishing log messages to tcp://*:5563
2026-06-12 12:29:18,222 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2026-06-12 12:29:19,293 INFO [make_forcing_links] nowcast+ 2026-06-12 forcing file links on arbutus.cloud-nowcast created
2026-06-12 12:29:19,297 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2026-06-12 12:29:19,475 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2026-06-12'], host='arbutus.cloud-nowcast')
2026-06-12 12:29:20,379 INFO [upload_forcing] nowcast+ 2026-06-12 forcing files upload to orcinus-nowcast-agrif completed
2026-06-12 12:29:20,384 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-06-12 12:29:21,593 INFO [run_NEMO] running in process 9048
2026-06-12 12:29:21,594 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-06-12 12:29:21,595 INFO [run_NEMO] publishing log messages to tcp://*:5556
2026-06-12 12:29:21,605 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2026-06-12 12:29:21,835 INFO [upload_forcing] nowcast+ 2026-06-12 forcing files upload to robot.nibi completed
2026-06-12 12:29:21,839 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-06-12 12:29:22,360 WARNING [run_NEMO] There are uncommitted changes in /nemoShare/MEOPAR/nowcast-sys/SS-run-sets
2026-06-12 12:29:22,676 INFO [run_NEMO] nowcast: launching /nemoShare/MEOPAR/nowcast-sys/runs/12jun26nowcast_2026-06-12T122921.648554-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2026-06-12 12:29:22,691 INFO [run_NEMO] nowcast NEMO run for 2026-06-12 on arbutus.cloud-nowcast started
2026-06-12 12:29:22,695 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2026-06-12 12:29:22,849 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'nowcast'], host='arbutus.cloud-nowcast')
2026-06-12 12:29:24,918 INFO [watch_NEMO] running in process 9321
2026-06-12 12:29:24,921 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-06-12 12:29:24,922 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2026-06-12 12:29:24,924 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2026-06-12 12:29:24,987 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2026-06-12 12:34:24,987 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 19291467 = 2026-06-12 05:37:20 UTC, 23.4% complete
2026-06-12 12:39:24,990 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 19292019 = 2026-06-12 11:45:20 UTC, 49.0% complete
2026-06-12 12:44:24,992 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 19292574 = 2026-06-12 17:55:20 UTC, 74.7% complete
2026-06-12 12:49:24,994 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 19293120 = 2026-06-12 23:59:20 UTC, 100.0% complete
2026-06-12 12:54:24,998 INFO [watch_NEMO] nowcast NEMO run on arbutus.cloud-nowcast completed
2026-06-12 12:54:25,002 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2026-06-12 12:54:25,311 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'ssh', '--run-date', '2026-06-12'], host='localhost')
2026-06-12 12:54:25,317 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2026-06-12'], host='localhost')
2026-06-12 12:54:26,543 INFO [download_results] running in process 79853
2026-06-12 12:54:26,545 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-06-12 12:54:26,546 INFO [download_results] publishing log messages to tcp://*:5563
2026-06-12 12:54:26,547 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2026-06-12 12:54:26,548 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/nowcast/12jun26 to /results/SalishSea/nowcast-blue.202111
2026-06-12 12:54:26,652 INFO [make_forcing_links] running in process 79852
2026-06-12 12:54:26,653 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-06-12 12:54:26,655 INFO [make_forcing_links] publishing log messages to tcp://*:5564
2026-06-12 12:54:26,656 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2026-06-12 12:54:27,673 INFO [make_forcing_links] ssh 2026-06-12 forcing file links on arbutus.cloud-nowcast created
2026-06-12 12:54:27,678 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2026-06-12 12:54:27,845 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2026-06-12'], host='arbutus.cloud-nowcast')
2026-06-12 12:54:30,324 INFO [run_NEMO] running in process 9584
2026-06-12 12:54:30,325 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-06-12 12:54:30,329 INFO [run_NEMO] publishing log messages to tcp://*:5556
2026-06-12 12:54:30,332 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2026-06-12 12:54:31,126 WARNING [run_NEMO] There are uncommitted changes in /nemoShare/MEOPAR/nowcast-sys/SS-run-sets
2026-06-12 12:54:31,454 INFO [run_NEMO] forecast: launching /nemoShare/MEOPAR/nowcast-sys/runs/12jun26forecast_2026-06-12T125430.406368-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2026-06-12 12:54:31,469 INFO [run_NEMO] forecast NEMO run for 2026-06-12 on arbutus.cloud-nowcast started
2026-06-12 12:54:31,480 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2026-06-12 12:54:31,638 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'forecast'], host='arbutus.cloud-nowcast')
2026-06-12 12:54:33,784 INFO [watch_NEMO] running in process 9860
2026-06-12 12:54:33,785 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-06-12 12:54:33,786 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2026-06-12 12:54:33,787 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2026-06-12 12:54:33,882 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2026-06-12 12:54:59,291 INFO [download_results] nowcast 2026-06-12 results files from arbutus.cloud-nowcast downloaded
2026-06-12 12:54:59,294 INFO [manager] checklist updated with [results files] items from download_results worker
2026-06-12 12:54:59,453 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast', 'research', '--run-date', '2026-06-12'], host='localhost')
2026-06-12 12:54:59,458 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast', 'comparison', '--run-date', '2026-06-11'], host='localhost')
2026-06-12 12:54:59,464 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['nowcast', '--run-date', '2026-06-12'], host='localhost')
2026-06-12 12:55:01,530 INFO [make_CHS_currents_file] running in process 79883
2026-06-12 12:55:01,531 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-06-12 12:55:01,531 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5563
2026-06-12 12:55:01,532 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555
2026-06-12 12:55:02,676 INFO [make_plots] running in process 79880
2026-06-12 12:55:02,677 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-06-12 12:55:02,678 INFO [make_plots] publishing log messages to tcp://*:5564
2026-06-12 12:55:02,679 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2026-06-12 12:55:02,753 INFO [make_plots] running in process 79882
2026-06-12 12:55:02,754 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-06-12 12:55:02,755 INFO [make_plots] publishing log messages to tcp://*:5565
2026-06-12 12:55:02,756 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2026-06-12 12:55:03,096 INFO [make_plots] preparing render list for 2026-06-12 NEMO nowcast-blue research figures
2026-06-12 12:55:03,160 INFO [make_plots] preparing render list for 2026-06-11 NEMO nowcast-blue comparison figures
2026-06-12 12:55:03,207 INFO [make_plots] added SH_wind figure to 2026-06-11 NEMO nowcast-blue comparison render list
2026-06-12 12:55:03,208 INFO [make_plots] added Compare_VENUS_East figure to 2026-06-11 NEMO nowcast-blue comparison render list
2026-06-12 12:55:03,209 INFO [make_plots] added Compare_VENUS_Central figure to 2026-06-11 NEMO nowcast-blue comparison render list
2026-06-12 12:55:03,210 INFO [make_plots] starting to render nemo nowcast comparison 11jun26 figures
2026-06-12 12:55:04,505 INFO [make_plots] added salinity_thalweg_and_surface figures to 2026-06-12 NEMO nowcast-blue research render list
2026-06-12 12:55:05,188 INFO [make_plots] added temperature_thalweg_and_surface figures to 2026-06-12 NEMO nowcast-blue research render list
2026-06-12 12:55:05,190 INFO [make_plots] added Currents_sections_and_surface figure to 2026-06-12 NEMO nowcast-blue research render list
2026-06-12 12:55:05,191 INFO [make_plots] added Currents_at_VENUS_Central figure to 2026-06-12 NEMO nowcast-blue research render list
2026-06-12 12:55:05,192 INFO [make_plots] added Currents_at_VENUS_East figure to 2026-06-12 NEMO nowcast-blue research render list
2026-06-12 12:55:05,194 INFO [make_plots] starting to render nemo nowcast research 12jun26 figures
2026-06-12 12:55:32,498 INFO [make_CHS_currents_file] Made CHS currents file for 2026-06-12 for nowcast
2026-06-12 12:55:32,501 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker
2026-06-12 12:56:04,066 INFO [make_plots] finished rendering nemo nowcast research 12jun26 figures
2026-06-12 12:56:04,069 INFO [make_plots] nemo research plots for 2026-06-12 nowcast completed
2026-06-12 12:56:04,088 INFO [manager] checklist updated with [plots] items from make_plots worker
2026-06-12 12:56:30,241 INFO [make_plots] finished rendering nemo nowcast comparison 11jun26 figures
2026-06-12 12:56:30,243 INFO [make_plots] nemo comparison plots for 2026-06-11 nowcast completed
2026-06-12 12:56:30,248 INFO [manager] checklist updated with [plots] items from make_plots worker
2026-06-12 12:59:33,879 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 19293608 = 2026-06-13 05:24:40 UTC, 15.0% complete
2026-06-12 13:04:33,880 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 19294125 = 2026-06-13 11:09:20 UTC, 31.0% complete
2026-06-12 13:09:33,881 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 19294664 = 2026-06-13 17:08:40 UTC, 47.6% complete
2026-06-12 13:14:33,904 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 19295176 = 2026-06-13 22:50:00 UTC, 63.4% complete
2026-06-12 13:19:33,891 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 19295706 = 2026-06-14 04:43:20 UTC, 79.8% complete
2026-06-12 13:24:33,886 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 19296273 = 2026-06-14 11:01:20 UTC, 97.3% complete
2026-06-12 13:29:33,890 INFO [watch_NEMO] forecast NEMO run on arbutus.cloud-nowcast completed
2026-06-12 13:29:33,894 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2026-06-12 13:29:34,231 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_wind_file', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2026-06-12'], host='arbutus.cloud-nowcast')
2026-06-12 13:29:34,241 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_current_file', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2026-06-12'], host='arbutus.cloud-nowcast')
2026-06-12 13:29:34,245 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'turbidity', '--run-date', '2026-06-12'], host='localhost')
2026-06-12 13:29:34,252 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'turbidity', '--run-date', '2026-06-12'], host='localhost')
2026-06-12 13:29:34,257 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.nibi', 'turbidity', '--run-date', '2026-06-12'], host='localhost')
2026-06-12 13:29:34,263 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'turbidity', '--run-date', '2026-06-12'], host='localhost')
2026-06-12 13:29:34,268 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2026-06-12'], host='localhost')
2026-06-12 13:29:35,596 INFO [download_results] running in process 80728
2026-06-12 13:29:35,598 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-06-12 13:29:35,599 INFO [download_results] publishing log messages to tcp://*:5563
2026-06-12 13:29:35,601 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2026-06-12 13:29:35,602 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/forecast/12jun26 to /results/SalishSea/forecast.202111
2026-06-12 13:29:35,625 INFO [upload_forcing] running in process 80726
2026-06-12 13:29:35,627 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-06-12 13:29:35,628 INFO [upload_forcing] publishing log messages to tcp://*:5564
2026-06-12 13:29:35,629 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-06-12 13:29:35,649 INFO [upload_forcing] running in process 80724
2026-06-12 13:29:35,650 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-06-12 13:29:35,652 INFO [upload_forcing] publishing log messages to tcp://*:5565
2026-06-12 13:29:35,653 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-06-12 13:29:35,654 INFO [upload_forcing] running in process 80727
2026-06-12 13:29:35,656 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-06-12 13:29:35,657 INFO [upload_forcing] publishing log messages to tcp://*:5566
2026-06-12 13:29:35,658 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-06-12 13:29:35,685 INFO [upload_forcing] running in process 80725
2026-06-12 13:29:35,686 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-06-12 13:29:35,687 INFO [upload_forcing] publishing log messages to tcp://*:5567
2026-06-12 13:29:35,688 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-06-12 13:29:36,292 INFO [upload_forcing] turbidity 2026-06-12 forcing files upload to optimum-hindcast completed
2026-06-12 13:29:36,296 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-06-12 13:29:36,783 INFO [upload_forcing] turbidity 2026-06-12 forcing files upload to arbutus.cloud-nowcast completed
2026-06-12 13:29:36,787 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-06-12 13:29:36,944 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2026-06-12'], host='localhost')
2026-06-12 13:29:37,023 INFO [make_ww3_current_file] running in process 10205
2026-06-12 13:29:37,024 INFO [make_ww3_current_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-06-12 13:29:37,025 INFO [make_ww3_current_file] publishing log messages to tcp://*:5571
2026-06-12 13:29:37,026 INFO [make_ww3_wind_file] running in process 10209
2026-06-12 13:29:37,027 INFO [make_ww3_wind_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-06-12 13:29:37,028 INFO [make_ww3_wind_file] publishing log messages to tcp://*:5570
2026-06-12 13:29:37,035 INFO [make_ww3_current_file] connected to skookum.eos.ubc.ca port 5555
2026-06-12 13:29:37,036 INFO [make_ww3_wind_file] connected to skookum.eos.ubc.ca port 5555
2026-06-12 13:29:37,037 INFO [make_ww3_current_file] Creating wwatch3 currents forcing file for 2026-06-12 forecast run
2026-06-12 13:29:37,038 INFO [make_ww3_wind_file] Creating wwatch3 wind forcing file for 2026-06-12 forecast run
2026-06-12 13:29:38,303 INFO [make_forcing_links] running in process 80758
2026-06-12 13:29:38,305 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-06-12 13:29:38,307 INFO [make_forcing_links] publishing log messages to tcp://*:5565
2026-06-12 13:29:38,308 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2026-06-12 13:29:39,438 INFO [make_forcing_links] nowcast-green 2026-06-12 forcing file links on arbutus.cloud-nowcast created
2026-06-12 13:29:39,442 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2026-06-12 13:29:39,600 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2026-06-12'], host='arbutus.cloud-nowcast')
2026-06-12 13:29:41,618 INFO [upload_forcing] turbidity 2026-06-12 forcing files upload to orcinus-nowcast-agrif completed
2026-06-12 13:29:41,623 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-06-12 13:29:42,443 INFO [run_NEMO] running in process 10560
2026-06-12 13:29:42,445 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-06-12 13:29:42,446 INFO [run_NEMO] publishing log messages to tcp://*:5556
2026-06-12 13:29:42,448 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2026-06-12 13:29:44,892 WARNING [run_NEMO] There are uncommitted changes in /nemoShare/MEOPAR/nowcast-sys/SS-run-sets
2026-06-12 13:29:45,069 INFO [upload_forcing] turbidity 2026-06-12 forcing files upload to robot.nibi completed
2026-06-12 13:29:45,073 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-06-12 13:29:45,938 INFO [run_NEMO] nowcast-green: launching /nemoShare/MEOPAR/nowcast-sys/runs/12jun26nowcast-green_2026-06-12T132942.534876-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2026-06-12 13:29:46,013 INFO [run_NEMO] nowcast-green NEMO run for 2026-06-12 on arbutus.cloud-nowcast started
2026-06-12 13:29:46,019 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2026-06-12 13:29:46,186 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'nowcast-green'], host='arbutus.cloud-nowcast')
2026-06-12 13:29:48,896 INFO [watch_NEMO] running in process 11119
2026-06-12 13:29:48,898 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-06-12 13:29:48,899 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2026-06-12 13:29:48,901 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2026-06-12 13:29:48,973 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2026-06-12 13:29:49,459 INFO [make_ww3_wind_file] wwatch3 wind forcing file created on arbutus.cloud-nowcast for 2026-06-12 forecast run
2026-06-12 13:29:49,462 INFO [manager] checklist updated with [WW3 wind forcing] items from make_ww3_wind_file worker
2026-06-12 13:30:43,572 INFO [download_results] forecast 2026-06-12 results files from arbutus.cloud-nowcast downloaded
2026-06-12 13:30:43,576 INFO [manager] checklist updated with [results files] items from download_results worker
2026-06-12 13:30:43,735 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['forecast', '--run-date', '2026-06-12'], host='localhost')
2026-06-12 13:30:45,943 INFO [make_CHS_currents_file] running in process 80814
2026-06-12 13:30:45,945 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-06-12 13:30:45,946 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5563
2026-06-12 13:30:45,947 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555
2026-06-12 13:31:27,816 INFO [make_CHS_currents_file] Made CHS currents file for 2026-06-12 for forecast
2026-06-12 13:31:27,819 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker
2026-06-12 13:31:27,977 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['nemo', 'forecast', '--run-date', '2026-06-12'], host='localhost')
2026-06-12 13:31:29,115 INFO [update_forecast_datasets] running in process 80853
2026-06-12 13:31:29,116 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-06-12 13:31:29,118 INFO [update_forecast_datasets] publishing log messages to tcp://*:5563
2026-06-12 13:31:29,119 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2026-06-12 13:31:29,120 INFO [update_forecast_datasets] updating nemo forecast directory for forecast/12jun26 run
2026-06-12 13:31:29,409 INFO [update_forecast_datasets] updated nemo forecast directory for forecast/12jun26 run: /results/SalishSea/rolling-forecasts/nemo
2026-06-12 13:31:29,410 INFO [update_forecast_datasets] nemo 2026-06-12 forecast rolling forecast datasets updated
2026-06-12 13:31:29,413 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2026-06-12 13:31:29,541 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nemo-forecast'], host='localhost')
2026-06-12 13:31:29,547 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'forecast', 'publish', '--run-date', '2026-06-12'], host='localhost')
2026-06-12 13:31:29,551 INFO [manager] launching NextWorker(module='nowcast.workers.make_surface_current_tiles', args=['forecast', '--run-date', '2026-06-12'], host='localhost')
2026-06-12 13:31:30,710 INFO [ping_erddap] running in process 80861
2026-06-12 13:31:30,711 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-06-12 13:31:30,712 INFO [ping_erddap] publishing log messages to tcp://*:5563
2026-06-12 13:31:30,713 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2026-06-12 13:31:30,723 INFO [ping_erddap] nemo-forecast ERDDAP dataset flag file(s) created
2026-06-12 13:31:30,725 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2026-06-12 13:31:31,866 INFO [make_surface_current_tiles] running in process 80864
2026-06-12 13:31:31,868 INFO [make_surface_current_tiles] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-06-12 13:31:31,868 INFO [make_surface_current_tiles] publishing log messages to tcp://*:5563
2026-06-12 13:31:31,869 INFO [make_surface_current_tiles] connected to skookum.eos.ubc.ca port 5555
2026-06-12 13:31:31,871 INFO [make_surface_current_tiles] starting rendering of tiles for 2026-06-12 forecast in 6 processes into /results/nowcast-sys/figures/surface_currents/forecast/12jun26
2026-06-12 13:31:32,988 INFO [make_plots] running in process 80863
2026-06-12 13:31:32,992 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-06-12 13:31:32,994 INFO [make_plots] publishing log messages to tcp://*:5564
2026-06-12 13:31:32,996 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2026-06-12 13:31:33,448 INFO [make_plots] preparing render list for 2026-06-12 NEMO forecast publish figures
2026-06-12 13:31:33,680 INFO [make_plots] added Website_thumbnail figure to 2026-06-12 NEMO forecast publish render list
2026-06-12 13:31:33,681 INFO [make_plots] added Threshold_website figure to 2026-06-12 NEMO forecast publish render list
2026-06-12 13:31:33,681 INFO [make_plots] added PA_tidal_predictions figure to 2026-06-12 NEMO forecast publish render list
2026-06-12 13:31:33,682 INFO [make_plots] added Boundary Bay figure to 2026-06-12 NEMO forecast publish render list
2026-06-12 13:31:33,683 INFO [make_plots] added Campbell River figure to 2026-06-12 NEMO forecast publish render list
2026-06-12 13:31:33,683 INFO [make_plots] added Cherry Point figure to 2026-06-12 NEMO forecast publish render list
2026-06-12 13:31:33,684 INFO [make_plots] added Friday Harbor figure to 2026-06-12 NEMO forecast publish render list
2026-06-12 13:31:33,685 INFO [make_plots] added Halfmoon Bay figure to 2026-06-12 NEMO forecast publish render list
2026-06-12 13:31:33,686 INFO [make_plots] added Nanaimo figure to 2026-06-12 NEMO forecast publish render list
2026-06-12 13:31:33,686 INFO [make_plots] added Neah Bay figure to 2026-06-12 NEMO forecast publish render list
2026-06-12 13:31:33,687 INFO [make_plots] added New Westminster figure to 2026-06-12 NEMO forecast publish render list
2026-06-12 13:31:33,688 INFO [make_plots] added Patricia Bay figure to 2026-06-12 NEMO forecast publish render list
2026-06-12 13:31:33,688 INFO [make_plots] added Point Atkinson figure to 2026-06-12 NEMO forecast publish render list
2026-06-12 13:31:33,689 INFO [make_plots] added Port Renfrew figure to 2026-06-12 NEMO forecast publish render list
2026-06-12 13:31:33,690 INFO [make_plots] added Sand Heads figure to 2026-06-12 NEMO forecast publish render list
2026-06-12 13:31:33,690 INFO [make_plots] added Sandy Cove figure to 2026-06-12 NEMO forecast publish render list
2026-06-12 13:31:33,691 INFO [make_plots] added Squamish figure to 2026-06-12 NEMO forecast publish render list
2026-06-12 13:31:33,692 INFO [make_plots] added Victoria figure to 2026-06-12 NEMO forecast publish render list
2026-06-12 13:31:33,693 INFO [make_plots] added Woodwards Landing figure to 2026-06-12 NEMO forecast publish render list
2026-06-12 13:31:33,694 INFO [make_plots] starting to render nemo forecast publish 12jun26 figures
2026-06-12 13:32:18,482 INFO [manager] checklist updated with [WW3 currents forcing] items from make_ww3_current_file worker
2026-06-12 13:32:18,491 INFO [make_ww3_current_file] wwatch3 currents forcing file created on arbutus.cloud-nowcast for 2026-06-12 forecast run
2026-06-12 13:32:18,784 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2026-06-12'], host='arbutus.cloud-nowcast')
2026-06-12 13:32:21,701 INFO [run_ww3] running in process 11247
2026-06-12 13:32:21,702 INFO [run_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-06-12 13:32:21,703 INFO [run_ww3] publishing log messages to tcp://*:5572
2026-06-12 13:32:21,704 INFO [run_ww3] connected to skookum.eos.ubc.ca port 5555
2026-06-12 13:32:21,706 INFO [run_ww3] Created run directory /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/nowcast_2026-06-12T133221.698913-0700
2026-06-12 13:32:21,708 INFO [run_ww3] nowcast: launching /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/nowcast_2026-06-12T133221.698913-0700/SoGWW3.sh on arbutus.cloud-nowcast
2026-06-12 13:32:21,722 INFO [run_ww3] nowcast WaveWatch3 run for 2026-06-12 on arbutus.cloud-nowcast started
2026-06-12 13:32:21,725 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker
2026-06-12 13:32:22,045 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'nowcast'], host='arbutus.cloud-nowcast')
2026-06-12 13:32:24,085 INFO [watch_ww3] running in process 11335
2026-06-12 13:32:24,085 INFO [watch_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-06-12 13:32:24,086 INFO [watch_ww3] publishing log messages to tcp://*:5573
2026-06-12 13:32:24,087 INFO [watch_ww3] connected to skookum.eos.ubc.ca port 5555
2026-06-12 13:32:24,119 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2026-06-12 13:34:48,974 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19291135 = 2026-06-12 01:56:00 UTC, 8.1% complete
2026-06-12 13:37:24,119 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 90, 20.8% complete
2026-06-12 13:39:11,988 WARNING [make_plots] invalid station number for Boundary Bay station: None
2026-06-12 13:39:11,989 WARNING [make_plots] can't resolve a valid CHS station code for Boundary Bay
2026-06-12 13:39:33,827 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb996/data for station 08074 Campbell River from 2026-06-07 00:00:00Z to 2026-06-14 11:55:00Z
2026-06-12 13:39:48,974 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19291327 = 2026-06-12 04:04:00 UTC, 17.0% complete
2026-06-12 13:40:39,955 WARNING [make_plots] invalid station number for Halfmoon Bay station: None
2026-06-12 13:40:39,956 WARNING [make_plots] can't resolve a valid CHS station code for Halfmoon Bay
2026-06-12 13:41:01,058 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb96d/data for station 07917 Nanaimo from 2026-06-07 00:00:00Z to 2026-06-14 11:55:00Z
2026-06-12 13:41:42,735 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e13d0f4a073c4bbf93/data for station 07654 New Westminster from 2026-06-07 00:00:00Z to 2026-06-14 11:55:00Z
2026-06-12 13:42:00,218 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd2d/data for station 07277 Patricia Bay from 2026-06-07 00:00:00Z to 2026-06-14 11:55:00Z
2026-06-12 13:42:06,832 INFO [make_surface_current_tiles] starting PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast/12jun26
2026-06-12 13:42:17,102 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94c/data for station 07795 Point Atkinson from 2026-06-07 00:00:00Z to 2026-06-14 11:55:00Z
2026-06-12 13:42:24,120 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 192, 44.4% complete
2026-06-12 13:42:33,721 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e23d0f4a073c4bc060/data for station 08525 Port Renfrew from 2026-06-07 00:00:00Z to 2026-06-14 11:55:00Z
2026-06-12 13:42:49,735 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb93c/data for station 07594 Sand Heads from 2026-06-07 00:00:00Z to 2026-06-14 11:55:00Z
2026-06-12 13:43:06,062 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc43b/data for station 07786 Sandy Cove from 2026-06-07 00:00:00Z to 2026-06-14 11:55:00Z
2026-06-12 13:43:23,151 WARNING [make_plots] invalid station number for Squamish station: None
2026-06-12 13:43:23,152 WARNING [make_plots] can't resolve a valid CHS station code for Squamish
2026-06-12 13:43:39,107 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd1e/data for station 07120 Victoria from 2026-06-07 00:00:00Z to 2026-06-14 11:55:00Z
2026-06-12 13:43:55,562 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc42d/data for station 07610 Woodwards Landing from 2026-06-07 00:00:00Z to 2026-06-14 11:55:00Z
2026-06-12 13:43:55,965 INFO [make_plots] no obs water level data available from 5cebf1e43d0f4a073c4bc42d during 2026-06-07 00:00:00Z to 2026-06-14 11:55:00Z
2026-06-12 13:44:11,919 INFO [make_plots] finished rendering nemo forecast publish 12jun26 figures
2026-06-12 13:44:11,922 INFO [make_plots] nemo publish plots for 2026-06-12 forecast completed
2026-06-12 13:44:11,931 INFO [manager] checklist updated with [plots] items from make_plots worker
2026-06-12 13:44:12,091 INFO [manager] launching NextWorker(module='nowcast.workers.make_feeds', args=['forecast', '--run-date', '2026-06-12'], host='localhost')
2026-06-12 13:44:15,062 INFO [make_feeds] running in process 85662
2026-06-12 13:44:15,225 INFO [make_feeds] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-06-12 13:44:15,229 INFO [make_feeds] publishing log messages to tcp://*:5564
2026-06-12 13:44:15,229 INFO [make_feeds] connected to skookum.eos.ubc.ca port 5555
2026-06-12 13:44:28,916 INFO [make_feeds] ATOM feeds for 2026-06-12 forecast run completed
2026-06-12 13:44:28,919 INFO [manager] checklist updated with [storm surge alert feeds] items from make_feeds worker
2026-06-12 13:44:48,975 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19291520 = 2026-06-12 06:12:40 UTC, 25.9% complete
2026-06-12 13:46:46,223 INFO [make_surface_current_tiles] finished PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast/12jun26
2026-06-12 13:46:46,243 INFO [make_surface_current_tiles] finished rendering of tiles for 2026-06-12 forecast into /results/nowcast-sys/figures/surface_currents/forecast/12jun26
2026-06-12 13:46:46,244 INFO [make_surface_current_tiles] surface current tile figures for 2026-06-12 forecast completed
2026-06-12 13:46:46,709 INFO [manager] checklist updated with [surface current tiles] items from make_surface_current_tiles worker
2026-06-12 13:47:24,121 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 297, 68.8% complete
2026-06-12 13:49:48,976 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19291714 = 2026-06-12 08:22:00 UTC, 34.9% complete
2026-06-12 13:52:24,125 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 405, 93.8% complete
2026-06-12 13:54:48,977 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19291907 = 2026-06-12 10:30:40 UTC, 43.8% complete
2026-06-12 13:57:24,122 INFO [watch_ww3] nowcast WWATCH3 run on arbutus.cloud-nowcast completed
2026-06-12 13:57:24,126 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2026-06-12 13:57:24,602 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2026-06-12'], host='localhost')
2026-06-12 13:57:24,609 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2026-06-12'], host='arbutus.cloud-nowcast')
2026-06-12 13:57:25,810 INFO [download_wwatch3_results] running in process 85944
2026-06-12 13:57:25,812 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-06-12 13:57:25,813 INFO [download_wwatch3_results] publishing log messages to tcp://*:5563
2026-06-12 13:57:25,814 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2026-06-12 13:57:26,608 INFO [run_ww3] running in process 11505
2026-06-12 13:57:26,609 INFO [run_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-06-12 13:57:26,610 INFO [run_ww3] publishing log messages to tcp://*:5572
2026-06-12 13:57:26,611 INFO [run_ww3] connected to skookum.eos.ubc.ca port 5555
2026-06-12 13:57:26,615 INFO [run_ww3] Created run directory /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast_2026-06-12T135726.607659-0700
2026-06-12 13:57:26,616 INFO [run_ww3] forecast: launching /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast_2026-06-12T135726.607659-0700/SoGWW3.sh on arbutus.cloud-nowcast
2026-06-12 13:57:26,630 INFO [run_ww3] forecast WaveWatch3 run for 2026-06-12 on arbutus.cloud-nowcast started
2026-06-12 13:57:26,634 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker
2026-06-12 13:57:26,947 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'forecast'], host='arbutus.cloud-nowcast')
2026-06-12 13:57:28,984 INFO [watch_ww3] running in process 11660
2026-06-12 13:57:28,986 INFO [watch_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-06-12 13:57:28,987 INFO [watch_ww3] publishing log messages to tcp://*:5573
2026-06-12 13:57:28,988 INFO [watch_ww3] connected to skookum.eos.ubc.ca port 5555
2026-06-12 13:57:29,021 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2026-06-12 13:57:57,912 INFO [download_wwatch3_results] nowcast 2026-06-12 results files from arbutus.cloud-nowcast downloaded
2026-06-12 13:57:57,915 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2026-06-12 13:59:48,978 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19292099 = 2026-06-12 12:38:40 UTC, 52.7% complete
2026-06-12 14:02:29,022 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 93, 14.4% complete
2026-06-12 14:04:48,980 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19292292 = 2026-06-12 14:47:20 UTC, 61.6% complete
2026-06-12 14:07:29,024 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 192, 29.6% complete
2026-06-12 14:09:48,980 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19292485 = 2026-06-12 16:56:00 UTC, 70.6% complete
2026-06-12 14:12:29,024 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 300, 46.3% complete
2026-06-12 14:14:48,981 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19292676 = 2026-06-12 19:03:20 UTC, 79.4% complete
2026-06-12 14:17:29,032 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 399, 61.6% complete
2026-06-12 14:19:48,984 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19292869 = 2026-06-12 21:12:00 UTC, 88.4% complete
2026-06-12 14:22:29,027 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 504, 77.8% complete
2026-06-12 14:24:48,983 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19293062 = 2026-06-12 23:20:40 UTC, 97.3% complete
2026-06-12 14:27:29,034 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 606, 93.5% complete
2026-06-12 14:29:48,985 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19293120 = 2026-06-12 23:59:20 UTC, 100.0% complete
2026-06-12 14:32:29,030 INFO [watch_ww3] forecast WWATCH3 run on arbutus.cloud-nowcast completed
2026-06-12 14:32:29,033 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2026-06-12 14:32:29,509 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2026-06-12'], host='localhost')
2026-06-12 14:32:30,776 INFO [download_wwatch3_results] running in process 86835
2026-06-12 14:32:30,777 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-06-12 14:32:30,779 INFO [download_wwatch3_results] publishing log messages to tcp://*:5563
2026-06-12 14:32:30,780 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2026-06-12 14:33:08,830 INFO [download_wwatch3_results] forecast 2026-06-12 results files from arbutus.cloud-nowcast downloaded
2026-06-12 14:33:08,834 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2026-06-12 14:33:09,145 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['wwatch3', 'forecast', '--run-date', '2026-06-12'], host='localhost')
2026-06-12 14:33:10,271 INFO [update_forecast_datasets] running in process 86852
2026-06-12 14:33:10,272 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-06-12 14:33:10,274 INFO [update_forecast_datasets] publishing log messages to tcp://*:5563
2026-06-12 14:33:10,275 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2026-06-12 14:33:10,276 INFO [update_forecast_datasets] updating wwatch3 most_recent_forecast directory from forecast/12jun26 run
2026-06-12 14:33:10,281 INFO [update_forecast_datasets] updated wwatch3 most_recent_forecast directory from forecast/12jun26 run
2026-06-12 14:33:10,282 INFO [update_forecast_datasets] updating wwatch3 forecast directory for forecast/12jun26 run
2026-06-12 14:33:10,326 INFO [update_forecast_datasets] updated wwatch3 forecast directory for forecast/12jun26 run: /results/SalishSea/rolling-forecasts/wwatch3
2026-06-12 14:33:10,328 INFO [update_forecast_datasets] wwatch3 2026-06-12 forecast rolling forecast datasets updated
2026-06-12 14:33:10,330 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2026-06-12 14:33:10,641 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['wwatch3-forecast'], host='localhost')
2026-06-12 14:33:11,717 INFO [ping_erddap] running in process 86858
2026-06-12 14:33:11,718 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-06-12 14:33:11,720 INFO [ping_erddap] publishing log messages to tcp://*:5563
2026-06-12 14:33:11,721 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2026-06-12 14:33:11,724 INFO [ping_erddap] wwatch3-forecast ERDDAP dataset flag file(s) created
2026-06-12 14:33:11,726 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2026-06-12 14:33:12,022 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['wwatch3', 'forecast', 'publish', '--run-date', '2026-06-12'], host='localhost')
2026-06-12 14:33:15,256 INFO [make_plots] running in process 86863
2026-06-12 14:33:15,259 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-06-12 14:33:15,260 INFO [make_plots] publishing log messages to tcp://*:5563
2026-06-12 14:33:15,261 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2026-06-12 14:33:15,262 INFO [make_plots] preparing render list for 2026-06-12 WaveWatch3 forecast publish figures
2026-06-12 14:33:15,264 INFO [make_plots] added Halibut Bank figure to 2026-06-12 WaveWatch3 forecast publish render list
2026-06-12 14:33:15,265 INFO [make_plots] added Sentry Shoal figure to 2026-06-12 WaveWatch3 forecast publish render list
2026-06-12 14:33:15,266 INFO [make_plots] starting to render wwatch3 forecast publish 12jun26 figures
2026-06-12 14:33:15,808 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46146 Halibut Bank
2026-06-12 14:33:21,959 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46131 Sentry Shoal
2026-06-12 14:33:26,931 INFO [make_plots] finished rendering wwatch3 forecast publish 12jun26 figures
2026-06-12 14:33:26,933 INFO [make_plots] wwatch3 publish plots for 2026-06-12 forecast completed
2026-06-12 14:33:26,936 INFO [manager] checklist updated with [plots] items from make_plots worker
2026-06-12 14:34:48,993 INFO [watch_NEMO] nowcast-green NEMO run on arbutus.cloud-nowcast completed
2026-06-12 14:34:48,996 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2026-06-12 14:34:49,473 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2026-06-12'], host='localhost')
2026-06-12 14:34:50,749 INFO [download_results] running in process 86925
2026-06-12 14:34:50,750 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-06-12 14:34:50,751 INFO [download_results] publishing log messages to tcp://*:5563
2026-06-12 14:34:50,752 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2026-06-12 14:34:50,753 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/nowcast-green/12jun26 to /results2/SalishSea/nowcast-green.202111
2026-06-12 14:37:18,138 INFO [download_results] nowcast-green 2026-06-12 results files from arbutus.cloud-nowcast downloaded
2026-06-12 14:37:18,144 INFO [manager] checklist updated with [results files] items from download_results worker
2026-06-12 14:37:18,465 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast-green', 'research', '--run-date', '2026-06-12'], host='localhost')
2026-06-12 14:37:18,471 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nowcast-green'], host='localhost')
2026-06-12 14:37:18,476 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'chemistry', '--run-date', '2026-06-12'], host='localhost')
2026-06-12 14:37:18,482 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'biology', '--run-date', '2026-06-12'], host='localhost')
2026-06-12 14:37:18,486 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'physics', '--run-date', '2026-06-12'], host='localhost')
2026-06-12 14:37:20,008 INFO [ping_erddap] running in process 87025
2026-06-12 14:37:20,009 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-06-12 14:37:20,010 INFO [ping_erddap] publishing log messages to tcp://*:5563
2026-06-12 14:37:20,011 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2026-06-12 14:37:20,017 INFO [ping_erddap] nowcast-green ERDDAP dataset flag file(s) created
2026-06-12 14:37:20,019 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2026-06-12 14:37:21,066 INFO [make_averaged_dataset] running in process 87027
2026-06-12 14:37:21,068 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-06-12 14:37:21,069 INFO [make_averaged_dataset] publishing log messages to tcp://*:5563
2026-06-12 14:37:21,070 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555
2026-06-12 14:37:21,071 INFO [make_averaged_dataset] creating day-averaged dataset for 12-Jun-2026 biology
2026-06-12 14:37:21,072 INFO [make_averaged_dataset] running in process 87026
2026-06-12 14:37:21,074 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-06-12 14:37:21,075 INFO [make_averaged_dataset] publishing log messages to tcp://*:5564
2026-06-12 14:37:21,076 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555
2026-06-12 14:37:21,077 INFO [make_averaged_dataset] creating day-averaged dataset for 12-Jun-2026 chemistry
2026-06-12 14:37:21,079 INFO [make_averaged_dataset] running in process 87028
2026-06-12 14:37:21,080 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-06-12 14:37:21,081 INFO [make_averaged_dataset] publishing log messages to tcp://*:5565
2026-06-12 14:37:21,082 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555
2026-06-12 14:37:21,083 INFO [make_averaged_dataset] creating day-averaged dataset for 12-Jun-2026 physics
2026-06-12 14:37:22,001 INFO [make_plots] running in process 87023
2026-06-12 14:37:22,004 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-06-12 14:37:22,005 INFO [make_plots] publishing log messages to tcp://*:5566
2026-06-12 14:37:22,007 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2026-06-12 14:37:22,365 INFO [make_plots] preparing render list for 2026-06-12 NEMO nowcast-green research figures
2026-06-12 14:37:24,152 INFO [make_plots] added temperature_thalweg_and_surface figures to 2026-06-12 NEMO nowcast-green research render list
2026-06-12 14:37:24,930 INFO [make_plots] added nitrate_thalweg_and_surface figures to 2026-06-12 NEMO nowcast-green research render list
2026-06-12 14:37:25,680 INFO [make_plots] added ammonium_thalweg_and_surface figures to 2026-06-12 NEMO nowcast-green research render list
2026-06-12 14:37:26,421 INFO [make_plots] added silicon_thalweg_and_surface figures to 2026-06-12 NEMO nowcast-green research render list
2026-06-12 14:37:27,188 INFO [make_plots] added dissolved_organic_nitrogen_thalweg_and_surface figures to 2026-06-12 NEMO nowcast-green research render list
2026-06-12 14:37:27,962 INFO [make_plots] added particulate_organic_nitrogen_thalweg_and_surface figures to 2026-06-12 NEMO nowcast-green research render list
2026-06-12 14:37:28,701 INFO [make_plots] added biogenic_silicon_thalweg_and_surface figures to 2026-06-12 NEMO nowcast-green research render list
2026-06-12 14:37:29,538 INFO [make_plots] added diatoms_thalweg_and_surface figures to 2026-06-12 NEMO nowcast-green research render list
2026-06-12 14:37:30,380 INFO [make_plots] added flagellates_thalweg_and_surface figures to 2026-06-12 NEMO nowcast-green research render list
2026-06-12 14:37:31,329 INFO [make_plots] added microzooplankton_thalweg_and_surface figures to 2026-06-12 NEMO nowcast-green research render list
2026-06-12 14:37:32,234 INFO [make_plots] added mesozooplankton_thalweg_and_surface figures to 2026-06-12 NEMO nowcast-green research render list
2026-06-12 14:37:33,046 INFO [make_plots] added turbidity_thalweg_and_surface figures to 2026-06-12 NEMO nowcast-green research render list
2026-06-12 14:37:34,173 INFO [make_plots] added temperature_salinity_timeseries figure to 2026-06-12 NEMO nowcast-green research render list
2026-06-12 14:37:34,174 INFO [make_plots] added nitrate_diatoms_timeseries figure to 2026-06-12 NEMO nowcast-green research render list
2026-06-12 14:37:34,175 INFO [make_plots] added diatoms_flagellates_timeseries figure to 2026-06-12 NEMO nowcast-green research render list
2026-06-12 14:37:34,175 INFO [make_plots] added z1_z2_zooplankton_timeseries figure to 2026-06-12 NEMO nowcast-green research render list
2026-06-12 14:37:34,176 INFO [make_plots] starting to render nemo nowcast-green research 12jun26 figures
2026-06-12 14:37:52,173 INFO [make_averaged_dataset] day-averaged dataset for 12-Jun-2026 physics created
2026-06-12 14:37:52,177 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker
2026-06-12 14:38:16,156 INFO [make_averaged_dataset] day-averaged dataset for 12-Jun-2026 chemistry created
2026-06-12 14:38:16,159 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker
2026-06-12 14:38:28,459 INFO [make_averaged_dataset] day-averaged dataset for 12-Jun-2026 biology created
2026-06-12 14:38:28,462 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker
2026-06-12 14:49:30,091 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/18/ to /results/forcing/atmospheric/continental2.5/GRIB/20260612/18/
2026-06-12 14:49:30,092 INFO [collect_weather] 2026-06-12 2.5km weather forecast 18 collection complete
2026-06-12 14:49:30,096 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2026-06-12 14:49:30,582 INFO [manager] launching NextWorker(module='nowcast.workers.download_weather', args=['00', '1km'], host='localhost')
2026-06-12 14:49:30,591 INFO [manager] launching NextWorker(module='nowcast.workers.download_weather', args=['12', '1km'], host='localhost')
2026-06-12 14:49:30,597 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['00', '2.5km'], host='localhost')
2026-06-12 14:49:30,601 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['00', '--fcst-date', '2026-06-13'], host='localhost')
2026-06-12 14:49:31,023 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20260612/18/
2026-06-12 14:49:31,024 INFO [crop_gribs] 2026-06-12 18 GRIBs cropping complete
2026-06-12 14:49:31,027 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2026-06-12 14:49:32,800 INFO [download_weather] running in process 87420
2026-06-12 14:49:32,803 INFO [download_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-06-12 14:49:32,812 INFO [download_weather] publishing log messages to tcp://*:5562
2026-06-12 14:49:32,822 INFO [download_weather] connected to skookum.eos.ubc.ca port 5555
2026-06-12 14:49:32,840 INFO [download_weather] downloading 12 1 km forecast GRIB2 files for 20260612
2026-06-12 14:49:32,876 INFO [download_weather] running in process 87418
2026-06-12 14:49:32,882 INFO [collect_weather] running in process 87421
2026-06-12 14:49:32,884 INFO [download_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-06-12 14:49:32,885 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-06-12 14:49:32,886 INFO [download_weather] publishing log messages to tcp://*:5563
2026-06-12 14:49:32,887 INFO [collect_weather] publishing log messages to tcp://*:5564
2026-06-12 14:49:32,888 INFO [download_weather] connected to skookum.eos.ubc.ca port 5555
2026-06-12 14:49:32,889 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2026-06-12 14:49:32,890 INFO [download_weather] downloading 00 1 km forecast GRIB2 files for 20260612
2026-06-12 14:49:32,894 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/00/
2026-06-12 14:49:34,103 INFO [crop_gribs] running in process 87422
2026-06-12 14:49:34,105 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-06-12 14:49:34,106 INFO [crop_gribs] publishing log messages to tcp://*:5565
2026-06-12 14:49:34,108 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2026-06-12 14:49:34,109 INFO [crop_gribs] cropping 2026-06-13 ECCC HRDPS 2.5km continental 00Z GRIB files to SalishSeaCast subdomain
2026-06-12 14:49:34,119 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20260613/00/
2026-06-12 14:53:08,050 INFO [download_weather] 2026-06-12 1km weather forecast 12 downloads complete
2026-06-12 14:53:08,095 INFO [manager] checklist updated with [weather forecast] items from download_weather worker
2026-06-12 14:53:08,606 INFO [download_weather] 2026-06-12 1km weather forecast 00 downloads complete
2026-06-12 14:53:08,609 INFO [manager] checklist updated with [weather forecast] items from download_weather worker
2026-06-12 15:39:55,320 INFO [make_plots] finished rendering nemo nowcast-green research 12jun26 figures
2026-06-12 15:39:55,331 INFO [make_plots] nemo research plots for 2026-06-12 nowcast-green completed
2026-06-12 15:39:55,390 INFO [manager] checklist updated with [plots] items from make_plots worker
2026-06-12 20:49:06,519 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/00/ to /results/forcing/atmospheric/continental2.5/GRIB/20260613/00/
2026-06-12 20:49:06,521 INFO [collect_weather] 2026-06-13 2.5km weather forecast 00 collection complete
2026-06-12 20:49:06,523 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2026-06-12 20:49:07,081 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['06', '2.5km'], host='localhost')
2026-06-12 20:49:07,089 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['06', '--fcst-date', '2026-06-13'], host='localhost')
2026-06-12 20:49:07,735 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20260613/00/
2026-06-12 20:49:07,736 INFO [crop_gribs] 2026-06-13 00 GRIBs cropping complete
2026-06-12 20:49:07,738 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2026-06-12 20:49:08,674 INFO [collect_weather] running in process 96080
2026-06-12 20:49:08,675 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-06-12 20:49:08,676 INFO [collect_weather] publishing log messages to tcp://*:5562
2026-06-12 20:49:08,677 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2026-06-12 20:49:08,694 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/06/
2026-06-12 20:49:09,953 INFO [crop_gribs] running in process 96082
2026-06-12 20:49:09,954 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-06-12 20:49:09,956 INFO [crop_gribs] publishing log messages to tcp://*:5563
2026-06-12 20:49:09,957 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2026-06-12 20:49:09,959 INFO [crop_gribs] cropping 2026-06-13 ECCC HRDPS 2.5km continental 06Z GRIB files to SalishSeaCast subdomain
2026-06-12 20:49:09,969 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20260613/06/
2026-06-13 02:39:27,275 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/06/ to /results/forcing/atmospheric/continental2.5/GRIB/20260613/06/
2026-06-13 02:39:27,276 INFO [collect_weather] 2026-06-13 2.5km weather forecast 06 collection complete
2026-06-13 02:39:27,279 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2026-06-13 02:39:27,756 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Capilano', '--data-date', '2026-06-12'], host='localhost')
2026-06-13 02:39:27,763 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'ChilliwackVedder', '--data-date', '2026-06-12'], host='localhost')
2026-06-13 02:39:27,769 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'ClowhomClowhomLake', '--data-date', '2026-06-12'], host='localhost')
2026-06-13 02:39:27,773 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Englishman', '--data-date', '2026-06-12'], host='localhost')
2026-06-13 02:39:27,779 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Fraser', '--data-date', '2026-06-12'], host='localhost')
2026-06-13 02:39:27,783 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'HomathkoMouth', '--data-date', '2026-06-12'], host='localhost')
2026-06-13 02:39:27,793 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'NicomeklLangley', '--data-date', '2026-06-12'], host='localhost')
2026-06-13 02:39:27,798 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'RobertsCreek', '--data-date', '2026-06-12'], host='localhost')
2026-06-13 02:39:27,806 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SalmonSayward', '--data-date', '2026-06-12'], host='localhost')
2026-06-13 02:39:27,834 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SanJuanPortRenfrew', '--data-date', '2026-06-12'], host='localhost')
2026-06-13 02:39:27,845 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SquamishBrackendale', '--data-date', '2026-06-12'], host='localhost')
2026-06-13 02:39:27,875 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaScotty', '--data-date', '2026-06-12'], host='localhost')
2026-06-13 02:39:27,884 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaBypass', '--data-date', '2026-06-12'], host='localhost')
2026-06-13 02:39:27,899 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaDiversion', '--data-date', '2026-06-12'], host='localhost')
2026-06-13 02:39:27,914 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ctd', args=['SCVIP'], host='localhost')
2026-06-13 02:39:27,928 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ctd', args=['SEVIP'], host='localhost')
2026-06-13 02:39:27,942 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ferry', args=['TWDP'], host='localhost')
2026-06-13 02:39:27,969 INFO [manager] launching NextWorker(module='nowcast.workers.get_vfpa_hadcp', args=['--data-date', '2026-06-12'], host='localhost')
2026-06-13 02:39:27,987 INFO [manager] launching NextWorker(module='nowcast.workers.collect_NeahBay_ssh', args=['00'], host='localhost')
2026-06-13 02:39:28,000 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['12', '2.5km'], host='localhost')
2026-06-13 02:39:28,017 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['12'], host='localhost')
2026-06-13 02:39:31,073 INFO [collect_weather] running in process 104502
2026-06-13 02:39:31,074 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-06-13 02:39:31,075 INFO [collect_weather] publishing log messages to tcp://*:5562
2026-06-13 02:39:31,076 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2026-06-13 02:39:31,092 INFO [collect_NeahBay_ssh] running in process 104501
2026-06-13 02:39:31,094 INFO [collect_NeahBay_ssh] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-06-13 02:39:31,096 INFO [collect_NeahBay_ssh] publishing log messages to tcp://*:5564
2026-06-13 02:39:31,097 INFO [collect_NeahBay_ssh] connected to skookum.eos.ubc.ca port 5555
2026-06-13 02:39:31,098 INFO [collect_NeahBay_ssh] collecting Neah Bay ssh 00Z obs/forecast for 2026-06-13
2026-06-13 02:39:31,101 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/12/
2026-06-13 02:39:31,941 INFO [collect_river_data] running in process 104487
2026-06-13 02:39:31,944 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-06-13 02:39:31,944 INFO [collect_river_data] publishing log messages to tcp://*:5565
2026-06-13 02:39:31,945 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-06-13 02:39:31,946 INFO [collect_river_data] Collecting ECCC HomathkoMouth river data for 2026-06-12
2026-06-13 02:39:31,974 INFO [collect_river_data] running in process 104494
2026-06-13 02:39:31,975 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-06-13 02:39:31,976 INFO [collect_river_data] publishing log messages to tcp://*:5566
2026-06-13 02:39:31,977 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-06-13 02:39:31,978 INFO [collect_river_data] Collecting ECCC TheodosiaBypass river data for 2026-06-12
2026-06-13 02:39:31,983 INFO [collect_river_data] running in process 104481
2026-06-13 02:39:31,983 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-06-13 02:39:31,985 INFO [collect_river_data] publishing log messages to tcp://*:5567
2026-06-13 02:39:31,986 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-06-13 02:39:31,987 INFO [collect_river_data] Collecting ECCC Capilano river data for 2026-06-12
2026-06-13 02:39:32,007 INFO [collect_river_data] running in process 104483
2026-06-13 02:39:32,008 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-06-13 02:39:32,009 INFO [collect_river_data] publishing log messages to tcp://*:5568
2026-06-13 02:39:32,010 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-06-13 02:39:32,010 INFO [collect_river_data] Collecting ECCC ChilliwackVedder river data for 2026-06-12
2026-06-13 02:39:32,017 INFO [collect_river_data] Appended ECCC TheodosiaBypass river average discharge for 2026-06-12 to: /results/forcing/rivers/observations/Theodosia_Bypass_flow
2026-06-13 02:39:32,018 INFO [collect_river_data] ECCC TheodosiaBypass river data collection for 2026-06-12 completed
2026-06-13 02:39:32,019 INFO [collect_river_data] running in process 104488
2026-06-13 02:39:32,020 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-06-13 02:39:32,021 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-06-13 02:39:32,023 INFO [collect_river_data] publishing log messages to tcp://*:5569
2026-06-13 02:39:32,024 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-06-13 02:39:32,025 INFO [collect_river_data] Collecting ECCC NicomeklLangley river data for 2026-06-12
2026-06-13 02:39:32,026 INFO [collect_river_data] running in process 104484
2026-06-13 02:39:32,027 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-06-13 02:39:32,027 INFO [collect_river_data] publishing log messages to tcp://*:5570
2026-06-13 02:39:32,029 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-06-13 02:39:32,031 INFO [collect_river_data] Collecting ECCC ClowhomClowhomLake river data for 2026-06-12
2026-06-13 02:39:32,033 INFO [collect_river_data] running in process 104486
2026-06-13 02:39:32,033 INFO [collect_river_data] Appended ECCC HomathkoMouth river average discharge for 2026-06-12 to: /results/forcing/rivers/observations/Homathko_Mouth_flow
2026-06-13 02:39:32,034 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-06-13 02:39:32,035 INFO [collect_river_data] ECCC HomathkoMouth river data collection for 2026-06-12 completed
2026-06-13 02:39:32,036 INFO [collect_river_data] publishing log messages to tcp://*:5571
2026-06-13 02:39:32,037 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-06-13 02:39:32,038 INFO [collect_river_data] Collecting ECCC Fraser river data for 2026-06-12
2026-06-13 02:39:32,053 INFO [collect_river_data] Appended ECCC Capilano river average discharge for 2026-06-12 to: /opp/observations/rivers/Capilano/Caplilano_08GA010_day_avg_flow
2026-06-13 02:39:32,054 INFO [collect_river_data] ECCC Capilano river data collection for 2026-06-12 completed
2026-06-13 02:39:32,060 INFO [collect_river_data] Appended ECCC NicomeklLangley river average discharge for 2026-06-12 to: /results/forcing/rivers/observations/Nicomekl_Langley_flow
2026-06-13 02:39:32,061 INFO [collect_river_data] ECCC NicomeklLangley river data collection for 2026-06-12 completed
2026-06-13 02:39:32,064 INFO [collect_river_data] Appended ECCC Fraser river average discharge for 2026-06-12 to: /data/dlatorne/SOG-projects/SOG-forcing/ECget/Fraser_flow
2026-06-13 02:39:32,070 INFO [collect_river_data] ECCC Fraser river data collection for 2026-06-12 completed
2026-06-13 02:39:32,073 INFO [collect_river_data] running in process 104485
2026-06-13 02:39:32,075 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-06-13 02:39:32,076 INFO [collect_river_data] Appended ECCC ChilliwackVedder river average discharge for 2026-06-12 to: /results/forcing/rivers/observations/Chilliwack_Vedder_flow
2026-06-13 02:39:32,077 INFO [collect_river_data] publishing log messages to tcp://*:5572
2026-06-13 02:39:32,078 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-06-13 02:39:32,078 INFO [collect_river_data] Collecting ECCC Englishman river data for 2026-06-12
2026-06-13 02:39:32,234 INFO [collect_river_data] ECCC ChilliwackVedder river data collection for 2026-06-12 completed
2026-06-13 02:39:32,239 INFO [collect_river_data] running in process 104489
2026-06-13 02:39:32,241 INFO [collect_river_data] running in process 104490
2026-06-13 02:39:32,241 INFO [collect_river_data] running in process 104493
2026-06-13 02:39:32,242 INFO [collect_river_data] running in process 104492
2026-06-13 02:39:32,243 INFO [collect_river_data] running in process 104491
2026-06-13 02:39:32,243 INFO [collect_river_data] Appended ECCC ClowhomClowhomLake river average discharge for 2026-06-12 to: /results/forcing/rivers/observations/Clowhom_ClowhomLake_flow
2026-06-13 02:39:32,248 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-06-13 02:39:32,250 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-06-13 02:39:32,250 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-06-13 02:39:32,251 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-06-13 02:39:32,252 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-06-13 02:39:32,253 INFO [collect_river_data] ECCC ClowhomClowhomLake river data collection for 2026-06-12 completed
2026-06-13 02:39:32,254 INFO [collect_river_data] publishing log messages to tcp://*:5573
2026-06-13 02:39:32,255 INFO [collect_river_data] Appended ECCC Englishman river average discharge for 2026-06-12 to: /data/dlatorne/SOG-projects/SOG-forcing/ECget/Englishman_flow
2026-06-13 02:39:32,256 INFO [collect_river_data] publishing log messages to tcp://*:5574
2026-06-13 02:39:32,257 INFO [collect_river_data] publishing log messages to tcp://*:5575
2026-06-13 02:39:32,258 INFO [collect_river_data] publishing log messages to tcp://*:5576
2026-06-13 02:39:32,259 INFO [collect_river_data] publishing log messages to tcp://*:5577
2026-06-13 02:39:32,260 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-06-13 02:39:32,261 INFO [collect_river_data] ECCC Englishman river data collection for 2026-06-12 completed
2026-06-13 02:39:32,262 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-06-13 02:39:32,264 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-06-13 02:39:32,265 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-06-13 02:39:32,266 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-06-13 02:39:32,266 INFO [collect_river_data] Collecting ECCC SanJuanPortRenfrew river data for 2026-06-12
2026-06-13 02:39:32,267 INFO [collect_river_data] Collecting ECCC RobertsCreek river data for 2026-06-12
2026-06-13 02:39:32,269 INFO [collect_river_data] Collecting ECCC SalmonSayward river data for 2026-06-12
2026-06-13 02:39:32,270 INFO [collect_river_data] Collecting ECCC TheodosiaScotty river data for 2026-06-12
2026-06-13 02:39:32,271 INFO [collect_river_data] Collecting ECCC SquamishBrackendale river data for 2026-06-12
2026-06-13 02:39:32,275 INFO [collect_river_data] Appended ECCC SanJuanPortRenfrew river average discharge for 2026-06-12 to: /results/forcing/rivers/observations/SanJuan_PortRenfrew_flow
2026-06-13 02:39:32,276 INFO [collect_river_data] Appended ECCC RobertsCreek river average discharge for 2026-06-12 to: /results/forcing/rivers/observations/RobertsCreek_flow
2026-06-13 02:39:32,277 INFO [collect_river_data] Appended ECCC SquamishBrackendale river average discharge for 2026-06-12 to: /results/forcing/rivers/observations/Squamish_Brackendale_flow
2026-06-13 02:39:32,278 INFO [collect_river_data] Appended ECCC SalmonSayward river average discharge for 2026-06-12 to: /results/forcing/rivers/observations/Salmon_Sayward_flow
2026-06-13 02:39:32,279 INFO [collect_river_data] Appended ECCC TheodosiaScotty river average discharge for 2026-06-12 to: /results/forcing/rivers/observations/Theodosia_Scotty_flow
2026-06-13 02:39:32,279 INFO [collect_river_data] ECCC SanJuanPortRenfrew river data collection for 2026-06-12 completed
2026-06-13 02:39:32,280 INFO [collect_river_data] ECCC RobertsCreek river data collection for 2026-06-12 completed
2026-06-13 02:39:32,281 INFO [collect_river_data] ECCC SquamishBrackendale river data collection for 2026-06-12 completed
2026-06-13 02:39:32,282 INFO [collect_river_data] ECCC SalmonSayward river data collection for 2026-06-12 completed
2026-06-13 02:39:32,283 INFO [collect_river_data] ECCC TheodosiaScotty river data collection for 2026-06-12 completed
2026-06-13 02:39:32,284 CRITICAL [collect_river_data] unhandled exception:
Traceback (most recent call last):
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/pandas/core/indexes/datetimes.py", line 610, in get_loc
    return self._partial_date_slice(reso, parsed)
           ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/pandas/core/indexes/datetimelike.py", line 333, in _partial_date_slice
    raise KeyError
KeyError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/SalishSeaCast/NEMO_Nowcast/nemo_nowcast/worker.py", line 387, in _do_work
    checklist = self.worker_func(
        self._parsed_args, self.config, self.tell_manager
    )
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/collect_river_data.py", line 109, in collect_river_data
    day_avg_discharge = day_avg_discharge_funcs[data_src](river_name, data_date, config)
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/collect_river_data.py", line 142, in _calc_eccc_day_avg_discharge
    day_avg_discharge = df.loc[f"{data_date.format('YYYY-MM-DD')}"].mean()[
                        ~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/pandas/core/indexing.py", line 1192, in __getitem__
    return self._getitem_axis(maybe_callable, axis=axis)
           ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/pandas/core/indexing.py", line 1432, in _getitem_axis
    return self._get_label(key, axis=axis)
           ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/pandas/core/indexing.py", line 1382, in _get_label
    return self.obj.xs(label, axis=axis)
           ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/pandas/core/generic.py", line 4323, in xs
    loc = index.get_loc(key)
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/pandas/core/indexes/datetimes.py", line 612, in get_loc
    raise KeyError(key) from err
KeyError: '2026-06-12'
2026-06-13 02:39:32,547 INFO [get_vfpa_hadcp] running in process 104499
2026-06-13 02:39:32,549 INFO [get_vfpa_hadcp] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-06-13 02:39:32,551 INFO [get_vfpa_hadcp] publishing log messages to tcp://*:5579
2026-06-13 02:39:32,552 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-06-13 02:39:32,553 INFO [get_vfpa_hadcp] connected to skookum.eos.ubc.ca port 5555
2026-06-13 02:39:32,554 INFO [get_vfpa_hadcp] processing VFPA HADCP data from 2nd Narrows Rail Bridge for 2026-06-12
2026-06-13 02:39:32,555 WARNING [get_vfpa_hadcp] file not found: /opp/observations/AISDATA/20260612T000000Z-20260612T010000Z.csv
2026-06-13 02:39:32,556 CRITICAL [get_vfpa_hadcp] unhandled exception:
Traceback (most recent call last):
  File "/SalishSeaCast/NEMO_Nowcast/nemo_nowcast/worker.py", line 387, in _do_work
    checklist = self.worker_func(
        self._parsed_args, self.config, self.tell_manager
    )
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/get_vfpa_hadcp.py", line 105, in get_vfpa_hadcp
    ds = _make_hour_dataset(csv_dir, data_date, place)
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/get_vfpa_hadcp.py", line 170, in _make_hour_dataset
    ds = _csv_to_dataset(csv_dir / csv_filename, place)
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/get_vfpa_hadcp.py", line 246, in _csv_to_dataset
    df = pandas.read_csv(csv_file, skiprows=3)
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/pandas/io/parsers/readers.py", line 1026, in read_csv
    return _read(filepath_or_buffer, kwds)
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/pandas/io/parsers/readers.py", line 620, in _read
    parser = TextFileReader(filepath_or_buffer, **kwds)
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/pandas/io/parsers/readers.py", line 1620, in __init__
    self._engine = self._make_engine(f, self.engine)
                   ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/pandas/io/parsers/readers.py", line 1880, in _make_engine
    self.handles = get_handle(
                   ~~~~~~~~~~^
        f,
        ^^
    ...<6 lines>...
        storage_options=self.options.get("storage_options", None),
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/pandas/io/common.py", line 873, in get_handle
    handle = open(
        handle,
    ...<3 lines>...
        newline="",
    )
FileNotFoundError: [Errno 2] No such file or directory: '/opp/observations/AISDATA/20260612T000000Z-20260612T010000Z.csv'
2026-06-13 02:39:32,595 INFO [crop_gribs] running in process 104503
2026-06-13 02:39:32,597 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-06-13 02:39:32,598 INFO [crop_gribs] publishing log messages to tcp://*:5590
2026-06-13 02:39:32,599 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2026-06-13 02:39:32,600 INFO [crop_gribs] cropping 2026-06-13 ECCC HRDPS 2.5km continental 12Z GRIB files to SalishSeaCast subdomain
2026-06-13 02:39:32,618 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20260613/12/
2026-06-13 02:39:32,780 INFO [collect_NeahBay_ssh] 2026-06-13T00:00:00-07:00 Neah Bay ssh 00Z obs/forecast data collection complete
2026-06-13 02:39:33,010 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-06-13 02:39:33,270 INFO [get_onc_ctd] running in process 104497
2026-06-13 02:39:33,272 INFO [get_onc_ctd] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-06-13 02:39:33,273 INFO [get_onc_ctd] publishing log messages to tcp://*:5565
2026-06-13 02:39:33,275 INFO [get_onc_ctd] connected to skookum.eos.ubc.ca port 5555
2026-06-13 02:39:33,276 INFO [get_onc_ctd] requesting ONC SEVIP CTD T&S data for 2026-06-12
2026-06-13 02:39:33,277 INFO [get_onc_ferry] running in process 104498
2026-06-13 02:39:33,278 INFO [get_onc_ferry] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-06-13 02:39:33,279 INFO [get_onc_ferry] publishing log messages to tcp://*:5566
2026-06-13 02:39:33,280 INFO [get_onc_ferry] connected to skookum.eos.ubc.ca port 5555
2026-06-13 02:39:33,281 INFO [get_onc_ferry] requesting ONC TWDP.N1 NAV data for 2026-06-12
2026-06-13 02:39:33,331 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-06-13 02:39:33,358 INFO [get_onc_ctd] running in process 104496
2026-06-13 02:39:33,359 INFO [get_onc_ctd] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-06-13 02:39:33,360 INFO [get_onc_ctd] publishing log messages to tcp://*:5591
2026-06-13 02:39:33,362 INFO [get_onc_ctd] connected to skookum.eos.ubc.ca port 5555
2026-06-13 02:39:33,363 INFO [get_onc_ctd] requesting ONC SCVIP CTD T&S data for 2026-06-12
2026-06-13 02:39:33,621 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-06-13 02:39:33,896 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-06-13 02:39:34,176 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-06-13 02:39:34,451 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-06-13 02:39:34,734 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-06-13 02:39:35,031 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-06-13 02:39:35,311 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-06-13 02:39:35,598 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-06-13 02:39:35,907 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-06-13 02:39:36,243 INFO [manager] checklist updated with [Neah Bay ssh data] items from collect_NeahBay_ssh worker
2026-06-13 02:39:36,639 INFO [manager] launching NextWorker(module='nowcast.workers.make_ssh_files', args=['forecast2', '--run-date', '2026-06-13'], host='localhost')
2026-06-13 02:39:39,869 INFO [make_ssh_files] running in process 104828
2026-06-13 02:39:39,870 INFO [make_ssh_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-06-13 02:39:39,871 INFO [make_ssh_files] publishing log messages to tcp://*:5564
2026-06-13 02:39:39,872 INFO [make_ssh_files] connected to skookum.eos.ubc.ca port 5555
2026-06-13 02:39:39,873 INFO [make_ssh_files] building forecast2 sea surface height boundary conditions file(s) from 2026-06-13 NOAA Neah Bay 00Z observation and forecast values
2026-06-13 02:39:42,029 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2026m06d08.nc
2026-06-13 02:39:42,203 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2026m06d09.nc
2026-06-13 02:39:42,376 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2026m06d10.nc
2026-06-13 02:39:42,557 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2026m06d11.nc
2026-06-13 02:39:42,737 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2026m06d12.nc
2026-06-13 02:39:42,982 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2026m06d13.nc
2026-06-13 02:39:43,186 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2026m06d14.nc
2026-06-13 02:39:43,412 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2026m06d15.nc
2026-06-13 02:39:43,598 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2026m06d16.nc
2026-06-13 02:39:43,743 INFO [make_ssh_files] sea surface height boundary file for 2026-06-13 forecast2 run created
2026-06-13 02:39:43,747 INFO [manager] checklist updated with [sea surface height forcing] items from make_ssh_files worker
2026-06-13 02:39:44,088 INFO [manager] launching NextWorker(module='nowcast.workers.make_runoff_file', args=['v202108'], host='localhost')
2026-06-13 02:39:45,951 INFO [make_runoff_file] running in process 104872
2026-06-13 02:39:45,952 INFO [make_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-06-13 02:39:45,954 INFO [make_runoff_file] publishing log messages to tcp://*:5564
2026-06-13 02:39:45,955 INFO [make_runoff_file] connected to skookum.eos.ubc.ca port 5555
2026-06-13 02:39:45,958 INFO [make_runoff_file] calculating NEMO runoff forcing for v202108 bathymetry for 2026-06-12
2026-06-13 02:39:46,942 ERROR [make_runoff_file] no 2026-06-12 discharge obs for SkagitMountVernon: patching
2026-06-13 02:39:47,362 ERROR [make_runoff_file] no 2026-06-12 discharge obs for SnohomishMonroe: patching
2026-06-13 02:39:47,566 ERROR [make_runoff_file] no 2026-06-12 discharge obs for NisquallyMcKenna: patching
2026-06-13 02:39:47,668 ERROR [make_runoff_file] no 2026-06-12 discharge obs for GreenwaterGreenwater: patching
2026-06-13 02:39:47,957 ERROR [make_runoff_file] no 2026-06-12 discharge obs for Theodosia: patching
2026-06-13 02:39:49,310 INFO [make_runoff_file] stored NEMO runoff forcing for v202108 bathymetry for 2026-06-12: /results/forcing/rivers/R202108Dailies_y2026m06d12.nc
2026-06-13 02:39:49,311 INFO [make_runoff_file] 2026-06-12 runoff file creation completed
2026-06-13 02:39:49,472 INFO [manager] checklist updated with [rivers forcing] items from make_runoff_file worker
2026-06-13 02:39:50,471 INFO [get_onc_ctd] 2026-06-12 ONC SEVIP CTD T&S file created
2026-06-13 02:39:50,473 INFO [get_onc_ctd] 2026-06-12 ONC SCVIP CTD T&S file created
2026-06-13 02:39:50,476 INFO [manager] checklist updated with [ONC CTD data] items from get_onc_ctd worker
2026-06-13 02:39:50,814 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['SEVIP-CTD'], host='localhost')
2026-06-13 02:39:50,821 INFO [manager] checklist updated with [ONC CTD data] items from get_onc_ctd worker
2026-06-13 02:39:51,141 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['SCVIP-CTD'], host='localhost')
2026-06-13 02:39:51,637 INFO [get_onc_ferry] requesting ONC TWDP TSG data for 2026-06-12
2026-06-13 02:39:51,914 INFO [ping_erddap] running in process 104898
2026-06-13 02:39:51,915 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-06-13 02:39:51,917 INFO [ping_erddap] publishing log messages to tcp://*:5564
2026-06-13 02:39:51,918 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2026-06-13 02:39:51,920 INFO [ping_erddap] SEVIP-CTD ERDDAP dataset flag file(s) created
2026-06-13 02:39:51,922 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2026-06-13 02:39:52,147 INFO [ping_erddap] running in process 104902
2026-06-13 02:39:52,148 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-06-13 02:39:52,149 INFO [ping_erddap] publishing log messages to tcp://*:5565
2026-06-13 02:39:52,150 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2026-06-13 02:39:52,152 INFO [ping_erddap] SCVIP-CTD ERDDAP dataset flag file(s) created
2026-06-13 02:39:52,204 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2026-06-13 02:40:13,444 INFO [get_onc_ferry] requesting ONC TWDP OXYSENSOR data for 2026-06-12
2026-06-13 02:40:36,483 INFO [get_onc_ferry] requesting ONC TWDP TURBCHLFL data for 2026-06-12
2026-06-13 02:40:57,941 INFO [get_onc_ferry] requesting ONC TWDP CO2SENSOR data for 2026-06-12
2026-06-13 02:40:58,012 WARNING [get_onc_ferry] No ONC TWDP CO2SENSOR partial_pressure,co2 data for 2026-06-12; substituting empty dataset
2026-06-13 02:40:58,017 WARNING [get_onc_ferry] No ONC TWDP CO2SENSOR partial_pressure data for 2026-06-12; substituting empty dataset
2026-06-13 02:40:58,019 WARNING [get_onc_ferry] No ONC TWDP CO2SENSOR co2 data for 2026-06-12; substituting empty dataset
2026-06-13 02:40:58,020 INFO [get_onc_ferry] requesting ONC TWDP TEMPHUMID data for 2026-06-12
2026-06-13 02:40:58,091 WARNING [get_onc_ferry] No ONC TWDP TEMPHUMID air_temperature,rel_humidity data for 2026-06-12; substituting empty dataset
2026-06-13 02:40:58,097 WARNING [get_onc_ferry] No ONC TWDP TEMPHUMID air_temperature data for 2026-06-12; substituting empty dataset
2026-06-13 02:40:58,098 WARNING [get_onc_ferry] No ONC TWDP TEMPHUMID rel_humidity data for 2026-06-12; substituting empty dataset
2026-06-13 02:40:58,100 INFO [get_onc_ferry] requesting ONC TWDP BARPRESS data for 2026-06-12
2026-06-13 02:40:58,179 WARNING [get_onc_ferry] No ONC TWDP BARPRESS barometric_pressure data for 2026-06-12; substituting empty dataset
2026-06-13 02:40:58,183 WARNING [get_onc_ferry] No ONC TWDP BARPRESS barometric_pressure data for 2026-06-12; substituting empty dataset
2026-06-13 02:40:58,184 INFO [get_onc_ferry] requesting ONC TWDP PYRANOMETER data for 2026-06-12
2026-06-13 02:40:58,241 WARNING [get_onc_ferry] No ONC TWDP PYRANOMETER solar_radiation data for 2026-06-12; substituting empty dataset
2026-06-13 02:40:58,244 WARNING [get_onc_ferry] No ONC TWDP PYRANOMETER solar_radiation data for 2026-06-12; substituting empty dataset
2026-06-13 02:40:58,245 INFO [get_onc_ferry] requesting ONC TWDP PYRGEOMETER data for 2026-06-12
2026-06-13 02:40:58,311 WARNING [get_onc_ferry] No ONC TWDP PYRGEOMETER downward_radiation data for 2026-06-12; substituting empty dataset
2026-06-13 02:40:58,315 WARNING [get_onc_ferry] No ONC TWDP PYRGEOMETER downward_radiation data for 2026-06-12; substituting empty dataset
2026-06-13 02:40:58,510 WARNING [get_onc_ferry] ONC TWDP CO2SENSOR partial_pressure data for 2026-06-12 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2026-06-13 02:40:58,528 WARNING [get_onc_ferry] ONC TWDP CO2SENSOR co2 data for 2026-06-12 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2026-06-13 02:40:58,543 WARNING [get_onc_ferry] ONC TWDP TEMPHUMID air_temperature data for 2026-06-12 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2026-06-13 02:40:58,558 WARNING [get_onc_ferry] ONC TWDP TEMPHUMID rel_humidity data for 2026-06-12 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2026-06-13 02:40:58,573 WARNING [get_onc_ferry] ONC TWDP BARPRESS barometric_pressure data for 2026-06-12 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2026-06-13 02:40:58,589 WARNING [get_onc_ferry] ONC TWDP PYRANOMETER solar_radiation data for 2026-06-12 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2026-06-13 02:40:58,604 WARNING [get_onc_ferry] ONC TWDP PYRGEOMETER downward_radiation data for 2026-06-12 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2026-06-13 02:41:14,290 INFO [get_onc_ferry] 2026-06-12 ONC TWDP ferry data file created
2026-06-13 02:41:14,293 INFO [manager] checklist updated with [ONC ferry data] items from get_onc_ferry worker
2026-06-13 02:41:14,611 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['TWDP-ferry'], host='localhost')
2026-06-13 02:41:15,809 INFO [ping_erddap] running in process 104945
2026-06-13 02:41:15,811 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-06-13 02:41:15,812 INFO [ping_erddap] publishing log messages to tcp://*:5564
2026-06-13 02:41:15,813 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2026-06-13 02:41:15,816 INFO [ping_erddap] TWDP-ferry ERDDAP dataset flag file(s) created
2026-06-13 02:41:15,818 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2026-06-13 04:49:11,300 INFO [crop_gribs] crop_gribs 06 has watched for 8h; retrying remaining 1 file(s)
2026-06-13 04:49:13,171 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20260613/06/
2026-06-13 04:49:13,173 INFO [crop_gribs] 2026-06-13 06 GRIBs cropping complete
2026-06-13 04:49:13,175 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2026-06-13 04:49:13,691 INFO [manager] launching NextWorker(module='nowcast.workers.grib_to_netcdf', args=['forecast2'], host='localhost')
2026-06-13 04:49:15,678 INFO [grib_to_netcdf] running in process 107501
2026-06-13 04:49:15,679 INFO [grib_to_netcdf] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-06-13 04:49:15,680 INFO [grib_to_netcdf] publishing log messages to tcp://*:5563
2026-06-13 04:49:15,682 INFO [grib_to_netcdf] connected to skookum.eos.ubc.ca port 5555
2026-06-13 04:49:15,683 INFO [grib_to_netcdf] creating NEMO-atmos forcing files for 2026-06-13 forecast2 run
2026-06-13 04:49:25,588 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2026m06d14.nc
2026-06-13 04:49:28,325 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2026m06d15.nc
2026-06-13 04:49:28,331 INFO [grib_to_netcdf] 2026-06-13 NEMO-atmos forcing file for forecast2 created
2026-06-13 04:49:28,334 INFO [manager] checklist updated with [weather forcing] items from grib_to_netcdf worker
2026-06-13 04:49:28,677 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'forecast2'], host='localhost')
2026-06-13 04:49:28,681 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'forecast2'], host='localhost')
2026-06-13 04:49:28,686 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.nibi', 'forecast2'], host='localhost')
2026-06-13 04:49:28,690 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'forecast2'], host='localhost')
2026-06-13 04:49:30,142 INFO [upload_forcing] running in process 107548
2026-06-13 04:49:30,143 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-06-13 04:49:30,145 INFO [upload_forcing] publishing log messages to tcp://*:5563
2026-06-13 04:49:30,146 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-06-13 04:49:30,147 INFO [upload_forcing] running in process 107552
2026-06-13 04:49:30,148 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-06-13 04:49:30,149 INFO [upload_forcing] publishing log messages to tcp://*:5564
2026-06-13 04:49:30,150 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-06-13 04:49:30,165 INFO [upload_forcing] running in process 107551
2026-06-13 04:49:30,167 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-06-13 04:49:30,168 INFO [upload_forcing] publishing log messages to tcp://*:5565
2026-06-13 04:49:30,168 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-06-13 04:49:30,177 INFO [upload_forcing] running in process 107550
2026-06-13 04:49:30,179 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-06-13 04:49:30,180 INFO [upload_forcing] publishing log messages to tcp://*:5566
2026-06-13 04:49:30,181 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-06-13 04:49:31,393 INFO [upload_forcing] LiveOcean boundary conditions file not found; created symlink to /results/forcing/LiveOcean/boundary_conditions/LiveOcean_v201905_y2026m06d12.nc
2026-06-13 04:49:31,473 INFO [upload_forcing] forecast2 2026-06-13 forcing files upload to optimum-hindcast completed
2026-06-13 04:49:31,476 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-06-13 04:49:33,085 INFO [upload_forcing] forecast2 2026-06-13 forcing files upload to arbutus.cloud-nowcast completed
2026-06-13 04:49:33,090 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-06-13 04:49:33,426 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2026-06-13'], host='localhost')
2026-06-13 04:49:34,682 INFO [make_forcing_links] running in process 107583
2026-06-13 04:49:34,684 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-06-13 04:49:34,685 INFO [make_forcing_links] publishing log messages to tcp://*:5563
2026-06-13 04:49:34,687 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2026-06-13 04:49:35,943 INFO [make_forcing_links] forecast2 2026-06-13 forcing file links on arbutus.cloud-nowcast created
2026-06-13 04:49:35,948 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2026-06-13 04:49:36,306 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2026-06-13'], host='arbutus.cloud-nowcast')
2026-06-13 04:49:36,762 INFO [upload_forcing] forecast2 2026-06-13 forcing files upload to orcinus-nowcast-agrif completed
2026-06-13 04:49:36,766 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-06-13 04:49:37,358 INFO [upload_forcing] forecast2 2026-06-13 forcing files upload to robot.nibi completed
2026-06-13 04:49:37,362 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-06-13 04:49:38,454 INFO [run_NEMO] running in process 18766
2026-06-13 04:49:38,455 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-06-13 04:49:38,456 INFO [run_NEMO] publishing log messages to tcp://*:5556
2026-06-13 04:49:38,515 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2026-06-13 04:49:39,269 WARNING [run_NEMO] There are uncommitted changes in /nemoShare/MEOPAR/nowcast-sys/SS-run-sets
2026-06-13 04:49:39,574 INFO [run_NEMO] forecast2: launching /nemoShare/MEOPAR/nowcast-sys/runs/12jun26forecast2_2026-06-13T044938.583189-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2026-06-13 04:49:39,589 INFO [run_NEMO] forecast2 NEMO run for 2026-06-13 on arbutus.cloud-nowcast started
2026-06-13 04:49:39,593 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2026-06-13 04:49:39,942 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'forecast2'], host='arbutus.cloud-nowcast')
2026-06-13 04:49:41,972 INFO [watch_NEMO] running in process 19040
2026-06-13 04:49:41,973 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-06-13 04:49:41,974 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2026-06-13 04:49:41,977 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2026-06-13 04:49:42,079 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2026-06-13 04:54:42,076 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 19295788 = 2026-06-14 05:38:00 UTC, 18.8% complete
2026-06-13 04:59:42,077 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 19296349 = 2026-06-14 11:52:00 UTC, 39.6% complete
2026-06-13 05:04:42,078 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 19296914 = 2026-06-14 18:08:40 UTC, 60.5% complete
2026-06-13 05:09:42,079 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 19297499 = 2026-06-15 00:38:40 UTC, 82.2% complete
2026-06-13 05:14:42,082 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 19297980 = 2026-06-15 05:59:20 UTC, 100.0% complete
2026-06-13 05:19:42,085 INFO [watch_NEMO] forecast2 NEMO run on arbutus.cloud-nowcast completed
2026-06-13 05:19:42,088 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2026-06-13 05:19:42,656 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_wind_file', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2026-06-13'], host='arbutus.cloud-nowcast')
2026-06-13 05:19:42,666 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_current_file', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2026-06-13'], host='arbutus.cloud-nowcast')
2026-06-13 05:19:42,670 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2026-06-12'], host='localhost')
2026-06-13 05:19:43,954 INFO [download_results] running in process 108177
2026-06-13 05:19:43,955 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-06-13 05:19:43,956 INFO [download_results] publishing log messages to tcp://*:5563
2026-06-13 05:19:43,958 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2026-06-13 05:19:43,959 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/forecast2/12jun26 to /results/SalishSea/forecast2.202111
2026-06-13 05:19:45,436 INFO [make_ww3_wind_file] running in process 19342
2026-06-13 05:19:45,437 INFO [make_ww3_wind_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-06-13 05:19:45,439 INFO [make_ww3_wind_file] publishing log messages to tcp://*:5570
2026-06-13 05:19:45,440 INFO [make_ww3_wind_file] connected to skookum.eos.ubc.ca port 5555
2026-06-13 05:19:45,441 INFO [make_ww3_wind_file] Creating wwatch3 wind forcing file for 2026-06-13 forecast2 run
2026-06-13 05:19:54,694 INFO [make_ww3_wind_file] wwatch3 wind forcing file created on arbutus.cloud-nowcast for 2026-06-13 forecast2 run
2026-06-13 05:19:54,697 INFO [manager] checklist updated with [WW3 wind forcing] items from make_ww3_wind_file worker
2026-06-13 05:20:20,397 INFO [download_results] forecast2 2026-06-12 results files from arbutus.cloud-nowcast downloaded
2026-06-13 05:20:20,402 INFO [manager] checklist updated with [results files] items from download_results worker
2026-06-13 05:20:20,750 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['forecast2', '--run-date', '2026-06-12'], host='localhost')
2026-06-13 05:20:22,784 INFO [make_CHS_currents_file] running in process 108193
2026-06-13 05:20:22,786 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-06-13 05:20:22,787 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5563
2026-06-13 05:20:22,789 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555
2026-06-13 05:20:58,681 INFO [make_CHS_currents_file] Made CHS currents file for 2026-06-12 for forecast2
2026-06-13 05:20:58,683 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker
2026-06-13 05:20:58,961 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['nemo', 'forecast2', '--run-date', '2026-06-12'], host='localhost')
2026-06-13 05:21:00,732 INFO [update_forecast_datasets] running in process 108238
2026-06-13 05:21:00,734 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-06-13 05:21:00,735 INFO [update_forecast_datasets] publishing log messages to tcp://*:5563
2026-06-13 05:21:00,736 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2026-06-13 05:21:00,738 INFO [update_forecast_datasets] updating nemo forecast directory for forecast2/12jun26 run
2026-06-13 05:23:20,860 INFO [make_ww3_current_file] wwatch3 currents forcing file created on arbutus.cloud-nowcast for 2026-06-13 forecast2 run
2026-06-13 05:23:20,865 INFO [manager] checklist updated with [WW3 currents forcing] items from make_ww3_current_file worker
2026-06-13 05:23:21,205 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2026-06-13'], host='arbutus.cloud-nowcast')
2026-06-13 05:23:23,209 INFO [run_ww3] running in process 19887
2026-06-13 05:23:23,211 INFO [run_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-06-13 05:23:23,212 INFO [run_ww3] publishing log messages to tcp://*:5572
2026-06-13 05:23:23,213 INFO [run_ww3] connected to skookum.eos.ubc.ca port 5555
2026-06-13 05:23:23,218 INFO [run_ww3] Created run directory /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast2_2026-06-13T052323.207896-0700
2026-06-13 05:23:23,220 INFO [run_ww3] forecast2: launching /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast2_2026-06-13T052323.207896-0700/SoGWW3.sh on arbutus.cloud-nowcast
2026-06-13 05:23:23,229 INFO [run_ww3] forecast2 WaveWatch3 run for 2026-06-13 on arbutus.cloud-nowcast started
2026-06-13 05:23:23,233 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker
2026-06-13 05:23:23,563 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'forecast2'], host='arbutus.cloud-nowcast')
2026-06-13 05:23:25,589 INFO [watch_ww3] running in process 19974
2026-06-13 05:23:25,590 INFO [watch_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-06-13 05:23:25,592 INFO [watch_ww3] publishing log messages to tcp://*:5573
2026-06-13 05:23:25,593 INFO [watch_ww3] connected to skookum.eos.ubc.ca port 5555
2026-06-13 05:23:25,625 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2026-06-13 05:25:14,890 INFO [update_forecast_datasets] updated nemo forecast directory for forecast2/12jun26 run: /results/SalishSea/rolling-forecasts/nemo
2026-06-13 05:25:14,891 INFO [update_forecast_datasets] nemo 2026-06-12 forecast2 rolling forecast datasets updated
2026-06-13 05:25:14,894 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2026-06-13 05:25:15,242 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nemo-forecast'], host='localhost')
2026-06-13 05:25:15,249 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'forecast2', 'publish', '--run-date', '2026-06-12'], host='localhost')
2026-06-13 05:25:15,253 INFO [manager] launching NextWorker(module='nowcast.workers.make_surface_current_tiles', args=['forecast2', '--run-date', '2026-06-12'], host='localhost')
2026-06-13 05:25:16,898 INFO [ping_erddap] running in process 108391
2026-06-13 05:25:16,899 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-06-13 05:25:16,900 INFO [ping_erddap] publishing log messages to tcp://*:5563
2026-06-13 05:25:16,901 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2026-06-13 05:25:16,913 INFO [ping_erddap] nemo-forecast ERDDAP dataset flag file(s) created
2026-06-13 05:25:16,915 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2026-06-13 05:25:18,792 INFO [make_surface_current_tiles] running in process 108394
2026-06-13 05:25:18,794 INFO [make_surface_current_tiles] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-06-13 05:25:18,796 INFO [make_surface_current_tiles] publishing log messages to tcp://*:5563
2026-06-13 05:25:18,797 INFO [make_surface_current_tiles] connected to skookum.eos.ubc.ca port 5555
2026-06-13 05:25:18,815 INFO [make_surface_current_tiles] starting rendering of tiles for 2026-06-12 forecast2 in 6 processes into /results/nowcast-sys/figures/surface_currents/forecast2/12jun26
2026-06-13 05:25:20,631 INFO [make_plots] running in process 108393
2026-06-13 05:25:20,632 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-06-13 05:25:20,633 INFO [make_plots] publishing log messages to tcp://*:5564
2026-06-13 05:25:20,633 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2026-06-13 05:25:21,009 INFO [make_plots] preparing render list for 2026-06-12 NEMO forecast2 publish figures
2026-06-13 05:25:21,281 INFO [make_plots] added Website_thumbnail figure to 2026-06-12 NEMO forecast2 publish render list
2026-06-13 05:25:21,283 INFO [make_plots] added Threshold_website figure to 2026-06-12 NEMO forecast2 publish render list
2026-06-13 05:25:21,284 INFO [make_plots] added PA_tidal_predictions figure to 2026-06-12 NEMO forecast2 publish render list
2026-06-13 05:25:21,285 INFO [make_plots] added Boundary Bay figure to 2026-06-12 NEMO forecast2 publish render list
2026-06-13 05:25:21,286 INFO [make_plots] added Campbell River figure to 2026-06-12 NEMO forecast2 publish render list
2026-06-13 05:25:21,286 INFO [make_plots] added Cherry Point figure to 2026-06-12 NEMO forecast2 publish render list
2026-06-13 05:25:21,288 INFO [make_plots] added Friday Harbor figure to 2026-06-12 NEMO forecast2 publish render list
2026-06-13 05:25:21,288 INFO [make_plots] added Halfmoon Bay figure to 2026-06-12 NEMO forecast2 publish render list
2026-06-13 05:25:21,290 INFO [make_plots] added Nanaimo figure to 2026-06-12 NEMO forecast2 publish render list
2026-06-13 05:25:21,291 INFO [make_plots] added Neah Bay figure to 2026-06-12 NEMO forecast2 publish render list
2026-06-13 05:25:21,291 INFO [make_plots] added New Westminster figure to 2026-06-12 NEMO forecast2 publish render list
2026-06-13 05:25:21,293 INFO [make_plots] added Patricia Bay figure to 2026-06-12 NEMO forecast2 publish render list
2026-06-13 05:25:21,294 INFO [make_plots] added Point Atkinson figure to 2026-06-12 NEMO forecast2 publish render list
2026-06-13 05:25:21,295 INFO [make_plots] added Port Renfrew figure to 2026-06-12 NEMO forecast2 publish render list
2026-06-13 05:25:21,296 INFO [make_plots] added Sand Heads figure to 2026-06-12 NEMO forecast2 publish render list
2026-06-13 05:25:21,296 INFO [make_plots] added Sandy Cove figure to 2026-06-12 NEMO forecast2 publish render list
2026-06-13 05:25:21,297 INFO [make_plots] added Squamish figure to 2026-06-12 NEMO forecast2 publish render list
2026-06-13 05:25:21,298 INFO [make_plots] added Victoria figure to 2026-06-12 NEMO forecast2 publish render list
2026-06-13 05:25:21,299 INFO [make_plots] added Woodwards Landing figure to 2026-06-12 NEMO forecast2 publish render list
2026-06-13 05:25:21,300 INFO [make_plots] starting to render nemo forecast2 publish 12jun26 figures
2026-06-13 05:28:25,626 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 90, 16.7% complete
2026-06-13 05:33:05,385 WARNING [make_plots] invalid station number for Boundary Bay station: None
2026-06-13 05:33:05,386 WARNING [make_plots] can't resolve a valid CHS station code for Boundary Bay
2026-06-13 05:33:25,627 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 192, 35.6% complete
2026-06-13 05:33:33,317 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb996/data for station 08074 Campbell River from 2026-06-08 00:00:00Z to 2026-06-15 05:55:00Z
2026-06-13 05:34:42,627 WARNING [make_plots] invalid station number for Halfmoon Bay station: None
2026-06-13 05:34:42,628 WARNING [make_plots] can't resolve a valid CHS station code for Halfmoon Bay
2026-06-13 05:35:06,036 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb96d/data for station 07917 Nanaimo from 2026-06-08 00:00:00Z to 2026-06-15 05:55:00Z
2026-06-13 05:35:48,411 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e13d0f4a073c4bbf93/data for station 07654 New Westminster from 2026-06-08 00:00:00Z to 2026-06-15 05:55:00Z
2026-06-13 05:36:09,625 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd2d/data for station 07277 Patricia Bay from 2026-06-08 00:00:00Z to 2026-06-15 05:55:00Z
2026-06-13 05:36:18,598 INFO [make_surface_current_tiles] starting PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast2/12jun26
2026-06-13 05:36:28,003 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94c/data for station 07795 Point Atkinson from 2026-06-08 00:00:00Z to 2026-06-15 05:55:00Z
2026-06-13 05:36:45,732 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e23d0f4a073c4bc060/data for station 08525 Port Renfrew from 2026-06-08 00:00:00Z to 2026-06-15 05:55:00Z
2026-06-13 05:37:01,924 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb93c/data for station 07594 Sand Heads from 2026-06-08 00:00:00Z to 2026-06-15 05:55:00Z
2026-06-13 05:37:19,686 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc43b/data for station 07786 Sandy Cove from 2026-06-08 00:00:00Z to 2026-06-15 05:55:00Z
2026-06-13 05:37:37,382 WARNING [make_plots] invalid station number for Squamish station: None
2026-06-13 05:37:37,383 WARNING [make_plots] can't resolve a valid CHS station code for Squamish
2026-06-13 05:37:54,965 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd1e/data for station 07120 Victoria from 2026-06-08 00:00:00Z to 2026-06-15 05:55:00Z
2026-06-13 05:38:10,917 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc42d/data for station 07610 Woodwards Landing from 2026-06-08 00:00:00Z to 2026-06-15 05:55:00Z
2026-06-13 05:38:11,317 INFO [make_plots] no obs water level data available from 5cebf1e43d0f4a073c4bc42d during 2026-06-08 00:00:00Z to 2026-06-15 05:55:00Z
2026-06-13 05:38:25,628 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 297, 55.0% complete
2026-06-13 05:38:28,528 INFO [make_plots] finished rendering nemo forecast2 publish 12jun26 figures
2026-06-13 05:38:28,530 INFO [make_plots] nemo publish plots for 2026-06-12 forecast2 completed
2026-06-13 05:38:28,539 INFO [manager] checklist updated with [plots] items from make_plots worker
2026-06-13 05:38:28,873 INFO [manager] launching NextWorker(module='nowcast.workers.make_feeds', args=['forecast2', '--run-date', '2026-06-12'], host='localhost')
2026-06-13 05:38:33,701 INFO [make_feeds] running in process 113525
2026-06-13 05:38:33,702 INFO [make_feeds] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-06-13 05:38:33,703 INFO [make_feeds] publishing log messages to tcp://*:5564
2026-06-13 05:38:33,705 INFO [make_feeds] connected to skookum.eos.ubc.ca port 5555
2026-06-13 05:38:48,020 INFO [make_feeds] ATOM feeds for 2026-06-12 forecast2 run completed
2026-06-13 05:38:48,024 INFO [manager] checklist updated with [storm surge alert feeds] items from make_feeds worker
2026-06-13 05:38:48,372 INFO [manager] launching NextWorker(module='nemo_nowcast.workers.clear_checklist', args=[], host='localhost')
2026-06-13 05:38:49,585 INFO [clear_checklist] running in process 113551
2026-06-13 05:38:49,586 INFO [clear_checklist] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-06-13 05:38:49,587 INFO [clear_checklist] publishing log messages to tcp://*:5564
2026-06-13 05:38:49,589 INFO [clear_checklist] connected to skookum.eos.ubc.ca port 5555
2026-06-13 05:38:49,590 INFO [clear_checklist] requesting that manager clear system state checklist
2026-06-13 05:38:49,592 INFO [manager] writing checklist to log file
2026-06-13 05:38:49,638 INFO [manager] checklist cleared
2026-06-13 05:38:49,641 INFO [clear_checklist] nowcast system checklist cleared
2026-06-13 05:38:49,643 INFO [manager] launching NextWorker(module='nemo_nowcast.workers.rotate_logs', args=[], host='localhost')
2026-06-13 05:38:50,747 INFO [rotate_logs] running in process 113556
2026-06-13 05:38:50,748 INFO [rotate_logs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-06-13 05:38:50,749 INFO [rotate_logs] publishing log messages to tcp://*:5564
2026-06-13 05:38:50,750 INFO [rotate_logs] connected to skookum.eos.ubc.ca port 5555
2026-06-13 05:38:50,751 INFO [rotate_logs] rotating log files