2026-03-08 03:32:26,663 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/nowcast.log
2026-03-08 03:32:26,671 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/nowcast.debug.log
2026-03-08 03:32:26,677 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/checklist.log
2026-03-08 03:32:26,678 INFO [rotate_logs] log files rotated
2026-03-08 03:32:26,692 INFO [manager] checklist updated with [log rotation] items from rotate_logs worker
2026-03-08 03:34:07,645 INFO [make_surface_current_tiles] finished PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast2/07mar26
2026-03-08 03:34:07,669 INFO [make_surface_current_tiles] finished rendering of tiles for 2026-03-07 forecast2 into /results/nowcast-sys/figures/surface_currents/forecast2/07mar26
2026-03-08 03:34:07,670 INFO [make_surface_current_tiles] surface current tile figures for 2026-03-07 forecast2 completed
2026-03-08 03:34:08,145 INFO [manager] checklist updated with [surface current tiles] items from make_surface_current_tiles worker
2026-03-08 03:36:51,715 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 381, 70.6% complete
2026-03-08 03:41:51,716 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 486, 90.0% complete
2026-03-08 03:46:51,718 INFO [watch_ww3] forecast2 WWATCH3 run on arbutus.cloud-nowcast completed
2026-03-08 03:46:51,722 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2026-03-08 03:46:52,079 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2026-03-08'], host='localhost')
2026-03-08 03:46:53,349 INFO [download_wwatch3_results] running in process 1179584
2026-03-08 03:46:53,351 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-08 03:46:53,352 INFO [download_wwatch3_results] publishing log messages to tcp://*:5563
2026-03-08 03:46:53,353 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2026-03-08 03:47:30,377 INFO [download_wwatch3_results] forecast2 2026-03-08 results files from arbutus.cloud-nowcast downloaded
2026-03-08 03:47:30,380 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2026-03-08 03:47:30,526 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['wwatch3', 'forecast2', '--run-date', '2026-03-08'], host='localhost')
2026-03-08 03:47:31,734 INFO [update_forecast_datasets] running in process 1179598
2026-03-08 03:47:31,736 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-08 03:47:31,737 INFO [update_forecast_datasets] publishing log messages to tcp://*:5563
2026-03-08 03:47:31,738 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2026-03-08 03:47:31,739 INFO [update_forecast_datasets] updating wwatch3 most_recent_forecast directory from forecast2/08mar26 run
2026-03-08 03:47:31,744 INFO [update_forecast_datasets] updated wwatch3 most_recent_forecast directory from forecast2/08mar26 run
2026-03-08 03:47:31,745 INFO [update_forecast_datasets] updating wwatch3 forecast directory for forecast2/08mar26 run
2026-03-08 03:47:56,438 INFO [update_forecast_datasets] updated wwatch3 forecast directory for forecast2/08mar26 run: /results/SalishSea/rolling-forecasts/wwatch3
2026-03-08 03:47:56,439 INFO [update_forecast_datasets] wwatch3 2026-03-08 forecast2 rolling forecast datasets updated
2026-03-08 03:47:56,443 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2026-03-08 03:47:56,593 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['wwatch3-forecast'], host='localhost')
2026-03-08 03:47:57,873 INFO [ping_erddap] running in process 1179610
2026-03-08 03:47:57,874 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-08 03:47:57,876 INFO [ping_erddap] publishing log messages to tcp://*:5563
2026-03-08 03:47:57,877 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2026-03-08 03:47:57,879 INFO [ping_erddap] wwatch3-forecast ERDDAP dataset flag file(s) created
2026-03-08 03:47:57,882 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2026-03-08 03:47:58,024 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['wwatch3', 'forecast2', 'publish', '--run-date', '2026-03-08'], host='localhost')
2026-03-08 03:48:01,664 INFO [make_plots] running in process 1179615
2026-03-08 03:48:01,665 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-08 03:48:01,668 INFO [make_plots] publishing log messages to tcp://*:5563
2026-03-08 03:48:01,669 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2026-03-08 03:48:01,670 INFO [make_plots] preparing render list for 2026-03-08 WaveWatch3 forecast2 publish figures
2026-03-08 03:48:01,671 INFO [make_plots] added Halibut Bank figure to 2026-03-08 WaveWatch3 forecast2 publish render list
2026-03-08 03:48:01,672 INFO [make_plots] added Sentry Shoal figure to 2026-03-08 WaveWatch3 forecast2 publish render list
2026-03-08 03:48:01,673 INFO [make_plots] starting to render wwatch3 forecast2 publish 08mar26 figures
2026-03-08 03:48:02,193 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46146 Halibut Bank
2026-03-08 03:48:07,912 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46131 Sentry Shoal
2026-03-08 03:48:12,205 INFO [make_plots] finished rendering wwatch3 forecast2 publish 08mar26 figures
2026-03-08 03:48:12,206 INFO [make_plots] wwatch3 publish plots for 2026-03-08 forecast2 completed
2026-03-08 03:48:12,210 INFO [manager] checklist updated with [plots] items from make_plots worker
2026-03-08 08:41:16,849 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/12/ to /results/forcing/atmospheric/continental2.5/GRIB/20260308/12/
2026-03-08 08:41:16,850 INFO [collect_weather] 2026-03-08 2.5km weather forecast 12 collection complete
2026-03-08 08:41:16,853 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2026-03-08 08:41:17,171 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'SkagitMountVernon', '--data-date', '2026-03-07'], host='localhost')
2026-03-08 08:41:17,179 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'SnohomishMonroe', '--data-date', '2026-03-07'], host='localhost')
2026-03-08 08:41:17,183 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'NisquallyMcKenna', '--data-date', '2026-03-07'], host='localhost')
2026-03-08 08:41:17,190 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'GreenwaterGreenwater', '--data-date', '2026-03-07'], host='localhost')
2026-03-08 08:41:17,194 INFO [manager] launching NextWorker(module='nowcast.workers.make_turbidity_file', args=[], host='localhost')
2026-03-08 08:41:17,204 INFO [manager] launching NextWorker(module='nowcast.workers.collect_NeahBay_ssh', args=['06'], host='localhost')
2026-03-08 08:41:17,208 INFO [manager] launching NextWorker(module='nowcast.workers.download_live_ocean', args=[], host='localhost')
2026-03-08 08:41:17,212 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['18', '2.5km'], host='localhost')
2026-03-08 08:41:17,216 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['18'], host='localhost')
2026-03-08 08:41:17,621 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20260308/12/
2026-03-08 08:41:17,622 INFO [crop_gribs] 2026-03-08 12 GRIBs cropping complete
2026-03-08 08:41:17,627 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2026-03-08 08:41:18,208 INFO [manager] launching NextWorker(module='nowcast.workers.grib_to_netcdf', args=['nowcast+'], host='localhost')
2026-03-08 08:41:18,959 INFO [collect_NeahBay_ssh] running in process 1184315
2026-03-08 08:41:18,959 INFO [collect_NeahBay_ssh] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-08 08:41:18,960 INFO [collect_NeahBay_ssh] publishing log messages to tcp://*:5562
2026-03-08 08:41:18,961 INFO [collect_NeahBay_ssh] connected to skookum.eos.ubc.ca port 5555
2026-03-08 08:41:18,962 INFO [collect_NeahBay_ssh] collecting Neah Bay ssh 06Z obs/forecast for 2026-03-08
2026-03-08 08:41:18,963 INFO [collect_weather] running in process 1184317
2026-03-08 08:41:18,966 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-08 08:41:18,967 INFO [collect_weather] publishing log messages to tcp://*:5563
2026-03-08 08:41:18,968 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2026-03-08 08:41:18,984 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/18/
2026-03-08 08:41:19,125 INFO [download_live_ocean] running in process 1184316
2026-03-08 08:41:19,126 INFO [download_live_ocean] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-08 08:41:19,127 INFO [download_live_ocean] publishing log messages to tcp://*:5564
2026-03-08 08:41:19,128 INFO [download_live_ocean] connected to skookum.eos.ubc.ca port 5555
2026-03-08 08:41:19,129 INFO [download_live_ocean] downloading Salish Sea western boundary day-averaged LiveOcean file for 2026-03-08
2026-03-08 08:41:19,459 INFO [collect_river_data] running in process 1184312
2026-03-08 08:41:19,460 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-08 08:41:19,461 INFO [collect_river_data] publishing log messages to tcp://*:5565
2026-03-08 08:41:19,462 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-08 08:41:19,463 INFO [collect_river_data] Collecting USGS NisquallyMcKenna river data for 2026-03-07
2026-03-08 08:41:19,482 INFO [collect_river_data] running in process 1184309
2026-03-08 08:41:19,484 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-08 08:41:19,484 INFO [collect_river_data] publishing log messages to tcp://*:5566
2026-03-08 08:41:19,485 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-08 08:41:19,486 INFO [collect_river_data] Collecting USGS SkagitMountVernon river data for 2026-03-07
2026-03-08 08:41:19,563 INFO [collect_river_data] running in process 1184311
2026-03-08 08:41:19,565 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-08 08:41:19,566 INFO [collect_river_data] publishing log messages to tcp://*:5567
2026-03-08 08:41:19,567 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-08 08:41:19,567 INFO [collect_river_data] Collecting USGS SnohomishMonroe river data for 2026-03-07
2026-03-08 08:41:19,736 INFO [collect_river_data] running in process 1184313
2026-03-08 08:41:19,742 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-08 08:41:19,743 INFO [collect_river_data] publishing log messages to tcp://*:5568
2026-03-08 08:41:19,744 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-08 08:41:19,744 INFO [collect_river_data] Collecting USGS GreenwaterGreenwater river data for 2026-03-07
2026-03-08 08:41:19,748 INFO [make_turbidity_file] running in process 1184314
2026-03-08 08:41:19,750 INFO [make_turbidity_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-08 08:41:19,751 INFO [make_turbidity_file] publishing log messages to tcp://*:5569
2026-03-08 08:41:19,752 INFO [make_turbidity_file] connected to skookum.eos.ubc.ca port 5555
2026-03-08 08:41:19,752 INFO [make_turbidity_file] Creating Fraser River turbidity forcing file for 2026-03-08
2026-03-08 08:41:19,881 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12150800&startDT=2026-03-07&endDT=2026-03-07 "HTTP/1.1 200 "
2026-03-08 08:41:19,884 INFO [collect_river_data] Appended USGS SnohomishMonroe river average discharge for 2026-03-07 to: /results/forcing/rivers/observations/Snohomish_Monroe_flow
2026-03-08 08:41:19,885 INFO [collect_river_data] USGS SnohomishMonroe river data collection for 2026-03-07 completed
2026-03-08 08:41:19,889 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-08 08:41:19,896 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12089500&startDT=2026-03-07&endDT=2026-03-07 "HTTP/1.1 200 "
2026-03-08 08:41:19,899 INFO [collect_river_data] Appended USGS NisquallyMcKenna river average discharge for 2026-03-07 to: /results/forcing/rivers/observations/Nisqually_McKenna_flow
2026-03-08 08:41:19,900 INFO [collect_river_data] USGS NisquallyMcKenna river data collection for 2026-03-07 completed
2026-03-08 08:41:19,901 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12200500&startDT=2026-03-07&endDT=2026-03-07 "HTTP/1.1 200 "
2026-03-08 08:41:19,904 INFO [collect_river_data] Appended USGS SkagitMountVernon river average discharge for 2026-03-07 to: /results/forcing/rivers/observations/Skagit_MountVernon_flow
2026-03-08 08:41:19,905 INFO [collect_river_data] USGS SkagitMountVernon river data collection for 2026-03-07 completed
2026-03-08 08:41:19,907 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12097500&startDT=2026-03-07&endDT=2026-03-07 "HTTP/1.1 200 "
2026-03-08 08:41:19,911 INFO [collect_river_data] Appended USGS GreenwaterGreenwater river average discharge for 2026-03-07 to: /results/forcing/rivers/observations/Greenwater_Greenwater_flow
2026-03-08 08:41:19,912 INFO [collect_river_data] USGS GreenwaterGreenwater river data collection for 2026-03-07 completed
2026-03-08 08:41:19,925 INFO [crop_gribs] running in process 1184318
2026-03-08 08:41:19,925 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-08 08:41:19,926 INFO [crop_gribs] publishing log messages to tcp://*:5570
2026-03-08 08:41:19,927 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2026-03-08 08:41:19,928 INFO [crop_gribs] cropping 2026-03-08 ECCC HRDPS 2.5km continental 18Z GRIB files to SalishSeaCast subdomain
2026-03-08 08:41:19,938 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20260308/18/
2026-03-08 08:41:20,079 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-08 08:41:20,141 INFO [collect_NeahBay_ssh] 2026-03-08T00:00:00-08:00 Neah Bay ssh 06Z obs/forecast data collection complete
2026-03-08 08:41:20,212 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-08 08:41:20,296 INFO [grib_to_netcdf] running in process 1184396
2026-03-08 08:41:20,297 INFO [grib_to_netcdf] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-08 08:41:20,298 INFO [grib_to_netcdf] publishing log messages to tcp://*:5571
2026-03-08 08:41:20,299 INFO [grib_to_netcdf] connected to skookum.eos.ubc.ca port 5555
2026-03-08 08:41:20,300 INFO [grib_to_netcdf] creating NEMO-atmos forcing files for 2026-03-08 nowcast and forecast runs
2026-03-08 08:41:20,379 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-08 08:41:20,515 INFO [manager] checklist updated with [Neah Bay ssh data] items from collect_NeahBay_ssh worker
2026-03-08 08:41:20,695 INFO [manager] launching NextWorker(module='nowcast.workers.make_ssh_files', args=['nowcast', '--run-date', '2026-03-08'], host='localhost')
2026-03-08 08:41:22,040 INFO [make_turbidity_file] 2026-03-08 Fraser River turbidity file creation complete
2026-03-08 08:41:22,042 INFO [manager] checklist updated with [Fraser River turbidity] items from make_turbidity_file worker
2026-03-08 08:41:23,610 INFO [make_ssh_files] running in process 1184453
2026-03-08 08:41:23,611 INFO [make_ssh_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-08 08:41:23,612 INFO [make_ssh_files] publishing log messages to tcp://*:5562
2026-03-08 08:41:23,614 INFO [make_ssh_files] connected to skookum.eos.ubc.ca port 5555
2026-03-08 08:41:23,614 INFO [make_ssh_files] building nowcast sea surface height boundary conditions file(s) from 2026-03-08 NOAA Neah Bay 06Z observation and forecast values
2026-03-08 08:41:25,211 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2026m03d04.nc
2026-03-08 08:41:25,408 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2026m03d05.nc
2026-03-08 08:41:25,597 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2026m03d06.nc
2026-03-08 08:41:25,786 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2026m03d07.nc
2026-03-08 08:41:25,994 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2026m03d08.nc
2026-03-08 08:41:26,184 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2026m03d09.nc
2026-03-08 08:41:26,359 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2026m03d10.nc
2026-03-08 08:41:26,555 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2026m03d11.nc
2026-03-08 08:41:26,674 INFO [make_ssh_files] sea surface height boundary file for 2026-03-08 nowcast run created
2026-03-08 08:41:26,677 INFO [manager] checklist updated with [sea surface height forcing] items from make_ssh_files worker
2026-03-08 08:41:26,848 INFO [manager] launching NextWorker(module='nowcast.workers.make_runoff_file', args=['v202108'], host='localhost')
2026-03-08 08:41:26,853 INFO [manager] launching NextWorker(module='nowcast.workers.make_201702_runoff_file', args=[], host='localhost')
2026-03-08 08:41:28,143 INFO [make_201702_runoff_file] running in process 1184508
2026-03-08 08:41:28,144 INFO [make_201702_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-08 08:41:28,145 INFO [make_201702_runoff_file] publishing log messages to tcp://*:5562
2026-03-08 08:41:28,146 INFO [make_201702_runoff_file] connected to skookum.eos.ubc.ca port 5555
2026-03-08 08:41:28,170 INFO [make_201702_runoff_file] calculating runoff forcing for b201702 bathymetry
2026-03-08 08:41:28,275 INFO [make_201702_runoff_file] 2026-03-08 runoff file creation from Fraser at Hope and climatology elsewhere complete
2026-03-08 08:41:28,277 INFO [manager] checklist updated with [rivers forcing] items from make_201702_runoff_file worker
2026-03-08 08:41:28,783 INFO [make_runoff_file] running in process 1184506
2026-03-08 08:41:28,784 INFO [make_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-08 08:41:28,786 INFO [make_runoff_file] publishing log messages to tcp://*:5562
2026-03-08 08:41:28,787 INFO [make_runoff_file] connected to skookum.eos.ubc.ca port 5555
2026-03-08 08:41:28,788 INFO [make_runoff_file] calculating NEMO runoff forcing for v202108 bathymetry for 2026-03-07
2026-03-08 08:41:31,245 INFO [make_runoff_file] stored NEMO runoff forcing for v202108 bathymetry for 2026-03-07: /results/forcing/rivers/R202108Dailies_y2026m03d07.nc
2026-03-08 08:41:31,246 INFO [make_runoff_file] 2026-03-07 runoff file creation completed
2026-03-08 08:41:31,248 INFO [manager] checklist updated with [rivers forcing] items from make_runoff_file worker
2026-03-08 08:41:32,768 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/hrdps_y2026m03d08.nc
2026-03-08 08:41:41,673 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2026m03d09.nc
2026-03-08 08:41:46,562 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2026m03d10.nc
2026-03-08 08:41:46,576 INFO [grib_to_netcdf] 2026-03-08 NEMO-atmos forcing file for nowcast+ created
2026-03-08 08:41:46,578 INFO [manager] checklist updated with [weather forcing] items from grib_to_netcdf worker
2026-03-08 09:21:20,581 INFO [download_live_ocean] downloaded 10196978 bytes from https://s3.kopah.uw.edu/liveocean-share/f2026.03.08/ubc.nc to /results/forcing/LiveOcean/downloaded/20260308/low_passed_UBC.nc
2026-03-08 09:21:20,582 INFO [download_live_ocean] Deflating in up to 1 concurrent sub-processes
2026-03-08 09:21:21,612 INFO [download_live_ocean] netCDF4 deflated /results/forcing/LiveOcean/downloaded/20260308/low_passed_UBC.nc
2026-03-08 09:21:21,613 INFO [download_live_ocean] 2026-03-08 Live Ocean file for Salish Sea western boundary downloaded
2026-03-08 09:21:21,615 INFO [manager] checklist updated with [Live Ocean products] items from download_live_ocean worker
2026-03-08 09:21:21,927 INFO [manager] launching NextWorker(module='nowcast.workers.make_live_ocean_files', args=['--run-date', '2026-03-08'], host='localhost')
2026-03-08 09:21:24,660 INFO [make_live_ocean_files] running in process 1185098
2026-03-08 09:21:24,661 INFO [make_live_ocean_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-08 09:21:24,662 INFO [make_live_ocean_files] publishing log messages to tcp://*:5562
2026-03-08 09:21:24,664 INFO [make_live_ocean_files] connected to skookum.eos.ubc.ca port 5555
2026-03-08 09:21:24,665 INFO [make_live_ocean_files] Creating T&S western boundary conditions file from 2026-03-08 Live Ocean run
2026-03-08 09:21:59,566 INFO [make_live_ocean_files] Stored T&S western boundary conditions file: /results/forcing/LiveOcean/boundary_conditions/LiveOcean_v201905_y2026m03d08.nc
2026-03-08 09:21:59,567 INFO [make_live_ocean_files] 2026-03-08 Live Ocean western boundary conditions files created
2026-03-08 09:21:59,569 INFO [manager] checklist updated with [Live Ocean boundary conditions] items from make_live_ocean_files worker
2026-03-08 09:21:59,865 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['weather'], host='localhost')
2026-03-08 09:21:59,866 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'nowcast+'], host='localhost')
2026-03-08 09:21:59,867 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'nowcast+'], host='localhost')
2026-03-08 09:21:59,868 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.nibi', 'nowcast+'], host='localhost')
2026-03-08 09:21:59,870 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'nowcast+'], host='localhost')
2026-03-08 09:22:00,911 INFO [ping_erddap] running in process 1185131
2026-03-08 09:22:00,912 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-08 09:22:00,913 INFO [ping_erddap] publishing log messages to tcp://*:5562
2026-03-08 09:22:00,915 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2026-03-08 09:22:00,916 INFO [ping_erddap] weather ERDDAP dataset flag file(s) created
2026-03-08 09:22:00,918 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2026-03-08 09:22:00,958 INFO [upload_forcing] running in process 1185135
2026-03-08 09:22:00,959 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-08 09:22:00,960 INFO [upload_forcing] publishing log messages to tcp://*:5564
2026-03-08 09:22:00,962 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-03-08 09:22:00,984 INFO [upload_forcing] running in process 1185133
2026-03-08 09:22:00,986 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-08 09:22:00,987 INFO [upload_forcing] publishing log messages to tcp://*:5565
2026-03-08 09:22:00,988 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-03-08 09:22:01,038 INFO [upload_forcing] running in process 1185136
2026-03-08 09:22:01,039 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-08 09:22:01,040 INFO [upload_forcing] publishing log messages to tcp://*:5566
2026-03-08 09:22:01,041 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-03-08 09:22:01,086 INFO [upload_forcing] running in process 1185134
2026-03-08 09:22:01,087 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-08 09:22:01,088 INFO [upload_forcing] publishing log messages to tcp://*:5567
2026-03-08 09:22:01,089 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-03-08 09:22:02,792 INFO [upload_forcing] nowcast+ 2026-03-08 forcing files upload to optimum-hindcast completed
2026-03-08 09:22:02,795 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-03-08 09:22:05,191 INFO [upload_forcing] nowcast+ 2026-03-08 forcing files upload to arbutus.cloud-nowcast completed
2026-03-08 09:22:05,196 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-03-08 09:22:05,348 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'nowcast+', '--run-date', '2026-03-08'], host='localhost')
2026-03-08 09:22:06,638 INFO [make_forcing_links] running in process 1185165
2026-03-08 09:22:06,639 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-08 09:22:06,640 INFO [make_forcing_links] publishing log messages to tcp://*:5562
2026-03-08 09:22:06,641 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2026-03-08 09:22:07,733 INFO [make_forcing_links] nowcast+ 2026-03-08 forcing file links on arbutus.cloud-nowcast created
2026-03-08 09:22:07,737 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2026-03-08 09:22:07,895 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2026-03-08'], host='arbutus.cloud-nowcast')
2026-03-08 09:22:08,593 INFO [upload_forcing] nowcast+ 2026-03-08 forcing files upload to orcinus-nowcast-agrif completed
2026-03-08 09:22:08,597 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-03-08 09:22:09,974 INFO [run_NEMO] running in process 10014
2026-03-08 09:22:09,976 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-08 09:22:09,977 INFO [run_NEMO] publishing log messages to tcp://*:5556
2026-03-08 09:22:10,036 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2026-03-08 09:22:10,813 WARNING [run_NEMO] There are uncommitted changes in /nemoShare/MEOPAR/nowcast-sys/SS-run-sets
2026-03-08 09:22:11,124 INFO [run_NEMO] nowcast: launching /nemoShare/MEOPAR/nowcast-sys/runs/08mar26nowcast_2026-03-08T092210.025113-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2026-03-08 09:22:11,139 INFO [run_NEMO] nowcast NEMO run for 2026-03-08 on arbutus.cloud-nowcast started
2026-03-08 09:22:11,143 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2026-03-08 09:22:11,298 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'nowcast'], host='arbutus.cloud-nowcast')
2026-03-08 09:22:11,531 INFO [upload_forcing] nowcast+ 2026-03-08 forcing files upload to robot.nibi completed
2026-03-08 09:22:11,535 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-03-08 09:22:13,323 INFO [watch_NEMO] running in process 10288
2026-03-08 09:22:13,325 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-08 09:22:13,326 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2026-03-08 09:22:13,328 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2026-03-08 09:22:13,562 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2026-03-08 09:27:13,559 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 19084090 = 2026-03-08 05:26:00 UTC, 22.6% complete
2026-03-08 09:32:13,560 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 19084636 = 2026-03-08 11:30:00 UTC, 47.9% complete
2026-03-08 09:37:13,561 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 19085184 = 2026-03-08 17:35:20 UTC, 73.3% complete
2026-03-08 09:42:13,562 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 19085715 = 2026-03-08 23:29:20 UTC, 97.9% complete
2026-03-08 09:47:13,567 INFO [watch_NEMO] nowcast NEMO run on arbutus.cloud-nowcast completed
2026-03-08 09:47:13,570 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2026-03-08 09:47:13,945 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'ssh', '--run-date', '2026-03-08'], host='localhost')
2026-03-08 09:47:13,952 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2026-03-08'], host='localhost')
2026-03-08 09:47:15,125 INFO [download_results] running in process 1185508
2026-03-08 09:47:15,127 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-08 09:47:15,127 INFO [download_results] publishing log messages to tcp://*:5562
2026-03-08 09:47:15,129 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2026-03-08 09:47:15,130 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/nowcast/08mar26 to /results/SalishSea/nowcast-blue.202111
2026-03-08 09:47:15,258 INFO [make_forcing_links] running in process 1185507
2026-03-08 09:47:15,259 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-08 09:47:15,260 INFO [make_forcing_links] publishing log messages to tcp://*:5564
2026-03-08 09:47:15,261 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2026-03-08 09:47:16,224 INFO [make_forcing_links] ssh 2026-03-08 forcing file links on arbutus.cloud-nowcast created
2026-03-08 09:47:16,228 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2026-03-08 09:47:16,441 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2026-03-08'], host='arbutus.cloud-nowcast')
2026-03-08 09:47:18,595 INFO [run_NEMO] running in process 10641
2026-03-08 09:47:18,599 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-08 09:47:18,600 INFO [run_NEMO] publishing log messages to tcp://*:5556
2026-03-08 09:47:18,602 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2026-03-08 09:47:19,355 WARNING [run_NEMO] There are uncommitted changes in /nemoShare/MEOPAR/nowcast-sys/SS-run-sets
2026-03-08 09:47:19,675 INFO [run_NEMO] forecast: launching /nemoShare/MEOPAR/nowcast-sys/runs/08mar26forecast_2026-03-08T094718.668902-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2026-03-08 09:47:19,689 INFO [run_NEMO] forecast NEMO run for 2026-03-08 on arbutus.cloud-nowcast started
2026-03-08 09:47:19,693 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2026-03-08 09:47:19,849 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'forecast'], host='arbutus.cloud-nowcast')
2026-03-08 09:47:21,975 INFO [watch_NEMO] running in process 10914
2026-03-08 09:47:21,977 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-08 09:47:21,978 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2026-03-08 09:47:21,980 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2026-03-08 09:47:22,056 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2026-03-08 09:47:53,933 INFO [download_results] nowcast 2026-03-08 results files from arbutus.cloud-nowcast downloaded
2026-03-08 09:47:53,936 INFO [manager] checklist updated with [results files] items from download_results worker
2026-03-08 09:47:54,095 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast', 'research', '--run-date', '2026-03-08'], host='localhost')
2026-03-08 09:47:54,101 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast', 'comparison', '--run-date', '2026-03-07'], host='localhost')
2026-03-08 09:47:54,106 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['nowcast', '--run-date', '2026-03-08'], host='localhost')
2026-03-08 09:47:56,077 INFO [make_CHS_currents_file] running in process 1185531
2026-03-08 09:47:56,079 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-08 09:47:56,079 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5562
2026-03-08 09:47:56,080 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555
2026-03-08 09:47:57,062 INFO [make_plots] running in process 1185530
2026-03-08 09:47:57,064 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-08 09:47:57,065 INFO [make_plots] publishing log messages to tcp://*:5564
2026-03-08 09:47:57,066 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2026-03-08 09:47:57,067 INFO [make_plots] running in process 1185528
2026-03-08 09:47:57,068 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-08 09:47:57,069 INFO [make_plots] publishing log messages to tcp://*:5565
2026-03-08 09:47:57,070 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2026-03-08 09:47:57,430 INFO [make_plots] preparing render list for 2026-03-08 NEMO nowcast-blue research figures
2026-03-08 09:47:57,431 INFO [make_plots] preparing render list for 2026-03-07 NEMO nowcast-blue comparison figures
2026-03-08 09:47:57,476 INFO [make_plots] added SH_wind figure to 2026-03-07 NEMO nowcast-blue comparison render list
2026-03-08 09:47:57,477 INFO [make_plots] added Compare_VENUS_East figure to 2026-03-07 NEMO nowcast-blue comparison render list
2026-03-08 09:47:57,478 INFO [make_plots] added Compare_VENUS_Central figure to 2026-03-07 NEMO nowcast-blue comparison render list
2026-03-08 09:47:57,479 INFO [make_plots] starting to render nemo nowcast comparison 07mar26 figures
2026-03-08 09:47:58,948 INFO [make_plots] added salinity_thalweg_and_surface figures to 2026-03-08 NEMO nowcast-blue research render list
2026-03-08 09:47:59,711 INFO [make_plots] added temperature_thalweg_and_surface figures to 2026-03-08 NEMO nowcast-blue research render list
2026-03-08 09:47:59,712 INFO [make_plots] added Currents_sections_and_surface figure to 2026-03-08 NEMO nowcast-blue research render list
2026-03-08 09:47:59,714 INFO [make_plots] added Currents_at_VENUS_Central figure to 2026-03-08 NEMO nowcast-blue research render list
2026-03-08 09:47:59,715 INFO [make_plots] added Currents_at_VENUS_East figure to 2026-03-08 NEMO nowcast-blue research render list
2026-03-08 09:47:59,716 INFO [make_plots] starting to render nemo nowcast research 08mar26 figures
2026-03-08 09:48:27,245 INFO [make_CHS_currents_file] Made CHS currents file for 2026-03-08 for nowcast
2026-03-08 09:48:27,248 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker
2026-03-08 09:48:58,381 INFO [make_plots] finished rendering nemo nowcast research 08mar26 figures
2026-03-08 09:48:58,385 INFO [make_plots] nemo research plots for 2026-03-08 nowcast completed
2026-03-08 09:48:58,388 INFO [manager] checklist updated with [plots] items from make_plots worker
2026-03-08 09:49:23,215 INFO [make_plots] finished rendering nemo nowcast comparison 07mar26 figures
2026-03-08 09:49:23,218 INFO [make_plots] nemo comparison plots for 2026-03-07 nowcast completed
2026-03-08 09:49:23,222 INFO [manager] checklist updated with [plots] items from make_plots worker
2026-03-08 09:52:22,052 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 19086260 = 2026-03-09 05:32:40 UTC, 15.4% complete
2026-03-08 09:57:22,054 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 19086797 = 2026-03-09 11:30:40 UTC, 32.0% complete
2026-03-08 10:02:22,056 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 19087304 = 2026-03-09 17:08:40 UTC, 47.6% complete
2026-03-08 10:07:22,057 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 19087844 = 2026-03-09 23:08:40 UTC, 64.3% complete
2026-03-08 10:12:22,058 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 19088362 = 2026-03-10 04:54:00 UTC, 80.3% complete
2026-03-08 10:17:22,060 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 19088897 = 2026-03-10 10:50:40 UTC, 96.8% complete
2026-03-08 10:22:22,064 INFO [watch_NEMO] forecast NEMO run on arbutus.cloud-nowcast completed
2026-03-08 10:22:22,068 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2026-03-08 10:22:22,389 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_wind_file', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2026-03-08'], host='arbutus.cloud-nowcast')
2026-03-08 10:22:22,406 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_current_file', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2026-03-08'], host='arbutus.cloud-nowcast')
2026-03-08 10:22:22,411 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'turbidity', '--run-date', '2026-03-08'], host='localhost')
2026-03-08 10:22:22,417 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'turbidity', '--run-date', '2026-03-08'], host='localhost')
2026-03-08 10:22:22,423 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.nibi', 'turbidity', '--run-date', '2026-03-08'], host='localhost')
2026-03-08 10:22:22,429 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'turbidity', '--run-date', '2026-03-08'], host='localhost')
2026-03-08 10:22:22,433 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2026-03-08'], host='localhost')
2026-03-08 10:22:24,186 INFO [download_results] running in process 1185940
2026-03-08 10:22:24,188 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-08 10:22:24,189 INFO [download_results] publishing log messages to tcp://*:5562
2026-03-08 10:22:24,190 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2026-03-08 10:22:24,191 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/forecast/08mar26 to /results/SalishSea/forecast.202111
2026-03-08 10:22:24,276 INFO [upload_forcing] running in process 1185938
2026-03-08 10:22:24,277 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-08 10:22:24,278 INFO [upload_forcing] publishing log messages to tcp://*:5564
2026-03-08 10:22:24,280 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-03-08 10:22:24,310 INFO [upload_forcing] running in process 1185937
2026-03-08 10:22:24,311 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-08 10:22:24,313 INFO [upload_forcing] publishing log messages to tcp://*:5565
2026-03-08 10:22:24,314 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-03-08 10:22:24,315 INFO [upload_forcing] running in process 1185935
2026-03-08 10:22:24,316 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-08 10:22:24,317 INFO [upload_forcing] publishing log messages to tcp://*:5566
2026-03-08 10:22:24,318 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-03-08 10:22:24,320 INFO [upload_forcing] running in process 1185936
2026-03-08 10:22:24,321 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-08 10:22:24,321 INFO [upload_forcing] publishing log messages to tcp://*:5567
2026-03-08 10:22:24,322 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-03-08 10:22:24,929 INFO [upload_forcing] turbidity 2026-03-08 forcing files upload to optimum-hindcast completed
2026-03-08 10:22:24,933 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-03-08 10:22:25,261 INFO [make_ww3_wind_file] running in process 11482
2026-03-08 10:22:25,262 INFO [make_ww3_wind_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-08 10:22:25,263 INFO [make_ww3_wind_file] publishing log messages to tcp://*:5570
2026-03-08 10:22:25,264 INFO [make_ww3_wind_file] connected to skookum.eos.ubc.ca port 5555
2026-03-08 10:22:25,358 INFO [make_ww3_wind_file] Creating wwatch3 wind forcing file for 2026-03-08 forecast run
2026-03-08 10:22:25,369 INFO [upload_forcing] turbidity 2026-03-08 forcing files upload to arbutus.cloud-nowcast completed
2026-03-08 10:22:25,370 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-03-08 10:22:25,384 INFO [make_ww3_current_file] running in process 11484
2026-03-08 10:22:25,386 INFO [make_ww3_current_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-08 10:22:25,387 INFO [make_ww3_current_file] publishing log messages to tcp://*:5571
2026-03-08 10:22:25,389 INFO [make_ww3_current_file] connected to skookum.eos.ubc.ca port 5555
2026-03-08 10:22:25,391 INFO [make_ww3_current_file] Creating wwatch3 currents forcing file for 2026-03-08 forecast run
2026-03-08 10:22:25,535 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2026-03-08'], host='localhost')
2026-03-08 10:22:28,121 INFO [make_forcing_links] running in process 1185973
2026-03-08 10:22:28,123 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-08 10:22:28,124 INFO [make_forcing_links] publishing log messages to tcp://*:5564
2026-03-08 10:22:28,125 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2026-03-08 10:22:28,953 INFO [upload_forcing] turbidity 2026-03-08 forcing files upload to robot.nibi completed
2026-03-08 10:22:28,958 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-03-08 10:22:30,375 INFO [upload_forcing] turbidity 2026-03-08 forcing files upload to orcinus-nowcast-agrif completed
2026-03-08 10:22:30,379 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-03-08 10:22:30,645 INFO [make_forcing_links] nowcast-green 2026-03-08 forcing file links on arbutus.cloud-nowcast created
2026-03-08 10:22:30,648 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2026-03-08 10:22:30,830 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2026-03-08'], host='arbutus.cloud-nowcast')
2026-03-08 10:22:35,056 INFO [run_NEMO] running in process 12111
2026-03-08 10:22:35,058 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-08 10:22:35,060 INFO [run_NEMO] publishing log messages to tcp://*:5556
2026-03-08 10:22:35,061 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2026-03-08 10:22:36,258 WARNING [run_NEMO] There are uncommitted changes in /nemoShare/MEOPAR/nowcast-sys/SS-run-sets
2026-03-08 10:22:36,612 INFO [run_NEMO] nowcast-green: launching /nemoShare/MEOPAR/nowcast-sys/runs/08mar26nowcast-green_2026-03-08T102235.163200-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2026-03-08 10:22:36,629 INFO [run_NEMO] nowcast-green NEMO run for 2026-03-08 on arbutus.cloud-nowcast started
2026-03-08 10:22:36,633 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2026-03-08 10:22:36,804 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'nowcast-green'], host='arbutus.cloud-nowcast')
2026-03-08 10:22:37,919 INFO [make_ww3_wind_file] wwatch3 wind forcing file created on arbutus.cloud-nowcast for 2026-03-08 forecast run
2026-03-08 10:22:37,924 INFO [manager] checklist updated with [WW3 wind forcing] items from make_ww3_wind_file worker
2026-03-08 10:22:39,237 INFO [watch_NEMO] running in process 12405
2026-03-08 10:22:39,238 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-08 10:22:39,240 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2026-03-08 10:22:39,242 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2026-03-08 10:22:39,321 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2026-03-08 10:23:01,983 INFO [download_results] forecast 2026-03-08 results files from arbutus.cloud-nowcast downloaded
2026-03-08 10:23:01,986 INFO [manager] checklist updated with [results files] items from download_results worker
2026-03-08 10:23:02,141 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['forecast', '--run-date', '2026-03-08'], host='localhost')
2026-03-08 10:23:05,033 INFO [make_CHS_currents_file] running in process 1186026
2026-03-08 10:23:05,034 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-08 10:23:05,035 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5562
2026-03-08 10:23:05,036 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555
2026-03-08 10:23:47,302 INFO [make_CHS_currents_file] Made CHS currents file for 2026-03-08 for forecast
2026-03-08 10:23:47,305 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker
2026-03-08 10:23:47,464 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['nemo', 'forecast', '--run-date', '2026-03-08'], host='localhost')
2026-03-08 10:23:48,632 INFO [update_forecast_datasets] running in process 1186057
2026-03-08 10:23:48,633 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-08 10:23:48,635 INFO [update_forecast_datasets] publishing log messages to tcp://*:5562
2026-03-08 10:23:48,636 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2026-03-08 10:23:48,637 INFO [update_forecast_datasets] updating nemo forecast directory for forecast/08mar26 run
2026-03-08 10:23:48,941 INFO [update_forecast_datasets] updated nemo forecast directory for forecast/08mar26 run: /results/SalishSea/rolling-forecasts/nemo
2026-03-08 10:23:48,943 INFO [update_forecast_datasets] nemo 2026-03-08 forecast rolling forecast datasets updated
2026-03-08 10:23:48,944 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2026-03-08 10:23:49,102 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nemo-forecast'], host='localhost')
2026-03-08 10:23:49,107 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'forecast', 'publish', '--run-date', '2026-03-08'], host='localhost')
2026-03-08 10:23:49,112 INFO [manager] launching NextWorker(module='nowcast.workers.make_surface_current_tiles', args=['forecast', '--run-date', '2026-03-08'], host='localhost')
2026-03-08 10:23:50,247 INFO [ping_erddap] running in process 1186064
2026-03-08 10:23:50,248 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-08 10:23:50,249 INFO [ping_erddap] publishing log messages to tcp://*:5562
2026-03-08 10:23:50,251 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2026-03-08 10:23:50,260 INFO [ping_erddap] nemo-forecast ERDDAP dataset flag file(s) created
2026-03-08 10:23:50,261 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2026-03-08 10:23:51,643 INFO [make_surface_current_tiles] running in process 1186067
2026-03-08 10:23:51,644 INFO [make_surface_current_tiles] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-08 10:23:51,646 INFO [make_surface_current_tiles] publishing log messages to tcp://*:5562
2026-03-08 10:23:51,647 INFO [make_surface_current_tiles] connected to skookum.eos.ubc.ca port 5555
2026-03-08 10:23:51,649 INFO [make_surface_current_tiles] starting rendering of tiles for 2026-03-08 forecast in 6 processes into /results/nowcast-sys/figures/surface_currents/forecast/08mar26
2026-03-08 10:23:53,689 INFO [make_plots] running in process 1186066
2026-03-08 10:23:53,690 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-08 10:23:53,693 INFO [make_plots] publishing log messages to tcp://*:5564
2026-03-08 10:23:53,694 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2026-03-08 10:23:54,142 INFO [make_plots] preparing render list for 2026-03-08 NEMO forecast publish figures
2026-03-08 10:23:54,397 INFO [make_plots] added Website_thumbnail figure to 2026-03-08 NEMO forecast publish render list
2026-03-08 10:23:54,398 INFO [make_plots] added Threshold_website figure to 2026-03-08 NEMO forecast publish render list
2026-03-08 10:23:54,399 INFO [make_plots] added PA_tidal_predictions figure to 2026-03-08 NEMO forecast publish render list
2026-03-08 10:23:54,400 INFO [make_plots] added Boundary Bay figure to 2026-03-08 NEMO forecast publish render list
2026-03-08 10:23:54,401 INFO [make_plots] added Campbell River figure to 2026-03-08 NEMO forecast publish render list
2026-03-08 10:23:54,402 INFO [make_plots] added Cherry Point figure to 2026-03-08 NEMO forecast publish render list
2026-03-08 10:23:54,403 INFO [make_plots] added Friday Harbor figure to 2026-03-08 NEMO forecast publish render list
2026-03-08 10:23:54,404 INFO [make_plots] added Halfmoon Bay figure to 2026-03-08 NEMO forecast publish render list
2026-03-08 10:23:54,405 INFO [make_plots] added Nanaimo figure to 2026-03-08 NEMO forecast publish render list
2026-03-08 10:23:54,406 INFO [make_plots] added Neah Bay figure to 2026-03-08 NEMO forecast publish render list
2026-03-08 10:23:54,407 INFO [make_plots] added New Westminster figure to 2026-03-08 NEMO forecast publish render list
2026-03-08 10:23:54,407 INFO [make_plots] added Patricia Bay figure to 2026-03-08 NEMO forecast publish render list
2026-03-08 10:23:54,408 INFO [make_plots] added Point Atkinson figure to 2026-03-08 NEMO forecast publish render list
2026-03-08 10:23:54,409 INFO [make_plots] added Port Renfrew figure to 2026-03-08 NEMO forecast publish render list
2026-03-08 10:23:54,410 INFO [make_plots] added Sand Heads figure to 2026-03-08 NEMO forecast publish render list
2026-03-08 10:23:54,411 INFO [make_plots] added Sandy Cove figure to 2026-03-08 NEMO forecast publish render list
2026-03-08 10:23:54,412 INFO [make_plots] added Squamish figure to 2026-03-08 NEMO forecast publish render list
2026-03-08 10:23:54,413 INFO [make_plots] added Victoria figure to 2026-03-08 NEMO forecast publish render list
2026-03-08 10:23:54,413 INFO [make_plots] added Woodwards Landing figure to 2026-03-08 NEMO forecast publish render list
2026-03-08 10:23:54,414 INFO [make_plots] starting to render nemo forecast publish 08mar26 figures
2026-03-08 10:25:02,065 INFO [make_ww3_current_file] wwatch3 currents forcing file created on arbutus.cloud-nowcast for 2026-03-08 forecast run
2026-03-08 10:25:02,069 INFO [manager] checklist updated with [WW3 currents forcing] items from make_ww3_current_file worker
2026-03-08 10:25:02,379 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2026-03-08'], host='arbutus.cloud-nowcast')
2026-03-08 10:25:04,387 INFO [run_ww3] running in process 12505
2026-03-08 10:25:04,388 INFO [run_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-08 10:25:04,389 INFO [run_ww3] publishing log messages to tcp://*:5572
2026-03-08 10:25:04,389 INFO [run_ww3] connected to skookum.eos.ubc.ca port 5555
2026-03-08 10:25:04,393 INFO [run_ww3] Created run directory /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/nowcast_2026-03-08T102504.389855-0700
2026-03-08 10:25:04,394 INFO [run_ww3] nowcast: launching /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/nowcast_2026-03-08T102504.389855-0700/SoGWW3.sh on arbutus.cloud-nowcast
2026-03-08 10:25:04,561 INFO [run_ww3] nowcast WaveWatch3 run for 2026-03-08 on arbutus.cloud-nowcast started
2026-03-08 10:25:04,562 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker
2026-03-08 10:25:04,720 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'nowcast'], host='arbutus.cloud-nowcast')
2026-03-08 10:25:06,736 INFO [watch_ww3] running in process 12592
2026-03-08 10:25:06,737 INFO [watch_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-08 10:25:06,738 INFO [watch_ww3] publishing log messages to tcp://*:5573
2026-03-08 10:25:06,739 INFO [watch_ww3] connected to skookum.eos.ubc.ca port 5555
2026-03-08 10:25:06,773 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2026-03-08 10:27:39,322 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19083774 = 2026-03-08 01:55:20 UTC, 8.0% complete
2026-03-08 10:30:06,775 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 93, 21.5% complete
2026-03-08 10:31:38,681 WARNING [make_plots] invalid station number for Boundary Bay station: None
2026-03-08 10:31:38,682 WARNING [make_plots] can't resolve a valid CHS station code for Boundary Bay
2026-03-08 10:32:01,996 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb996/data for station 08074 Campbell River from 2026-03-03 00:00:00Z to 2026-03-10 11:55:00Z
2026-03-08 10:32:39,324 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19083966 = 2026-03-08 04:03:20 UTC, 16.9% complete
2026-03-08 10:33:04,343 WARNING [make_plots] invalid station number for Halfmoon Bay station: None
2026-03-08 10:33:04,344 WARNING [make_plots] can't resolve a valid CHS station code for Halfmoon Bay
2026-03-08 10:33:26,064 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb96d/data for station 07917 Nanaimo from 2026-03-03 00:00:00Z to 2026-03-10 11:55:00Z
2026-03-08 10:33:59,871 INFO [make_surface_current_tiles] starting PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast/08mar26
2026-03-08 10:34:02,509 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e13d0f4a073c4bbf93/data for station 07654 New Westminster from 2026-03-03 00:00:00Z to 2026-03-10 11:55:00Z
2026-03-08 10:34:20,117 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd2d/data for station 07277 Patricia Bay from 2026-03-03 00:00:00Z to 2026-03-10 11:55:00Z
2026-03-08 10:34:37,485 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94c/data for station 07795 Point Atkinson from 2026-03-03 00:00:00Z to 2026-03-10 11:55:00Z
2026-03-08 10:34:54,168 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e23d0f4a073c4bc060/data for station 08525 Port Renfrew from 2026-03-03 00:00:00Z to 2026-03-10 11:55:00Z
2026-03-08 10:35:06,777 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 192, 44.4% complete
2026-03-08 10:35:12,399 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb93c/data for station 07594 Sand Heads from 2026-03-03 00:00:00Z to 2026-03-10 11:55:00Z
2026-03-08 10:35:29,265 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc43b/data for station 07786 Sandy Cove from 2026-03-03 00:00:00Z to 2026-03-10 11:55:00Z
2026-03-08 10:35:45,452 WARNING [make_plots] invalid station number for Squamish station: None
2026-03-08 10:35:45,453 WARNING [make_plots] can't resolve a valid CHS station code for Squamish
2026-03-08 10:36:03,213 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd1e/data for station 07120 Victoria from 2026-03-03 00:00:00Z to 2026-03-10 11:55:00Z
2026-03-08 10:36:20,430 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc42d/data for station 07610 Woodwards Landing from 2026-03-03 00:00:00Z to 2026-03-10 11:55:00Z
2026-03-08 10:36:36,511 INFO [make_plots] finished rendering nemo forecast publish 08mar26 figures
2026-03-08 10:36:36,514 INFO [make_plots] nemo publish plots for 2026-03-08 forecast completed
2026-03-08 10:36:36,521 INFO [manager] checklist updated with [plots] items from make_plots worker
2026-03-08 10:36:36,660 INFO [manager] launching NextWorker(module='nowcast.workers.make_feeds', args=['forecast', '--run-date', '2026-03-08'], host='localhost')
2026-03-08 10:36:39,842 INFO [make_feeds] running in process 1190759
2026-03-08 10:36:39,844 INFO [make_feeds] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-08 10:36:39,845 INFO [make_feeds] publishing log messages to tcp://*:5564
2026-03-08 10:36:39,846 INFO [make_feeds] connected to skookum.eos.ubc.ca port 5555
2026-03-08 10:36:54,156 INFO [make_feeds] ATOM feeds for 2026-03-08 forecast run completed
2026-03-08 10:36:54,158 INFO [manager] checklist updated with [storm surge alert feeds] items from make_feeds worker
2026-03-08 10:37:39,326 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19084159 = 2026-03-08 06:12:00 UTC, 25.8% complete
2026-03-08 10:38:42,107 INFO [make_surface_current_tiles] finished PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast/08mar26
2026-03-08 10:38:42,132 INFO [make_surface_current_tiles] finished rendering of tiles for 2026-03-08 forecast into /results/nowcast-sys/figures/surface_currents/forecast/08mar26
2026-03-08 10:38:42,133 INFO [make_surface_current_tiles] surface current tile figures for 2026-03-08 forecast completed
2026-03-08 10:38:42,587 INFO [manager] checklist updated with [surface current tiles] items from make_surface_current_tiles worker
2026-03-08 10:40:06,779 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 291, 67.4% complete
2026-03-08 10:42:39,328 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19084353 = 2026-03-08 08:21:20 UTC, 34.8% complete
2026-03-08 10:45:06,780 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 396, 91.7% complete
2026-03-08 10:47:39,330 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19084547 = 2026-03-08 10:30:40 UTC, 43.8% complete
2026-03-08 10:50:06,783 INFO [watch_ww3] nowcast WWATCH3 run on arbutus.cloud-nowcast completed
2026-03-08 10:50:06,786 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2026-03-08 10:50:07,218 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2026-03-08'], host='localhost')
2026-03-08 10:50:07,224 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2026-03-08'], host='arbutus.cloud-nowcast')
2026-03-08 10:50:08,423 INFO [download_wwatch3_results] running in process 1190888
2026-03-08 10:50:08,424 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-08 10:50:08,425 INFO [download_wwatch3_results] publishing log messages to tcp://*:5562
2026-03-08 10:50:08,426 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2026-03-08 10:50:09,285 INFO [run_ww3] running in process 12883
2026-03-08 10:50:09,287 INFO [run_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-08 10:50:09,288 INFO [run_ww3] publishing log messages to tcp://*:5572
2026-03-08 10:50:09,289 INFO [run_ww3] connected to skookum.eos.ubc.ca port 5555
2026-03-08 10:50:09,293 INFO [run_ww3] Created run directory /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast_2026-03-08T105009.284959-0700
2026-03-08 10:50:09,295 INFO [run_ww3] forecast: launching /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast_2026-03-08T105009.284959-0700/SoGWW3.sh on arbutus.cloud-nowcast
2026-03-08 10:50:09,306 INFO [run_ww3] forecast WaveWatch3 run for 2026-03-08 on arbutus.cloud-nowcast started
2026-03-08 10:50:09,311 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker
2026-03-08 10:50:09,626 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'forecast'], host='arbutus.cloud-nowcast')
2026-03-08 10:50:11,692 INFO [watch_ww3] running in process 13038
2026-03-08 10:50:11,693 INFO [watch_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-08 10:50:11,694 INFO [watch_ww3] publishing log messages to tcp://*:5573
2026-03-08 10:50:11,696 INFO [watch_ww3] connected to skookum.eos.ubc.ca port 5555
2026-03-08 10:50:11,727 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2026-03-08 10:50:40,793 INFO [download_wwatch3_results] nowcast 2026-03-08 results files from arbutus.cloud-nowcast downloaded
2026-03-08 10:50:40,797 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2026-03-08 10:52:39,331 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19084739 = 2026-03-08 12:38:40 UTC, 52.7% complete
2026-03-08 10:55:11,729 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 90, 13.9% complete
2026-03-08 10:57:39,333 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19084933 = 2026-03-08 14:48:00 UTC, 61.7% complete
2026-03-08 11:00:11,731 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 189, 29.2% complete
2026-03-08 11:02:39,334 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19085127 = 2026-03-08 16:57:20 UTC, 70.7% complete
2026-03-08 11:05:11,797 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 291, 44.9% complete
2026-03-08 11:07:39,335 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19085319 = 2026-03-08 19:05:20 UTC, 79.6% complete
2026-03-08 11:10:11,733 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 387, 59.7% complete
2026-03-08 11:12:39,336 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19085513 = 2026-03-08 21:14:40 UTC, 88.6% complete
2026-03-08 11:15:11,734 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 486, 75.0% complete
2026-03-08 11:17:39,337 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19085706 = 2026-03-08 23:23:20 UTC, 97.5% complete
2026-03-08 11:20:11,735 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 588, 90.7% complete
2026-03-08 11:22:39,339 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19085760 = 2026-03-08 23:59:20 UTC, 100.0% complete
2026-03-08 11:25:11,736 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2026-03-08 11:27:39,346 INFO [watch_NEMO] nowcast-green NEMO run on arbutus.cloud-nowcast completed
2026-03-08 11:27:39,350 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2026-03-08 11:27:39,854 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2026-03-08'], host='localhost')
2026-03-08 11:27:41,154 INFO [download_results] running in process 1191202
2026-03-08 11:27:41,155 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-08 11:27:41,156 INFO [download_results] publishing log messages to tcp://*:5562
2026-03-08 11:27:41,157 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2026-03-08 11:27:41,158 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/nowcast-green/08mar26 to /results2/SalishSea/nowcast-green.202111
2026-03-08 11:29:32,500 INFO [download_results] nowcast-green 2026-03-08 results files from arbutus.cloud-nowcast downloaded
2026-03-08 11:29:32,506 INFO [manager] checklist updated with [results files] items from download_results worker
2026-03-08 11:29:32,824 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast-green', 'research', '--run-date', '2026-03-08'], host='localhost')
2026-03-08 11:29:32,830 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nowcast-green'], host='localhost')
2026-03-08 11:29:32,835 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'physics', '--run-date', '2026-03-08'], host='localhost')
2026-03-08 11:29:32,841 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'biology', '--run-date', '2026-03-08'], host='localhost')
2026-03-08 11:29:32,846 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'chemistry', '--run-date', '2026-03-08'], host='localhost')
2026-03-08 11:29:34,099 INFO [ping_erddap] running in process 1191223
2026-03-08 11:29:34,100 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-08 11:29:34,101 INFO [ping_erddap] publishing log messages to tcp://*:5562
2026-03-08 11:29:34,101 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2026-03-08 11:29:34,107 INFO [ping_erddap] nowcast-green ERDDAP dataset flag file(s) created
2026-03-08 11:29:34,108 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2026-03-08 11:29:35,294 INFO [make_averaged_dataset] running in process 1191226
2026-03-08 11:29:35,295 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-08 11:29:35,296 INFO [make_averaged_dataset] publishing log messages to tcp://*:5562
2026-03-08 11:29:35,297 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555
2026-03-08 11:29:35,298 INFO [make_averaged_dataset] creating day-averaged dataset for 08-Mar-2026 chemistry
2026-03-08 11:29:35,313 INFO [make_averaged_dataset] running in process 1191224
2026-03-08 11:29:35,315 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-08 11:29:35,315 INFO [make_averaged_dataset] publishing log messages to tcp://*:5564
2026-03-08 11:29:35,316 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555
2026-03-08 11:29:35,317 INFO [make_averaged_dataset] creating day-averaged dataset for 08-Mar-2026 physics
2026-03-08 11:29:35,335 INFO [make_averaged_dataset] running in process 1191225
2026-03-08 11:29:35,336 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-08 11:29:35,337 INFO [make_averaged_dataset] publishing log messages to tcp://*:5565
2026-03-08 11:29:35,337 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555
2026-03-08 11:29:35,338 INFO [make_averaged_dataset] creating day-averaged dataset for 08-Mar-2026 biology
2026-03-08 11:29:36,243 INFO [make_plots] running in process 1191221
2026-03-08 11:29:36,244 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-08 11:29:36,245 INFO [make_plots] publishing log messages to tcp://*:5566
2026-03-08 11:29:36,246 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2026-03-08 11:29:36,614 INFO [make_plots] preparing render list for 2026-03-08 NEMO nowcast-green research figures
2026-03-08 11:29:38,446 INFO [make_plots] added temperature_thalweg_and_surface figures to 2026-03-08 NEMO nowcast-green research render list
2026-03-08 11:29:39,244 INFO [make_plots] added nitrate_thalweg_and_surface figures to 2026-03-08 NEMO nowcast-green research render list
2026-03-08 11:29:39,976 INFO [make_plots] added ammonium_thalweg_and_surface figures to 2026-03-08 NEMO nowcast-green research render list
2026-03-08 11:29:40,706 INFO [make_plots] added silicon_thalweg_and_surface figures to 2026-03-08 NEMO nowcast-green research render list
2026-03-08 11:29:41,447 INFO [make_plots] added dissolved_organic_nitrogen_thalweg_and_surface figures to 2026-03-08 NEMO nowcast-green research render list
2026-03-08 11:29:42,173 INFO [make_plots] added particulate_organic_nitrogen_thalweg_and_surface figures to 2026-03-08 NEMO nowcast-green research render list
2026-03-08 11:29:42,926 INFO [make_plots] added biogenic_silicon_thalweg_and_surface figures to 2026-03-08 NEMO nowcast-green research render list
2026-03-08 11:29:43,753 INFO [make_plots] added diatoms_thalweg_and_surface figures to 2026-03-08 NEMO nowcast-green research render list
2026-03-08 11:29:44,588 INFO [make_plots] added flagellates_thalweg_and_surface figures to 2026-03-08 NEMO nowcast-green research render list
2026-03-08 11:29:45,428 INFO [make_plots] added microzooplankton_thalweg_and_surface figures to 2026-03-08 NEMO nowcast-green research render list
2026-03-08 11:29:46,233 INFO [make_plots] added mesozooplankton_thalweg_and_surface figures to 2026-03-08 NEMO nowcast-green research render list
2026-03-08 11:29:46,991 INFO [make_plots] added turbidity_thalweg_and_surface figures to 2026-03-08 NEMO nowcast-green research render list
2026-03-08 11:29:48,212 INFO [make_plots] added temperature_salinity_timeseries figure to 2026-03-08 NEMO nowcast-green research render list
2026-03-08 11:29:48,212 INFO [make_plots] added nitrate_diatoms_timeseries figure to 2026-03-08 NEMO nowcast-green research render list
2026-03-08 11:29:48,213 INFO [make_plots] added diatoms_flagellates_timeseries figure to 2026-03-08 NEMO nowcast-green research render list
2026-03-08 11:29:48,214 INFO [make_plots] added z1_z2_zooplankton_timeseries figure to 2026-03-08 NEMO nowcast-green research render list
2026-03-08 11:29:48,215 INFO [make_plots] starting to render nemo nowcast-green research 08mar26 figures
2026-03-08 11:30:08,297 INFO [make_averaged_dataset] day-averaged dataset for 08-Mar-2026 chemistry created
2026-03-08 11:30:08,300 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker
2026-03-08 11:30:11,737 INFO [watch_ww3] forecast WWATCH3 run on arbutus.cloud-nowcast completed
2026-03-08 11:30:11,740 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2026-03-08 11:30:12,218 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2026-03-08'], host='localhost')
2026-03-08 11:30:13,216 INFO [download_wwatch3_results] running in process 1191398
2026-03-08 11:30:13,218 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-08 11:30:13,219 INFO [download_wwatch3_results] publishing log messages to tcp://*:5562
2026-03-08 11:30:13,220 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2026-03-08 11:30:29,897 INFO [make_averaged_dataset] day-averaged dataset for 08-Mar-2026 physics created
2026-03-08 11:30:29,900 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker
2026-03-08 11:30:39,137 INFO [make_averaged_dataset] day-averaged dataset for 08-Mar-2026 biology created
2026-03-08 11:30:39,141 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker
2026-03-08 11:30:52,198 INFO [download_wwatch3_results] forecast 2026-03-08 results files from arbutus.cloud-nowcast downloaded
2026-03-08 11:30:52,202 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2026-03-08 11:30:52,519 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['wwatch3', 'forecast', '--run-date', '2026-03-08'], host='localhost')
2026-03-08 11:30:53,791 INFO [update_forecast_datasets] running in process 1191417
2026-03-08 11:30:53,794 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-08 11:30:53,795 INFO [update_forecast_datasets] publishing log messages to tcp://*:5562
2026-03-08 11:30:53,796 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2026-03-08 11:30:53,797 INFO [update_forecast_datasets] updating wwatch3 most_recent_forecast directory from forecast/08mar26 run
2026-03-08 11:30:53,799 INFO [update_forecast_datasets] updated wwatch3 most_recent_forecast directory from forecast/08mar26 run
2026-03-08 11:30:53,800 INFO [update_forecast_datasets] updating wwatch3 forecast directory for forecast/08mar26 run
2026-03-08 11:30:53,842 INFO [update_forecast_datasets] updated wwatch3 forecast directory for forecast/08mar26 run: /results/SalishSea/rolling-forecasts/wwatch3
2026-03-08 11:30:53,843 INFO [update_forecast_datasets] wwatch3 2026-03-08 forecast rolling forecast datasets updated
2026-03-08 11:30:53,845 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2026-03-08 11:30:54,154 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['wwatch3-forecast'], host='localhost')
2026-03-08 11:30:55,322 INFO [ping_erddap] running in process 1191422
2026-03-08 11:30:55,324 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-08 11:30:55,325 INFO [ping_erddap] publishing log messages to tcp://*:5562
2026-03-08 11:30:55,326 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2026-03-08 11:30:55,327 INFO [ping_erddap] wwatch3-forecast ERDDAP dataset flag file(s) created
2026-03-08 11:30:55,349 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2026-03-08 11:30:55,652 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['wwatch3', 'forecast', 'publish', '--run-date', '2026-03-08'], host='localhost')
2026-03-08 11:30:59,142 INFO [make_plots] running in process 1191427
2026-03-08 11:30:59,144 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-08 11:30:59,145 INFO [make_plots] publishing log messages to tcp://*:5562
2026-03-08 11:30:59,146 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2026-03-08 11:30:59,147 INFO [make_plots] preparing render list for 2026-03-08 WaveWatch3 forecast publish figures
2026-03-08 11:30:59,148 INFO [make_plots] added Halibut Bank figure to 2026-03-08 WaveWatch3 forecast publish render list
2026-03-08 11:30:59,149 INFO [make_plots] added Sentry Shoal figure to 2026-03-08 WaveWatch3 forecast publish render list
2026-03-08 11:30:59,150 INFO [make_plots] starting to render wwatch3 forecast publish 08mar26 figures
2026-03-08 11:30:59,717 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46146 Halibut Bank
2026-03-08 11:31:03,992 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46131 Sentry Shoal
2026-03-08 11:31:08,016 INFO [make_plots] finished rendering wwatch3 forecast publish 08mar26 figures
2026-03-08 11:31:08,016 INFO [make_plots] wwatch3 publish plots for 2026-03-08 forecast completed
2026-03-08 11:31:08,020 INFO [manager] checklist updated with [plots] items from make_plots worker
2026-03-08 12:32:08,007 INFO [make_plots] finished rendering nemo nowcast-green research 08mar26 figures
2026-03-08 12:32:08,176 INFO [make_plots] nemo research plots for 2026-03-08 nowcast-green completed
2026-03-08 12:32:08,247 INFO [manager] checklist updated with [plots] items from make_plots worker
2026-03-08 14:47:20,632 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/18/ to /results/forcing/atmospheric/continental2.5/GRIB/20260308/18/
2026-03-08 14:47:20,634 INFO [collect_weather] 2026-03-08 2.5km weather forecast 18 collection complete
2026-03-08 14:47:20,636 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2026-03-08 14:47:20,824 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20260308/18/
2026-03-08 14:47:20,826 INFO [crop_gribs] 2026-03-08 18 GRIBs cropping complete
2026-03-08 14:47:21,295 INFO [manager] launching NextWorker(module='nowcast.workers.download_weather', args=['00', '1km'], host='localhost')
2026-03-08 14:47:21,304 INFO [manager] launching NextWorker(module='nowcast.workers.download_weather', args=['12', '1km'], host='localhost')
2026-03-08 14:47:21,311 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['00', '2.5km'], host='localhost')
2026-03-08 14:47:21,317 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['00', '--fcst-date', '2026-03-09'], host='localhost')
2026-03-08 14:47:21,324 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2026-03-08 14:47:23,075 INFO [download_weather] running in process 1193671
2026-03-08 14:47:23,076 INFO [download_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-08 14:47:23,077 INFO [download_weather] publishing log messages to tcp://*:5562
2026-03-08 14:47:23,078 INFO [download_weather] connected to skookum.eos.ubc.ca port 5555
2026-03-08 14:47:23,079 INFO [download_weather] downloading 12 1 km forecast GRIB2 files for 20260308
2026-03-08 14:47:23,080 INFO [download_weather] running in process 1193669
2026-03-08 14:47:23,082 INFO [collect_weather] running in process 1193672
2026-03-08 14:47:23,083 INFO [download_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-08 14:47:23,084 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-08 14:47:23,085 INFO [download_weather] publishing log messages to tcp://*:5563
2026-03-08 14:47:23,086 INFO [collect_weather] publishing log messages to tcp://*:5564
2026-03-08 14:47:23,087 INFO [download_weather] connected to skookum.eos.ubc.ca port 5555
2026-03-08 14:47:23,088 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2026-03-08 14:47:23,089 INFO [download_weather] downloading 00 1 km forecast GRIB2 files for 20260308
2026-03-08 14:47:23,100 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/00/
2026-03-08 14:47:24,469 INFO [crop_gribs] running in process 1193673
2026-03-08 14:47:24,470 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-08 14:47:24,471 INFO [crop_gribs] publishing log messages to tcp://*:5565
2026-03-08 14:47:24,472 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2026-03-08 14:47:24,473 INFO [crop_gribs] cropping 2026-03-09 ECCC HRDPS 2.5km continental 00Z GRIB files to SalishSeaCast subdomain
2026-03-08 14:47:24,487 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20260309/00/
2026-03-08 14:50:52,722 INFO [download_weather] 2026-03-08 1km weather forecast 12 downloads complete
2026-03-08 14:50:52,726 INFO [manager] checklist updated with [weather forecast] items from download_weather worker
2026-03-08 14:50:55,931 INFO [download_weather] 2026-03-08 1km weather forecast 00 downloads complete
2026-03-08 14:50:55,934 INFO [manager] checklist updated with [weather forecast] items from download_weather worker
2026-03-08 20:44:58,753 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/00/ to /results/forcing/atmospheric/continental2.5/GRIB/20260309/00/
2026-03-08 20:44:58,754 INFO [collect_weather] 2026-03-09 2.5km weather forecast 00 collection complete
2026-03-08 20:44:58,757 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2026-03-08 20:44:59,471 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['06', '2.5km'], host='localhost')
2026-03-08 20:44:59,480 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['06', '--fcst-date', '2026-03-09'], host='localhost')
2026-03-08 20:44:59,481 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20260309/00/
2026-03-08 20:44:59,482 INFO [crop_gribs] 2026-03-09 00 GRIBs cropping complete
2026-03-08 20:44:59,487 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2026-03-08 20:45:00,855 INFO [collect_weather] running in process 1197707
2026-03-08 20:45:00,856 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-08 20:45:00,857 INFO [collect_weather] publishing log messages to tcp://*:5562
2026-03-08 20:45:00,858 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2026-03-08 20:45:00,875 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/06/
2026-03-08 20:45:02,252 INFO [crop_gribs] running in process 1197709
2026-03-08 20:45:02,254 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-08 20:45:02,255 INFO [crop_gribs] publishing log messages to tcp://*:5563
2026-03-08 20:45:02,256 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2026-03-08 20:45:02,257 INFO [crop_gribs] cropping 2026-03-09 ECCC HRDPS 2.5km continental 06Z GRIB files to SalishSeaCast subdomain
2026-03-08 20:45:02,269 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20260309/06/
2026-03-09 02:42:37,505 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/06/ to /results/forcing/atmospheric/continental2.5/GRIB/20260309/06/
2026-03-09 02:42:37,506 INFO [collect_weather] 2026-03-09 2.5km weather forecast 06 collection complete
2026-03-09 02:42:37,509 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2026-03-09 02:42:37,985 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Capilano', '--data-date', '2026-03-08'], host='localhost')
2026-03-09 02:42:37,992 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'ChilliwackVedder', '--data-date', '2026-03-08'], host='localhost')
2026-03-09 02:42:37,996 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'ClowhomClowhomLake', '--data-date', '2026-03-08'], host='localhost')
2026-03-09 02:42:38,002 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Englishman', '--data-date', '2026-03-08'], host='localhost')
2026-03-09 02:42:38,007 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Fraser', '--data-date', '2026-03-08'], host='localhost')
2026-03-09 02:42:38,012 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'HomathkoMouth', '--data-date', '2026-03-08'], host='localhost')
2026-03-09 02:42:38,017 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'NicomeklLangley', '--data-date', '2026-03-08'], host='localhost')
2026-03-09 02:42:38,022 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'RobertsCreek', '--data-date', '2026-03-08'], host='localhost')
2026-03-09 02:42:38,027 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SalmonSayward', '--data-date', '2026-03-08'], host='localhost')
2026-03-09 02:42:38,032 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SanJuanPortRenfrew', '--data-date', '2026-03-08'], host='localhost')
2026-03-09 02:42:38,037 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SquamishBrackendale', '--data-date', '2026-03-08'], host='localhost')
2026-03-09 02:42:38,052 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaScotty', '--data-date', '2026-03-08'], host='localhost')
2026-03-09 02:42:38,066 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaBypass', '--data-date', '2026-03-08'], host='localhost')
2026-03-09 02:42:38,080 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaDiversion', '--data-date', '2026-03-08'], host='localhost')
2026-03-09 02:42:38,103 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ctd', args=['SCVIP'], host='localhost')
2026-03-09 02:42:38,124 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ctd', args=['SEVIP'], host='localhost')
2026-03-09 02:42:38,131 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ferry', args=['TWDP'], host='localhost')
2026-03-09 02:42:38,141 INFO [manager] launching NextWorker(module='nowcast.workers.get_vfpa_hadcp', args=['--data-date', '2026-03-08'], host='localhost')
2026-03-09 02:42:38,151 INFO [manager] launching NextWorker(module='nowcast.workers.collect_NeahBay_ssh', args=['00'], host='localhost')
2026-03-09 02:42:38,175 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['12', '2.5km'], host='localhost')
2026-03-09 02:42:38,190 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['12'], host='localhost')
2026-03-09 02:42:41,253 INFO [collect_weather] running in process 1202019
2026-03-09 02:42:41,255 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-09 02:42:41,256 INFO [collect_weather] publishing log messages to tcp://*:5562
2026-03-09 02:42:41,258 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2026-03-09 02:42:41,272 INFO [collect_NeahBay_ssh] running in process 1202018
2026-03-09 02:42:41,273 INFO [collect_NeahBay_ssh] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-09 02:42:41,273 INFO [collect_NeahBay_ssh] publishing log messages to tcp://*:5564
2026-03-09 02:42:41,276 INFO [collect_NeahBay_ssh] connected to skookum.eos.ubc.ca port 5555
2026-03-09 02:42:41,277 INFO [collect_NeahBay_ssh] collecting Neah Bay ssh 00Z obs/forecast for 2026-03-09
2026-03-09 02:42:41,344 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/12/
2026-03-09 02:42:42,013 INFO [collect_river_data] running in process 1201999
2026-03-09 02:42:42,013 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-09 02:42:42,014 INFO [collect_river_data] publishing log messages to tcp://*:5565
2026-03-09 02:42:42,015 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-09 02:42:42,018 INFO [collect_river_data] Collecting ECCC Capilano river data for 2026-03-08
2026-03-09 02:42:42,058 INFO [collect_river_data] Appended ECCC Capilano river average discharge for 2026-03-08 to: /opp/observations/rivers/Capilano/Caplilano_08GA010_day_avg_flow
2026-03-09 02:42:42,058 INFO [collect_river_data] ECCC Capilano river data collection for 2026-03-08 completed
2026-03-09 02:42:42,060 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-09 02:42:42,072 INFO [collect_river_data] running in process 1202001
2026-03-09 02:42:42,073 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-09 02:42:42,074 INFO [collect_river_data] publishing log messages to tcp://*:5566
2026-03-09 02:42:42,076 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-09 02:42:42,077 INFO [collect_river_data] Collecting ECCC ChilliwackVedder river data for 2026-03-08
2026-03-09 02:42:42,113 INFO [collect_river_data] running in process 1202012
2026-03-09 02:42:42,115 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-09 02:42:42,116 INFO [collect_river_data] publishing log messages to tcp://*:5567
2026-03-09 02:42:42,116 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-09 02:42:42,117 INFO [collect_river_data] Collecting ECCC TheodosiaBypass river data for 2026-03-08
2026-03-09 02:42:42,146 INFO [collect_river_data] Appended ECCC ChilliwackVedder river average discharge for 2026-03-08 to: /results/forcing/rivers/observations/Chilliwack_Vedder_flow
2026-03-09 02:42:42,146 INFO [collect_river_data] ECCC ChilliwackVedder river data collection for 2026-03-08 completed
2026-03-09 02:42:42,155 INFO [collect_river_data] running in process 1202004
2026-03-09 02:42:42,156 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-09 02:42:42,317 INFO [collect_river_data] Appended ECCC TheodosiaBypass river average discharge for 2026-03-08 to: /results/forcing/rivers/observations/Theodosia_Bypass_flow
2026-03-09 02:42:42,319 INFO [collect_river_data] publishing log messages to tcp://*:5568
2026-03-09 02:42:42,320 INFO [collect_river_data] ECCC TheodosiaBypass river data collection for 2026-03-08 completed
2026-03-09 02:42:42,322 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-09 02:42:42,323 INFO [collect_river_data] Collecting ECCC Fraser river data for 2026-03-08
2026-03-09 02:42:42,325 INFO [collect_river_data] Appended ECCC Fraser river average discharge for 2026-03-08 to: /data/dlatorne/SOG-projects/SOG-forcing/ECget/Fraser_flow
2026-03-09 02:42:42,327 INFO [collect_river_data] ECCC Fraser river data collection for 2026-03-08 completed
2026-03-09 02:42:42,330 INFO [collect_river_data] running in process 1202007
2026-03-09 02:42:42,331 INFO [collect_river_data] running in process 1202006
2026-03-09 02:42:42,332 INFO [collect_river_data] running in process 1202010
2026-03-09 02:42:42,333 INFO [collect_river_data] running in process 1202008
2026-03-09 02:42:42,333 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-09 02:42:42,335 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-09 02:42:42,335 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-09 02:42:42,336 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-09 02:42:42,337 INFO [collect_river_data] publishing log messages to tcp://*:5569
2026-03-09 02:42:42,338 INFO [collect_river_data] publishing log messages to tcp://*:5570
2026-03-09 02:42:42,339 INFO [collect_river_data] publishing log messages to tcp://*:5571
2026-03-09 02:42:42,341 INFO [collect_river_data] publishing log messages to tcp://*:5572
2026-03-09 02:42:42,343 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-09 02:42:42,344 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-09 02:42:42,345 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-09 02:42:42,346 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-09 02:42:42,348 INFO [collect_river_data] Collecting ECCC RobertsCreek river data for 2026-03-08
2026-03-09 02:42:42,348 INFO [collect_river_data] Collecting ECCC NicomeklLangley river data for 2026-03-08
2026-03-09 02:42:42,349 INFO [collect_river_data] Collecting ECCC SquamishBrackendale river data for 2026-03-08
2026-03-09 02:42:42,350 INFO [collect_river_data] Collecting ECCC SalmonSayward river data for 2026-03-08
2026-03-09 02:42:42,354 INFO [collect_river_data] Appended ECCC RobertsCreek river average discharge for 2026-03-08 to: /results/forcing/rivers/observations/RobertsCreek_flow
2026-03-09 02:42:42,355 INFO [collect_river_data] Appended ECCC NicomeklLangley river average discharge for 2026-03-08 to: /results/forcing/rivers/observations/Nicomekl_Langley_flow
2026-03-09 02:42:42,356 INFO [collect_river_data] Appended ECCC SquamishBrackendale river average discharge for 2026-03-08 to: /results/forcing/rivers/observations/Squamish_Brackendale_flow
2026-03-09 02:42:42,357 INFO [collect_river_data] ECCC RobertsCreek river data collection for 2026-03-08 completed
2026-03-09 02:42:42,358 INFO [collect_river_data] ECCC NicomeklLangley river data collection for 2026-03-08 completed
2026-03-09 02:42:42,358 INFO [collect_river_data] ECCC SquamishBrackendale river data collection for 2026-03-08 completed
2026-03-09 02:42:42,360 INFO [collect_river_data] running in process 1202005
2026-03-09 02:42:42,361 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-09 02:42:42,362 INFO [collect_river_data] publishing log messages to tcp://*:5573
2026-03-09 02:42:42,363 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-09 02:42:42,363 INFO [collect_river_data] Collecting ECCC HomathkoMouth river data for 2026-03-08
2026-03-09 02:42:42,364 INFO [collect_river_data] running in process 1202009
2026-03-09 02:42:42,365 INFO [collect_river_data] running in process 1202002
2026-03-09 02:42:42,366 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-09 02:42:42,367 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-09 02:42:42,367 INFO [collect_river_data] publishing log messages to tcp://*:5574
2026-03-09 02:42:42,368 INFO [collect_river_data] publishing log messages to tcp://*:5575
2026-03-09 02:42:42,369 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-09 02:42:42,369 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-09 02:42:42,370 INFO [collect_river_data] Collecting ECCC SanJuanPortRenfrew river data for 2026-03-08
2026-03-09 02:42:42,371 INFO [collect_river_data] Collecting ECCC ClowhomClowhomLake river data for 2026-03-08
2026-03-09 02:42:42,372 INFO [collect_river_data] running in process 1202003
2026-03-09 02:42:42,374 INFO [collect_river_data] Appended ECCC SalmonSayward river average discharge for 2026-03-08 to: /results/forcing/rivers/observations/Salmon_Sayward_flow
2026-03-09 02:42:42,375 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-09 02:42:42,375 INFO [collect_river_data] ECCC SalmonSayward river data collection for 2026-03-08 completed
2026-03-09 02:42:42,377 INFO [collect_river_data] publishing log messages to tcp://*:5576
2026-03-09 02:42:42,378 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-09 02:42:42,378 INFO [collect_river_data] Collecting ECCC Englishman river data for 2026-03-08
2026-03-09 02:42:42,379 INFO [collect_river_data] running in process 1202011
2026-03-09 02:42:42,380 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-09 02:42:42,381 INFO [collect_river_data] publishing log messages to tcp://*:5577
2026-03-09 02:42:42,382 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-09 02:42:42,382 INFO [collect_river_data] Collecting ECCC TheodosiaScotty river data for 2026-03-08
2026-03-09 02:42:42,390 INFO [collect_river_data] Appended ECCC HomathkoMouth river average discharge for 2026-03-08 to: /results/forcing/rivers/observations/Homathko_Mouth_flow
2026-03-09 02:42:42,391 INFO [collect_river_data] ECCC HomathkoMouth river data collection for 2026-03-08 completed
2026-03-09 02:42:42,392 INFO [collect_river_data] running in process 1202013
2026-03-09 02:42:42,394 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-09 02:42:42,395 INFO [collect_river_data] publishing log messages to tcp://*:5578
2026-03-09 02:42:42,396 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-09 02:42:42,399 INFO [collect_river_data] Collecting ECCC TheodosiaDiversion river data for 2026-03-08
2026-03-09 02:42:42,403 INFO [collect_river_data] Appended ECCC ClowhomClowhomLake river average discharge for 2026-03-08 to: /results/forcing/rivers/observations/Clowhom_ClowhomLake_flow
2026-03-09 02:42:42,404 INFO [collect_river_data] Appended ECCC Englishman river average discharge for 2026-03-08 to: /data/dlatorne/SOG-projects/SOG-forcing/ECget/Englishman_flow
2026-03-09 02:42:42,405 INFO [collect_river_data] ECCC ClowhomClowhomLake river data collection for 2026-03-08 completed
2026-03-09 02:42:42,405 INFO [collect_river_data] ECCC Englishman river data collection for 2026-03-08 completed
2026-03-09 02:42:42,412 INFO [collect_river_data] Appended ECCC TheodosiaScotty river average discharge for 2026-03-08 to: /results/forcing/rivers/observations/Theodosia_Scotty_flow
2026-03-09 02:42:42,412 INFO [collect_river_data] ECCC TheodosiaScotty river data collection for 2026-03-08 completed
2026-03-09 02:42:42,422 INFO [collect_river_data] Appended ECCC SanJuanPortRenfrew river average discharge for 2026-03-08 to: /results/forcing/rivers/observations/SanJuan_PortRenfrew_flow
2026-03-09 02:42:42,423 INFO [collect_river_data] ECCC SanJuanPortRenfrew river data collection for 2026-03-08 completed
2026-03-09 02:42:42,434 INFO [collect_river_data] Appended ECCC TheodosiaDiversion river average discharge for 2026-03-08 to: /results/forcing/rivers/observations/Theodosia_Diversion_flow
2026-03-09 02:42:42,434 INFO [collect_river_data] ECCC TheodosiaDiversion river data collection for 2026-03-08 completed
2026-03-09 02:42:42,534 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-09 02:42:42,551 INFO [get_vfpa_hadcp] running in process 1202017
2026-03-09 02:42:42,553 INFO [get_vfpa_hadcp] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-09 02:42:42,554 INFO [get_vfpa_hadcp] publishing log messages to tcp://*:5579
2026-03-09 02:42:42,555 INFO [get_vfpa_hadcp] connected to skookum.eos.ubc.ca port 5555
2026-03-09 02:42:42,556 INFO [get_vfpa_hadcp] processing VFPA HADCP data from 2nd Narrows Rail Bridge for 2026-03-08
2026-03-09 02:42:42,789 INFO [crop_gribs] running in process 1202020
2026-03-09 02:42:42,790 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-09 02:42:42,791 INFO [crop_gribs] publishing log messages to tcp://*:5590
2026-03-09 02:42:42,792 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2026-03-09 02:42:42,793 INFO [crop_gribs] cropping 2026-03-09 ECCC HRDPS 2.5km continental 12Z GRIB files to SalishSeaCast subdomain
2026-03-09 02:42:42,806 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20260309/12/
2026-03-09 02:42:42,921 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-09 02:42:43,389 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-09 02:42:43,467 INFO [collect_NeahBay_ssh] 2026-03-09T00:00:00-07:00 Neah Bay ssh 00Z obs/forecast data collection complete
2026-03-09 02:42:43,577 INFO [get_onc_ferry] running in process 1202016
2026-03-09 02:42:43,578 INFO [get_onc_ferry] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-09 02:42:43,579 INFO [get_onc_ferry] publishing log messages to tcp://*:5565
2026-03-09 02:42:43,580 INFO [get_onc_ferry] connected to skookum.eos.ubc.ca port 5555
2026-03-09 02:42:43,581 INFO [get_onc_ferry] requesting ONC TWDP.N1 NAV data for 2026-03-08
2026-03-09 02:42:43,582 INFO [get_onc_ctd] running in process 1202014
2026-03-09 02:42:43,583 INFO [get_onc_ctd] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-09 02:42:43,584 INFO [get_onc_ctd] publishing log messages to tcp://*:5566
2026-03-09 02:42:43,585 INFO [get_onc_ctd] connected to skookum.eos.ubc.ca port 5555
2026-03-09 02:42:43,586 INFO [get_onc_ctd] requesting ONC SCVIP CTD T&S data for 2026-03-08
2026-03-09 02:42:43,627 INFO [get_onc_ctd] running in process 1202015
2026-03-09 02:42:43,628 INFO [get_onc_ctd] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-09 02:42:43,629 INFO [get_onc_ctd] publishing log messages to tcp://*:5591
2026-03-09 02:42:43,630 INFO [get_onc_ctd] connected to skookum.eos.ubc.ca port 5555
2026-03-09 02:42:43,631 INFO [get_onc_ctd] requesting ONC SEVIP CTD T&S data for 2026-03-08
2026-03-09 02:42:43,712 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-09 02:42:44,037 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-09 02:42:44,227 WARNING [get_vfpa_hadcp] file not found: /opp/observations/AISDATA/20260308T090000Z-20260308T100000Z.csv
2026-03-09 02:42:44,351 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-09 02:42:44,665 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-09 02:42:44,993 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-09 02:42:45,143 INFO [get_vfpa_hadcp] added VFPA HADCP data from 2nd Narrows Rail Bridge for 2026-03-08 to /opp/observations/AISDATA/netcdf/VFPA_2ND_NARROWS_HADCP_2s_202603.nc
2026-03-09 02:42:45,144 INFO [get_vfpa_hadcp] VFPA HADCP observations added to 2026-03 netcdf file
2026-03-09 02:42:45,292 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-09 02:42:45,637 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-09 02:42:45,951 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-09 02:42:46,228 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-09 02:42:46,520 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-09 02:42:46,849 INFO [manager] checklist updated with [Neah Bay ssh data] items from collect_NeahBay_ssh worker
2026-03-09 02:42:47,159 INFO [manager] launching NextWorker(module='nowcast.workers.make_ssh_files', args=['forecast2', '--run-date', '2026-03-09'], host='localhost')
2026-03-09 02:42:47,167 INFO [manager] checklist updated with [VFPA HADCP data] items from get_vfpa_hadcp worker
2026-03-09 02:42:47,663 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['VFPA-HADCP'], host='localhost')
2026-03-09 02:42:48,896 INFO [ping_erddap] running in process 1202497
2026-03-09 02:42:48,897 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-09 02:42:48,898 INFO [ping_erddap] publishing log messages to tcp://*:5564
2026-03-09 02:42:48,899 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2026-03-09 02:42:48,900 INFO [ping_erddap] VFPA-HADCP ERDDAP dataset flag file(s) created
2026-03-09 02:42:48,901 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2026-03-09 02:42:50,657 INFO [make_ssh_files] running in process 1202484
2026-03-09 02:42:50,658 INFO [make_ssh_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-09 02:42:50,659 INFO [make_ssh_files] publishing log messages to tcp://*:5564
2026-03-09 02:42:50,660 INFO [make_ssh_files] connected to skookum.eos.ubc.ca port 5555
2026-03-09 02:42:50,661 INFO [make_ssh_files] building forecast2 sea surface height boundary conditions file(s) from 2026-03-09 NOAA Neah Bay 00Z observation and forecast values
2026-03-09 02:42:52,771 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2026m03d04.nc
2026-03-09 02:42:52,944 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2026m03d05.nc
2026-03-09 02:42:53,116 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2026m03d06.nc
2026-03-09 02:42:53,286 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2026m03d07.nc
2026-03-09 02:42:53,462 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2026m03d08.nc
2026-03-09 02:42:53,711 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2026m03d09.nc
2026-03-09 02:42:53,903 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2026m03d10.nc
2026-03-09 02:42:54,111 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2026m03d11.nc
2026-03-09 02:42:54,300 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2026m03d12.nc
2026-03-09 02:42:54,439 INFO [make_ssh_files] sea surface height boundary file for 2026-03-09 forecast2 run created
2026-03-09 02:42:54,442 INFO [manager] checklist updated with [sea surface height forcing] items from make_ssh_files worker
2026-03-09 02:42:54,789 INFO [manager] launching NextWorker(module='nowcast.workers.make_runoff_file', args=['v202108'], host='localhost')
2026-03-09 02:42:54,795 INFO [manager] launching NextWorker(module='nowcast.workers.make_201702_runoff_file', args=[], host='localhost')
2026-03-09 02:42:56,146 INFO [make_201702_runoff_file] running in process 1202536
2026-03-09 02:42:56,147 INFO [make_201702_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-09 02:42:56,148 INFO [make_201702_runoff_file] publishing log messages to tcp://*:5564
2026-03-09 02:42:56,149 INFO [make_201702_runoff_file] connected to skookum.eos.ubc.ca port 5555
2026-03-09 02:42:56,185 INFO [make_201702_runoff_file] calculating runoff forcing for b201702 bathymetry
2026-03-09 02:42:56,277 INFO [make_201702_runoff_file] 2026-03-09 runoff file creation from Fraser at Hope and climatology elsewhere complete
2026-03-09 02:42:56,280 INFO [manager] checklist updated with [rivers forcing] items from make_201702_runoff_file worker
2026-03-09 02:42:56,696 INFO [make_runoff_file] running in process 1202534
2026-03-09 02:42:56,698 INFO [make_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-09 02:42:56,699 INFO [make_runoff_file] publishing log messages to tcp://*:5567
2026-03-09 02:42:56,700 INFO [make_runoff_file] connected to skookum.eos.ubc.ca port 5555
2026-03-09 02:42:56,701 INFO [make_runoff_file] calculating NEMO runoff forcing for v202108 bathymetry for 2026-03-08
2026-03-09 02:42:57,716 ERROR [make_runoff_file] no 2026-03-08 discharge obs for SkagitMountVernon: patching
2026-03-09 02:42:58,269 ERROR [make_runoff_file] no 2026-03-08 discharge obs for SnohomishMonroe: patching
2026-03-09 02:42:58,527 ERROR [make_runoff_file] no 2026-03-08 discharge obs for NisquallyMcKenna: patching
2026-03-09 02:42:58,642 ERROR [make_runoff_file] no 2026-03-08 discharge obs for GreenwaterGreenwater: patching
2026-03-09 02:42:59,956 INFO [make_runoff_file] stored NEMO runoff forcing for v202108 bathymetry for 2026-03-08: /results/forcing/rivers/R202108Dailies_y2026m03d08.nc
2026-03-09 02:42:59,957 INFO [make_runoff_file] 2026-03-08 runoff file creation completed
2026-03-09 02:42:59,959 INFO [manager] checklist updated with [rivers forcing] items from make_runoff_file worker
2026-03-09 02:43:01,146 INFO [get_onc_ctd] 2026-03-08 ONC SCVIP CTD T&S file created
2026-03-09 02:43:01,149 INFO [manager] checklist updated with [ONC CTD data] items from get_onc_ctd worker
2026-03-09 02:43:01,455 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['SCVIP-CTD'], host='localhost')
2026-03-09 02:43:01,870 INFO [get_onc_ctd] 2026-03-08 ONC SEVIP CTD T&S file created
2026-03-09 02:43:01,872 INFO [manager] checklist updated with [ONC CTD data] items from get_onc_ctd worker
2026-03-09 02:43:02,180 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['SEVIP-CTD'], host='localhost')
2026-03-09 02:43:02,589 INFO [ping_erddap] running in process 1202576
2026-03-09 02:43:02,590 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-09 02:43:02,592 INFO [ping_erddap] publishing log messages to tcp://*:5564
2026-03-09 02:43:02,592 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2026-03-09 02:43:02,593 INFO [ping_erddap] SCVIP-CTD ERDDAP dataset flag file(s) created
2026-03-09 02:43:02,594 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2026-03-09 02:43:03,365 INFO [ping_erddap] running in process 1202581
2026-03-09 02:43:03,367 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-09 02:43:03,368 INFO [ping_erddap] publishing log messages to tcp://*:5564
2026-03-09 02:43:03,369 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2026-03-09 02:43:03,370 INFO [ping_erddap] SEVIP-CTD ERDDAP dataset flag file(s) created
2026-03-09 02:43:03,372 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2026-03-09 02:43:03,395 INFO [get_onc_ferry] requesting ONC TWDP TSG data for 2026-03-08
2026-03-09 02:43:25,427 INFO [get_onc_ferry] requesting ONC TWDP OXYSENSOR data for 2026-03-08
2026-03-09 02:43:47,279 INFO [get_onc_ferry] requesting ONC TWDP TURBCHLFL data for 2026-03-08
2026-03-09 02:44:08,897 INFO [get_onc_ferry] requesting ONC TWDP CO2SENSOR data for 2026-03-08
2026-03-09 02:44:08,970 WARNING [get_onc_ferry] No ONC TWDP CO2SENSOR partial_pressure,co2 data for 2026-03-08; substituting empty dataset
2026-03-09 02:44:08,976 WARNING [get_onc_ferry] No ONC TWDP CO2SENSOR partial_pressure data for 2026-03-08; substituting empty dataset
2026-03-09 02:44:08,978 WARNING [get_onc_ferry] No ONC TWDP CO2SENSOR co2 data for 2026-03-08; substituting empty dataset
2026-03-09 02:44:08,979 INFO [get_onc_ferry] requesting ONC TWDP TEMPHUMID data for 2026-03-08
2026-03-09 02:44:09,044 WARNING [get_onc_ferry] No ONC TWDP TEMPHUMID air_temperature,rel_humidity data for 2026-03-08; substituting empty dataset
2026-03-09 02:44:09,049 WARNING [get_onc_ferry] No ONC TWDP TEMPHUMID air_temperature data for 2026-03-08; substituting empty dataset
2026-03-09 02:44:09,051 WARNING [get_onc_ferry] No ONC TWDP TEMPHUMID rel_humidity data for 2026-03-08; substituting empty dataset
2026-03-09 02:44:09,052 INFO [get_onc_ferry] requesting ONC TWDP BARPRESS data for 2026-03-08
2026-03-09 02:44:09,110 WARNING [get_onc_ferry] No ONC TWDP BARPRESS barometric_pressure data for 2026-03-08; substituting empty dataset
2026-03-09 02:44:09,114 WARNING [get_onc_ferry] No ONC TWDP BARPRESS barometric_pressure data for 2026-03-08; substituting empty dataset
2026-03-09 02:44:09,115 INFO [get_onc_ferry] requesting ONC TWDP PYRANOMETER data for 2026-03-08
2026-03-09 02:44:09,167 WARNING [get_onc_ferry] No ONC TWDP PYRANOMETER solar_radiation data for 2026-03-08; substituting empty dataset
2026-03-09 02:44:09,169 WARNING [get_onc_ferry] No ONC TWDP PYRANOMETER solar_radiation data for 2026-03-08; substituting empty dataset
2026-03-09 02:44:09,170 INFO [get_onc_ferry] requesting ONC TWDP PYRGEOMETER data for 2026-03-08
2026-03-09 02:44:09,232 WARNING [get_onc_ferry] No ONC TWDP PYRGEOMETER downward_radiation data for 2026-03-08; substituting empty dataset
2026-03-09 02:44:09,236 WARNING [get_onc_ferry] No ONC TWDP PYRGEOMETER downward_radiation data for 2026-03-08; substituting empty dataset
2026-03-09 02:44:09,413 WARNING [get_onc_ferry] ONC TWDP CO2SENSOR partial_pressure data for 2026-03-08 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2026-03-09 02:44:09,428 WARNING [get_onc_ferry] ONC TWDP CO2SENSOR co2 data for 2026-03-08 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2026-03-09 02:44:09,443 WARNING [get_onc_ferry] ONC TWDP TEMPHUMID air_temperature data for 2026-03-08 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2026-03-09 02:44:09,458 WARNING [get_onc_ferry] ONC TWDP TEMPHUMID rel_humidity data for 2026-03-08 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2026-03-09 02:44:09,473 WARNING [get_onc_ferry] ONC TWDP BARPRESS barometric_pressure data for 2026-03-08 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2026-03-09 02:44:09,489 WARNING [get_onc_ferry] ONC TWDP PYRANOMETER solar_radiation data for 2026-03-08 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2026-03-09 02:44:09,503 WARNING [get_onc_ferry] ONC TWDP PYRGEOMETER downward_radiation data for 2026-03-08 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2026-03-09 02:44:27,262 INFO [get_onc_ferry] 2026-03-08 ONC TWDP ferry data file created
2026-03-09 02:44:27,265 INFO [manager] checklist updated with [ONC ferry data] items from get_onc_ferry worker
2026-03-09 02:44:27,594 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['TWDP-ferry'], host='localhost')
2026-03-09 02:44:28,787 INFO [ping_erddap] running in process 1202612
2026-03-09 02:44:28,788 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-09 02:44:28,790 INFO [ping_erddap] publishing log messages to tcp://*:5564
2026-03-09 02:44:28,791 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2026-03-09 02:44:28,793 INFO [ping_erddap] TWDP-ferry ERDDAP dataset flag file(s) created
2026-03-09 02:44:28,795 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2026-03-09 04:45:03,340 INFO [crop_gribs] crop_gribs 06 has watched for 8h; retrying remaining 1 file(s)
2026-03-09 04:45:05,012 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20260309/06/
2026-03-09 04:45:05,013 INFO [crop_gribs] 2026-03-09 06 GRIBs cropping complete
2026-03-09 04:45:05,015 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2026-03-09 04:45:05,581 INFO [manager] launching NextWorker(module='nowcast.workers.grib_to_netcdf', args=['forecast2'], host='localhost')
2026-03-09 04:45:07,656 INFO [grib_to_netcdf] running in process 1203849
2026-03-09 04:45:07,657 INFO [grib_to_netcdf] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-09 04:45:07,658 INFO [grib_to_netcdf] publishing log messages to tcp://*:5563
2026-03-09 04:45:07,659 INFO [grib_to_netcdf] connected to skookum.eos.ubc.ca port 5555
2026-03-09 04:45:07,660 INFO [grib_to_netcdf] creating NEMO-atmos forcing files for 2026-03-09 forecast2 run
2026-03-09 04:45:18,156 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2026m03d10.nc
2026-03-09 04:45:20,994 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2026m03d11.nc
2026-03-09 04:45:21,000 INFO [grib_to_netcdf] 2026-03-09 NEMO-atmos forcing file for forecast2 created
2026-03-09 04:45:21,002 INFO [manager] checklist updated with [weather forcing] items from grib_to_netcdf worker
2026-03-09 04:45:21,273 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'forecast2'], host='localhost')
2026-03-09 04:45:21,277 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'forecast2'], host='localhost')
2026-03-09 04:45:21,282 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.nibi', 'forecast2'], host='localhost')
2026-03-09 04:45:21,285 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'forecast2'], host='localhost')
2026-03-09 04:45:22,624 INFO [upload_forcing] running in process 1203896
2026-03-09 04:45:22,625 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-09 04:45:22,627 INFO [upload_forcing] publishing log messages to tcp://*:5563
2026-03-09 04:45:22,628 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-03-09 04:45:22,629 INFO [upload_forcing] running in process 1203899
2026-03-09 04:45:22,630 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-09 04:45:22,632 INFO [upload_forcing] publishing log messages to tcp://*:5564
2026-03-09 04:45:22,633 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-03-09 04:45:22,640 INFO [upload_forcing] running in process 1203900
2026-03-09 04:45:22,641 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-09 04:45:22,642 INFO [upload_forcing] publishing log messages to tcp://*:5565
2026-03-09 04:45:22,643 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-03-09 04:45:22,655 INFO [upload_forcing] running in process 1203898
2026-03-09 04:45:22,657 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-09 04:45:22,658 INFO [upload_forcing] publishing log messages to tcp://*:5566
2026-03-09 04:45:22,659 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-03-09 04:45:23,894 INFO [upload_forcing] LiveOcean boundary conditions file not found; created symlink to /results/forcing/LiveOcean/boundary_conditions/LiveOcean_v201905_y2026m03d08.nc
2026-03-09 04:45:23,961 INFO [upload_forcing] forecast2 2026-03-09 forcing files upload to optimum-hindcast completed
2026-03-09 04:45:23,964 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-03-09 04:45:25,161 INFO [upload_forcing] forecast2 2026-03-09 forcing files upload to arbutus.cloud-nowcast completed
2026-03-09 04:45:25,166 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-03-09 04:45:25,445 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2026-03-09'], host='localhost')
2026-03-09 04:45:26,816 INFO [make_forcing_links] running in process 1203927
2026-03-09 04:45:26,817 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-09 04:45:26,818 INFO [make_forcing_links] publishing log messages to tcp://*:5563
2026-03-09 04:45:26,819 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2026-03-09 04:45:27,948 INFO [make_forcing_links] forecast2 2026-03-09 forcing file links on arbutus.cloud-nowcast created
2026-03-09 04:45:27,953 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2026-03-09 04:45:28,307 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2026-03-09'], host='arbutus.cloud-nowcast')
2026-03-09 04:45:29,313 INFO [upload_forcing] forecast2 2026-03-09 forcing files upload to orcinus-nowcast-agrif completed
2026-03-09 04:45:29,317 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-03-09 04:45:30,377 INFO [run_NEMO] running in process 23318
2026-03-09 04:45:30,380 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-09 04:45:30,381 INFO [run_NEMO] publishing log messages to tcp://*:5556
2026-03-09 04:45:30,432 INFO [upload_forcing] forecast2 2026-03-09 forcing files upload to robot.nibi completed
2026-03-09 04:45:30,437 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-03-09 04:45:30,441 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2026-03-09 04:45:31,516 WARNING [run_NEMO] There are uncommitted changes in /nemoShare/MEOPAR/nowcast-sys/SS-run-sets
2026-03-09 04:45:31,856 INFO [run_NEMO] forecast2: launching /nemoShare/MEOPAR/nowcast-sys/runs/08mar26forecast2_2026-03-09T044530.827671-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2026-03-09 04:45:31,871 INFO [run_NEMO] forecast2 NEMO run for 2026-03-09 on arbutus.cloud-nowcast started
2026-03-09 04:45:31,876 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2026-03-09 04:45:32,222 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'forecast2'], host='arbutus.cloud-nowcast')
2026-03-09 04:45:34,279 INFO [watch_NEMO] running in process 23591
2026-03-09 04:45:34,280 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-09 04:45:34,282 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2026-03-09 04:45:34,285 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2026-03-09 04:45:34,370 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2026-03-09 04:50:34,365 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 19088434 = 2026-03-10 05:42:00 UTC, 19.0% complete
2026-03-09 04:55:34,366 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 19088959 = 2026-03-10 11:32:00 UTC, 38.5% complete
2026-03-09 05:00:34,367 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 19089485 = 2026-03-10 17:22:40 UTC, 57.9% complete
2026-03-09 05:05:34,376 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 19090049 = 2026-03-10 23:38:40 UTC, 78.8% complete
2026-03-09 05:10:34,370 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 19090592 = 2026-03-11 05:40:40 UTC, 99.0% complete
2026-03-09 05:15:34,374 INFO [watch_NEMO] forecast2 NEMO run on arbutus.cloud-nowcast completed
2026-03-09 05:15:34,378 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2026-03-09 05:15:35,034 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_wind_file', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2026-03-09'], host='arbutus.cloud-nowcast')
2026-03-09 05:15:35,046 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_current_file', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2026-03-09'], host='arbutus.cloud-nowcast')
2026-03-09 05:15:35,050 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2026-03-08'], host='localhost')
2026-03-09 05:15:36,392 INFO [download_results] running in process 1204239
2026-03-09 05:15:36,393 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-09 05:15:36,394 INFO [download_results] publishing log messages to tcp://*:5563
2026-03-09 05:15:36,396 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2026-03-09 05:15:36,397 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/forecast2/08mar26 to /results/SalishSea/forecast2.202111
2026-03-09 05:15:37,789 INFO [make_ww3_current_file] running in process 24104
2026-03-09 05:15:37,791 INFO [make_ww3_current_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-09 05:15:37,792 INFO [make_ww3_current_file] publishing log messages to tcp://*:5571
2026-03-09 05:15:37,793 INFO [make_ww3_current_file] connected to skookum.eos.ubc.ca port 5555
2026-03-09 05:15:37,794 INFO [make_ww3_current_file] Creating wwatch3 currents forcing file for 2026-03-09 forecast2 run
2026-03-09 05:15:37,806 INFO [make_ww3_wind_file] running in process 24103
2026-03-09 05:15:37,807 INFO [make_ww3_wind_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-09 05:15:37,808 INFO [make_ww3_wind_file] publishing log messages to tcp://*:5570
2026-03-09 05:15:37,809 INFO [make_ww3_wind_file] connected to skookum.eos.ubc.ca port 5555
2026-03-09 05:15:37,810 INFO [make_ww3_wind_file] Creating wwatch3 wind forcing file for 2026-03-09 forecast2 run
2026-03-09 05:15:46,583 INFO [make_ww3_wind_file] wwatch3 wind forcing file created on arbutus.cloud-nowcast for 2026-03-09 forecast2 run
2026-03-09 05:15:46,585 INFO [manager] checklist updated with [WW3 wind forcing] items from make_ww3_wind_file worker
2026-03-09 05:16:09,466 INFO [download_results] forecast2 2026-03-08 results files from arbutus.cloud-nowcast downloaded
2026-03-09 05:16:09,470 INFO [manager] checklist updated with [results files] items from download_results worker
2026-03-09 05:16:09,788 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['forecast2', '--run-date', '2026-03-08'], host='localhost')
2026-03-09 05:16:11,826 INFO [make_CHS_currents_file] running in process 1204247
2026-03-09 05:16:11,828 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-09 05:16:11,829 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5563
2026-03-09 05:16:11,830 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555
2026-03-09 05:16:47,861 INFO [make_CHS_currents_file] Made CHS currents file for 2026-03-08 for forecast2
2026-03-09 05:16:47,864 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker
2026-03-09 05:16:48,142 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['nemo', 'forecast2', '--run-date', '2026-03-08'], host='localhost')
2026-03-09 05:16:49,907 INFO [update_forecast_datasets] running in process 1204288
2026-03-09 05:16:49,908 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-09 05:16:49,909 INFO [update_forecast_datasets] publishing log messages to tcp://*:5563
2026-03-09 05:16:49,911 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2026-03-09 05:16:49,912 INFO [update_forecast_datasets] updating nemo forecast directory for forecast2/08mar26 run
2026-03-09 05:19:15,527 INFO [make_ww3_current_file] wwatch3 currents forcing file created on arbutus.cloud-nowcast for 2026-03-09 forecast2 run
2026-03-09 05:19:15,531 INFO [manager] checklist updated with [WW3 currents forcing] items from make_ww3_current_file worker
2026-03-09 05:19:15,863 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2026-03-09'], host='arbutus.cloud-nowcast')
2026-03-09 05:19:17,880 INFO [run_ww3] running in process 24684
2026-03-09 05:19:17,881 INFO [run_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-09 05:19:17,882 INFO [run_ww3] publishing log messages to tcp://*:5572
2026-03-09 05:19:17,883 INFO [run_ww3] connected to skookum.eos.ubc.ca port 5555
2026-03-09 05:19:17,886 INFO [run_ww3] Created run directory /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast2_2026-03-09T051917.878167-0700
2026-03-09 05:19:17,888 INFO [run_ww3] forecast2: launching /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast2_2026-03-09T051917.878167-0700/SoGWW3.sh on arbutus.cloud-nowcast
2026-03-09 05:19:17,900 INFO [run_ww3] forecast2 WaveWatch3 run for 2026-03-09 on arbutus.cloud-nowcast started
2026-03-09 05:19:17,903 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker
2026-03-09 05:19:18,217 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'forecast2'], host='arbutus.cloud-nowcast')
2026-03-09 05:19:20,253 INFO [watch_ww3] running in process 24771
2026-03-09 05:19:20,254 INFO [watch_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-09 05:19:20,255 INFO [watch_ww3] publishing log messages to tcp://*:5573
2026-03-09 05:19:20,256 INFO [watch_ww3] connected to skookum.eos.ubc.ca port 5555
2026-03-09 05:19:20,291 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2026-03-09 05:21:02,340 INFO [update_forecast_datasets] updated nemo forecast directory for forecast2/08mar26 run: /results/SalishSea/rolling-forecasts/nemo
2026-03-09 05:21:02,341 INFO [update_forecast_datasets] nemo 2026-03-08 forecast2 rolling forecast datasets updated
2026-03-09 05:21:02,344 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2026-03-09 05:21:02,683 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nemo-forecast'], host='localhost')
2026-03-09 05:21:02,689 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'forecast2', 'publish', '--run-date', '2026-03-08'], host='localhost')
2026-03-09 05:21:02,694 INFO [manager] launching NextWorker(module='nowcast.workers.make_surface_current_tiles', args=['forecast2', '--run-date', '2026-03-08'], host='localhost')
2026-03-09 05:21:04,340 INFO [ping_erddap] running in process 1204387
2026-03-09 05:21:04,340 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-09 05:21:04,341 INFO [ping_erddap] publishing log messages to tcp://*:5563
2026-03-09 05:21:04,342 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2026-03-09 05:21:04,351 INFO [ping_erddap] nemo-forecast ERDDAP dataset flag file(s) created
2026-03-09 05:21:04,353 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2026-03-09 05:21:06,091 INFO [make_surface_current_tiles] running in process 1204390
2026-03-09 05:21:06,093 INFO [make_surface_current_tiles] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-09 05:21:06,096 INFO [make_surface_current_tiles] publishing log messages to tcp://*:5563
2026-03-09 05:21:06,097 INFO [make_surface_current_tiles] connected to skookum.eos.ubc.ca port 5555
2026-03-09 05:21:06,098 INFO [make_surface_current_tiles] starting rendering of tiles for 2026-03-08 forecast2 in 6 processes into /results/nowcast-sys/figures/surface_currents/forecast2/08mar26
2026-03-09 05:21:07,996 INFO [make_plots] running in process 1204389
2026-03-09 05:21:07,997 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-09 05:21:07,998 INFO [make_plots] publishing log messages to tcp://*:5564
2026-03-09 05:21:07,998 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2026-03-09 05:21:08,425 INFO [make_plots] preparing render list for 2026-03-08 NEMO forecast2 publish figures
2026-03-09 05:21:08,699 INFO [make_plots] added Website_thumbnail figure to 2026-03-08 NEMO forecast2 publish render list
2026-03-09 05:21:08,701 INFO [make_plots] added Threshold_website figure to 2026-03-08 NEMO forecast2 publish render list
2026-03-09 05:21:08,702 INFO [make_plots] added PA_tidal_predictions figure to 2026-03-08 NEMO forecast2 publish render list
2026-03-09 05:21:08,702 INFO [make_plots] added Boundary Bay figure to 2026-03-08 NEMO forecast2 publish render list
2026-03-09 05:21:08,703 INFO [make_plots] added Campbell River figure to 2026-03-08 NEMO forecast2 publish render list
2026-03-09 05:21:08,705 INFO [make_plots] added Cherry Point figure to 2026-03-08 NEMO forecast2 publish render list
2026-03-09 05:21:08,705 INFO [make_plots] added Friday Harbor figure to 2026-03-08 NEMO forecast2 publish render list
2026-03-09 05:21:08,706 INFO [make_plots] added Halfmoon Bay figure to 2026-03-08 NEMO forecast2 publish render list
2026-03-09 05:21:08,707 INFO [make_plots] added Nanaimo figure to 2026-03-08 NEMO forecast2 publish render list
2026-03-09 05:21:08,708 INFO [make_plots] added Neah Bay figure to 2026-03-08 NEMO forecast2 publish render list
2026-03-09 05:21:08,709 INFO [make_plots] added New Westminster figure to 2026-03-08 NEMO forecast2 publish render list
2026-03-09 05:21:08,709 INFO [make_plots] added Patricia Bay figure to 2026-03-08 NEMO forecast2 publish render list
2026-03-09 05:21:08,710 INFO [make_plots] added Point Atkinson figure to 2026-03-08 NEMO forecast2 publish render list
2026-03-09 05:21:08,711 INFO [make_plots] added Port Renfrew figure to 2026-03-08 NEMO forecast2 publish render list
2026-03-09 05:21:08,712 INFO [make_plots] added Sand Heads figure to 2026-03-08 NEMO forecast2 publish render list
2026-03-09 05:21:08,712 INFO [make_plots] added Sandy Cove figure to 2026-03-08 NEMO forecast2 publish render list
2026-03-09 05:21:08,713 INFO [make_plots] added Squamish figure to 2026-03-08 NEMO forecast2 publish render list
2026-03-09 05:21:08,714 INFO [make_plots] added Victoria figure to 2026-03-08 NEMO forecast2 publish render list
2026-03-09 05:21:08,715 INFO [make_plots] added Woodwards Landing figure to 2026-03-08 NEMO forecast2 publish render list
2026-03-09 05:21:08,715 INFO [make_plots] starting to render nemo forecast2 publish 08mar26 figures
2026-03-09 05:24:20,291 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 93, 17.2% complete
2026-03-09 05:29:03,329 WARNING [make_plots] invalid station number for Boundary Bay station: None
2026-03-09 05:29:03,329 WARNING [make_plots] can't resolve a valid CHS station code for Boundary Bay
2026-03-09 05:29:20,293 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 192, 35.6% complete
2026-03-09 05:29:29,018 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb996/data for station 08074 Campbell River from 2026-03-04 00:00:00Z to 2026-03-11 05:55:00Z
2026-03-09 05:30:37,769 WARNING [make_plots] invalid station number for Halfmoon Bay station: None
2026-03-09 05:30:37,770 WARNING [make_plots] can't resolve a valid CHS station code for Halfmoon Bay
2026-03-09 05:31:00,063 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb96d/data for station 07917 Nanaimo from 2026-03-04 00:00:00Z to 2026-03-11 05:55:00Z
2026-03-09 05:31:43,738 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e13d0f4a073c4bbf93/data for station 07654 New Westminster from 2026-03-04 00:00:00Z to 2026-03-11 05:55:00Z
2026-03-09 05:32:03,569 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd2d/data for station 07277 Patricia Bay from 2026-03-04 00:00:00Z to 2026-03-11 05:55:00Z
2026-03-09 05:32:13,564 INFO [make_surface_current_tiles] starting PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast2/08mar26
2026-03-09 05:32:20,585 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94c/data for station 07795 Point Atkinson from 2026-03-04 00:00:00Z to 2026-03-11 05:55:00Z
2026-03-09 05:32:37,174 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e23d0f4a073c4bc060/data for station 08525 Port Renfrew from 2026-03-04 00:00:00Z to 2026-03-11 05:55:00Z
2026-03-09 05:32:54,519 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb93c/data for station 07594 Sand Heads from 2026-03-04 00:00:00Z to 2026-03-11 05:55:00Z
2026-03-09 05:33:11,647 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc43b/data for station 07786 Sandy Cove from 2026-03-04 00:00:00Z to 2026-03-11 05:55:00Z
2026-03-09 05:33:27,667 WARNING [make_plots] invalid station number for Squamish station: None
2026-03-09 05:33:27,668 WARNING [make_plots] can't resolve a valid CHS station code for Squamish
2026-03-09 05:33:44,266 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd1e/data for station 07120 Victoria from 2026-03-04 00:00:00Z to 2026-03-11 05:55:00Z
2026-03-09 05:34:02,326 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc42d/data for station 07610 Woodwards Landing from 2026-03-04 00:00:00Z to 2026-03-11 05:55:00Z
2026-03-09 05:34:18,285 INFO [make_plots] finished rendering nemo forecast2 publish 08mar26 figures
2026-03-09 05:34:18,287 INFO [make_plots] nemo publish plots for 2026-03-08 forecast2 completed
2026-03-09 05:34:18,294 INFO [manager] checklist updated with [plots] items from make_plots worker
2026-03-09 05:34:18,607 INFO [manager] launching NextWorker(module='nowcast.workers.make_feeds', args=['forecast2', '--run-date', '2026-03-08'], host='localhost')
2026-03-09 05:34:20,295 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 282, 52.2% complete
2026-03-09 05:34:23,531 INFO [make_feeds] running in process 1209349
2026-03-09 05:34:23,535 INFO [make_feeds] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-09 05:34:23,536 INFO [make_feeds] publishing log messages to tcp://*:5564
2026-03-09 05:34:23,537 INFO [make_feeds] connected to skookum.eos.ubc.ca port 5555
2026-03-09 05:34:37,973 INFO [make_feeds] ATOM feeds for 2026-03-08 forecast2 run completed
2026-03-09 05:34:37,975 INFO [manager] checklist updated with [storm surge alert feeds] items from make_feeds worker
2026-03-09 05:34:38,295 INFO [manager] launching NextWorker(module='nemo_nowcast.workers.clear_checklist', args=[], host='localhost')
2026-03-09 05:34:39,537 INFO [clear_checklist] running in process 1209375
2026-03-09 05:34:39,538 INFO [clear_checklist] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-09 05:34:39,540 INFO [clear_checklist] publishing log messages to tcp://*:5564
2026-03-09 05:34:39,541 INFO [clear_checklist] connected to skookum.eos.ubc.ca port 5555
2026-03-09 05:34:39,542 INFO [clear_checklist] requesting that manager clear system state checklist
2026-03-09 05:34:39,544 INFO [manager] writing checklist to log file
2026-03-09 05:34:39,585 INFO [manager] checklist cleared
2026-03-09 05:34:39,587 INFO [clear_checklist] nowcast system checklist cleared
2026-03-09 05:34:39,590 INFO [manager] launching NextWorker(module='nemo_nowcast.workers.rotate_logs', args=[], host='localhost')
2026-03-09 05:34:40,775 INFO [rotate_logs] running in process 1209380
2026-03-09 05:34:40,776 INFO [rotate_logs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-09 05:34:40,777 INFO [rotate_logs] publishing log messages to tcp://*:5564
2026-03-09 05:34:40,778 INFO [rotate_logs] connected to skookum.eos.ubc.ca port 5555
2026-03-09 05:34:40,779 INFO [rotate_logs] rotating log files