2025-09-13 03:38:13,181 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/nowcast.log
2025-09-13 03:38:13,187 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/nowcast.debug.log
2025-09-13 03:38:13,192 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/checklist.log
2025-09-13 03:38:13,192 INFO [rotate_logs] log files rotated
2025-09-13 03:38:13,197 INFO [manager] checklist updated with [log rotation] items from rotate_logs worker
2025-09-13 03:40:09,101 INFO [make_surface_current_tiles] finished PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast2/12sep25
2025-09-13 03:40:09,125 INFO [make_surface_current_tiles] finished rendering of tiles for 2025-09-12 forecast2 into /results/nowcast-sys/figures/surface_currents/forecast2/12sep25
2025-09-13 03:40:09,126 INFO [make_surface_current_tiles] surface current tile figures for 2025-09-12 forecast2 completed
2025-09-13 03:40:09,437 INFO [manager] checklist updated with [surface current tiles] items from make_surface_current_tiles worker
2025-09-13 03:41:10,103 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 333, 61.7% complete
2025-09-13 03:46:10,105 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 426, 78.9% complete
2025-09-13 03:51:10,105 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 513, 95.0% complete
2025-09-13 03:56:10,109 INFO [watch_ww3] forecast2 WWATCH3 run on arbutus.cloud-nowcast completed
2025-09-13 03:56:10,121 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2025-09-13 03:56:10,463 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2025-09-13'], host='localhost')
2025-09-13 03:56:11,666 INFO [download_wwatch3_results] running in process 2889872
2025-09-13 03:56:11,668 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-09-13 03:56:11,669 INFO [download_wwatch3_results] publishing log messages to tcp://*:5562
2025-09-13 03:56:11,670 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2025-09-13 03:56:54,691 INFO [download_wwatch3_results] forecast2 2025-09-13 results files from arbutus.cloud-nowcast downloaded
2025-09-13 03:56:54,695 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2025-09-13 03:56:54,853 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['wwatch3', 'forecast2', '--run-date', '2025-09-13'], host='localhost')
2025-09-13 03:56:55,962 INFO [update_forecast_datasets] running in process 2889884
2025-09-13 03:56:55,964 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-09-13 03:56:55,965 INFO [update_forecast_datasets] publishing log messages to tcp://*:5562
2025-09-13 03:56:55,966 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2025-09-13 03:56:55,967 INFO [update_forecast_datasets] updating wwatch3 most_recent_forecast directory from forecast2/13sep25 run
2025-09-13 03:56:55,982 INFO [update_forecast_datasets] updated wwatch3 most_recent_forecast directory from forecast2/13sep25 run
2025-09-13 03:56:55,983 INFO [update_forecast_datasets] updating wwatch3 forecast directory for forecast2/13sep25 run
2025-09-13 03:57:24,560 INFO [update_forecast_datasets] updated wwatch3 forecast directory for forecast2/13sep25 run: /results/SalishSea/rolling-forecasts/wwatch3
2025-09-13 03:57:24,561 INFO [update_forecast_datasets] wwatch3 2025-09-13 forecast2 rolling forecast datasets updated
2025-09-13 03:57:24,565 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2025-09-13 03:57:24,696 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['wwatch3-forecast'], host='localhost')
2025-09-13 03:57:25,918 INFO [ping_erddap] running in process 2889895
2025-09-13 03:57:25,920 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-09-13 03:57:25,921 INFO [ping_erddap] publishing log messages to tcp://*:5562
2025-09-13 03:57:25,923 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2025-09-13 03:57:25,924 INFO [ping_erddap] wwatch3-forecast ERDDAP dataset flag file(s) created
2025-09-13 03:57:25,926 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2025-09-13 03:57:26,054 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['wwatch3', 'forecast2', 'publish', '--run-date', '2025-09-13'], host='localhost')
2025-09-13 03:57:28,881 INFO [make_plots] running in process 2889900
2025-09-13 03:57:28,883 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-09-13 03:57:28,883 INFO [make_plots] publishing log messages to tcp://*:5562
2025-09-13 03:57:28,884 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2025-09-13 03:57:28,885 INFO [make_plots] preparing render list for 2025-09-13 WaveWatch3 forecast2 publish figures
2025-09-13 03:57:28,886 INFO [make_plots] added Halibut Bank figure to 2025-09-13 WaveWatch3 forecast2 publish render list
2025-09-13 03:57:28,887 INFO [make_plots] added Sentry Shoal figure to 2025-09-13 WaveWatch3 forecast2 publish render list
2025-09-13 03:57:28,887 INFO [make_plots] starting to render wwatch3 forecast2 publish 13sep25 figures
2025-09-13 03:57:29,669 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46146 Halibut Bank
2025-09-13 03:57:36,039 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46131 Sentry Shoal
2025-09-13 03:57:40,557 INFO [make_plots] finished rendering wwatch3 forecast2 publish 13sep25 figures
2025-09-13 03:57:40,557 INFO [make_plots] wwatch3 publish plots for 2025-09-13 forecast2 completed
2025-09-13 03:57:40,560 INFO [manager] checklist updated with [plots] items from make_plots worker
2025-09-13 08:46:51,922 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/12/ to /results/forcing/atmospheric/continental2.5/GRIB/20250913/12/
2025-09-13 08:46:51,923 INFO [collect_weather] 2025-09-13 2.5km weather forecast 12 collection complete
2025-09-13 08:46:51,926 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2025-09-13 08:46:52,273 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'SkagitMountVernon', '--data-date', '2025-09-12'], host='localhost')
2025-09-13 08:46:52,279 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'SnohomishMonroe', '--data-date', '2025-09-12'], host='localhost')
2025-09-13 08:46:52,284 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'NisquallyMcKenna', '--data-date', '2025-09-12'], host='localhost')
2025-09-13 08:46:52,288 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'GreenwaterGreenwater', '--data-date', '2025-09-12'], host='localhost')
2025-09-13 08:46:52,292 INFO [manager] launching NextWorker(module='nowcast.workers.make_turbidity_file', args=[], host='localhost')
2025-09-13 08:46:52,299 INFO [manager] launching NextWorker(module='nowcast.workers.collect_NeahBay_ssh', args=['06'], host='localhost')
2025-09-13 08:46:52,302 INFO [manager] launching NextWorker(module='nowcast.workers.download_live_ocean', args=[], host='localhost')
2025-09-13 08:46:52,307 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['18', '2.5km'], host='localhost')
2025-09-13 08:46:52,312 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['18'], host='localhost')
2025-09-13 08:46:53,582 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20250913/12/
2025-09-13 08:46:53,582 INFO [crop_gribs] 2025-09-13 12 GRIBs cropping complete
2025-09-13 08:46:53,585 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2025-09-13 08:46:53,989 INFO [manager] launching NextWorker(module='nowcast.workers.grib_to_netcdf', args=['nowcast+'], host='localhost')
2025-09-13 08:46:53,994 INFO [collect_NeahBay_ssh] running in process 2894062
2025-09-13 08:46:53,995 INFO [collect_NeahBay_ssh] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-09-13 08:46:53,998 INFO [collect_NeahBay_ssh] publishing log messages to tcp://*:5562
2025-09-13 08:46:53,999 INFO [collect_NeahBay_ssh] connected to skookum.eos.ubc.ca port 5555
2025-09-13 08:46:53,999 INFO [collect_NeahBay_ssh] collecting Neah Bay ssh 06Z obs/forecast for 2025-09-13
2025-09-13 08:46:54,028 INFO [collect_weather] running in process 2894064
2025-09-13 08:46:54,029 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-09-13 08:46:54,030 INFO [collect_weather] publishing log messages to tcp://*:5563
2025-09-13 08:46:54,030 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2025-09-13 08:46:54,049 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/18/
2025-09-13 08:46:54,168 INFO [download_live_ocean] running in process 2894063
2025-09-13 08:46:54,173 INFO [download_live_ocean] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-09-13 08:46:54,174 INFO [download_live_ocean] publishing log messages to tcp://*:5564
2025-09-13 08:46:54,174 INFO [download_live_ocean] connected to skookum.eos.ubc.ca port 5555
2025-09-13 08:46:54,175 INFO [download_live_ocean] downloading Salish Sea western boundary day-averaged LiveOcean file for 2025-09-13
2025-09-13 08:46:54,531 INFO [collect_river_data] running in process 2894058
2025-09-13 08:46:54,532 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-09-13 08:46:54,533 INFO [collect_river_data] publishing log messages to tcp://*:5565
2025-09-13 08:46:54,534 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-09-13 08:46:54,535 INFO [collect_river_data] Collecting USGS SnohomishMonroe river data for 2025-09-12
2025-09-13 08:46:54,580 INFO [collect_river_data] running in process 2894056
2025-09-13 08:46:54,581 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-09-13 08:46:54,582 INFO [collect_river_data] publishing log messages to tcp://*:5566
2025-09-13 08:46:54,583 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-09-13 08:46:54,584 INFO [collect_river_data] Collecting USGS SkagitMountVernon river data for 2025-09-12
2025-09-13 08:46:54,584 INFO [collect_river_data] running in process 2894060
2025-09-13 08:46:54,585 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-09-13 08:46:54,586 INFO [collect_river_data] publishing log messages to tcp://*:5567
2025-09-13 08:46:54,587 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-09-13 08:46:54,587 INFO [collect_river_data] Collecting USGS GreenwaterGreenwater river data for 2025-09-12
2025-09-13 08:46:54,701 INFO [collect_river_data] running in process 2894059
2025-09-13 08:46:54,703 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-09-13 08:46:54,704 INFO [collect_river_data] publishing log messages to tcp://*:5568
2025-09-13 08:46:54,705 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-09-13 08:46:54,705 INFO [collect_river_data] Collecting USGS NisquallyMcKenna river data for 2025-09-12
2025-09-13 08:46:54,717 INFO [make_turbidity_file] running in process 2894061
2025-09-13 08:46:54,718 INFO [make_turbidity_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-09-13 08:46:54,719 INFO [make_turbidity_file] publishing log messages to tcp://*:5569
2025-09-13 08:46:54,720 INFO [make_turbidity_file] connected to skookum.eos.ubc.ca port 5555
2025-09-13 08:46:54,721 INFO [make_turbidity_file] Creating Fraser River turbidity forcing file for 2025-09-13
2025-09-13 08:46:54,929 INFO [crop_gribs] running in process 2894065
2025-09-13 08:46:54,931 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-09-13 08:46:54,931 INFO [crop_gribs] publishing log messages to tcp://*:5570
2025-09-13 08:46:54,932 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2025-09-13 08:46:54,933 INFO [crop_gribs] cropping 2025-09-13 ECCC HRDPS 2.5km continental 18Z GRIB files to SalishSeaCast subdomain
2025-09-13 08:46:54,947 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20250913/18/
2025-09-13 08:46:55,022 INFO [download_live_ocean] downloaded 10183324 bytes from https://s3.kopah.uw.edu/liveocean-share/f2025.09.13/ubc.nc to /results/forcing/LiveOcean/downloaded/20250913/low_passed_UBC.nc
2025-09-13 08:46:55,022 INFO [download_live_ocean] Deflating in up to 1 concurrent sub-processes
2025-09-13 08:46:55,050 INFO [collect_NeahBay_ssh] 2025-09-13T00:00:00-07:00 Neah Bay ssh 06Z obs/forecast data collection complete
2025-09-13 08:46:55,053 INFO [manager] checklist updated with [Neah Bay ssh data] items from collect_NeahBay_ssh worker
2025-09-13 08:46:55,060 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12089500&startDT=2025-09-12&endDT=2025-09-12 "HTTP/1.1 200 "
2025-09-13 08:46:55,088 INFO [collect_river_data] Appended USGS NisquallyMcKenna river average discharge for 2025-09-12 to: /results/forcing/rivers/observations/Nisqually_McKenna_flow
2025-09-13 08:46:55,089 INFO [collect_river_data] USGS NisquallyMcKenna river data collection for 2025-09-12 completed
2025-09-13 08:46:55,176 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12150800&startDT=2025-09-12&endDT=2025-09-12 "HTTP/1.1 200 "
2025-09-13 08:46:55,178 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12097500&startDT=2025-09-12&endDT=2025-09-12 "HTTP/1.1 200 "
2025-09-13 08:46:55,190 INFO [collect_river_data] Appended USGS SnohomishMonroe river average discharge for 2025-09-12 to: /results/forcing/rivers/observations/Snohomish_Monroe_flow
2025-09-13 08:46:55,191 INFO [collect_river_data] USGS SnohomishMonroe river data collection for 2025-09-12 completed
2025-09-13 08:46:55,193 INFO [manager] launching NextWorker(module='nowcast.workers.make_ssh_files', args=['nowcast', '--run-date', '2025-09-13'], host='localhost')
2025-09-13 08:46:55,196 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12200500&startDT=2025-09-12&endDT=2025-09-12 "HTTP/1.1 200 "
2025-09-13 08:46:55,197 INFO [collect_river_data] Appended USGS GreenwaterGreenwater river average discharge for 2025-09-12 to: /results/forcing/rivers/observations/Greenwater_Greenwater_flow
2025-09-13 08:46:55,198 INFO [collect_river_data] USGS GreenwaterGreenwater river data collection for 2025-09-12 completed
2025-09-13 08:46:55,200 INFO [collect_river_data] Appended USGS SkagitMountVernon river average discharge for 2025-09-12 to: /results/forcing/rivers/observations/Skagit_MountVernon_flow
2025-09-13 08:46:55,201 INFO [collect_river_data] USGS SkagitMountVernon river data collection for 2025-09-12 completed
2025-09-13 08:46:55,203 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-09-13 08:46:55,380 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-09-13 08:46:55,599 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-09-13 08:46:55,807 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-09-13 08:46:55,958 INFO [grib_to_netcdf] running in process 2894152
2025-09-13 08:46:55,959 INFO [grib_to_netcdf] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-09-13 08:46:55,960 INFO [grib_to_netcdf] publishing log messages to tcp://*:5562
2025-09-13 08:46:55,961 INFO [grib_to_netcdf] connected to skookum.eos.ubc.ca port 5555
2025-09-13 08:46:55,961 INFO [grib_to_netcdf] creating NEMO-atmos forcing files for 2025-09-13 nowcast and forecast runs
2025-09-13 08:46:56,062 INFO [download_live_ocean] netCDF4 deflated /results/forcing/LiveOcean/downloaded/20250913/low_passed_UBC.nc
2025-09-13 08:46:56,063 INFO [download_live_ocean] 2025-09-13 Live Ocean file for Salish Sea western boundary downloaded
2025-09-13 08:46:56,065 INFO [manager] checklist updated with [Live Ocean products] items from download_live_ocean worker
2025-09-13 08:46:56,426 INFO [manager] launching NextWorker(module='nowcast.workers.make_live_ocean_files', args=['--run-date', '2025-09-13'], host='localhost')
2025-09-13 08:46:57,127 INFO [make_turbidity_file] 2025-09-13 Fraser River turbidity file creation complete
2025-09-13 08:46:57,128 INFO [manager] checklist updated with [Fraser River turbidity] items from make_turbidity_file worker
2025-09-13 08:46:57,841 INFO [make_ssh_files] running in process 2894189
2025-09-13 08:46:57,842 INFO [make_ssh_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-09-13 08:46:57,843 INFO [make_ssh_files] publishing log messages to tcp://*:5564
2025-09-13 08:46:57,844 INFO [make_ssh_files] connected to skookum.eos.ubc.ca port 5555
2025-09-13 08:46:57,845 INFO [make_ssh_files] building nowcast sea surface height boundary conditions file(s) from 2025-09-13 NOAA Neah Bay 06Z observation and forecast values
2025-09-13 08:46:58,468 INFO [make_live_ocean_files] running in process 2894210
2025-09-13 08:46:58,470 INFO [make_live_ocean_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-09-13 08:46:58,471 INFO [make_live_ocean_files] publishing log messages to tcp://*:5565
2025-09-13 08:46:58,472 INFO [make_live_ocean_files] connected to skookum.eos.ubc.ca port 5555
2025-09-13 08:46:58,483 INFO [make_live_ocean_files] Creating T&S western boundary conditions file from 2025-09-13 Live Ocean run
2025-09-13 08:46:59,665 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2025m09d09.nc
2025-09-13 08:46:59,831 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2025m09d10.nc
2025-09-13 08:47:00,007 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2025m09d11.nc
2025-09-13 08:47:00,179 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2025m09d12.nc
2025-09-13 08:47:00,378 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2025m09d13.nc
2025-09-13 08:47:00,566 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2025m09d14.nc
2025-09-13 08:47:00,752 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2025m09d15.nc
2025-09-13 08:47:00,934 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2025m09d16.nc
2025-09-13 08:47:01,049 INFO [make_ssh_files] sea surface height boundary file for 2025-09-13 nowcast run created
2025-09-13 08:47:01,052 INFO [manager] checklist updated with [sea surface height forcing] items from make_ssh_files worker
2025-09-13 08:47:01,211 INFO [manager] launching NextWorker(module='nowcast.workers.make_runoff_file', args=['v202108'], host='localhost')
2025-09-13 08:47:01,215 INFO [manager] launching NextWorker(module='nowcast.workers.make_201702_runoff_file', args=[], host='localhost')
2025-09-13 08:47:02,525 INFO [make_201702_runoff_file] running in process 2894242
2025-09-13 08:47:02,526 INFO [make_201702_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-09-13 08:47:02,527 INFO [make_201702_runoff_file] publishing log messages to tcp://*:5564
2025-09-13 08:47:02,528 INFO [make_201702_runoff_file] connected to skookum.eos.ubc.ca port 5555
2025-09-13 08:47:02,556 INFO [make_201702_runoff_file] calculating runoff forcing for b201702 bathymetry
2025-09-13 08:47:02,638 INFO [make_201702_runoff_file] 2025-09-13 runoff file creation from Fraser at Hope and climatology elsewhere complete
2025-09-13 08:47:02,640 INFO [manager] checklist updated with [rivers forcing] items from make_201702_runoff_file worker
2025-09-13 08:47:02,988 INFO [make_runoff_file] running in process 2894240
2025-09-13 08:47:02,989 INFO [make_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-09-13 08:47:02,990 INFO [make_runoff_file] publishing log messages to tcp://*:5566
2025-09-13 08:47:02,991 INFO [make_runoff_file] connected to skookum.eos.ubc.ca port 5555
2025-09-13 08:47:02,992 INFO [make_runoff_file] calculating NEMO runoff forcing for v202108 bathymetry for 2025-09-12
2025-09-13 08:47:05,759 INFO [make_runoff_file] stored NEMO runoff forcing for v202108 bathymetry for 2025-09-12: /results/forcing/rivers/R202108Dailies_y2025m09d12.nc
2025-09-13 08:47:05,760 INFO [make_runoff_file] 2025-09-12 runoff file creation completed
2025-09-13 08:47:05,762 INFO [manager] checklist updated with [rivers forcing] items from make_runoff_file worker
2025-09-13 08:47:06,346 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/hrdps_y2025m09d13.nc
2025-09-13 08:47:13,829 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2025m09d14.nc
2025-09-13 08:47:18,239 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2025m09d15.nc
2025-09-13 08:47:18,253 INFO [grib_to_netcdf] 2025-09-13 NEMO-atmos forcing file for nowcast+ created
2025-09-13 08:47:18,256 INFO [manager] checklist updated with [weather forcing] items from grib_to_netcdf worker
2025-09-13 08:47:34,445 INFO [make_live_ocean_files] Stored T&S western boundary conditions file: /results/forcing/LiveOcean/boundary_conditions/LiveOcean_v201905_y2025m09d13.nc
2025-09-13 08:47:34,445 INFO [make_live_ocean_files] 2025-09-13 Live Ocean western boundary conditions files created
2025-09-13 08:47:34,447 INFO [manager] checklist updated with [Live Ocean boundary conditions] items from make_live_ocean_files worker
2025-09-13 08:47:34,605 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['weather'], host='localhost')
2025-09-13 08:47:34,609 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'nowcast+'], host='localhost')
2025-09-13 08:47:34,613 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'nowcast+'], host='localhost')
2025-09-13 08:47:34,616 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.graham', 'nowcast+'], host='localhost')
2025-09-13 08:47:34,621 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'nowcast+'], host='localhost')
2025-09-13 08:47:35,814 INFO [upload_forcing] running in process 2894295
2025-09-13 08:47:35,816 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-09-13 08:47:35,817 INFO [upload_forcing] publishing log messages to tcp://*:5562
2025-09-13 08:47:35,818 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-09-13 08:47:35,837 INFO [ping_erddap] running in process 2894293
2025-09-13 08:47:35,838 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-09-13 08:47:35,839 INFO [ping_erddap] publishing log messages to tcp://*:5564
2025-09-13 08:47:35,840 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2025-09-13 08:47:35,841 INFO [ping_erddap] weather ERDDAP dataset flag file(s) created
2025-09-13 08:47:35,842 INFO [upload_forcing] running in process 2894296
2025-09-13 08:47:35,843 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-09-13 08:47:35,844 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2025-09-13 08:47:35,845 INFO [upload_forcing] publishing log messages to tcp://*:5565
2025-09-13 08:47:35,846 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-09-13 08:47:35,856 INFO [upload_forcing] running in process 2894298
2025-09-13 08:47:35,857 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-09-13 08:47:35,858 INFO [upload_forcing] publishing log messages to tcp://*:5566
2025-09-13 08:47:35,858 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-09-13 08:47:35,869 INFO [upload_forcing] running in process 2894297
2025-09-13 08:47:35,870 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-09-13 08:47:35,870 INFO [upload_forcing] publishing log messages to tcp://*:5567
2025-09-13 08:47:35,871 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-09-13 08:47:37,799 INFO [upload_forcing] nowcast+ 2025-09-13 forcing files upload to optimum-hindcast completed
2025-09-13 08:47:37,803 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-09-13 08:47:43,327 INFO [upload_forcing] nowcast+ 2025-09-13 forcing files upload to arbutus.cloud-nowcast completed
2025-09-13 08:47:43,333 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-09-13 08:47:43,467 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'nowcast+', '--run-date', '2025-09-13'], host='localhost')
2025-09-13 08:47:44,555 INFO [make_forcing_links] running in process 2894328
2025-09-13 08:47:44,557 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-09-13 08:47:44,559 INFO [make_forcing_links] publishing log messages to tcp://*:5562
2025-09-13 08:47:44,560 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2025-09-13 08:47:45,905 INFO [upload_forcing] nowcast+ 2025-09-13 forcing files upload to robot.graham completed
2025-09-13 08:47:45,910 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-09-13 08:47:46,777 INFO [make_forcing_links] nowcast+ 2025-09-13 forcing file links on arbutus.cloud-nowcast created
2025-09-13 08:47:46,781 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2025-09-13 08:47:46,938 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2025-09-13'], host='arbutus.cloud-nowcast')
2025-09-13 08:47:49,033 INFO [run_NEMO] running in process 23146
2025-09-13 08:47:49,034 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-09-13 08:47:49,035 INFO [run_NEMO] publishing log messages to tcp://*:5556
2025-09-13 08:47:49,150 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2025-09-13 08:47:49,843 WARNING [run_NEMO] There are uncommitted changes in /nemoShare/MEOPAR/nowcast-sys/SS-run-sets
2025-09-13 08:47:50,156 INFO [run_NEMO] nowcast: launching /nemoShare/MEOPAR/nowcast-sys/runs/13sep25nowcast_2025-09-13T084749.081553-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2025-09-13 08:47:50,169 INFO [run_NEMO] nowcast NEMO run for 2025-09-13 on arbutus.cloud-nowcast started
2025-09-13 08:47:50,173 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2025-09-13 08:47:50,339 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'nowcast'], host='arbutus.cloud-nowcast')
2025-09-13 08:47:52,309 INFO [watch_NEMO] running in process 23421
2025-09-13 08:47:52,310 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-09-13 08:47:52,311 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2025-09-13 08:47:52,314 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2025-09-13 08:47:52,375 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2025-09-13 08:49:46,578 CRITICAL [upload_forcing] unhandled exception:
Traceback (most recent call last):
  File "/SalishSeaCast/NEMO_Nowcast/nemo_nowcast/worker.py", line 387, in _do_work
    checklist = self.worker_func(
        self._parsed_args, self.config, self.tell_manager
    )
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/upload_forcing.py", line 94, in upload_forcing
    ssh_client, sftp_client = ssh_sftp.sftp(host_name, ssh_key)
                              ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/ssh_sftp.py", line 144, in sftp
    ssh_client = ssh(host, key_filename, ssh_config_file)
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/ssh_sftp.py", line 75, in ssh
    ssh_client.connect(
    ~~~~~~~~~~~~~~~~~~^
        host["hostname"],
        ^^^^^^^^^^^^^^^^^
    ...<3 lines>...
        look_for_keys=False,
        ^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/paramiko/client.py", line 386, in connect
    sock.connect(addr)
    ~~~~~~~~~~~~^^^^^^
TimeoutError: [Errno 110] Connection timed out
2025-09-13 08:52:52,375 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 18703904 = 2025-09-13 05:08:40 UTC, 21.4% complete
2025-09-13 08:57:52,376 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 18704416 = 2025-09-13 10:50:00 UTC, 45.2% complete
2025-09-13 09:02:52,379 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 18704939 = 2025-09-13 16:38:40 UTC, 69.4% complete
2025-09-13 09:07:52,377 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 18705434 = 2025-09-13 22:08:40 UTC, 92.3% complete
2025-09-13 09:12:52,383 INFO [watch_NEMO] nowcast NEMO run on arbutus.cloud-nowcast completed
2025-09-13 09:12:52,386 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2025-09-13 09:12:52,721 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'ssh', '--run-date', '2025-09-13'], host='localhost')
2025-09-13 09:12:52,726 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2025-09-13'], host='localhost')
2025-09-13 09:12:53,881 INFO [download_results] running in process 2894642
2025-09-13 09:12:53,884 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-09-13 09:12:53,885 INFO [download_results] publishing log messages to tcp://*:5562
2025-09-13 09:12:53,886 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2025-09-13 09:12:53,887 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/nowcast/13sep25 to /results/SalishSea/nowcast-blue.202111
2025-09-13 09:12:53,962 INFO [make_forcing_links] running in process 2894641
2025-09-13 09:12:53,963 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-09-13 09:12:53,964 INFO [make_forcing_links] publishing log messages to tcp://*:5564
2025-09-13 09:12:53,965 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2025-09-13 09:12:54,862 INFO [make_forcing_links] ssh 2025-09-13 forcing file links on arbutus.cloud-nowcast created
2025-09-13 09:12:54,866 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2025-09-13 09:12:55,024 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2025-09-13'], host='arbutus.cloud-nowcast')
2025-09-13 09:12:57,276 INFO [run_NEMO] running in process 23908
2025-09-13 09:12:57,279 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-09-13 09:12:57,280 INFO [run_NEMO] publishing log messages to tcp://*:5556
2025-09-13 09:12:57,281 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2025-09-13 09:12:58,076 WARNING [run_NEMO] There are uncommitted changes in /nemoShare/MEOPAR/nowcast-sys/SS-run-sets
2025-09-13 09:12:58,403 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2025-09-13 09:12:58,405 INFO [run_NEMO] forecast: launching /nemoShare/MEOPAR/nowcast-sys/runs/13sep25forecast_2025-09-13T091257.351138-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2025-09-13 09:12:58,417 INFO [run_NEMO] forecast NEMO run for 2025-09-13 on arbutus.cloud-nowcast started
2025-09-13 09:12:58,568 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'forecast'], host='arbutus.cloud-nowcast')
2025-09-13 09:13:01,016 INFO [watch_NEMO] running in process 24185
2025-09-13 09:13:01,017 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-09-13 09:13:01,019 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2025-09-13 09:13:01,020 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2025-09-13 09:13:01,125 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2025-09-13 09:13:30,545 INFO [download_results] nowcast 2025-09-13 results files from arbutus.cloud-nowcast downloaded
2025-09-13 09:13:30,550 INFO [manager] checklist updated with [results files] items from download_results worker
2025-09-13 09:13:30,709 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast', 'research', '--run-date', '2025-09-13'], host='localhost')
2025-09-13 09:13:30,713 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast', 'comparison', '--run-date', '2025-09-12'], host='localhost')
2025-09-13 09:13:30,717 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['nowcast', '--run-date', '2025-09-13'], host='localhost')
2025-09-13 09:13:32,629 INFO [make_CHS_currents_file] running in process 2894663
2025-09-13 09:13:32,630 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-09-13 09:13:32,631 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5562
2025-09-13 09:13:32,632 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555
2025-09-13 09:13:33,614 INFO [make_plots] running in process 2894662
2025-09-13 09:13:33,615 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-09-13 09:13:33,616 INFO [make_plots] publishing log messages to tcp://*:5564
2025-09-13 09:13:33,617 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2025-09-13 09:13:33,643 INFO [make_plots] running in process 2894660
2025-09-13 09:13:33,658 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-09-13 09:13:33,665 INFO [make_plots] publishing log messages to tcp://*:5565
2025-09-13 09:13:33,666 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2025-09-13 09:13:33,999 INFO [make_plots] preparing render list for 2025-09-13 NEMO nowcast-blue research figures
2025-09-13 09:13:34,000 INFO [make_plots] preparing render list for 2025-09-12 NEMO nowcast-blue comparison figures
2025-09-13 09:13:34,042 INFO [make_plots] added SH_wind figure to 2025-09-12 NEMO nowcast-blue comparison render list
2025-09-13 09:13:34,044 INFO [make_plots] added Compare_VENUS_East figure to 2025-09-12 NEMO nowcast-blue comparison render list
2025-09-13 09:13:34,045 INFO [make_plots] added Compare_VENUS_Central figure to 2025-09-12 NEMO nowcast-blue comparison render list
2025-09-13 09:13:34,046 INFO [make_plots] starting to render nemo nowcast comparison 12sep25 figures
2025-09-13 09:13:35,153 INFO [make_plots] added salinity_thalweg_and_surface figures to 2025-09-13 NEMO nowcast-blue research render list
2025-09-13 09:13:36,025 INFO [make_plots] added temperature_thalweg_and_surface figures to 2025-09-13 NEMO nowcast-blue research render list
2025-09-13 09:13:36,025 INFO [make_plots] added Currents_sections_and_surface figure to 2025-09-13 NEMO nowcast-blue research render list
2025-09-13 09:13:36,026 INFO [make_plots] added Currents_at_VENUS_Central figure to 2025-09-13 NEMO nowcast-blue research render list
2025-09-13 09:13:36,027 INFO [make_plots] added Currents_at_VENUS_East figure to 2025-09-13 NEMO nowcast-blue research render list
2025-09-13 09:13:36,027 INFO [make_plots] starting to render nemo nowcast research 13sep25 figures
2025-09-13 09:14:05,336 INFO [make_CHS_currents_file] Made CHS currents file for 2025-09-13 for nowcast
2025-09-13 09:14:05,339 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker
2025-09-13 09:14:44,004 INFO [make_plots] finished rendering nemo nowcast research 13sep25 figures
2025-09-13 09:14:44,007 INFO [make_plots] nemo research plots for 2025-09-13 nowcast completed
2025-09-13 09:14:44,024 INFO [manager] checklist updated with [plots] items from make_plots worker
2025-09-13 09:15:02,307 INFO [make_plots] finished rendering nemo nowcast comparison 12sep25 figures
2025-09-13 09:15:02,310 INFO [make_plots] nemo comparison plots for 2025-09-12 nowcast completed
2025-09-13 09:15:02,315 INFO [manager] checklist updated with [plots] items from make_plots worker
2025-09-13 09:18:01,121 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 18706065 = 2025-09-14 05:09:20 UTC, 14.3% complete
2025-09-13 09:23:01,123 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 18706580 = 2025-09-14 10:52:40 UTC, 30.2% complete
2025-09-13 09:28:01,125 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 18707095 = 2025-09-14 16:36:00 UTC, 46.1% complete
2025-09-13 09:33:01,127 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 18707598 = 2025-09-14 22:11:20 UTC, 61.7% complete
2025-09-13 09:38:01,128 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 18708115 = 2025-09-15 03:56:00 UTC, 77.6% complete
2025-09-13 09:43:01,127 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 18708594 = 2025-09-15 09:15:20 UTC, 92.4% complete
2025-09-13 09:48:01,129 INFO [watch_NEMO] forecast NEMO run on arbutus.cloud-nowcast completed
2025-09-13 09:48:01,134 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2025-09-13 09:48:01,810 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_wind_file', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2025-09-13'], host='arbutus.cloud-nowcast')
2025-09-13 09:48:01,819 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_current_file', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2025-09-13'], host='arbutus.cloud-nowcast')
2025-09-13 09:48:01,822 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'turbidity', '--run-date', '2025-09-13'], host='localhost')
2025-09-13 09:48:01,828 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'turbidity', '--run-date', '2025-09-13'], host='localhost')
2025-09-13 09:48:01,831 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.graham', 'turbidity', '--run-date', '2025-09-13'], host='localhost')
2025-09-13 09:48:01,835 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'turbidity', '--run-date', '2025-09-13'], host='localhost')
2025-09-13 09:48:01,841 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2025-09-13'], host='localhost')
2025-09-13 09:48:03,067 INFO [upload_forcing] running in process 2895090
2025-09-13 09:48:03,069 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-09-13 09:48:03,070 INFO [upload_forcing] publishing log messages to tcp://*:5562
2025-09-13 09:48:03,071 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-09-13 09:48:03,086 INFO [upload_forcing] running in process 2895086
2025-09-13 09:48:03,087 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-09-13 09:48:03,088 INFO [upload_forcing] publishing log messages to tcp://*:5564
2025-09-13 09:48:03,089 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-09-13 09:48:03,090 INFO [upload_forcing] running in process 2895088
2025-09-13 09:48:03,091 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-09-13 09:48:03,091 INFO [upload_forcing] publishing log messages to tcp://*:5565
2025-09-13 09:48:03,092 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-09-13 09:48:03,093 INFO [download_results] running in process 2895091
2025-09-13 09:48:03,093 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-09-13 09:48:03,094 INFO [download_results] publishing log messages to tcp://*:5566
2025-09-13 09:48:03,095 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2025-09-13 09:48:03,096 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/forecast/13sep25 to /results/SalishSea/forecast.202111
2025-09-13 09:48:03,103 INFO [upload_forcing] running in process 2895089
2025-09-13 09:48:03,105 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-09-13 09:48:03,106 INFO [upload_forcing] publishing log messages to tcp://*:5567
2025-09-13 09:48:03,107 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-09-13 09:48:03,728 INFO [upload_forcing] turbidity 2025-09-13 forcing files upload to optimum-hindcast completed
2025-09-13 09:48:03,732 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-09-13 09:48:04,224 INFO [upload_forcing] turbidity 2025-09-13 forcing files upload to arbutus.cloud-nowcast completed
2025-09-13 09:48:04,228 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-09-13 09:48:04,389 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2025-09-13'], host='localhost')
2025-09-13 09:48:05,103 INFO [make_ww3_wind_file] running in process 25486
2025-09-13 09:48:05,105 INFO [make_ww3_wind_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-09-13 09:48:05,105 INFO [make_ww3_wind_file] publishing log messages to tcp://*:5570
2025-09-13 09:48:05,107 INFO [make_ww3_current_file] running in process 25488
2025-09-13 09:48:05,107 INFO [make_ww3_current_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-09-13 09:48:05,108 INFO [make_ww3_current_file] publishing log messages to tcp://*:5571
2025-09-13 09:48:05,112 INFO [make_ww3_current_file] connected to skookum.eos.ubc.ca port 5555
2025-09-13 09:48:05,113 INFO [make_ww3_current_file] Creating wwatch3 currents forcing file for 2025-09-13 forecast run
2025-09-13 09:48:05,128 INFO [make_ww3_wind_file] connected to skookum.eos.ubc.ca port 5555
2025-09-13 09:48:05,129 INFO [make_ww3_wind_file] Creating wwatch3 wind forcing file for 2025-09-13 forecast run
2025-09-13 09:48:05,596 INFO [make_forcing_links] running in process 2895118
2025-09-13 09:48:05,598 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-09-13 09:48:05,599 INFO [make_forcing_links] publishing log messages to tcp://*:5562
2025-09-13 09:48:05,601 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2025-09-13 09:48:06,790 INFO [make_forcing_links] nowcast-green 2025-09-13 forcing file links on arbutus.cloud-nowcast created
2025-09-13 09:48:06,795 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2025-09-13 09:48:06,966 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2025-09-13'], host='arbutus.cloud-nowcast')
2025-09-13 09:48:07,234 INFO [upload_forcing] turbidity 2025-09-13 forcing files upload to robot.graham completed
2025-09-13 09:48:07,238 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-09-13 09:48:08,989 INFO [run_NEMO] running in process 25802
2025-09-13 09:48:08,990 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-09-13 09:48:08,991 INFO [run_NEMO] publishing log messages to tcp://*:5556
2025-09-13 09:48:08,992 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2025-09-13 09:48:11,365 WARNING [run_NEMO] There are uncommitted changes in /nemoShare/MEOPAR/nowcast-sys/SS-run-sets
2025-09-13 09:48:12,284 INFO [run_NEMO] nowcast-green: launching /nemoShare/MEOPAR/nowcast-sys/runs/13sep25nowcast-green_2025-09-13T094809.039537-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2025-09-13 09:48:12,359 INFO [run_NEMO] nowcast-green NEMO run for 2025-09-13 on arbutus.cloud-nowcast started
2025-09-13 09:48:12,363 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2025-09-13 09:48:12,531 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'nowcast-green'], host='arbutus.cloud-nowcast')
2025-09-13 09:48:16,251 INFO [watch_NEMO] running in process 26399
2025-09-13 09:48:16,254 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-09-13 09:48:16,255 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2025-09-13 09:48:16,255 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2025-09-13 09:48:16,320 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2025-09-13 09:48:17,566 INFO [manager] checklist updated with [WW3 wind forcing] items from make_ww3_wind_file worker
2025-09-13 09:48:17,568 INFO [make_ww3_wind_file] wwatch3 wind forcing file created on arbutus.cloud-nowcast for 2025-09-13 forecast run
2025-09-13 09:50:13,581 CRITICAL [upload_forcing] unhandled exception:
Traceback (most recent call last):
  File "/SalishSeaCast/NEMO_Nowcast/nemo_nowcast/worker.py", line 387, in _do_work
    checklist = self.worker_func(
        self._parsed_args, self.config, self.tell_manager
    )
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/upload_forcing.py", line 94, in upload_forcing
    ssh_client, sftp_client = ssh_sftp.sftp(host_name, ssh_key)
                              ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/ssh_sftp.py", line 144, in sftp
    ssh_client = ssh(host, key_filename, ssh_config_file)
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/ssh_sftp.py", line 75, in ssh
    ssh_client.connect(
    ~~~~~~~~~~~~~~~~~~^
        host["hostname"],
        ^^^^^^^^^^^^^^^^^
    ...<3 lines>...
        look_for_keys=False,
        ^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/paramiko/client.py", line 386, in connect
    sock.connect(addr)
    ~~~~~~~~~~~~^^^^^^
TimeoutError: [Errno 110] Connection timed out
2025-09-13 09:50:42,537 INFO [make_ww3_current_file] wwatch3 currents forcing file created on arbutus.cloud-nowcast for 2025-09-13 forecast run
2025-09-13 09:50:42,542 INFO [manager] checklist updated with [WW3 currents forcing] items from make_ww3_current_file worker
2025-09-13 09:50:42,706 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2025-09-13'], host='arbutus.cloud-nowcast')
2025-09-13 09:50:44,811 INFO [run_ww3] running in process 26552
2025-09-13 09:50:44,813 INFO [run_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-09-13 09:50:44,814 INFO [run_ww3] publishing log messages to tcp://*:5572
2025-09-13 09:50:44,816 INFO [run_ww3] connected to skookum.eos.ubc.ca port 5555
2025-09-13 09:50:44,820 INFO [run_ww3] Created run directory /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/nowcast_2025-09-13T095044.813070-0700
2025-09-13 09:50:44,822 INFO [run_ww3] nowcast: launching /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/nowcast_2025-09-13T095044.813070-0700/SoGWW3.sh on arbutus.cloud-nowcast
2025-09-13 09:50:44,853 INFO [run_ww3] nowcast WaveWatch3 run for 2025-09-13 on arbutus.cloud-nowcast started
2025-09-13 09:50:44,856 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker
2025-09-13 09:50:45,027 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'nowcast'], host='arbutus.cloud-nowcast')
2025-09-13 09:50:47,058 INFO [watch_ww3] running in process 26639
2025-09-13 09:50:47,059 INFO [watch_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-09-13 09:50:47,060 INFO [watch_ww3] publishing log messages to tcp://*:5573
2025-09-13 09:50:47,061 INFO [watch_ww3] connected to skookum.eos.ubc.ca port 5555
2025-09-13 09:50:47,129 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2025-09-13 09:53:16,326 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18703608 = 2025-09-13 01:51:20 UTC, 7.7% complete
2025-09-13 09:55:21,360 INFO [download_results] forecast 2025-09-13 results files from arbutus.cloud-nowcast downloaded
2025-09-13 09:55:21,364 INFO [manager] checklist updated with [results files] items from download_results worker
2025-09-13 09:55:21,533 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['forecast', '--run-date', '2025-09-13'], host='localhost')
2025-09-13 09:55:23,355 INFO [make_CHS_currents_file] running in process 2895228
2025-09-13 09:55:23,356 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-09-13 09:55:23,357 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5562
2025-09-13 09:55:23,358 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555
2025-09-13 09:55:47,129 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 93, 21.5% complete
2025-09-13 09:56:07,691 INFO [make_CHS_currents_file] Made CHS currents file for 2025-09-13 for forecast
2025-09-13 09:56:07,694 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker
2025-09-13 09:56:07,861 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['nemo', 'forecast', '--run-date', '2025-09-13'], host='localhost')
2025-09-13 09:56:08,893 INFO [update_forecast_datasets] running in process 2895257
2025-09-13 09:56:08,894 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-09-13 09:56:08,895 INFO [update_forecast_datasets] publishing log messages to tcp://*:5562
2025-09-13 09:56:08,896 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2025-09-13 09:56:08,897 INFO [update_forecast_datasets] updating nemo forecast directory for forecast/13sep25 run
2025-09-13 09:56:09,161 INFO [update_forecast_datasets] updated nemo forecast directory for forecast/13sep25 run: /results/SalishSea/rolling-forecasts/nemo
2025-09-13 09:56:09,165 INFO [update_forecast_datasets] nemo 2025-09-13 forecast rolling forecast datasets updated
2025-09-13 09:56:09,167 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2025-09-13 09:56:09,336 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nemo-forecast'], host='localhost')
2025-09-13 09:56:09,341 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'forecast', 'publish', '--run-date', '2025-09-13'], host='localhost')
2025-09-13 09:56:09,345 INFO [manager] launching NextWorker(module='nowcast.workers.make_surface_current_tiles', args=['forecast', '--run-date', '2025-09-13'], host='localhost')
2025-09-13 09:56:10,474 INFO [ping_erddap] running in process 2895264
2025-09-13 09:56:10,476 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-09-13 09:56:10,476 INFO [ping_erddap] publishing log messages to tcp://*:5562
2025-09-13 09:56:10,477 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2025-09-13 09:56:10,485 INFO [ping_erddap] nemo-forecast ERDDAP dataset flag file(s) created
2025-09-13 09:56:10,487 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2025-09-13 09:56:11,513 INFO [make_surface_current_tiles] running in process 2895267
2025-09-13 09:56:11,514 INFO [make_surface_current_tiles] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-09-13 09:56:11,515 INFO [make_surface_current_tiles] publishing log messages to tcp://*:5562
2025-09-13 09:56:11,516 INFO [make_surface_current_tiles] connected to skookum.eos.ubc.ca port 5555
2025-09-13 09:56:11,518 INFO [make_surface_current_tiles] starting rendering of tiles for 2025-09-13 forecast in 6 processes into /results/nowcast-sys/figures/surface_currents/forecast/13sep25
2025-09-13 09:56:12,198 INFO [make_plots] running in process 2895266
2025-09-13 09:56:12,199 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-09-13 09:56:12,200 INFO [make_plots] publishing log messages to tcp://*:5564
2025-09-13 09:56:12,200 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2025-09-13 09:56:12,653 INFO [make_plots] preparing render list for 2025-09-13 NEMO forecast publish figures
2025-09-13 09:56:12,798 INFO [make_plots] added Website_thumbnail figure to 2025-09-13 NEMO forecast publish render list
2025-09-13 09:56:12,803 INFO [make_plots] added Threshold_website figure to 2025-09-13 NEMO forecast publish render list
2025-09-13 09:56:12,804 INFO [make_plots] added PA_tidal_predictions figure to 2025-09-13 NEMO forecast publish render list
2025-09-13 09:56:12,804 INFO [make_plots] added Boundary Bay figure to 2025-09-13 NEMO forecast publish render list
2025-09-13 09:56:12,805 INFO [make_plots] added Campbell River figure to 2025-09-13 NEMO forecast publish render list
2025-09-13 09:56:12,805 INFO [make_plots] added Cherry Point figure to 2025-09-13 NEMO forecast publish render list
2025-09-13 09:56:12,806 INFO [make_plots] added Friday Harbor figure to 2025-09-13 NEMO forecast publish render list
2025-09-13 09:56:12,807 INFO [make_plots] added Halfmoon Bay figure to 2025-09-13 NEMO forecast publish render list
2025-09-13 09:56:12,807 INFO [make_plots] added Nanaimo figure to 2025-09-13 NEMO forecast publish render list
2025-09-13 09:56:12,808 INFO [make_plots] added Neah Bay figure to 2025-09-13 NEMO forecast publish render list
2025-09-13 09:56:12,809 INFO [make_plots] added New Westminster figure to 2025-09-13 NEMO forecast publish render list
2025-09-13 09:56:12,810 INFO [make_plots] added Patricia Bay figure to 2025-09-13 NEMO forecast publish render list
2025-09-13 09:56:12,810 INFO [make_plots] added Point Atkinson figure to 2025-09-13 NEMO forecast publish render list
2025-09-13 09:56:12,811 INFO [make_plots] added Port Renfrew figure to 2025-09-13 NEMO forecast publish render list
2025-09-13 09:56:12,812 INFO [make_plots] added Sand Heads figure to 2025-09-13 NEMO forecast publish render list
2025-09-13 09:56:12,813 INFO [make_plots] added Sandy Cove figure to 2025-09-13 NEMO forecast publish render list
2025-09-13 09:56:12,814 INFO [make_plots] added Squamish figure to 2025-09-13 NEMO forecast publish render list
2025-09-13 09:56:12,815 INFO [make_plots] added Victoria figure to 2025-09-13 NEMO forecast publish render list
2025-09-13 09:56:12,815 INFO [make_plots] added Woodwards Landing figure to 2025-09-13 NEMO forecast publish render list
2025-09-13 09:56:12,816 INFO [make_plots] starting to render nemo forecast publish 13sep25 figures
2025-09-13 09:58:16,326 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18703796 = 2025-09-13 03:56:40 UTC, 16.4% complete
2025-09-13 10:00:47,128 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 189, 43.8% complete
2025-09-13 10:03:16,337 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18703981 = 2025-09-13 06:00:00 UTC, 25.0% complete
2025-09-13 10:04:35,724 WARNING [make_plots] invalid station number for Boundary Bay station: None
2025-09-13 10:04:35,725 WARNING [make_plots] can't resolve a valid CHS station code for Boundary Bay
2025-09-13 10:05:00,256 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb996/data for station 08074 Campbell River from 2025-09-08 00:00:00Z to 2025-09-15 11:55:00Z
2025-09-13 10:05:47,128 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 288, 66.7% complete
2025-09-13 10:06:06,929 WARNING [make_plots] invalid station number for Halfmoon Bay station: None
2025-09-13 10:06:06,930 WARNING [make_plots] can't resolve a valid CHS station code for Halfmoon Bay
2025-09-13 10:06:27,145 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb96d/data for station 07917 Nanaimo from 2025-09-08 00:00:00Z to 2025-09-15 11:55:00Z
2025-09-13 10:06:42,626 INFO [make_surface_current_tiles] starting PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast/13sep25
2025-09-13 10:07:01,279 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e13d0f4a073c4bbf93/data for station 07654 New Westminster from 2025-09-08 00:00:00Z to 2025-09-15 11:55:00Z
2025-09-13 10:07:18,563 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd2d/data for station 07277 Patricia Bay from 2025-09-08 00:00:00Z to 2025-09-15 11:55:00Z
2025-09-13 10:07:35,329 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94c/data for station 07795 Point Atkinson from 2025-09-08 00:00:00Z to 2025-09-15 11:55:00Z
2025-09-13 10:07:52,566 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e23d0f4a073c4bc060/data for station 08525 Port Renfrew from 2025-09-08 00:00:00Z to 2025-09-15 11:55:00Z
2025-09-13 10:08:08,984 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb93c/data for station 07594 Sand Heads from 2025-09-08 00:00:00Z to 2025-09-15 11:55:00Z
2025-09-13 10:08:16,325 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18704169 = 2025-09-13 08:05:20 UTC, 33.7% complete
2025-09-13 10:08:26,043 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc43b/data for station 07786 Sandy Cove from 2025-09-08 00:00:00Z to 2025-09-15 11:55:00Z
2025-09-13 10:08:43,378 WARNING [make_plots] invalid station number for Squamish station: None
2025-09-13 10:08:43,379 WARNING [make_plots] can't resolve a valid CHS station code for Squamish
2025-09-13 10:08:59,901 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd1e/data for station 07120 Victoria from 2025-09-08 00:00:00Z to 2025-09-15 11:55:00Z
2025-09-13 10:09:15,875 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc42d/data for station 07610 Woodwards Landing from 2025-09-08 00:00:00Z to 2025-09-15 11:55:00Z
2025-09-13 10:09:32,748 INFO [make_plots] finished rendering nemo forecast publish 13sep25 figures
2025-09-13 10:09:32,750 INFO [make_plots] nemo publish plots for 2025-09-13 forecast completed
2025-09-13 10:09:32,754 INFO [manager] checklist updated with [plots] items from make_plots worker
2025-09-13 10:09:32,889 INFO [manager] launching NextWorker(module='nowcast.workers.make_feeds', args=['forecast', '--run-date', '2025-09-13'], host='localhost')
2025-09-13 10:09:35,577 INFO [make_feeds] running in process 2897329
2025-09-13 10:09:35,578 INFO [make_feeds] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-09-13 10:09:35,580 INFO [make_feeds] publishing log messages to tcp://*:5564
2025-09-13 10:09:35,581 INFO [make_feeds] connected to skookum.eos.ubc.ca port 5555
2025-09-13 10:09:49,876 INFO [make_feeds] ATOM feeds for 2025-09-13 forecast run completed
2025-09-13 10:09:49,879 INFO [manager] checklist updated with [storm surge alert feeds] items from make_feeds worker
2025-09-13 10:10:47,129 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 387, 89.6% complete
2025-09-13 10:11:23,172 INFO [make_surface_current_tiles] finished PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast/13sep25
2025-09-13 10:11:23,195 INFO [make_surface_current_tiles] finished rendering of tiles for 2025-09-13 forecast into /results/nowcast-sys/figures/surface_currents/forecast/13sep25
2025-09-13 10:11:23,196 INFO [make_surface_current_tiles] surface current tile figures for 2025-09-13 forecast completed
2025-09-13 10:11:23,517 INFO [manager] checklist updated with [surface current tiles] items from make_surface_current_tiles worker
2025-09-13 10:13:16,336 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18704360 = 2025-09-13 10:12:40 UTC, 42.6% complete
2025-09-13 10:15:47,129 INFO [watch_ww3] nowcast WWATCH3 run on arbutus.cloud-nowcast completed
2025-09-13 10:15:47,139 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2025-09-13 10:15:47,638 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2025-09-13'], host='localhost')
2025-09-13 10:15:47,644 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2025-09-13'], host='arbutus.cloud-nowcast')
2025-09-13 10:15:48,779 INFO [download_wwatch3_results] running in process 2897418
2025-09-13 10:15:48,781 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-09-13 10:15:48,782 INFO [download_wwatch3_results] publishing log messages to tcp://*:5562
2025-09-13 10:15:48,783 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2025-09-13 10:15:49,665 INFO [run_ww3] running in process 27081
2025-09-13 10:15:49,666 INFO [run_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-09-13 10:15:49,667 INFO [run_ww3] publishing log messages to tcp://*:5572
2025-09-13 10:15:49,668 INFO [run_ww3] connected to skookum.eos.ubc.ca port 5555
2025-09-13 10:15:49,673 INFO [run_ww3] Created run directory /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast_2025-09-13T101549.670568-0700
2025-09-13 10:15:49,674 INFO [run_ww3] forecast: launching /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast_2025-09-13T101549.670568-0700/SoGWW3.sh on arbutus.cloud-nowcast
2025-09-13 10:15:49,685 INFO [run_ww3] forecast WaveWatch3 run for 2025-09-13 on arbutus.cloud-nowcast started
2025-09-13 10:15:49,699 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker
2025-09-13 10:15:50,025 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'forecast'], host='arbutus.cloud-nowcast')
2025-09-13 10:15:52,294 INFO [watch_ww3] running in process 27236
2025-09-13 10:15:52,298 INFO [watch_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-09-13 10:15:52,299 INFO [watch_ww3] publishing log messages to tcp://*:5573
2025-09-13 10:15:52,300 INFO [watch_ww3] connected to skookum.eos.ubc.ca port 5555
2025-09-13 10:15:52,334 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2025-09-13 10:16:21,092 INFO [download_wwatch3_results] nowcast 2025-09-13 results files from arbutus.cloud-nowcast downloaded
2025-09-13 10:16:21,095 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2025-09-13 10:18:16,335 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18704544 = 2025-09-13 12:15:20 UTC, 51.1% complete
2025-09-13 10:20:52,336 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 93, 14.4% complete
2025-09-13 10:23:16,332 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18704727 = 2025-09-13 14:17:20 UTC, 59.6% complete
2025-09-13 10:25:52,337 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 192, 29.6% complete
2025-09-13 10:28:16,337 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18704913 = 2025-09-13 16:21:20 UTC, 68.2% complete
2025-09-13 10:30:52,337 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 297, 45.8% complete
2025-09-13 10:33:16,332 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18705096 = 2025-09-13 18:23:20 UTC, 76.7% complete
2025-09-13 10:35:52,337 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 399, 61.6% complete
2025-09-13 10:38:16,343 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18705282 = 2025-09-13 20:27:20 UTC, 85.3% complete
2025-09-13 10:40:52,345 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 504, 77.8% complete
2025-09-13 10:43:16,333 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18705466 = 2025-09-13 22:30:00 UTC, 93.8% complete
2025-09-13 10:45:52,346 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 606, 93.5% complete
2025-09-13 10:48:16,337 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18705600 = 2025-09-13 23:59:20 UTC, 100.0% complete
2025-09-13 10:50:52,350 INFO [watch_ww3] forecast WWATCH3 run on arbutus.cloud-nowcast completed
2025-09-13 10:50:52,352 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2025-09-13 10:50:52,835 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2025-09-13'], host='localhost')
2025-09-13 10:50:53,957 INFO [download_wwatch3_results] running in process 2897782
2025-09-13 10:50:53,959 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-09-13 10:50:53,960 INFO [download_wwatch3_results] publishing log messages to tcp://*:5562
2025-09-13 10:50:53,961 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2025-09-13 10:51:31,648 INFO [download_wwatch3_results] forecast 2025-09-13 results files from arbutus.cloud-nowcast downloaded
2025-09-13 10:51:31,652 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2025-09-13 10:51:31,977 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['wwatch3', 'forecast', '--run-date', '2025-09-13'], host='localhost')
2025-09-13 10:51:33,169 INFO [update_forecast_datasets] running in process 2897799
2025-09-13 10:51:33,171 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-09-13 10:51:33,172 INFO [update_forecast_datasets] publishing log messages to tcp://*:5562
2025-09-13 10:51:33,173 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2025-09-13 10:51:33,174 INFO [update_forecast_datasets] updating wwatch3 most_recent_forecast directory from forecast/13sep25 run
2025-09-13 10:51:33,177 INFO [update_forecast_datasets] updated wwatch3 most_recent_forecast directory from forecast/13sep25 run
2025-09-13 10:51:33,178 INFO [update_forecast_datasets] updating wwatch3 forecast directory for forecast/13sep25 run
2025-09-13 10:51:33,221 INFO [update_forecast_datasets] updated wwatch3 forecast directory for forecast/13sep25 run: /results/SalishSea/rolling-forecasts/wwatch3
2025-09-13 10:51:33,222 INFO [update_forecast_datasets] wwatch3 2025-09-13 forecast rolling forecast datasets updated
2025-09-13 10:51:33,225 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2025-09-13 10:51:33,549 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['wwatch3-forecast'], host='localhost')
2025-09-13 10:51:34,587 INFO [ping_erddap] running in process 2897805
2025-09-13 10:51:34,589 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-09-13 10:51:34,590 INFO [ping_erddap] publishing log messages to tcp://*:5562
2025-09-13 10:51:34,590 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2025-09-13 10:51:34,592 INFO [ping_erddap] wwatch3-forecast ERDDAP dataset flag file(s) created
2025-09-13 10:51:34,593 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2025-09-13 10:51:34,854 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['wwatch3', 'forecast', 'publish', '--run-date', '2025-09-13'], host='localhost')
2025-09-13 10:51:37,809 INFO [make_plots] running in process 2897811
2025-09-13 10:51:37,810 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-09-13 10:51:37,810 INFO [make_plots] publishing log messages to tcp://*:5562
2025-09-13 10:51:37,811 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2025-09-13 10:51:37,812 INFO [make_plots] preparing render list for 2025-09-13 WaveWatch3 forecast publish figures
2025-09-13 10:51:37,812 INFO [make_plots] added Halibut Bank figure to 2025-09-13 WaveWatch3 forecast publish render list
2025-09-13 10:51:37,813 INFO [make_plots] added Sentry Shoal figure to 2025-09-13 WaveWatch3 forecast publish render list
2025-09-13 10:51:37,814 INFO [make_plots] starting to render wwatch3 forecast publish 13sep25 figures
2025-09-13 10:51:38,327 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46146 Halibut Bank
2025-09-13 10:51:44,967 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46131 Sentry Shoal
2025-09-13 10:51:50,262 INFO [make_plots] finished rendering wwatch3 forecast publish 13sep25 figures
2025-09-13 10:51:50,263 INFO [make_plots] wwatch3 publish plots for 2025-09-13 forecast completed
2025-09-13 10:51:50,267 INFO [manager] checklist updated with [plots] items from make_plots worker
2025-09-13 10:53:16,473 INFO [watch_NEMO] nowcast-green NEMO run on arbutus.cloud-nowcast completed
2025-09-13 10:53:16,476 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2025-09-13 10:53:16,970 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2025-09-13'], host='localhost')
2025-09-13 10:53:18,136 INFO [download_results] running in process 2897848
2025-09-13 10:53:18,138 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-09-13 10:53:18,138 INFO [download_results] publishing log messages to tcp://*:5562
2025-09-13 10:53:18,139 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2025-09-13 10:53:18,140 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/nowcast-green/13sep25 to /results2/SalishSea/nowcast-green.202111
2025-09-13 10:57:52,196 INFO [download_results] nowcast-green 2025-09-13 results files from arbutus.cloud-nowcast downloaded
2025-09-13 10:57:52,202 INFO [manager] checklist updated with [results files] items from download_results worker
2025-09-13 10:57:52,539 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast-green', 'research', '--run-date', '2025-09-13'], host='localhost')
2025-09-13 10:57:52,544 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nowcast-green'], host='localhost')
2025-09-13 10:57:52,550 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'physics', '--run-date', '2025-09-13'], host='localhost')
2025-09-13 10:57:52,555 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'biology', '--run-date', '2025-09-13'], host='localhost')
2025-09-13 10:57:52,559 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'chemistry', '--run-date', '2025-09-13'], host='localhost')
2025-09-13 10:57:54,029 INFO [ping_erddap] running in process 2897895
2025-09-13 10:57:54,031 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-09-13 10:57:54,032 INFO [ping_erddap] publishing log messages to tcp://*:5562
2025-09-13 10:57:54,033 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2025-09-13 10:57:54,038 INFO [ping_erddap] nowcast-green ERDDAP dataset flag file(s) created
2025-09-13 10:57:54,040 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2025-09-13 10:57:55,091 INFO [make_averaged_dataset] running in process 2897896
2025-09-13 10:57:55,092 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-09-13 10:57:55,093 INFO [make_averaged_dataset] publishing log messages to tcp://*:5562
2025-09-13 10:57:55,094 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555
2025-09-13 10:57:55,095 INFO [make_averaged_dataset] creating day-averaged dataset for 13-Sep-2025 physics
2025-09-13 10:57:55,100 INFO [make_averaged_dataset] running in process 2897897
2025-09-13 10:57:55,101 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-09-13 10:57:55,103 INFO [make_averaged_dataset] publishing log messages to tcp://*:5564
2025-09-13 10:57:55,103 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555
2025-09-13 10:57:55,104 INFO [make_averaged_dataset] creating day-averaged dataset for 13-Sep-2025 biology
2025-09-13 10:57:55,112 INFO [make_averaged_dataset] running in process 2897898
2025-09-13 10:57:55,113 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-09-13 10:57:55,114 INFO [make_averaged_dataset] publishing log messages to tcp://*:5565
2025-09-13 10:57:55,115 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555
2025-09-13 10:57:55,116 INFO [make_averaged_dataset] creating day-averaged dataset for 13-Sep-2025 chemistry
2025-09-13 10:57:55,903 INFO [make_plots] running in process 2897893
2025-09-13 10:57:55,904 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-09-13 10:57:55,905 INFO [make_plots] publishing log messages to tcp://*:5566
2025-09-13 10:57:55,906 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2025-09-13 10:57:56,248 INFO [make_plots] preparing render list for 2025-09-13 NEMO nowcast-green research figures
2025-09-13 10:57:58,378 INFO [make_plots] added temperature_thalweg_and_surface figures to 2025-09-13 NEMO nowcast-green research render list
2025-09-13 10:57:59,160 INFO [make_plots] added nitrate_thalweg_and_surface figures to 2025-09-13 NEMO nowcast-green research render list
2025-09-13 10:57:59,933 INFO [make_plots] added ammonium_thalweg_and_surface figures to 2025-09-13 NEMO nowcast-green research render list
2025-09-13 10:58:00,718 INFO [make_plots] added silicon_thalweg_and_surface figures to 2025-09-13 NEMO nowcast-green research render list
2025-09-13 10:58:01,520 INFO [make_plots] added dissolved_organic_nitrogen_thalweg_and_surface figures to 2025-09-13 NEMO nowcast-green research render list
2025-09-13 10:58:02,313 INFO [make_plots] added particulate_organic_nitrogen_thalweg_and_surface figures to 2025-09-13 NEMO nowcast-green research render list
2025-09-13 10:58:03,094 INFO [make_plots] added biogenic_silicon_thalweg_and_surface figures to 2025-09-13 NEMO nowcast-green research render list
2025-09-13 10:58:04,012 INFO [make_plots] added diatoms_thalweg_and_surface figures to 2025-09-13 NEMO nowcast-green research render list
2025-09-13 10:58:04,952 INFO [make_plots] added flagellates_thalweg_and_surface figures to 2025-09-13 NEMO nowcast-green research render list
2025-09-13 10:58:05,859 INFO [make_plots] added microzooplankton_thalweg_and_surface figures to 2025-09-13 NEMO nowcast-green research render list
2025-09-13 10:58:06,829 INFO [make_plots] added mesozooplankton_thalweg_and_surface figures to 2025-09-13 NEMO nowcast-green research render list
2025-09-13 10:58:07,665 INFO [make_plots] added turbidity_thalweg_and_surface figures to 2025-09-13 NEMO nowcast-green research render list
2025-09-13 10:58:08,776 INFO [make_plots] added temperature_salinity_timeseries figure to 2025-09-13 NEMO nowcast-green research render list
2025-09-13 10:58:08,777 INFO [make_plots] added nitrate_diatoms_timeseries figure to 2025-09-13 NEMO nowcast-green research render list
2025-09-13 10:58:08,778 INFO [make_plots] added diatoms_flagellates_timeseries figure to 2025-09-13 NEMO nowcast-green research render list
2025-09-13 10:58:08,779 INFO [make_plots] added z1_z2_zooplankton_timeseries figure to 2025-09-13 NEMO nowcast-green research render list
2025-09-13 10:58:08,780 INFO [make_plots] starting to render nemo nowcast-green research 13sep25 figures
2025-09-13 10:59:15,158 INFO [make_averaged_dataset] day-averaged dataset for 13-Sep-2025 physics created
2025-09-13 10:59:15,162 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker
2025-09-13 10:59:16,143 INFO [make_averaged_dataset] day-averaged dataset for 13-Sep-2025 chemistry created
2025-09-13 10:59:16,147 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker
2025-09-13 10:59:49,407 INFO [make_averaged_dataset] day-averaged dataset for 13-Sep-2025 biology created
2025-09-13 10:59:49,410 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker
2025-09-13 11:53:25,578 INFO [make_plots] finished rendering nemo nowcast-green research 13sep25 figures
2025-09-13 11:53:25,589 INFO [make_plots] nemo research plots for 2025-09-13 nowcast-green completed
2025-09-13 11:53:25,656 INFO [manager] checklist updated with [plots] items from make_plots worker
2025-09-13 14:50:07,754 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/18/ to /results/forcing/atmospheric/continental2.5/GRIB/20250913/18/
2025-09-13 14:50:07,755 INFO [collect_weather] 2025-09-13 2.5km weather forecast 18 collection complete
2025-09-13 14:50:07,759 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2025-09-13 14:50:08,271 INFO [manager] launching NextWorker(module='nowcast.workers.download_weather', args=['00', '1km'], host='localhost')
2025-09-13 14:50:08,277 INFO [manager] launching NextWorker(module='nowcast.workers.download_weather', args=['12', '1km'], host='localhost')
2025-09-13 14:50:08,282 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['00', '2.5km'], host='localhost')
2025-09-13 14:50:08,287 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['00', '--fcst-date', '2025-09-14'], host='localhost')
2025-09-13 14:50:08,446 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20250913/18/
2025-09-13 14:50:08,447 INFO [crop_gribs] 2025-09-13 18 GRIBs cropping complete
2025-09-13 14:50:08,448 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2025-09-13 14:50:09,694 INFO [download_weather] running in process 2900835
2025-09-13 14:50:09,696 INFO [download_weather] running in process 2900837
2025-09-13 14:50:09,696 INFO [download_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-09-13 14:50:09,697 INFO [download_weather] publishing log messages to tcp://*:5562
2025-09-13 14:50:09,699 INFO [download_weather] connected to skookum.eos.ubc.ca port 5555
2025-09-13 14:50:09,700 INFO [download_weather] downloading 00 1 km forecast GRIB2 files for 20250913
2025-09-13 14:50:09,701 INFO [download_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-09-13 14:50:09,702 INFO [collect_weather] running in process 2900838
2025-09-13 14:50:09,703 INFO [download_weather] publishing log messages to tcp://*:5563
2025-09-13 14:50:09,703 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-09-13 14:50:09,704 INFO [download_weather] connected to skookum.eos.ubc.ca port 5555
2025-09-13 14:50:09,705 INFO [collect_weather] publishing log messages to tcp://*:5564
2025-09-13 14:50:09,705 INFO [download_weather] downloading 12 1 km forecast GRIB2 files for 20250913
2025-09-13 14:50:09,706 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2025-09-13 14:50:09,712 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/00/
2025-09-13 14:50:10,508 INFO [crop_gribs] running in process 2900839
2025-09-13 14:50:10,509 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-09-13 14:50:10,510 INFO [crop_gribs] publishing log messages to tcp://*:5565
2025-09-13 14:50:10,511 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2025-09-13 14:50:10,512 INFO [crop_gribs] cropping 2025-09-14 ECCC HRDPS 2.5km continental 00Z GRIB files to SalishSeaCast subdomain
2025-09-13 14:50:10,526 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20250914/00/
2025-09-13 14:53:39,048 INFO [download_weather] 2025-09-13 1km weather forecast 00 downloads complete
2025-09-13 14:53:39,052 INFO [manager] checklist updated with [weather forecast] items from download_weather worker
2025-09-13 14:53:45,458 INFO [download_weather] 2025-09-13 1km weather forecast 12 downloads complete
2025-09-13 14:53:45,462 INFO [manager] checklist updated with [weather forecast] items from download_weather worker
2025-09-13 20:47:03,414 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/00/ to /results/forcing/atmospheric/continental2.5/GRIB/20250914/00/
2025-09-13 20:47:03,415 INFO [collect_weather] 2025-09-14 2.5km weather forecast 00 collection complete
2025-09-13 20:47:03,418 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2025-09-13 20:47:03,947 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['06', '2.5km'], host='localhost')
2025-09-13 20:47:03,952 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['06', '--fcst-date', '2025-09-14'], host='localhost')
2025-09-13 20:47:05,606 INFO [collect_weather] running in process 2904692
2025-09-13 20:47:05,608 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-09-13 20:47:05,608 INFO [collect_weather] publishing log messages to tcp://*:5562
2025-09-13 20:47:05,609 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2025-09-13 20:47:05,626 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/06/
2025-09-13 20:47:05,945 INFO [crop_gribs] running in process 2904694
2025-09-13 20:47:05,947 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-09-13 20:47:05,948 INFO [crop_gribs] publishing log messages to tcp://*:5563
2025-09-13 20:47:05,949 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2025-09-13 20:47:05,950 INFO [crop_gribs] cropping 2025-09-14 ECCC HRDPS 2.5km continental 06Z GRIB files to SalishSeaCast subdomain
2025-09-13 20:47:05,963 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20250914/06/
2025-09-13 22:50:11,675 INFO [crop_gribs] crop_gribs 00 has watched for 8h; retrying remaining 1 file(s)
2025-09-13 22:50:13,434 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20250914/00/
2025-09-13 22:50:13,435 INFO [crop_gribs] 2025-09-14 00 GRIBs cropping complete
2025-09-13 22:50:13,437 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2025-09-14 02:49:24,358 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/06/ to /results/forcing/atmospheric/continental2.5/GRIB/20250914/06/
2025-09-14 02:49:24,359 INFO [collect_weather] 2025-09-14 2.5km weather forecast 06 collection complete
2025-09-14 02:49:24,362 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2025-09-14 02:49:24,876 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Capilano', '--data-date', '2025-09-13'], host='localhost')
2025-09-14 02:49:24,882 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'ChilliwackVedder', '--data-date', '2025-09-13'], host='localhost')
2025-09-14 02:49:24,886 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'ClowhomClowhomLake', '--data-date', '2025-09-13'], host='localhost')
2025-09-14 02:49:24,890 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Englishman', '--data-date', '2025-09-13'], host='localhost')
2025-09-14 02:49:24,894 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Fraser', '--data-date', '2025-09-13'], host='localhost')
2025-09-14 02:49:24,898 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'HomathkoMouth', '--data-date', '2025-09-13'], host='localhost')
2025-09-14 02:49:24,902 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'NicomeklLangley', '--data-date', '2025-09-13'], host='localhost')
2025-09-14 02:49:24,906 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'RobertsCreek', '--data-date', '2025-09-13'], host='localhost')
2025-09-14 02:49:24,912 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SalmonSayward', '--data-date', '2025-09-13'], host='localhost')
2025-09-14 02:49:24,917 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SanJuanPortRenfrew', '--data-date', '2025-09-13'], host='localhost')
2025-09-14 02:49:24,921 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SquamishBrackendale', '--data-date', '2025-09-13'], host='localhost')
2025-09-14 02:49:24,927 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaScotty', '--data-date', '2025-09-13'], host='localhost')
2025-09-14 02:49:24,952 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaBypass', '--data-date', '2025-09-13'], host='localhost')
2025-09-14 02:49:24,965 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaDiversion', '--data-date', '2025-09-13'], host='localhost')
2025-09-14 02:49:24,976 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ctd', args=['SCVIP'], host='localhost')
2025-09-14 02:49:24,986 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20250914/06/
2025-09-14 02:49:24,990 INFO [crop_gribs] 2025-09-14 06 GRIBs cropping complete
2025-09-14 02:49:24,993 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ctd', args=['SEVIP'], host='localhost')
2025-09-14 02:49:25,005 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ferry', args=['TWDP'], host='localhost')
2025-09-14 02:49:25,014 INFO [manager] launching NextWorker(module='nowcast.workers.get_vfpa_hadcp', args=['--data-date', '2025-09-13'], host='localhost')
2025-09-14 02:49:25,042 INFO [manager] launching NextWorker(module='nowcast.workers.collect_NeahBay_ssh', args=['00'], host='localhost')
2025-09-14 02:49:25,055 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['12', '2.5km'], host='localhost')
2025-09-14 02:49:25,069 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['12'], host='localhost')
2025-09-14 02:49:25,085 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2025-09-14 02:49:26,493 INFO [manager] launching NextWorker(module='nowcast.workers.grib_to_netcdf', args=['forecast2'], host='localhost')
2025-09-14 02:49:28,291 INFO [collect_NeahBay_ssh] running in process 2908473
2025-09-14 02:49:28,293 INFO [collect_NeahBay_ssh] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-09-14 02:49:28,294 INFO [collect_NeahBay_ssh] publishing log messages to tcp://*:5562
2025-09-14 02:49:28,294 INFO [collect_NeahBay_ssh] connected to skookum.eos.ubc.ca port 5555
2025-09-14 02:49:28,295 INFO [collect_NeahBay_ssh] collecting Neah Bay ssh 00Z obs/forecast for 2025-09-14
2025-09-14 02:49:28,302 INFO [collect_weather] running in process 2908474
2025-09-14 02:49:28,303 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-09-14 02:49:28,304 INFO [collect_weather] publishing log messages to tcp://*:5563
2025-09-14 02:49:28,305 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2025-09-14 02:49:28,346 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/12/
2025-09-14 02:49:29,321 INFO [collect_river_data] running in process 2908464
2025-09-14 02:49:29,322 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-09-14 02:49:29,323 INFO [collect_river_data] publishing log messages to tcp://*:5564
2025-09-14 02:49:29,325 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-09-14 02:49:29,326 INFO [collect_river_data] Collecting ECCC SanJuanPortRenfrew river data for 2025-09-13
2025-09-14 02:49:29,379 INFO [collect_river_data] Appended ECCC SanJuanPortRenfrew river average discharge for 2025-09-13 to: /results/forcing/rivers/observations/SanJuan_PortRenfrew_flow
2025-09-14 02:49:29,380 INFO [collect_river_data] ECCC SanJuanPortRenfrew river data collection for 2025-09-13 completed
2025-09-14 02:49:29,381 INFO [collect_river_data] running in process 2908465
2025-09-14 02:49:29,383 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-09-14 02:49:29,384 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-09-14 02:49:29,384 INFO [collect_river_data] publishing log messages to tcp://*:5565
2025-09-14 02:49:29,385 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-09-14 02:49:29,386 INFO [collect_river_data] Collecting ECCC SquamishBrackendale river data for 2025-09-13
2025-09-14 02:49:29,387 INFO [collect_river_data] running in process 2908455
2025-09-14 02:49:29,388 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-09-14 02:49:29,389 INFO [collect_river_data] publishing log messages to tcp://*:5566
2025-09-14 02:49:29,389 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-09-14 02:49:29,390 INFO [collect_river_data] Collecting ECCC ChilliwackVedder river data for 2025-09-13
2025-09-14 02:49:29,394 INFO [collect_river_data] running in process 2908456
2025-09-14 02:49:29,395 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-09-14 02:49:29,396 INFO [collect_river_data] publishing log messages to tcp://*:5567
2025-09-14 02:49:29,398 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-09-14 02:49:29,399 INFO [collect_river_data] Collecting ECCC ClowhomClowhomLake river data for 2025-09-13
2025-09-14 02:49:29,418 INFO [collect_river_data] running in process 2908462
2025-09-14 02:49:29,419 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-09-14 02:49:29,420 INFO [collect_river_data] publishing log messages to tcp://*:5568
2025-09-14 02:49:29,421 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-09-14 02:49:29,421 INFO [collect_river_data] Collecting ECCC RobertsCreek river data for 2025-09-13
2025-09-14 02:49:29,431 INFO [collect_river_data] Appended ECCC SquamishBrackendale river average discharge for 2025-09-13 to: /results/forcing/rivers/observations/Squamish_Brackendale_flow
2025-09-14 02:49:29,431 INFO [collect_river_data] ECCC SquamishBrackendale river data collection for 2025-09-13 completed
2025-09-14 02:49:29,435 INFO [collect_river_data] Appended ECCC ChilliwackVedder river average discharge for 2025-09-13 to: /results/forcing/rivers/observations/Chilliwack_Vedder_flow
2025-09-14 02:49:29,435 INFO [collect_river_data] ECCC ChilliwackVedder river data collection for 2025-09-13 completed
2025-09-14 02:49:29,444 INFO [collect_river_data] Appended ECCC ClowhomClowhomLake river average discharge for 2025-09-13 to: /results/forcing/rivers/observations/Clowhom_ClowhomLake_flow
2025-09-14 02:49:29,445 INFO [collect_river_data] ECCC ClowhomClowhomLake river data collection for 2025-09-13 completed
2025-09-14 02:49:29,446 INFO [collect_river_data] running in process 2908458
2025-09-14 02:49:29,447 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-09-14 02:49:29,448 INFO [collect_river_data] publishing log messages to tcp://*:5569
2025-09-14 02:49:29,449 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-09-14 02:49:29,449 INFO [collect_river_data] Collecting ECCC Fraser river data for 2025-09-13
2025-09-14 02:49:29,465 INFO [collect_river_data] running in process 2908461
2025-09-14 02:49:29,467 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-09-14 02:49:29,467 INFO [collect_river_data] publishing log messages to tcp://*:5570
2025-09-14 02:49:29,468 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-09-14 02:49:29,469 INFO [collect_river_data] Collecting ECCC NicomeklLangley river data for 2025-09-13
2025-09-14 02:49:29,472 INFO [collect_river_data] Appended ECCC Fraser river average discharge for 2025-09-13 to: /data/dlatorne/SOG-projects/SOG-forcing/ECget/Fraser_flow
2025-09-14 02:49:29,473 INFO [collect_river_data] ECCC Fraser river data collection for 2025-09-13 completed
2025-09-14 02:49:29,475 INFO [collect_river_data] Appended ECCC RobertsCreek river average discharge for 2025-09-13 to: /results/forcing/rivers/observations/RobertsCreek_flow
2025-09-14 02:49:29,475 INFO [collect_river_data] ECCC RobertsCreek river data collection for 2025-09-13 completed
2025-09-14 02:49:29,482 INFO [collect_river_data] running in process 2908457
2025-09-14 02:49:29,483 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-09-14 02:49:29,484 INFO [collect_river_data] publishing log messages to tcp://*:5571
2025-09-14 02:49:29,485 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-09-14 02:49:29,486 INFO [collect_river_data] Collecting ECCC Englishman river data for 2025-09-13
2025-09-14 02:49:29,490 INFO [collect_river_data] running in process 2908463
2025-09-14 02:49:29,492 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-09-14 02:49:29,493 INFO [collect_river_data] publishing log messages to tcp://*:5572
2025-09-14 02:49:29,494 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-09-14 02:49:29,495 INFO [collect_river_data] Collecting ECCC SalmonSayward river data for 2025-09-13
2025-09-14 02:49:29,510 INFO [collect_river_data] running in process 2908459
2025-09-14 02:49:29,511 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-09-14 02:49:29,511 INFO [collect_river_data] publishing log messages to tcp://*:5573
2025-09-14 02:49:29,513 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-09-14 02:49:29,513 INFO [collect_river_data] Collecting ECCC HomathkoMouth river data for 2025-09-13
2025-09-14 02:49:29,516 INFO [collect_river_data] running in process 2908453
2025-09-14 02:49:29,516 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-09-14 02:49:29,517 INFO [collect_river_data] publishing log messages to tcp://*:5574
2025-09-14 02:49:29,518 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-09-14 02:49:29,519 INFO [collect_river_data] Collecting ECCC Capilano river data for 2025-09-13
2025-09-14 02:49:29,520 INFO [collect_river_data] Appended ECCC NicomeklLangley river average discharge for 2025-09-13 to: /results/forcing/rivers/observations/Nicomekl_Langley_flow
2025-09-14 02:49:29,521 INFO [collect_river_data] ECCC NicomeklLangley river data collection for 2025-09-13 completed
2025-09-14 02:49:29,523 INFO [collect_river_data] Appended ECCC Englishman river average discharge for 2025-09-13 to: /data/dlatorne/SOG-projects/SOG-forcing/ECget/Englishman_flow
2025-09-14 02:49:29,524 INFO [collect_river_data] ECCC Englishman river data collection for 2025-09-13 completed
2025-09-14 02:49:29,529 INFO [collect_river_data] running in process 2908466
2025-09-14 02:49:29,530 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-09-14 02:49:29,531 INFO [collect_river_data] publishing log messages to tcp://*:5575
2025-09-14 02:49:29,531 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-09-14 02:49:29,532 INFO [collect_river_data] Collecting ECCC TheodosiaScotty river data for 2025-09-13
2025-09-14 02:49:29,538 INFO [collect_river_data] Appended ECCC SalmonSayward river average discharge for 2025-09-13 to: /results/forcing/rivers/observations/Salmon_Sayward_flow
2025-09-14 02:49:29,539 INFO [collect_river_data] ECCC SalmonSayward river data collection for 2025-09-13 completed
2025-09-14 02:49:29,548 INFO [collect_river_data] running in process 2908468
2025-09-14 02:49:29,549 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-09-14 02:49:29,550 INFO [collect_river_data] publishing log messages to tcp://*:5576
2025-09-14 02:49:29,551 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-09-14 02:49:29,552 INFO [collect_river_data] Collecting ECCC TheodosiaDiversion river data for 2025-09-13
2025-09-14 02:49:29,556 INFO [collect_river_data] running in process 2908467
2025-09-14 02:49:29,558 INFO [collect_river_data] Appended ECCC Capilano river average discharge for 2025-09-13 to: /opp/observations/rivers/Capilano/Caplilano_08GA010_day_avg_flow
2025-09-14 02:49:29,559 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-09-14 02:49:29,560 INFO [collect_river_data] ECCC Capilano river data collection for 2025-09-13 completed
2025-09-14 02:49:29,561 INFO [collect_river_data] publishing log messages to tcp://*:5577
2025-09-14 02:49:29,563 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-09-14 02:49:29,563 INFO [collect_river_data] Collecting ECCC TheodosiaBypass river data for 2025-09-13
2025-09-14 02:49:29,564 INFO [collect_river_data] Appended ECCC HomathkoMouth river average discharge for 2025-09-13 to: /results/forcing/rivers/observations/Homathko_Mouth_flow
2025-09-14 02:49:29,565 INFO [collect_river_data] ECCC HomathkoMouth river data collection for 2025-09-13 completed
2025-09-14 02:49:29,573 INFO [collect_river_data] Appended ECCC TheodosiaScotty river average discharge for 2025-09-13 to: /results/forcing/rivers/observations/Theodosia_Scotty_flow
2025-09-14 02:49:29,574 INFO [collect_river_data] ECCC TheodosiaScotty river data collection for 2025-09-13 completed
2025-09-14 02:49:29,578 INFO [collect_NeahBay_ssh] 2025-09-14T00:00:00-07:00 Neah Bay ssh 00Z obs/forecast data collection complete
2025-09-14 02:49:29,581 INFO [collect_river_data] Appended ECCC TheodosiaBypass river average discharge for 2025-09-13 to: /results/forcing/rivers/observations/Theodosia_Bypass_flow
2025-09-14 02:49:29,582 INFO [collect_river_data] ECCC TheodosiaBypass river data collection for 2025-09-13 completed
2025-09-14 02:49:29,584 INFO [collect_river_data] Appended ECCC TheodosiaDiversion river average discharge for 2025-09-13 to: /results/forcing/rivers/observations/Theodosia_Diversion_flow
2025-09-14 02:49:29,585 INFO [collect_river_data] ECCC TheodosiaDiversion river data collection for 2025-09-13 completed
2025-09-14 02:49:29,699 INFO [get_vfpa_hadcp] running in process 2908472
2025-09-14 02:49:29,701 INFO [get_vfpa_hadcp] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-09-14 02:49:29,701 INFO [get_vfpa_hadcp] publishing log messages to tcp://*:5578
2025-09-14 02:49:29,702 INFO [get_vfpa_hadcp] connected to skookum.eos.ubc.ca port 5555
2025-09-14 02:49:29,703 INFO [get_vfpa_hadcp] processing VFPA HADCP data from 2nd Narrows Rail Bridge for 2025-09-13
2025-09-14 02:49:29,819 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-09-14 02:49:30,007 INFO [crop_gribs] running in process 2908475
2025-09-14 02:49:30,008 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-09-14 02:49:30,008 INFO [crop_gribs] publishing log messages to tcp://*:5579
2025-09-14 02:49:30,009 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2025-09-14 02:49:30,010 INFO [crop_gribs] cropping 2025-09-14 ECCC HRDPS 2.5km continental 12Z GRIB files to SalishSeaCast subdomain
2025-09-14 02:49:30,021 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20250914/12/
2025-09-14 02:49:30,186 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-09-14 02:49:30,228 INFO [grib_to_netcdf] running in process 2908695
2025-09-14 02:49:30,229 INFO [grib_to_netcdf] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-09-14 02:49:30,230 INFO [grib_to_netcdf] publishing log messages to tcp://*:5564
2025-09-14 02:49:30,232 INFO [grib_to_netcdf] connected to skookum.eos.ubc.ca port 5555
2025-09-14 02:49:30,233 INFO [grib_to_netcdf] creating NEMO-atmos forcing files for 2025-09-14 forecast2 run
2025-09-14 02:49:30,370 INFO [get_onc_ctd] running in process 2908469
2025-09-14 02:49:30,372 INFO [get_onc_ctd] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-09-14 02:49:30,373 INFO [get_onc_ctd] publishing log messages to tcp://*:5590
2025-09-14 02:49:30,374 INFO [get_onc_ctd] connected to skookum.eos.ubc.ca port 5555
2025-09-14 02:49:30,375 INFO [get_onc_ctd] requesting ONC SCVIP CTD T&S data for 2025-09-13
2025-09-14 02:49:30,386 INFO [get_onc_ctd] running in process 2908470
2025-09-14 02:49:30,386 INFO [get_onc_ctd] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-09-14 02:49:30,387 INFO [get_onc_ctd] publishing log messages to tcp://*:5591
2025-09-14 02:49:30,389 INFO [get_onc_ctd] connected to skookum.eos.ubc.ca port 5555
2025-09-14 02:49:30,390 INFO [get_onc_ctd] requesting ONC SEVIP CTD T&S data for 2025-09-13
2025-09-14 02:49:30,447 INFO [get_onc_ferry] running in process 2908471
2025-09-14 02:49:30,448 INFO [get_onc_ferry] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-09-14 02:49:30,449 INFO [get_onc_ferry] publishing log messages to tcp://*:5565
2025-09-14 02:49:30,450 INFO [get_onc_ferry] connected to skookum.eos.ubc.ca port 5555
2025-09-14 02:49:30,451 INFO [get_onc_ferry] requesting ONC TWDP.N1 NAV data for 2025-09-13
2025-09-14 02:49:30,543 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-09-14 02:49:30,575 ERROR [get_onc_ferry] request for ONC TWDP.N1 NAV data for 2025-09-13 failed: 400 Client Error:  for url: https://data.oceannetworks.ca/api/scalardata?method=getByLocation&token=8efe44eb-0fb5-4413-bba3-ae3dbe0fbb1d&locationCode=TWDP.N1&deviceCategoryCode=NAV&sensorCategoryCodes=longitude%2Clatitude&dateFrom=2025-09-13T00:00:00.000Z&dateTo=2025-09-13T23:59:00.000Z&resampleType=avg&resamplePeriod=1
2025-09-14 02:49:30,991 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-09-14 02:49:31,112 CRITICAL [get_onc_ferry] 2025-09-13 ONC TWDP ferry data file creation failed
2025-09-14 02:49:31,518 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-09-14 02:49:31,584 INFO [get_vfpa_hadcp] added VFPA HADCP data from 2nd Narrows Rail Bridge for 2025-09-13 to /opp/observations/AISDATA/netcdf/VFPA_2ND_NARROWS_HADCP_2s_202509.nc
2025-09-14 02:49:31,586 INFO [get_vfpa_hadcp] VFPA HADCP observations added to 2025-09 netcdf file
2025-09-14 02:49:31,692 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-09-14 02:49:32,013 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-09-14 02:49:32,388 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-09-14 02:49:32,698 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-09-14 02:49:33,006 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-09-14 02:49:33,330 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-09-14 02:49:33,661 INFO [manager] checklist updated with [Neah Bay ssh data] items from collect_NeahBay_ssh worker
2025-09-14 02:49:33,988 INFO [manager] launching NextWorker(module='nowcast.workers.make_ssh_files', args=['forecast2', '--run-date', '2025-09-14'], host='localhost')
2025-09-14 02:49:33,993 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-09-14 02:49:34,450 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-09-14 02:49:34,807 INFO [manager] checklist updated with [VFPA HADCP data] items from get_vfpa_hadcp worker
2025-09-14 02:49:35,222 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['VFPA-HADCP'], host='localhost')
2025-09-14 02:49:36,384 INFO [ping_erddap] running in process 2908831
2025-09-14 02:49:36,386 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-09-14 02:49:36,387 INFO [ping_erddap] publishing log messages to tcp://*:5562
2025-09-14 02:49:36,388 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2025-09-14 02:49:36,389 INFO [ping_erddap] VFPA-HADCP ERDDAP dataset flag file(s) created
2025-09-14 02:49:36,391 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2025-09-14 02:49:37,165 INFO [make_ssh_files] running in process 2908816
2025-09-14 02:49:37,166 INFO [make_ssh_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-09-14 02:49:37,167 INFO [make_ssh_files] publishing log messages to tcp://*:5562
2025-09-14 02:49:37,168 INFO [make_ssh_files] connected to skookum.eos.ubc.ca port 5555
2025-09-14 02:49:37,169 INFO [make_ssh_files] building forecast2 sea surface height boundary conditions file(s) from 2025-09-14 NOAA Neah Bay 00Z observation and forecast values
2025-09-14 02:49:39,510 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2025m09d09.nc
2025-09-14 02:49:39,798 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2025m09d10.nc
2025-09-14 02:49:39,996 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2025m09d15.nc
2025-09-14 02:49:40,056 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2025m09d11.nc
2025-09-14 02:49:40,330 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2025m09d12.nc
2025-09-14 02:49:40,607 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2025m09d13.nc
2025-09-14 02:49:40,885 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2025m09d14.nc
2025-09-14 02:49:41,162 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2025m09d15.nc
2025-09-14 02:49:41,441 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2025m09d16.nc
2025-09-14 02:49:41,645 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2025m09d17.nc
2025-09-14 02:49:41,798 INFO [make_ssh_files] sea surface height boundary file for 2025-09-14 forecast2 run created
2025-09-14 02:49:41,802 INFO [manager] checklist updated with [sea surface height forcing] items from make_ssh_files worker
2025-09-14 02:49:42,512 INFO [manager] launching NextWorker(module='nowcast.workers.make_runoff_file', args=['v202108'], host='localhost')
2025-09-14 02:49:42,517 INFO [manager] launching NextWorker(module='nowcast.workers.make_201702_runoff_file', args=[], host='localhost')
2025-09-14 02:49:42,955 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2025m09d16.nc
2025-09-14 02:49:42,962 INFO [grib_to_netcdf] 2025-09-14 NEMO-atmos forcing file for forecast2 created
2025-09-14 02:49:42,965 INFO [manager] checklist updated with [weather forcing] items from grib_to_netcdf worker
2025-09-14 02:49:43,857 INFO [make_201702_runoff_file] running in process 2908866
2025-09-14 02:49:43,859 INFO [make_201702_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-09-14 02:49:43,860 INFO [make_201702_runoff_file] publishing log messages to tcp://*:5562
2025-09-14 02:49:43,861 INFO [make_201702_runoff_file] connected to skookum.eos.ubc.ca port 5555
2025-09-14 02:49:43,885 INFO [make_201702_runoff_file] calculating runoff forcing for b201702 bathymetry
2025-09-14 02:49:43,978 INFO [make_201702_runoff_file] 2025-09-14 runoff file creation from Fraser at Hope and climatology elsewhere complete
2025-09-14 02:49:43,981 INFO [manager] checklist updated with [rivers forcing] items from make_201702_runoff_file worker
2025-09-14 02:49:44,420 INFO [make_runoff_file] running in process 2908864
2025-09-14 02:49:44,421 INFO [make_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-09-14 02:49:44,422 INFO [make_runoff_file] publishing log messages to tcp://*:5564
2025-09-14 02:49:44,423 INFO [make_runoff_file] connected to skookum.eos.ubc.ca port 5555
2025-09-14 02:49:44,424 INFO [make_runoff_file] calculating NEMO runoff forcing for v202108 bathymetry for 2025-09-13
2025-09-14 02:49:45,440 ERROR [make_runoff_file] no 2025-09-13 discharge obs for SkagitMountVernon: patching
2025-09-14 02:49:45,842 ERROR [make_runoff_file] no 2025-09-13 discharge obs for SnohomishMonroe: patching
2025-09-14 02:49:46,057 ERROR [make_runoff_file] no 2025-09-13 discharge obs for NisquallyMcKenna: patching
2025-09-14 02:49:46,151 ERROR [make_runoff_file] no 2025-09-13 discharge obs for GreenwaterGreenwater: patching
2025-09-14 02:49:47,404 INFO [make_runoff_file] stored NEMO runoff forcing for v202108 bathymetry for 2025-09-13: /results/forcing/rivers/R202108Dailies_y2025m09d13.nc
2025-09-14 02:49:47,404 INFO [make_runoff_file] 2025-09-13 runoff file creation completed
2025-09-14 02:49:47,406 INFO [manager] checklist updated with [rivers forcing] items from make_runoff_file worker
2025-09-14 02:49:47,719 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'forecast2'], host='localhost')
2025-09-14 02:49:47,723 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'forecast2'], host='localhost')
2025-09-14 02:49:47,727 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.graham', 'forecast2'], host='localhost')
2025-09-14 02:49:47,732 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'forecast2'], host='localhost')
2025-09-14 02:49:49,040 INFO [upload_forcing] running in process 2908901
2025-09-14 02:49:49,042 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-09-14 02:49:49,043 INFO [upload_forcing] publishing log messages to tcp://*:5562
2025-09-14 02:49:49,044 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-09-14 02:49:49,091 INFO [upload_forcing] running in process 2908900
2025-09-14 02:49:49,093 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-09-14 02:49:49,094 INFO [upload_forcing] publishing log messages to tcp://*:5564
2025-09-14 02:49:49,094 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-09-14 02:49:49,224 INFO [upload_forcing] running in process 2908899
2025-09-14 02:49:49,225 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-09-14 02:49:49,226 INFO [upload_forcing] publishing log messages to tcp://*:5565
2025-09-14 02:49:49,226 INFO [upload_forcing] running in process 2908902
2025-09-14 02:49:49,227 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-09-14 02:49:49,228 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-09-14 02:49:49,228 INFO [upload_forcing] publishing log messages to tcp://*:5566
2025-09-14 02:49:49,229 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-09-14 02:49:49,247 INFO [get_onc_ctd] 2025-09-13 ONC SEVIP CTD T&S file created
2025-09-14 02:49:49,249 INFO [manager] checklist updated with [ONC CTD data] items from get_onc_ctd worker
2025-09-14 02:49:49,565 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['SEVIP-CTD'], host='localhost')
2025-09-14 02:49:49,584 INFO [get_onc_ctd] 2025-09-13 ONC SCVIP CTD T&S file created
2025-09-14 02:49:49,587 INFO [manager] checklist updated with [ONC CTD data] items from get_onc_ctd worker
2025-09-14 02:49:49,943 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['SCVIP-CTD'], host='localhost')
2025-09-14 02:49:50,684 INFO [upload_forcing] LiveOcean boundary conditions file not found; created symlink to /results/forcing/LiveOcean/boundary_conditions/LiveOcean_v201905_y2025m09d13.nc
2025-09-14 02:49:50,718 INFO [ping_erddap] running in process 2908922
2025-09-14 02:49:50,719 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-09-14 02:49:50,720 INFO [ping_erddap] publishing log messages to tcp://*:5567
2025-09-14 02:49:50,720 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2025-09-14 02:49:50,721 INFO [ping_erddap] SEVIP-CTD ERDDAP dataset flag file(s) created
2025-09-14 02:49:50,723 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2025-09-14 02:49:50,742 INFO [upload_forcing] forecast2 2025-09-14 forcing files upload to optimum-hindcast completed
2025-09-14 02:49:51,046 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-09-14 02:49:51,108 INFO [ping_erddap] running in process 2908925
2025-09-14 02:49:51,109 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-09-14 02:49:51,110 INFO [ping_erddap] publishing log messages to tcp://*:5568
2025-09-14 02:49:51,111 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2025-09-14 02:49:51,113 INFO [ping_erddap] SCVIP-CTD ERDDAP dataset flag file(s) created
2025-09-14 02:49:51,357 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2025-09-14 02:49:53,225 INFO [upload_forcing] forecast2 2025-09-14 forcing files upload to arbutus.cloud-nowcast completed
2025-09-14 02:49:53,230 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-09-14 02:49:53,619 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2025-09-14'], host='localhost')
2025-09-14 02:49:54,830 INFO [make_forcing_links] running in process 2908940
2025-09-14 02:49:54,831 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-09-14 02:49:54,832 INFO [make_forcing_links] publishing log messages to tcp://*:5565
2025-09-14 02:49:54,833 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2025-09-14 02:49:56,006 INFO [make_forcing_links] forecast2 2025-09-14 forcing file links on arbutus.cloud-nowcast created
2025-09-14 02:49:56,010 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2025-09-14 02:49:56,364 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2025-09-14'], host='arbutus.cloud-nowcast')
2025-09-14 02:49:57,208 INFO [upload_forcing] forecast2 2025-09-14 forcing files upload to robot.graham completed
2025-09-14 02:49:57,212 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-09-14 02:49:58,601 INFO [run_NEMO] running in process 6290
2025-09-14 02:49:58,602 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-09-14 02:49:58,602 INFO [run_NEMO] publishing log messages to tcp://*:5556
2025-09-14 02:49:58,728 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2025-09-14 02:49:59,811 WARNING [run_NEMO] There are uncommitted changes in /nemoShare/MEOPAR/nowcast-sys/SS-run-sets
2025-09-14 02:50:00,120 INFO [run_NEMO] forecast2: launching /nemoShare/MEOPAR/nowcast-sys/runs/13sep25forecast2_2025-09-14T024958.791969-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2025-09-14 02:50:00,131 INFO [run_NEMO] forecast2 NEMO run for 2025-09-14 on arbutus.cloud-nowcast started
2025-09-14 02:50:00,136 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2025-09-14 02:50:00,460 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'forecast2'], host='arbutus.cloud-nowcast')
2025-09-14 02:50:02,448 INFO [watch_NEMO] running in process 6564
2025-09-14 02:50:02,449 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-09-14 02:50:02,450 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2025-09-14 02:50:02,451 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2025-09-14 02:50:02,514 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2025-09-14 02:51:58,415 CRITICAL [upload_forcing] unhandled exception:
Traceback (most recent call last):
  File "/SalishSeaCast/NEMO_Nowcast/nemo_nowcast/worker.py", line 387, in _do_work
    checklist = self.worker_func(
        self._parsed_args, self.config, self.tell_manager
    )
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/upload_forcing.py", line 94, in upload_forcing
    ssh_client, sftp_client = ssh_sftp.sftp(host_name, ssh_key)
                              ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/ssh_sftp.py", line 144, in sftp
    ssh_client = ssh(host, key_filename, ssh_config_file)
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/ssh_sftp.py", line 75, in ssh
    ssh_client.connect(
    ~~~~~~~~~~~~~~~~~~^
        host["hostname"],
        ^^^^^^^^^^^^^^^^^
    ...<3 lines>...
        look_for_keys=False,
        ^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/paramiko/client.py", line 386, in connect
    sock.connect(addr)
    ~~~~~~~~~~~~^^^^^^
TimeoutError: [Errno 110] Connection timed out
2025-09-14 02:55:02,516 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 18708214 = 2025-09-15 05:02:00 UTC, 16.8% complete
2025-09-14 03:00:02,517 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 18708674 = 2025-09-15 10:08:40 UTC, 33.8% complete
2025-09-14 03:05:02,517 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 18709165 = 2025-09-15 15:36:00 UTC, 52.0% complete
2025-09-14 03:10:02,518 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 18709675 = 2025-09-15 21:16:00 UTC, 70.9% complete
2025-09-14 03:15:02,518 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 18710191 = 2025-09-16 03:00:00 UTC, 90.0% complete
2025-09-14 03:20:02,522 INFO [watch_NEMO] forecast2 NEMO run on arbutus.cloud-nowcast completed
2025-09-14 03:20:02,526 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2025-09-14 03:20:03,035 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_wind_file', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2025-09-14'], host='arbutus.cloud-nowcast')
2025-09-14 03:20:03,117 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_current_file', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2025-09-14'], host='arbutus.cloud-nowcast')
2025-09-14 03:20:03,121 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2025-09-13'], host='localhost')
2025-09-14 03:20:04,372 INFO [download_results] running in process 2909337
2025-09-14 03:20:04,374 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-09-14 03:20:04,375 INFO [download_results] publishing log messages to tcp://*:5562
2025-09-14 03:20:04,376 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2025-09-14 03:20:04,377 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/forecast2/13sep25 to /results/SalishSea/forecast2.202111
2025-09-14 03:20:05,967 INFO [make_ww3_wind_file] running in process 6921
2025-09-14 03:20:05,968 INFO [make_ww3_wind_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-09-14 03:20:05,971 INFO [make_ww3_wind_file] publishing log messages to tcp://*:5570
2025-09-14 03:20:05,973 INFO [make_ww3_wind_file] connected to skookum.eos.ubc.ca port 5555
2025-09-14 03:20:05,975 INFO [make_ww3_wind_file] Creating wwatch3 wind forcing file for 2025-09-14 forecast2 run
2025-09-14 03:20:06,192 INFO [make_ww3_current_file] running in process 6919
2025-09-14 03:20:06,193 INFO [make_ww3_current_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-09-14 03:20:06,194 INFO [make_ww3_current_file] publishing log messages to tcp://*:5571
2025-09-14 03:20:06,195 INFO [make_ww3_current_file] connected to skookum.eos.ubc.ca port 5555
2025-09-14 03:20:06,196 INFO [make_ww3_current_file] Creating wwatch3 currents forcing file for 2025-09-14 forecast2 run
2025-09-14 03:20:15,992 INFO [make_ww3_wind_file] wwatch3 wind forcing file created on arbutus.cloud-nowcast for 2025-09-14 forecast2 run
2025-09-14 03:20:15,995 INFO [manager] checklist updated with [WW3 wind forcing] items from make_ww3_wind_file worker
2025-09-14 03:20:54,036 INFO [download_results] forecast2 2025-09-13 results files from arbutus.cloud-nowcast downloaded
2025-09-14 03:20:54,041 INFO [manager] checklist updated with [results files] items from download_results worker
2025-09-14 03:20:54,410 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['forecast2', '--run-date', '2025-09-13'], host='localhost')
2025-09-14 03:20:56,089 INFO [make_CHS_currents_file] running in process 2909353
2025-09-14 03:20:56,090 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-09-14 03:20:56,091 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5562
2025-09-14 03:20:56,092 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555
2025-09-14 03:21:36,301 INFO [make_CHS_currents_file] Made CHS currents file for 2025-09-13 for forecast2
2025-09-14 03:21:36,304 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker
2025-09-14 03:21:36,662 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['nemo', 'forecast2', '--run-date', '2025-09-13'], host='localhost')
2025-09-14 03:21:37,950 INFO [update_forecast_datasets] running in process 2909379
2025-09-14 03:21:37,952 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-09-14 03:21:37,953 INFO [update_forecast_datasets] publishing log messages to tcp://*:5562
2025-09-14 03:21:37,954 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2025-09-14 03:21:37,955 INFO [update_forecast_datasets] updating nemo forecast directory for forecast2/13sep25 run
2025-09-14 03:23:35,825 INFO [make_ww3_current_file] wwatch3 currents forcing file created on arbutus.cloud-nowcast for 2025-09-14 forecast2 run
2025-09-14 03:23:35,829 INFO [manager] checklist updated with [WW3 currents forcing] items from make_ww3_current_file worker
2025-09-14 03:23:36,190 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2025-09-14'], host='arbutus.cloud-nowcast')
2025-09-14 03:23:38,286 INFO [run_ww3] running in process 7477
2025-09-14 03:23:38,287 INFO [run_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-09-14 03:23:38,288 INFO [run_ww3] publishing log messages to tcp://*:5572
2025-09-14 03:23:38,288 INFO [run_ww3] connected to skookum.eos.ubc.ca port 5555
2025-09-14 03:23:38,291 INFO [run_ww3] Created run directory /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast2_2025-09-14T032338.285256-0700
2025-09-14 03:23:38,292 INFO [run_ww3] forecast2: launching /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast2_2025-09-14T032338.285256-0700/SoGWW3.sh on arbutus.cloud-nowcast
2025-09-14 03:23:38,303 INFO [run_ww3] forecast2 WaveWatch3 run for 2025-09-14 on arbutus.cloud-nowcast started
2025-09-14 03:23:38,308 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker
2025-09-14 03:23:38,645 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'forecast2'], host='arbutus.cloud-nowcast')
2025-09-14 03:23:40,743 INFO [watch_ww3] running in process 7566
2025-09-14 03:23:40,744 INFO [watch_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-09-14 03:23:40,745 INFO [watch_ww3] publishing log messages to tcp://*:5573
2025-09-14 03:23:40,748 INFO [watch_ww3] connected to skookum.eos.ubc.ca port 5555
2025-09-14 03:23:40,781 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2025-09-14 03:25:53,675 INFO [update_forecast_datasets] updated nemo forecast directory for forecast2/13sep25 run: /results/SalishSea/rolling-forecasts/nemo
2025-09-14 03:25:53,677 INFO [update_forecast_datasets] nemo 2025-09-13 forecast2 rolling forecast datasets updated
2025-09-14 03:25:53,680 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2025-09-14 03:25:54,029 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nemo-forecast'], host='localhost')
2025-09-14 03:25:54,034 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'forecast2', 'publish', '--run-date', '2025-09-13'], host='localhost')
2025-09-14 03:25:54,039 INFO [manager] launching NextWorker(module='nowcast.workers.make_surface_current_tiles', args=['forecast2', '--run-date', '2025-09-13'], host='localhost')
2025-09-14 03:25:55,391 INFO [ping_erddap] running in process 2909479
2025-09-14 03:25:55,392 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-09-14 03:25:55,393 INFO [ping_erddap] publishing log messages to tcp://*:5562
2025-09-14 03:25:55,394 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2025-09-14 03:25:55,403 INFO [ping_erddap] nemo-forecast ERDDAP dataset flag file(s) created
2025-09-14 03:25:55,405 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2025-09-14 03:25:56,473 INFO [make_surface_current_tiles] running in process 2909482
2025-09-14 03:25:56,475 INFO [make_surface_current_tiles] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-09-14 03:25:56,476 INFO [make_surface_current_tiles] publishing log messages to tcp://*:5562
2025-09-14 03:25:56,477 INFO [make_surface_current_tiles] connected to skookum.eos.ubc.ca port 5555
2025-09-14 03:25:56,478 INFO [make_surface_current_tiles] starting rendering of tiles for 2025-09-13 forecast2 in 6 processes into /results/nowcast-sys/figures/surface_currents/forecast2/13sep25
2025-09-14 03:25:57,338 INFO [make_plots] running in process 2909481
2025-09-14 03:25:57,340 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-09-14 03:25:57,342 INFO [make_plots] publishing log messages to tcp://*:5564
2025-09-14 03:25:57,343 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2025-09-14 03:25:57,819 INFO [make_plots] preparing render list for 2025-09-13 NEMO forecast2 publish figures
2025-09-14 03:25:57,936 INFO [make_plots] added Website_thumbnail figure to 2025-09-13 NEMO forecast2 publish render list
2025-09-14 03:25:57,937 INFO [make_plots] added Threshold_website figure to 2025-09-13 NEMO forecast2 publish render list
2025-09-14 03:25:57,938 INFO [make_plots] added PA_tidal_predictions figure to 2025-09-13 NEMO forecast2 publish render list
2025-09-14 03:25:57,939 INFO [make_plots] added Boundary Bay figure to 2025-09-13 NEMO forecast2 publish render list
2025-09-14 03:25:57,940 INFO [make_plots] added Campbell River figure to 2025-09-13 NEMO forecast2 publish render list
2025-09-14 03:25:57,941 INFO [make_plots] added Cherry Point figure to 2025-09-13 NEMO forecast2 publish render list
2025-09-14 03:25:57,942 INFO [make_plots] added Friday Harbor figure to 2025-09-13 NEMO forecast2 publish render list
2025-09-14 03:25:57,943 INFO [make_plots] added Halfmoon Bay figure to 2025-09-13 NEMO forecast2 publish render list
2025-09-14 03:25:57,943 INFO [make_plots] added Nanaimo figure to 2025-09-13 NEMO forecast2 publish render list
2025-09-14 03:25:57,944 INFO [make_plots] added Neah Bay figure to 2025-09-13 NEMO forecast2 publish render list
2025-09-14 03:25:57,945 INFO [make_plots] added New Westminster figure to 2025-09-13 NEMO forecast2 publish render list
2025-09-14 03:25:57,945 INFO [make_plots] added Patricia Bay figure to 2025-09-13 NEMO forecast2 publish render list
2025-09-14 03:25:57,946 INFO [make_plots] added Point Atkinson figure to 2025-09-13 NEMO forecast2 publish render list
2025-09-14 03:25:57,947 INFO [make_plots] added Port Renfrew figure to 2025-09-13 NEMO forecast2 publish render list
2025-09-14 03:25:57,947 INFO [make_plots] added Sand Heads figure to 2025-09-13 NEMO forecast2 publish render list
2025-09-14 03:25:57,948 INFO [make_plots] added Sandy Cove figure to 2025-09-13 NEMO forecast2 publish render list
2025-09-14 03:25:57,949 INFO [make_plots] added Squamish figure to 2025-09-13 NEMO forecast2 publish render list
2025-09-14 03:25:57,949 INFO [make_plots] added Victoria figure to 2025-09-13 NEMO forecast2 publish render list
2025-09-14 03:25:57,950 INFO [make_plots] added Woodwards Landing figure to 2025-09-13 NEMO forecast2 publish render list
2025-09-14 03:25:57,951 INFO [make_plots] starting to render nemo forecast2 publish 13sep25 figures
2025-09-14 03:28:40,781 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 75, 13.9% complete
2025-09-14 03:33:37,737 WARNING [make_plots] invalid station number for Boundary Bay station: None
2025-09-14 03:33:37,739 WARNING [make_plots] can't resolve a valid CHS station code for Boundary Bay
2025-09-14 03:33:40,785 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 156, 28.9% complete
2025-09-14 03:34:01,312 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb996/data for station 08074 Campbell River from 2025-09-09 00:00:00Z to 2025-09-16 05:55:00Z
2025-09-14 03:35:14,308 WARNING [make_plots] invalid station number for Halfmoon Bay station: None
2025-09-14 03:35:14,309 WARNING [make_plots] can't resolve a valid CHS station code for Halfmoon Bay
2025-09-14 03:35:36,700 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb96d/data for station 07917 Nanaimo from 2025-09-09 00:00:00Z to 2025-09-16 05:55:00Z
2025-09-14 03:36:19,687 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e13d0f4a073c4bbf93/data for station 07654 New Westminster from 2025-09-09 00:00:00Z to 2025-09-16 05:55:00Z
2025-09-14 03:36:42,294 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd2d/data for station 07277 Patricia Bay from 2025-09-09 00:00:00Z to 2025-09-16 05:55:00Z
2025-09-14 03:37:04,270 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94c/data for station 07795 Point Atkinson from 2025-09-09 00:00:00Z to 2025-09-16 05:55:00Z
2025-09-14 03:37:20,816 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e23d0f4a073c4bc060/data for station 08525 Port Renfrew from 2025-09-09 00:00:00Z to 2025-09-16 05:55:00Z
2025-09-14 03:37:22,002 INFO [make_surface_current_tiles] starting PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast2/13sep25
2025-09-14 03:37:37,234 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb93c/data for station 07594 Sand Heads from 2025-09-09 00:00:00Z to 2025-09-16 05:55:00Z
2025-09-14 03:37:53,205 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc43b/data for station 07786 Sandy Cove from 2025-09-09 00:00:00Z to 2025-09-16 05:55:00Z
2025-09-14 03:38:09,984 WARNING [make_plots] invalid station number for Squamish station: None
2025-09-14 03:38:09,984 WARNING [make_plots] can't resolve a valid CHS station code for Squamish
2025-09-14 03:38:25,141 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd1e/data for station 07120 Victoria from 2025-09-09 00:00:00Z to 2025-09-16 05:55:00Z
2025-09-14 03:38:40,784 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 234, 43.3% complete
2025-09-14 03:38:41,657 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc42d/data for station 07610 Woodwards Landing from 2025-09-09 00:00:00Z to 2025-09-16 05:55:00Z
2025-09-14 03:38:57,238 INFO [make_plots] finished rendering nemo forecast2 publish 13sep25 figures
2025-09-14 03:38:57,240 INFO [make_plots] nemo publish plots for 2025-09-13 forecast2 completed
2025-09-14 03:38:57,246 INFO [manager] checklist updated with [plots] items from make_plots worker
2025-09-14 03:38:57,538 INFO [manager] launching NextWorker(module='nowcast.workers.make_feeds', args=['forecast2', '--run-date', '2025-09-13'], host='localhost')
2025-09-14 03:39:00,739 INFO [make_feeds] running in process 2911504
2025-09-14 03:39:00,740 INFO [make_feeds] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-09-14 03:39:00,741 INFO [make_feeds] publishing log messages to tcp://*:5564
2025-09-14 03:39:00,742 INFO [make_feeds] connected to skookum.eos.ubc.ca port 5555
2025-09-14 03:39:14,965 INFO [make_feeds] ATOM feeds for 2025-09-13 forecast2 run completed
2025-09-14 03:39:14,968 INFO [manager] checklist updated with [storm surge alert feeds] items from make_feeds worker
2025-09-14 03:39:15,326 INFO [manager] launching NextWorker(module='nemo_nowcast.workers.clear_checklist', args=[], host='localhost')
2025-09-14 03:39:16,388 INFO [clear_checklist] running in process 2911523
2025-09-14 03:39:16,389 INFO [clear_checklist] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-09-14 03:39:16,389 INFO [clear_checklist] publishing log messages to tcp://*:5564
2025-09-14 03:39:16,390 INFO [clear_checklist] connected to skookum.eos.ubc.ca port 5555
2025-09-14 03:39:16,392 INFO [clear_checklist] requesting that manager clear system state checklist
2025-09-14 03:39:16,394 INFO [manager] writing checklist to log file
2025-09-14 03:39:16,437 INFO [manager] checklist cleared
2025-09-14 03:39:16,439 INFO [clear_checklist] nowcast system checklist cleared
2025-09-14 03:39:16,442 INFO [manager] launching NextWorker(module='nemo_nowcast.workers.rotate_logs', args=[], host='localhost')
2025-09-14 03:39:17,485 INFO [rotate_logs] running in process 2911528
2025-09-14 03:39:17,487 INFO [rotate_logs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-09-14 03:39:17,487 INFO [rotate_logs] publishing log messages to tcp://*:5564
2025-09-14 03:39:17,488 INFO [rotate_logs] connected to skookum.eos.ubc.ca port 5555
2025-09-14 03:39:17,488 INFO [rotate_logs] rotating log files