2025-09-12 03:39:50,701 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/nowcast.log 2025-09-12 03:39:50,705 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/nowcast.debug.log 2025-09-12 03:39:50,709 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/checklist.log 2025-09-12 03:39:50,710 INFO [rotate_logs] log files rotated 2025-09-12 03:39:50,715 INFO [manager] checklist updated with [log rotation] items from rotate_logs worker 2025-09-12 03:41:00,621 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 315, 58.3% complete 2025-09-12 03:41:33,471 INFO [make_surface_current_tiles] finished PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast2/11sep25 2025-09-12 03:41:33,495 INFO [make_surface_current_tiles] finished rendering of tiles for 2025-09-11 forecast2 into /results/nowcast-sys/figures/surface_currents/forecast2/11sep25 2025-09-12 03:41:33,496 INFO [make_surface_current_tiles] surface current tile figures for 2025-09-11 forecast2 completed 2025-09-12 03:41:33,802 INFO [manager] checklist updated with [surface current tiles] items from make_surface_current_tiles worker 2025-09-12 03:46:00,623 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 399, 73.9% complete 2025-09-12 03:51:00,625 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 489, 90.6% complete 2025-09-12 03:56:00,626 INFO [watch_ww3] forecast2 WWATCH3 run on arbutus.cloud-nowcast completed 2025-09-12 03:56:00,633 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker 2025-09-12 03:56:00,973 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2025-09-12'], host='localhost') 2025-09-12 03:56:02,147 INFO [download_wwatch3_results] running in process 2867415 2025-09-12 03:56:02,149 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-09-12 03:56:02,150 INFO [download_wwatch3_results] publishing log messages to tcp://*:5563 2025-09-12 03:56:02,151 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555 2025-09-12 03:56:35,250 INFO [download_wwatch3_results] forecast2 2025-09-12 results files from arbutus.cloud-nowcast downloaded 2025-09-12 03:56:35,253 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker 2025-09-12 03:56:35,409 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['wwatch3', 'forecast2', '--run-date', '2025-09-12'], host='localhost') 2025-09-12 03:56:36,471 INFO [update_forecast_datasets] running in process 2867430 2025-09-12 03:56:36,474 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-09-12 03:56:36,475 INFO [update_forecast_datasets] publishing log messages to tcp://*:5563 2025-09-12 03:56:36,476 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555 2025-09-12 03:56:36,477 INFO [update_forecast_datasets] updating wwatch3 most_recent_forecast directory from forecast2/12sep25 run 2025-09-12 03:56:36,479 INFO [update_forecast_datasets] updated wwatch3 most_recent_forecast directory from forecast2/12sep25 run 2025-09-12 03:56:36,480 INFO [update_forecast_datasets] updating wwatch3 forecast directory for forecast2/12sep25 run 2025-09-12 03:56:58,042 INFO [update_forecast_datasets] updated wwatch3 forecast directory for forecast2/12sep25 run: /results/SalishSea/rolling-forecasts/wwatch3 2025-09-12 03:56:58,043 INFO [update_forecast_datasets] wwatch3 2025-09-12 forecast2 rolling forecast datasets updated 2025-09-12 03:56:58,046 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker 2025-09-12 03:56:58,207 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['wwatch3-forecast'], host='localhost') 2025-09-12 03:56:59,215 INFO [ping_erddap] running in process 2867443 2025-09-12 03:56:59,216 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-09-12 03:56:59,217 INFO [ping_erddap] publishing log messages to tcp://*:5563 2025-09-12 03:56:59,218 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555 2025-09-12 03:56:59,219 INFO [ping_erddap] wwatch3-forecast ERDDAP dataset flag file(s) created 2025-09-12 03:56:59,221 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker 2025-09-12 03:56:59,379 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['wwatch3', 'forecast2', 'publish', '--run-date', '2025-09-12'], host='localhost') 2025-09-12 03:57:02,215 INFO [make_plots] running in process 2867448 2025-09-12 03:57:02,216 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-09-12 03:57:02,217 INFO [make_plots] publishing log messages to tcp://*:5563 2025-09-12 03:57:02,218 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555 2025-09-12 03:57:02,219 INFO [make_plots] preparing render list for 2025-09-12 WaveWatch3 forecast2 publish figures 2025-09-12 03:57:02,220 INFO [make_plots] added Halibut Bank figure to 2025-09-12 WaveWatch3 forecast2 publish render list 2025-09-12 03:57:02,220 INFO [make_plots] added Sentry Shoal figure to 2025-09-12 WaveWatch3 forecast2 publish render list 2025-09-12 03:57:02,221 INFO [make_plots] starting to render wwatch3 forecast2 publish 12sep25 figures 2025-09-12 03:57:03,004 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46146 Halibut Bank 2025-09-12 03:57:07,765 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46131 Sentry Shoal 2025-09-12 03:57:12,339 INFO [make_plots] finished rendering wwatch3 forecast2 publish 12sep25 figures 2025-09-12 03:57:12,340 INFO [make_plots] wwatch3 publish plots for 2025-09-12 forecast2 completed 2025-09-12 03:57:12,345 INFO [manager] checklist updated with [plots] items from make_plots worker 2025-09-12 08:45:29,638 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/12/ to /results/forcing/atmospheric/continental2.5/GRIB/20250912/12/ 2025-09-12 08:45:29,639 INFO [collect_weather] 2025-09-12 2.5km weather forecast 12 collection complete 2025-09-12 08:45:29,642 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker 2025-09-12 08:45:30,005 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'SkagitMountVernon', '--data-date', '2025-09-11'], host='localhost') 2025-09-12 08:45:30,010 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'SnohomishMonroe', '--data-date', '2025-09-11'], host='localhost') 2025-09-12 08:45:30,016 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'NisquallyMcKenna', '--data-date', '2025-09-11'], host='localhost') 2025-09-12 08:45:30,019 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'GreenwaterGreenwater', '--data-date', '2025-09-11'], host='localhost') 2025-09-12 08:45:30,024 INFO [manager] launching NextWorker(module='nowcast.workers.make_turbidity_file', args=[], host='localhost') 2025-09-12 08:45:30,029 INFO [manager] launching NextWorker(module='nowcast.workers.collect_NeahBay_ssh', args=['06'], host='localhost') 2025-09-12 08:45:30,035 INFO [manager] launching NextWorker(module='nowcast.workers.download_live_ocean', args=[], host='localhost') 2025-09-12 08:45:30,041 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['18', '2.5km'], host='localhost') 2025-09-12 08:45:30,062 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['18'], host='localhost') 2025-09-12 08:45:31,391 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20250912/12/ 2025-09-12 08:45:31,392 INFO [crop_gribs] 2025-09-12 12 GRIBs cropping complete 2025-09-12 08:45:31,393 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker 2025-09-12 08:45:31,762 INFO [collect_weather] running in process 2871872 2025-09-12 08:45:31,763 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-09-12 08:45:31,764 INFO [collect_weather] publishing log messages to tcp://*:5562 2025-09-12 08:45:31,765 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555 2025-09-12 08:45:31,783 INFO [collect_NeahBay_ssh] running in process 2871870 2025-09-12 08:45:31,784 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/18/ 2025-09-12 08:45:31,786 INFO [collect_NeahBay_ssh] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-09-12 08:45:31,786 INFO [collect_NeahBay_ssh] publishing log messages to tcp://*:5563 2025-09-12 08:45:31,787 INFO [collect_NeahBay_ssh] connected to skookum.eos.ubc.ca port 5555 2025-09-12 08:45:31,789 INFO [collect_NeahBay_ssh] collecting Neah Bay ssh 06Z obs/forecast for 2025-09-12 2025-09-12 08:45:31,823 INFO [manager] launching NextWorker(module='nowcast.workers.grib_to_netcdf', args=['nowcast+'], host='localhost') 2025-09-12 08:45:31,898 INFO [download_live_ocean] running in process 2871871 2025-09-12 08:45:31,899 INFO [download_live_ocean] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-09-12 08:45:31,899 INFO [download_live_ocean] publishing log messages to tcp://*:5564 2025-09-12 08:45:31,901 INFO [download_live_ocean] connected to skookum.eos.ubc.ca port 5555 2025-09-12 08:45:31,902 INFO [download_live_ocean] downloading Salish Sea western boundary day-averaged LiveOcean file for 2025-09-12 2025-09-12 08:45:32,246 INFO [collect_river_data] running in process 2871866 2025-09-12 08:45:32,247 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-09-12 08:45:32,248 INFO [collect_river_data] publishing log messages to tcp://*:5565 2025-09-12 08:45:32,248 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2025-09-12 08:45:32,249 INFO [collect_river_data] Collecting USGS SnohomishMonroe river data for 2025-09-11 2025-09-12 08:45:32,292 INFO [collect_river_data] running in process 2871868 2025-09-12 08:45:32,292 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-09-12 08:45:32,293 INFO [collect_river_data] publishing log messages to tcp://*:5566 2025-09-12 08:45:32,294 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2025-09-12 08:45:32,294 INFO [collect_river_data] Collecting USGS GreenwaterGreenwater river data for 2025-09-11 2025-09-12 08:45:32,313 INFO [collect_river_data] running in process 2871867 2025-09-12 08:45:32,314 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-09-12 08:45:32,315 INFO [collect_river_data] publishing log messages to tcp://*:5567 2025-09-12 08:45:32,316 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2025-09-12 08:45:32,316 INFO [collect_river_data] Collecting USGS NisquallyMcKenna river data for 2025-09-11 2025-09-12 08:45:32,399 INFO [make_turbidity_file] running in process 2871869 2025-09-12 08:45:32,400 INFO [make_turbidity_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-09-12 08:45:32,401 INFO [make_turbidity_file] publishing log messages to tcp://*:5568 2025-09-12 08:45:32,402 INFO [make_turbidity_file] connected to skookum.eos.ubc.ca port 5555 2025-09-12 08:45:32,403 INFO [make_turbidity_file] Creating Fraser River turbidity forcing file for 2025-09-12 2025-09-12 08:45:32,432 INFO [collect_river_data] running in process 2871864 2025-09-12 08:45:32,433 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-09-12 08:45:32,434 INFO [collect_river_data] publishing log messages to tcp://*:5569 2025-09-12 08:45:32,435 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2025-09-12 08:45:32,436 INFO [collect_river_data] Collecting USGS SkagitMountVernon river data for 2025-09-11 2025-09-12 08:45:32,567 INFO [download_live_ocean] downloaded 10195742 bytes from https://s3.kopah.uw.edu/liveocean-share/f2025.09.12/ubc.nc to /results/forcing/LiveOcean/downloaded/20250912/low_passed_UBC.nc 2025-09-12 08:45:32,568 INFO [download_live_ocean] Deflating in up to 1 concurrent sub-processes 2025-09-12 08:45:32,772 INFO [crop_gribs] running in process 2871874 2025-09-12 08:45:32,774 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-09-12 08:45:32,774 INFO [crop_gribs] publishing log messages to tcp://*:5570 2025-09-12 08:45:32,777 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555 2025-09-12 08:45:32,778 INFO [crop_gribs] cropping 2025-09-12 ECCC HRDPS 2.5km continental 18Z GRIB files to SalishSeaCast subdomain 2025-09-12 08:45:32,788 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20250912/18/ 2025-09-12 08:45:32,858 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json¶meterCd=00060&sites=12097500&startDT=2025-09-11&endDT=2025-09-11 "HTTP/1.1 200 " 2025-09-12 08:45:32,863 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json¶meterCd=00060&sites=12089500&startDT=2025-09-11&endDT=2025-09-11 "HTTP/1.1 200 " 2025-09-12 08:45:32,867 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json¶meterCd=00060&sites=12150800&startDT=2025-09-11&endDT=2025-09-11 "HTTP/1.1 200 " 2025-09-12 08:45:32,877 INFO [collect_river_data] Appended USGS GreenwaterGreenwater river average discharge for 2025-09-11 to: /results/forcing/rivers/observations/Greenwater_Greenwater_flow 2025-09-12 08:45:32,878 INFO [collect_river_data] USGS GreenwaterGreenwater river data collection for 2025-09-11 completed 2025-09-12 08:45:32,880 INFO [collect_river_data] Appended USGS NisquallyMcKenna river average discharge for 2025-09-11 to: /results/forcing/rivers/observations/Nisqually_McKenna_flow 2025-09-12 08:45:32,881 INFO [collect_river_data] USGS NisquallyMcKenna river data collection for 2025-09-11 completed 2025-09-12 08:45:32,883 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2025-09-12 08:45:32,884 INFO [collect_river_data] Appended USGS SnohomishMonroe river average discharge for 2025-09-11 to: /results/forcing/rivers/observations/Snohomish_Monroe_flow 2025-09-12 08:45:32,885 INFO [collect_river_data] USGS SnohomishMonroe river data collection for 2025-09-11 completed 2025-09-12 08:45:32,929 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json¶meterCd=00060&sites=12200500&startDT=2025-09-11&endDT=2025-09-11 "HTTP/1.1 200 " 2025-09-12 08:45:32,934 INFO [collect_river_data] Appended USGS SkagitMountVernon river average discharge for 2025-09-11 to: /results/forcing/rivers/observations/Skagit_MountVernon_flow 2025-09-12 08:45:32,935 INFO [collect_river_data] USGS SkagitMountVernon river data collection for 2025-09-11 completed 2025-09-12 08:45:32,941 INFO [collect_NeahBay_ssh] 2025-09-12T00:00:00-07:00 Neah Bay ssh 06Z obs/forecast data collection complete 2025-09-12 08:45:33,043 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2025-09-12 08:45:33,250 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2025-09-12 08:45:33,445 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2025-09-12 08:45:33,589 INFO [manager] checklist updated with [Neah Bay ssh data] items from collect_NeahBay_ssh worker 2025-09-12 08:45:33,592 INFO [download_live_ocean] netCDF4 deflated /results/forcing/LiveOcean/downloaded/20250912/low_passed_UBC.nc 2025-09-12 08:45:33,593 INFO [download_live_ocean] 2025-09-12 Live Ocean file for Salish Sea western boundary downloaded 2025-09-12 08:45:33,737 INFO [manager] launching NextWorker(module='nowcast.workers.make_ssh_files', args=['nowcast', '--run-date', '2025-09-12'], host='localhost') 2025-09-12 08:45:33,742 INFO [manager] checklist updated with [Live Ocean products] items from download_live_ocean worker 2025-09-12 08:45:33,830 INFO [grib_to_netcdf] running in process 2871966 2025-09-12 08:45:33,831 INFO [grib_to_netcdf] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-09-12 08:45:33,832 INFO [grib_to_netcdf] publishing log messages to tcp://*:5565 2025-09-12 08:45:33,833 INFO [grib_to_netcdf] connected to skookum.eos.ubc.ca port 5555 2025-09-12 08:45:33,834 INFO [grib_to_netcdf] creating NEMO-atmos forcing files for 2025-09-12 nowcast and forecast runs 2025-09-12 08:45:34,066 INFO [manager] launching NextWorker(module='nowcast.workers.make_live_ocean_files', args=['--run-date', '2025-09-12'], host='localhost') 2025-09-12 08:45:34,729 INFO [make_turbidity_file] 2025-09-12 Fraser River turbidity file creation complete 2025-09-12 08:45:34,730 INFO [manager] checklist updated with [Fraser River turbidity] items from make_turbidity_file worker 2025-09-12 08:45:36,389 INFO [make_live_ocean_files] running in process 2872017 2025-09-12 08:45:36,390 INFO [make_live_ocean_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-09-12 08:45:36,391 INFO [make_live_ocean_files] publishing log messages to tcp://*:5563 2025-09-12 08:45:36,392 INFO [make_live_ocean_files] connected to skookum.eos.ubc.ca port 5555 2025-09-12 08:45:36,395 INFO [make_live_ocean_files] Creating T&S western boundary conditions file from 2025-09-12 Live Ocean run 2025-09-12 08:45:36,405 INFO [make_ssh_files] running in process 2872004 2025-09-12 08:45:36,425 INFO [make_ssh_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-09-12 08:45:36,426 INFO [make_ssh_files] publishing log messages to tcp://*:5564 2025-09-12 08:45:36,427 INFO [make_ssh_files] connected to skookum.eos.ubc.ca port 5555 2025-09-12 08:45:36,429 INFO [make_ssh_files] building nowcast sea surface height boundary conditions file(s) from 2025-09-12 NOAA Neah Bay 06Z observation and forecast values 2025-09-12 08:45:38,237 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2025m09d08.nc 2025-09-12 08:45:38,408 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2025m09d09.nc 2025-09-12 08:45:38,589 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2025m09d10.nc 2025-09-12 08:45:38,768 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2025m09d11.nc 2025-09-12 08:45:38,941 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2025m09d12.nc 2025-09-12 08:45:39,103 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2025m09d13.nc 2025-09-12 08:45:39,260 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2025m09d14.nc 2025-09-12 08:45:39,437 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2025m09d15.nc 2025-09-12 08:45:39,538 INFO [make_ssh_files] sea surface height boundary file for 2025-09-12 nowcast run created 2025-09-12 08:45:39,541 INFO [manager] checklist updated with [sea surface height forcing] items from make_ssh_files worker 2025-09-12 08:45:39,675 INFO [manager] launching NextWorker(module='nowcast.workers.make_runoff_file', args=['v202108'], host='localhost') 2025-09-12 08:45:39,679 INFO [manager] launching NextWorker(module='nowcast.workers.make_201702_runoff_file', args=[], host='localhost') 2025-09-12 08:45:41,055 INFO [make_201702_runoff_file] running in process 2872050 2025-09-12 08:45:41,056 INFO [make_201702_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-09-12 08:45:41,057 INFO [make_201702_runoff_file] publishing log messages to tcp://*:5564 2025-09-12 08:45:41,057 INFO [make_201702_runoff_file] connected to skookum.eos.ubc.ca port 5555 2025-09-12 08:45:41,095 INFO [make_201702_runoff_file] calculating runoff forcing for b201702 bathymetry 2025-09-12 08:45:41,199 INFO [make_201702_runoff_file] 2025-09-12 runoff file creation from Fraser at Hope and climatology elsewhere complete 2025-09-12 08:45:41,201 INFO [manager] checklist updated with [rivers forcing] items from make_201702_runoff_file worker 2025-09-12 08:45:41,593 INFO [make_runoff_file] running in process 2872048 2025-09-12 08:45:41,595 INFO [make_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-09-12 08:45:41,595 INFO [make_runoff_file] publishing log messages to tcp://*:5566 2025-09-12 08:45:41,596 INFO [make_runoff_file] connected to skookum.eos.ubc.ca port 5555 2025-09-12 08:45:41,599 INFO [make_runoff_file] calculating NEMO runoff forcing for v202108 bathymetry for 2025-09-11 2025-09-12 08:45:44,074 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/hrdps_y2025m09d12.nc 2025-09-12 08:45:44,460 INFO [make_runoff_file] stored NEMO runoff forcing for v202108 bathymetry for 2025-09-11: /results/forcing/rivers/R202108Dailies_y2025m09d11.nc 2025-09-12 08:45:44,461 INFO [make_runoff_file] 2025-09-11 runoff file creation completed 2025-09-12 08:45:44,463 INFO [manager] checklist updated with [rivers forcing] items from make_runoff_file worker 2025-09-12 08:45:51,374 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2025m09d13.nc 2025-09-12 08:45:55,625 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2025m09d14.nc 2025-09-12 08:45:55,639 INFO [grib_to_netcdf] 2025-09-12 NEMO-atmos forcing file for nowcast+ created 2025-09-12 08:45:55,642 INFO [manager] checklist updated with [weather forcing] items from grib_to_netcdf worker 2025-09-12 08:46:12,755 INFO [make_live_ocean_files] Stored T&S western boundary conditions file: /results/forcing/LiveOcean/boundary_conditions/LiveOcean_v201905_y2025m09d12.nc 2025-09-12 08:46:12,755 INFO [make_live_ocean_files] 2025-09-12 Live Ocean western boundary conditions files created 2025-09-12 08:46:12,758 INFO [manager] checklist updated with [Live Ocean boundary conditions] items from make_live_ocean_files worker 2025-09-12 08:46:13,051 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['weather'], host='localhost') 2025-09-12 08:46:13,055 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'nowcast+'], host='localhost') 2025-09-12 08:46:13,056 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'nowcast+'], host='localhost') 2025-09-12 08:46:13,057 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.graham', 'nowcast+'], host='localhost') 2025-09-12 08:46:13,058 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'nowcast+'], host='localhost') 2025-09-12 08:46:14,052 INFO [ping_erddap] running in process 2872094 2025-09-12 08:46:14,054 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-09-12 08:46:14,055 INFO [ping_erddap] publishing log messages to tcp://*:5563 2025-09-12 08:46:14,056 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555 2025-09-12 08:46:14,059 INFO [ping_erddap] weather ERDDAP dataset flag file(s) created 2025-09-12 08:46:14,060 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker 2025-09-12 08:46:14,129 INFO [upload_forcing] running in process 2872099 2025-09-12 08:46:14,131 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-09-12 08:46:14,132 INFO [upload_forcing] publishing log messages to tcp://*:5564 2025-09-12 08:46:14,133 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2025-09-12 08:46:14,135 INFO [upload_forcing] running in process 2872096 2025-09-12 08:46:14,136 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-09-12 08:46:14,137 INFO [upload_forcing] publishing log messages to tcp://*:5565 2025-09-12 08:46:14,138 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2025-09-12 08:46:14,151 INFO [upload_forcing] running in process 2872097 2025-09-12 08:46:14,152 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-09-12 08:46:14,153 INFO [upload_forcing] publishing log messages to tcp://*:5566 2025-09-12 08:46:14,154 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2025-09-12 08:46:14,191 INFO [upload_forcing] running in process 2872098 2025-09-12 08:46:14,192 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-09-12 08:46:14,193 INFO [upload_forcing] publishing log messages to tcp://*:5567 2025-09-12 08:46:14,194 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2025-09-12 08:46:16,105 INFO [upload_forcing] nowcast+ 2025-09-12 forcing files upload to optimum-hindcast completed 2025-09-12 08:46:16,108 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2025-09-12 08:46:20,558 INFO [upload_forcing] nowcast+ 2025-09-12 forcing files upload to arbutus.cloud-nowcast completed 2025-09-12 08:46:20,563 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2025-09-12 08:46:20,742 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'nowcast+', '--run-date', '2025-09-12'], host='localhost') 2025-09-12 08:46:22,176 INFO [make_forcing_links] running in process 2872128 2025-09-12 08:46:22,178 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-09-12 08:46:22,179 INFO [make_forcing_links] publishing log messages to tcp://*:5563 2025-09-12 08:46:22,180 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555 2025-09-12 08:46:23,592 INFO [make_forcing_links] nowcast+ 2025-09-12 forcing file links on arbutus.cloud-nowcast created 2025-09-12 08:46:23,596 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker 2025-09-12 08:46:23,763 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2025-09-12'], host='arbutus.cloud-nowcast') 2025-09-12 08:46:23,989 INFO [upload_forcing] nowcast+ 2025-09-12 forcing files upload to robot.graham completed 2025-09-12 08:46:23,994 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2025-09-12 08:46:25,837 INFO [run_NEMO] running in process 3069 2025-09-12 08:46:25,838 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2025-09-12 08:46:25,840 INFO [run_NEMO] publishing log messages to tcp://*:5556 2025-09-12 08:46:26,752 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555 2025-09-12 08:46:26,777 WARNING [run_NEMO] There are uncommitted changes in /nemoShare/MEOPAR/nowcast-sys/SS-run-sets 2025-09-12 08:46:26,854 INFO [run_NEMO] nowcast: launching /nemoShare/MEOPAR/nowcast-sys/runs/12sep25nowcast_2025-09-12T084625.874659-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast 2025-09-12 08:46:26,864 INFO [run_NEMO] nowcast NEMO run for 2025-09-12 on arbutus.cloud-nowcast started 2025-09-12 08:46:26,868 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker 2025-09-12 08:46:27,027 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'nowcast'], host='arbutus.cloud-nowcast') 2025-09-12 08:46:29,000 INFO [watch_NEMO] running in process 3342 2025-09-12 08:46:29,000 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2025-09-12 08:46:29,001 INFO [watch_NEMO] publishing log messages to tcp://*:5557 2025-09-12 08:46:29,002 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555 2025-09-12 08:46:29,097 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: time.step not found; continuing to watch... 2025-09-12 08:48:23,631 CRITICAL [upload_forcing] unhandled exception: Traceback (most recent call last): File "/SalishSeaCast/NEMO_Nowcast/nemo_nowcast/worker.py", line 387, in _do_work checklist = self.worker_func( self._parsed_args, self.config, self.tell_manager ) File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/upload_forcing.py", line 94, in upload_forcing ssh_client, sftp_client = ssh_sftp.sftp(host_name, ssh_key) ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/SalishSeaNowcast/nowcast/ssh_sftp.py", line 144, in sftp ssh_client = ssh(host, key_filename, ssh_config_file) File "/SalishSeaCast/SalishSeaNowcast/nowcast/ssh_sftp.py", line 75, in ssh ssh_client.connect( ~~~~~~~~~~~~~~~~~~^ host["hostname"], ^^^^^^^^^^^^^^^^^ ...<3 lines>... look_for_keys=False, ^^^^^^^^^^^^^^^^^^^^ ) ^ File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/paramiko/client.py", line 386, in connect sock.connect(addr) ~~~~~~~~~~~~^^^^^^ TimeoutError: [Errno 110] Connection timed out 2025-09-12 08:51:29,097 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 18701744 = 2025-09-12 05:08:40 UTC, 21.4% complete 2025-09-12 08:56:29,098 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 18702262 = 2025-09-12 10:54:00 UTC, 45.4% complete 2025-09-12 09:01:29,099 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 18702757 = 2025-09-12 16:24:00 UTC, 68.4% complete 2025-09-12 09:06:29,099 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 18703272 = 2025-09-12 22:07:20 UTC, 92.2% complete 2025-09-12 09:11:29,103 INFO [watch_NEMO] nowcast NEMO run on arbutus.cloud-nowcast completed 2025-09-12 09:11:29,105 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker 2025-09-12 09:11:29,445 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'ssh', '--run-date', '2025-09-12'], host='localhost') 2025-09-12 09:11:29,450 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2025-09-12'], host='localhost') 2025-09-12 09:11:30,576 INFO [download_results] running in process 2872444 2025-09-12 09:11:30,578 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-09-12 09:11:30,579 INFO [download_results] publishing log messages to tcp://*:5563 2025-09-12 09:11:30,580 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2025-09-12 09:11:30,582 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/nowcast/12sep25 to /results/SalishSea/nowcast-blue.202111 2025-09-12 09:11:30,593 INFO [make_forcing_links] running in process 2872443 2025-09-12 09:11:30,595 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-09-12 09:11:30,596 INFO [make_forcing_links] publishing log messages to tcp://*:5564 2025-09-12 09:11:30,597 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555 2025-09-12 09:11:31,588 INFO [make_forcing_links] ssh 2025-09-12 forcing file links on arbutus.cloud-nowcast created 2025-09-12 09:11:31,593 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker 2025-09-12 09:11:31,755 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2025-09-12'], host='arbutus.cloud-nowcast') 2025-09-12 09:11:33,842 INFO [run_NEMO] running in process 3718 2025-09-12 09:11:33,844 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2025-09-12 09:11:33,845 INFO [run_NEMO] publishing log messages to tcp://*:5556 2025-09-12 09:11:33,846 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555 2025-09-12 09:11:34,648 WARNING [run_NEMO] There are uncommitted changes in /nemoShare/MEOPAR/nowcast-sys/SS-run-sets 2025-09-12 09:11:34,978 INFO [run_NEMO] forecast: launching /nemoShare/MEOPAR/nowcast-sys/runs/12sep25forecast_2025-09-12T091133.975972-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast 2025-09-12 09:11:34,988 INFO [run_NEMO] forecast NEMO run for 2025-09-12 on arbutus.cloud-nowcast started 2025-09-12 09:11:35,009 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker 2025-09-12 09:11:35,176 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'forecast'], host='arbutus.cloud-nowcast') 2025-09-12 09:11:37,255 INFO [watch_NEMO] running in process 3994 2025-09-12 09:11:37,258 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2025-09-12 09:11:37,259 INFO [watch_NEMO] publishing log messages to tcp://*:5557 2025-09-12 09:11:37,260 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555 2025-09-12 09:11:37,340 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: time.step not found; continuing to watch... 2025-09-12 09:12:20,056 INFO [download_results] nowcast 2025-09-12 results files from arbutus.cloud-nowcast downloaded 2025-09-12 09:12:20,060 INFO [manager] checklist updated with [results files] items from download_results worker 2025-09-12 09:12:20,219 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast', 'research', '--run-date', '2025-09-12'], host='localhost') 2025-09-12 09:12:20,403 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast', 'comparison', '--run-date', '2025-09-11'], host='localhost') 2025-09-12 09:12:20,404 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['nowcast', '--run-date', '2025-09-12'], host='localhost') 2025-09-12 09:12:22,070 INFO [make_CHS_currents_file] running in process 2872468 2025-09-12 09:12:22,071 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-09-12 09:12:22,072 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5563 2025-09-12 09:12:22,073 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555 2025-09-12 09:12:23,025 INFO [make_plots] running in process 2872467 2025-09-12 09:12:23,027 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-09-12 09:12:23,028 INFO [make_plots] publishing log messages to tcp://*:5564 2025-09-12 09:12:23,029 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555 2025-09-12 09:12:23,050 INFO [make_plots] running in process 2872465 2025-09-12 09:12:23,069 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-09-12 09:12:23,076 INFO [make_plots] publishing log messages to tcp://*:5565 2025-09-12 09:12:23,077 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555 2025-09-12 09:12:23,384 INFO [make_plots] preparing render list for 2025-09-12 NEMO nowcast-blue research figures 2025-09-12 09:12:23,409 INFO [make_plots] preparing render list for 2025-09-11 NEMO nowcast-blue comparison figures 2025-09-12 09:12:23,524 INFO [make_plots] added SH_wind figure to 2025-09-11 NEMO nowcast-blue comparison render list 2025-09-12 09:12:23,525 INFO [make_plots] added Compare_VENUS_East figure to 2025-09-11 NEMO nowcast-blue comparison render list 2025-09-12 09:12:23,525 INFO [make_plots] added Compare_VENUS_Central figure to 2025-09-11 NEMO nowcast-blue comparison render list 2025-09-12 09:12:23,526 INFO [make_plots] starting to render nemo nowcast comparison 11sep25 figures 2025-09-12 09:12:24,737 INFO [make_plots] added salinity_thalweg_and_surface figures to 2025-09-12 NEMO nowcast-blue research render list 2025-09-12 09:12:25,842 INFO [make_plots] added temperature_thalweg_and_surface figures to 2025-09-12 NEMO nowcast-blue research render list 2025-09-12 09:12:25,842 INFO [make_plots] added Currents_sections_and_surface figure to 2025-09-12 NEMO nowcast-blue research render list 2025-09-12 09:12:25,843 INFO [make_plots] added Currents_at_VENUS_Central figure to 2025-09-12 NEMO nowcast-blue research render list 2025-09-12 09:12:25,844 INFO [make_plots] added Currents_at_VENUS_East figure to 2025-09-12 NEMO nowcast-blue research render list 2025-09-12 09:12:25,845 INFO [make_plots] starting to render nemo nowcast research 12sep25 figures 2025-09-12 09:12:54,798 INFO [make_CHS_currents_file] Made CHS currents file for 2025-09-12 for nowcast 2025-09-12 09:12:54,801 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker 2025-09-12 09:13:33,380 INFO [make_plots] finished rendering nemo nowcast research 12sep25 figures 2025-09-12 09:13:33,382 INFO [make_plots] nemo research plots for 2025-09-12 nowcast completed 2025-09-12 09:13:33,393 INFO [manager] checklist updated with [plots] items from make_plots worker 2025-09-12 09:13:49,397 INFO [make_plots] finished rendering nemo nowcast comparison 11sep25 figures 2025-09-12 09:13:49,400 INFO [make_plots] nemo comparison plots for 2025-09-11 nowcast completed 2025-09-12 09:13:49,405 INFO [manager] checklist updated with [plots] items from make_plots worker 2025-09-12 09:16:37,344 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 18703890 = 2025-09-13 04:59:20 UTC, 13.9% complete 2025-09-12 09:21:37,337 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 18704354 = 2025-09-13 10:08:40 UTC, 28.2% complete 2025-09-12 09:26:37,339 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 18704845 = 2025-09-13 15:36:00 UTC, 43.3% complete 2025-09-12 09:31:37,340 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 18705344 = 2025-09-13 21:08:40 UTC, 58.8% complete 2025-09-12 09:36:37,342 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 18705841 = 2025-09-14 02:40:00 UTC, 74.1% complete 2025-09-12 09:41:37,341 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 18706338 = 2025-09-14 08:11:20 UTC, 89.4% complete 2025-09-12 09:46:37,345 INFO [watch_NEMO] forecast NEMO run on arbutus.cloud-nowcast completed 2025-09-12 09:46:37,348 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker 2025-09-12 09:46:37,717 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_wind_file', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2025-09-12'], host='arbutus.cloud-nowcast') 2025-09-12 09:46:37,724 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_current_file', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2025-09-12'], host='arbutus.cloud-nowcast') 2025-09-12 09:46:37,729 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'turbidity', '--run-date', '2025-09-12'], host='localhost') 2025-09-12 09:46:37,736 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'turbidity', '--run-date', '2025-09-12'], host='localhost') 2025-09-12 09:46:37,739 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.graham', 'turbidity', '--run-date', '2025-09-12'], host='localhost') 2025-09-12 09:46:37,743 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'turbidity', '--run-date', '2025-09-12'], host='localhost') 2025-09-12 09:46:37,747 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2025-09-12'], host='localhost') 2025-09-12 09:46:39,141 INFO [upload_forcing] running in process 2872818 2025-09-12 09:46:39,142 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-09-12 09:46:39,143 INFO [upload_forcing] publishing log messages to tcp://*:5563 2025-09-12 09:46:39,144 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2025-09-12 09:46:39,145 INFO [upload_forcing] running in process 2872821 2025-09-12 09:46:39,147 INFO [upload_forcing] running in process 2872819 2025-09-12 09:46:39,148 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-09-12 09:46:39,149 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-09-12 09:46:39,150 INFO [upload_forcing] publishing log messages to tcp://*:5564 2025-09-12 09:46:39,150 INFO [upload_forcing] publishing log messages to tcp://*:5565 2025-09-12 09:46:39,151 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2025-09-12 09:46:39,152 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2025-09-12 09:46:39,153 INFO [download_results] running in process 2872822 2025-09-12 09:46:39,153 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-09-12 09:46:39,154 INFO [download_results] publishing log messages to tcp://*:5566 2025-09-12 09:46:39,155 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2025-09-12 09:46:39,155 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/forecast/12sep25 to /results/SalishSea/forecast.202111 2025-09-12 09:46:39,159 INFO [upload_forcing] running in process 2872820 2025-09-12 09:46:39,160 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-09-12 09:46:39,161 INFO [upload_forcing] publishing log messages to tcp://*:5567 2025-09-12 09:46:39,161 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2025-09-12 09:46:39,783 INFO [upload_forcing] turbidity 2025-09-12 forcing files upload to optimum-hindcast completed 2025-09-12 09:46:39,786 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2025-09-12 09:46:40,246 INFO [upload_forcing] turbidity 2025-09-12 forcing files upload to arbutus.cloud-nowcast completed 2025-09-12 09:46:40,250 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2025-09-12 09:46:40,399 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2025-09-12'], host='localhost') 2025-09-12 09:46:40,573 INFO [make_ww3_wind_file] running in process 4939 2025-09-12 09:46:40,576 INFO [make_ww3_wind_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2025-09-12 09:46:40,577 INFO [make_ww3_wind_file] publishing log messages to tcp://*:5570 2025-09-12 09:46:40,590 INFO [make_ww3_wind_file] connected to skookum.eos.ubc.ca port 5555 2025-09-12 09:46:40,591 INFO [make_ww3_wind_file] Creating wwatch3 wind forcing file for 2025-09-12 forecast run 2025-09-12 09:46:40,595 INFO [make_ww3_current_file] running in process 4941 2025-09-12 09:46:40,596 INFO [make_ww3_current_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2025-09-12 09:46:40,598 INFO [make_ww3_current_file] publishing log messages to tcp://*:5571 2025-09-12 09:46:40,600 INFO [make_ww3_current_file] connected to skookum.eos.ubc.ca port 5555 2025-09-12 09:46:40,602 INFO [make_ww3_current_file] Creating wwatch3 currents forcing file for 2025-09-12 forecast run 2025-09-12 09:46:41,779 INFO [make_forcing_links] running in process 2872850 2025-09-12 09:46:41,780 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-09-12 09:46:41,782 INFO [make_forcing_links] publishing log messages to tcp://*:5563 2025-09-12 09:46:41,783 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555 2025-09-12 09:46:43,328 INFO [make_forcing_links] nowcast-green 2025-09-12 forcing file links on arbutus.cloud-nowcast created 2025-09-12 09:46:43,333 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker 2025-09-12 09:46:43,348 INFO [upload_forcing] turbidity 2025-09-12 forcing files upload to robot.graham completed 2025-09-12 09:46:43,499 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2025-09-12'], host='arbutus.cloud-nowcast') 2025-09-12 09:46:43,504 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2025-09-12 09:46:47,091 INFO [run_NEMO] running in process 5554 2025-09-12 09:46:47,092 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2025-09-12 09:46:47,092 INFO [run_NEMO] publishing log messages to tcp://*:5556 2025-09-12 09:46:47,094 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555 2025-09-12 09:46:49,630 WARNING [run_NEMO] There are uncommitted changes in /nemoShare/MEOPAR/nowcast-sys/SS-run-sets 2025-09-12 09:46:50,222 INFO [run_NEMO] nowcast-green: launching /nemoShare/MEOPAR/nowcast-sys/runs/12sep25nowcast-green_2025-09-12T094647.228468-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast 2025-09-12 09:46:50,242 INFO [run_NEMO] nowcast-green NEMO run for 2025-09-12 on arbutus.cloud-nowcast started 2025-09-12 09:46:50,244 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker 2025-09-12 09:46:50,411 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'nowcast-green'], host='arbutus.cloud-nowcast') 2025-09-12 09:46:52,460 INFO [make_ww3_wind_file] wwatch3 wind forcing file created on arbutus.cloud-nowcast for 2025-09-12 forecast run 2025-09-12 09:46:52,463 INFO [manager] checklist updated with [WW3 wind forcing] items from make_ww3_wind_file worker 2025-09-12 09:46:52,630 INFO [watch_NEMO] running in process 5849 2025-09-12 09:46:52,634 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2025-09-12 09:46:52,634 INFO [watch_NEMO] publishing log messages to tcp://*:5557 2025-09-12 09:46:52,635 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555 2025-09-12 09:46:52,711 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: time.step not found; continuing to watch... 2025-09-12 09:48:09,233 INFO [download_results] forecast 2025-09-12 results files from arbutus.cloud-nowcast downloaded 2025-09-12 09:48:09,237 INFO [manager] checklist updated with [results files] items from download_results worker 2025-09-12 09:48:09,403 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['forecast', '--run-date', '2025-09-12'], host='localhost') 2025-09-12 09:48:11,241 INFO [make_CHS_currents_file] running in process 2872870 2025-09-12 09:48:11,242 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-09-12 09:48:11,243 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5563 2025-09-12 09:48:11,244 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555 2025-09-12 09:48:48,588 CRITICAL [upload_forcing] unhandled exception: Traceback (most recent call last): File "/SalishSeaCast/NEMO_Nowcast/nemo_nowcast/worker.py", line 387, in _do_work checklist = self.worker_func( self._parsed_args, self.config, self.tell_manager ) File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/upload_forcing.py", line 94, in upload_forcing ssh_client, sftp_client = ssh_sftp.sftp(host_name, ssh_key) ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/SalishSeaNowcast/nowcast/ssh_sftp.py", line 144, in sftp ssh_client = ssh(host, key_filename, ssh_config_file) File "/SalishSeaCast/SalishSeaNowcast/nowcast/ssh_sftp.py", line 75, in ssh ssh_client.connect( ~~~~~~~~~~~~~~~~~~^ host["hostname"], ^^^^^^^^^^^^^^^^^ ...<3 lines>... look_for_keys=False, ^^^^^^^^^^^^^^^^^^^^ ) ^ File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/paramiko/client.py", line 386, in connect sock.connect(addr) ~~~~~~~~~~~~^^^^^^ TimeoutError: [Errno 110] Connection timed out 2025-09-12 09:49:00,346 INFO [make_CHS_currents_file] Made CHS currents file for 2025-09-12 for forecast 2025-09-12 09:49:00,348 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker 2025-09-12 09:49:00,512 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['nemo', 'forecast', '--run-date', '2025-09-12'], host='localhost') 2025-09-12 09:49:01,614 INFO [update_forecast_datasets] running in process 2872904 2025-09-12 09:49:01,615 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-09-12 09:49:01,616 INFO [update_forecast_datasets] publishing log messages to tcp://*:5563 2025-09-12 09:49:01,617 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555 2025-09-12 09:49:01,617 INFO [update_forecast_datasets] updating nemo forecast directory for forecast/12sep25 run 2025-09-12 09:49:01,875 INFO [update_forecast_datasets] updated nemo forecast directory for forecast/12sep25 run: /results/SalishSea/rolling-forecasts/nemo 2025-09-12 09:49:01,876 INFO [update_forecast_datasets] nemo 2025-09-12 forecast rolling forecast datasets updated 2025-09-12 09:49:01,879 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker 2025-09-12 09:49:02,043 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nemo-forecast'], host='localhost') 2025-09-12 09:49:02,047 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'forecast', 'publish', '--run-date', '2025-09-12'], host='localhost') 2025-09-12 09:49:02,051 INFO [manager] launching NextWorker(module='nowcast.workers.make_surface_current_tiles', args=['forecast', '--run-date', '2025-09-12'], host='localhost') 2025-09-12 09:49:03,238 INFO [ping_erddap] running in process 2872911 2025-09-12 09:49:03,240 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-09-12 09:49:03,240 INFO [ping_erddap] publishing log messages to tcp://*:5563 2025-09-12 09:49:03,241 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555 2025-09-12 09:49:03,251 INFO [ping_erddap] nemo-forecast ERDDAP dataset flag file(s) created 2025-09-12 09:49:03,252 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker 2025-09-12 09:49:04,256 INFO [make_surface_current_tiles] running in process 2872914 2025-09-12 09:49:04,259 INFO [make_surface_current_tiles] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-09-12 09:49:04,260 INFO [make_surface_current_tiles] publishing log messages to tcp://*:5563 2025-09-12 09:49:04,261 INFO [make_surface_current_tiles] connected to skookum.eos.ubc.ca port 5555 2025-09-12 09:49:04,262 INFO [make_surface_current_tiles] starting rendering of tiles for 2025-09-12 forecast in 6 processes into /results/nowcast-sys/figures/surface_currents/forecast/12sep25 2025-09-12 09:49:04,934 INFO [make_plots] running in process 2872913 2025-09-12 09:49:04,937 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-09-12 09:49:04,940 INFO [make_plots] publishing log messages to tcp://*:5564 2025-09-12 09:49:04,941 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555 2025-09-12 09:49:05,399 INFO [make_plots] preparing render list for 2025-09-12 NEMO forecast publish figures 2025-09-12 09:49:05,533 INFO [make_plots] added Website_thumbnail figure to 2025-09-12 NEMO forecast publish render list 2025-09-12 09:49:05,536 INFO [make_plots] added Threshold_website figure to 2025-09-12 NEMO forecast publish render list 2025-09-12 09:49:05,540 INFO [make_plots] added PA_tidal_predictions figure to 2025-09-12 NEMO forecast publish render list 2025-09-12 09:49:05,545 INFO [make_plots] added Boundary Bay figure to 2025-09-12 NEMO forecast publish render list 2025-09-12 09:49:05,547 INFO [make_plots] added Campbell River figure to 2025-09-12 NEMO forecast publish render list 2025-09-12 09:49:05,549 INFO [make_plots] added Cherry Point figure to 2025-09-12 NEMO forecast publish render list 2025-09-12 09:49:05,552 INFO [make_plots] added Friday Harbor figure to 2025-09-12 NEMO forecast publish render list 2025-09-12 09:49:05,556 INFO [make_plots] added Halfmoon Bay figure to 2025-09-12 NEMO forecast publish render list 2025-09-12 09:49:05,559 INFO [make_plots] added Nanaimo figure to 2025-09-12 NEMO forecast publish render list 2025-09-12 09:49:05,562 INFO [make_plots] added Neah Bay figure to 2025-09-12 NEMO forecast publish render list 2025-09-12 09:49:05,564 INFO [make_plots] added New Westminster figure to 2025-09-12 NEMO forecast publish render list 2025-09-12 09:49:05,565 INFO [make_plots] added Patricia Bay figure to 2025-09-12 NEMO forecast publish render list 2025-09-12 09:49:05,570 INFO [make_plots] added Point Atkinson figure to 2025-09-12 NEMO forecast publish render list 2025-09-12 09:49:05,580 INFO [make_plots] added Port Renfrew figure to 2025-09-12 NEMO forecast publish render list 2025-09-12 09:49:05,581 INFO [make_plots] added Sand Heads figure to 2025-09-12 NEMO forecast publish render list 2025-09-12 09:49:05,581 INFO [make_plots] added Sandy Cove figure to 2025-09-12 NEMO forecast publish render list 2025-09-12 09:49:05,582 INFO [make_plots] added Squamish figure to 2025-09-12 NEMO forecast publish render list 2025-09-12 09:49:05,583 INFO [make_plots] added Victoria figure to 2025-09-12 NEMO forecast publish render list 2025-09-12 09:49:05,584 INFO [make_plots] added Woodwards Landing figure to 2025-09-12 NEMO forecast publish render list 2025-09-12 09:49:05,585 INFO [make_plots] starting to render nemo forecast publish 12sep25 figures 2025-09-12 09:49:06,116 INFO [make_ww3_current_file] wwatch3 currents forcing file created on arbutus.cloud-nowcast for 2025-09-12 forecast run 2025-09-12 09:49:06,120 INFO [manager] checklist updated with [WW3 currents forcing] items from make_ww3_current_file worker 2025-09-12 09:49:06,348 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2025-09-12'], host='arbutus.cloud-nowcast') 2025-09-12 09:49:08,990 INFO [run_ww3] running in process 5992 2025-09-12 09:49:08,992 INFO [run_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2025-09-12 09:49:08,995 INFO [run_ww3] publishing log messages to tcp://*:5572 2025-09-12 09:49:08,998 INFO [run_ww3] connected to skookum.eos.ubc.ca port 5555 2025-09-12 09:49:09,002 INFO [run_ww3] Created run directory /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/nowcast_2025-09-12T094908.986435-0700 2025-09-12 09:49:09,003 INFO [run_ww3] nowcast: launching /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/nowcast_2025-09-12T094908.986435-0700/SoGWW3.sh on arbutus.cloud-nowcast 2025-09-12 09:49:09,008 INFO [run_ww3] nowcast WaveWatch3 run for 2025-09-12 on arbutus.cloud-nowcast started 2025-09-12 09:49:09,076 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker 2025-09-12 09:49:09,379 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'nowcast'], host='arbutus.cloud-nowcast') 2025-09-12 09:49:11,266 INFO [watch_ww3] running in process 6080 2025-09-12 09:49:11,268 INFO [watch_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2025-09-12 09:49:11,269 INFO [watch_ww3] publishing log messages to tcp://*:5573 2025-09-12 09:49:11,269 INFO [watch_ww3] connected to skookum.eos.ubc.ca port 5555 2025-09-12 09:49:11,345 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch... 2025-09-12 09:51:52,711 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18701445 = 2025-09-12 01:49:20 UTC, 7.6% complete 2025-09-12 09:54:11,345 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 75, 17.4% complete 2025-09-12 09:56:52,715 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18701633 = 2025-09-12 03:54:40 UTC, 16.3% complete 2025-09-12 09:57:02,553 WARNING [make_plots] invalid station number for Boundary Bay station: None 2025-09-12 09:57:02,554 WARNING [make_plots] can't resolve a valid CHS station code for Boundary Bay 2025-09-12 09:57:22,936 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb996/data for station 08074 Campbell River from 2025-09-07 00:00:00Z to 2025-09-14 11:55:00Z 2025-09-12 09:58:24,999 WARNING [make_plots] invalid station number for Halfmoon Bay station: None 2025-09-12 09:58:24,999 WARNING [make_plots] can't resolve a valid CHS station code for Halfmoon Bay 2025-09-12 09:58:45,742 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb96d/data for station 07917 Nanaimo from 2025-09-07 00:00:00Z to 2025-09-14 11:55:00Z 2025-09-12 09:59:11,346 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 165, 38.2% complete 2025-09-12 09:59:24,186 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e13d0f4a073c4bbf93/data for station 07654 New Westminster from 2025-09-07 00:00:00Z to 2025-09-14 11:55:00Z 2025-09-12 09:59:40,871 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd2d/data for station 07277 Patricia Bay from 2025-09-07 00:00:00Z to 2025-09-14 11:55:00Z 2025-09-12 09:59:44,934 INFO [make_surface_current_tiles] starting PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast/12sep25 2025-09-12 09:59:57,033 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94c/data for station 07795 Point Atkinson from 2025-09-07 00:00:00Z to 2025-09-14 11:55:00Z 2025-09-12 10:00:13,480 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e23d0f4a073c4bc060/data for station 08525 Port Renfrew from 2025-09-07 00:00:00Z to 2025-09-14 11:55:00Z 2025-09-12 10:00:29,181 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb93c/data for station 07594 Sand Heads from 2025-09-07 00:00:00Z to 2025-09-14 11:55:00Z 2025-09-12 10:00:45,786 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc43b/data for station 07786 Sandy Cove from 2025-09-07 00:00:00Z to 2025-09-14 11:55:00Z 2025-09-12 10:01:02,449 WARNING [make_plots] invalid station number for Squamish station: None 2025-09-12 10:01:02,450 WARNING [make_plots] can't resolve a valid CHS station code for Squamish 2025-09-12 10:01:18,200 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd1e/data for station 07120 Victoria from 2025-09-07 00:00:00Z to 2025-09-14 11:55:00Z 2025-09-12 10:01:34,162 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc42d/data for station 07610 Woodwards Landing from 2025-09-07 00:00:00Z to 2025-09-14 11:55:00Z 2025-09-12 10:01:50,841 INFO [make_plots] finished rendering nemo forecast publish 12sep25 figures 2025-09-12 10:01:50,843 INFO [make_plots] nemo publish plots for 2025-09-12 forecast completed 2025-09-12 10:01:50,851 INFO [manager] checklist updated with [plots] items from make_plots worker 2025-09-12 10:01:51,016 INFO [manager] launching NextWorker(module='nowcast.workers.make_feeds', args=['forecast', '--run-date', '2025-09-12'], host='localhost') 2025-09-12 10:01:52,714 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18701822 = 2025-09-12 06:00:40 UTC, 25.1% complete 2025-09-12 10:01:54,032 INFO [make_feeds] running in process 2874805 2025-09-12 10:01:54,033 INFO [make_feeds] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-09-12 10:01:54,035 INFO [make_feeds] publishing log messages to tcp://*:5564 2025-09-12 10:01:54,035 INFO [make_feeds] connected to skookum.eos.ubc.ca port 5555 2025-09-12 10:02:08,485 INFO [make_feeds] ATOM feeds for 2025-09-12 forecast run completed 2025-09-12 10:02:08,488 INFO [manager] checklist updated with [storm surge alert feeds] items from make_feeds worker 2025-09-12 10:04:11,349 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 252, 58.3% complete 2025-09-12 10:04:23,380 INFO [make_surface_current_tiles] finished PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast/12sep25 2025-09-12 10:04:23,405 INFO [make_surface_current_tiles] finished rendering of tiles for 2025-09-12 forecast into /results/nowcast-sys/figures/surface_currents/forecast/12sep25 2025-09-12 10:04:23,406 INFO [make_surface_current_tiles] surface current tile figures for 2025-09-12 forecast completed 2025-09-12 10:04:23,729 INFO [manager] checklist updated with [surface current tiles] items from make_surface_current_tiles worker 2025-09-12 10:06:52,717 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18702011 = 2025-09-12 08:06:40 UTC, 33.8% complete 2025-09-12 10:09:11,351 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 336, 77.8% complete 2025-09-12 10:11:52,730 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18702200 = 2025-09-12 10:12:40 UTC, 42.6% complete 2025-09-12 10:14:11,353 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 426, 98.6% complete 2025-09-12 10:16:52,727 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18702384 = 2025-09-12 12:15:20 UTC, 51.1% complete 2025-09-12 10:19:11,355 INFO [watch_ww3] nowcast WWATCH3 run on arbutus.cloud-nowcast completed 2025-09-12 10:19:11,358 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker 2025-09-12 10:19:11,868 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2025-09-12'], host='localhost') 2025-09-12 10:19:11,874 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2025-09-12'], host='arbutus.cloud-nowcast') 2025-09-12 10:19:12,984 INFO [download_wwatch3_results] running in process 2875096 2025-09-12 10:19:12,985 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-09-12 10:19:12,986 INFO [download_wwatch3_results] publishing log messages to tcp://*:5563 2025-09-12 10:19:12,987 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555 2025-09-12 10:19:13,921 INFO [run_ww3] running in process 6482 2025-09-12 10:19:13,922 INFO [run_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2025-09-12 10:19:13,923 INFO [run_ww3] publishing log messages to tcp://*:5572 2025-09-12 10:19:13,924 INFO [run_ww3] connected to skookum.eos.ubc.ca port 5555 2025-09-12 10:19:13,928 INFO [run_ww3] Created run directory /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast_2025-09-12T101913.914096-0700 2025-09-12 10:19:13,930 INFO [run_ww3] forecast: launching /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast_2025-09-12T101913.914096-0700/SoGWW3.sh on arbutus.cloud-nowcast 2025-09-12 10:19:13,940 INFO [run_ww3] forecast WaveWatch3 run for 2025-09-12 on arbutus.cloud-nowcast started 2025-09-12 10:19:13,957 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker 2025-09-12 10:19:14,282 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'forecast'], host='arbutus.cloud-nowcast') 2025-09-12 10:19:16,377 INFO [watch_ww3] running in process 6637 2025-09-12 10:19:16,380 INFO [watch_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2025-09-12 10:19:16,381 INFO [watch_ww3] publishing log messages to tcp://*:5573 2025-09-12 10:19:16,382 INFO [watch_ww3] connected to skookum.eos.ubc.ca port 5555 2025-09-12 10:19:16,461 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch... 2025-09-12 10:19:44,635 INFO [download_wwatch3_results] nowcast 2025-09-12 results files from arbutus.cloud-nowcast downloaded 2025-09-12 10:19:44,638 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker 2025-09-12 10:21:52,723 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18702573 = 2025-09-12 14:21:20 UTC, 59.8% complete 2025-09-12 10:24:16,462 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 75, 11.6% complete 2025-09-12 10:26:52,727 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18702760 = 2025-09-12 16:26:00 UTC, 68.5% complete 2025-09-12 10:29:16,463 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 165, 25.5% complete 2025-09-12 10:31:52,723 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18702947 = 2025-09-12 18:30:40 UTC, 77.2% complete 2025-09-12 10:34:16,461 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 252, 38.9% complete 2025-09-12 10:36:52,722 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18703132 = 2025-09-12 20:34:00 UTC, 85.7% complete 2025-09-12 10:39:16,461 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 336, 51.9% complete 2025-09-12 10:41:52,723 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18703319 = 2025-09-12 22:38:40 UTC, 94.4% complete 2025-09-12 10:44:16,460 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 426, 65.7% complete 2025-09-12 10:46:52,722 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18703440 = 2025-09-12 23:59:20 UTC, 100.0% complete 2025-09-12 10:49:16,461 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 516, 79.6% complete 2025-09-12 10:51:52,728 INFO [watch_NEMO] nowcast-green NEMO run on arbutus.cloud-nowcast completed 2025-09-12 10:51:52,731 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker 2025-09-12 10:51:53,253 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2025-09-12'], host='localhost') 2025-09-12 10:51:54,356 INFO [download_results] running in process 2876649 2025-09-12 10:51:54,357 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-09-12 10:51:54,358 INFO [download_results] publishing log messages to tcp://*:5563 2025-09-12 10:51:54,359 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2025-09-12 10:51:54,360 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/nowcast-green/12sep25 to /results2/SalishSea/nowcast-green.202111 2025-09-12 10:54:16,463 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 606, 93.5% complete 2025-09-12 10:55:08,098 INFO [download_results] nowcast-green 2025-09-12 results files from arbutus.cloud-nowcast downloaded 2025-09-12 10:55:08,103 INFO [manager] checklist updated with [results files] items from download_results worker 2025-09-12 10:55:08,437 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast-green', 'research', '--run-date', '2025-09-12'], host='localhost') 2025-09-12 10:55:08,442 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nowcast-green'], host='localhost') 2025-09-12 10:55:08,447 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'physics', '--run-date', '2025-09-12'], host='localhost') 2025-09-12 10:55:08,452 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'biology', '--run-date', '2025-09-12'], host='localhost') 2025-09-12 10:55:08,456 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'chemistry', '--run-date', '2025-09-12'], host='localhost') 2025-09-12 10:55:09,873 INFO [ping_erddap] running in process 2876704 2025-09-12 10:55:09,874 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-09-12 10:55:09,875 INFO [ping_erddap] publishing log messages to tcp://*:5563 2025-09-12 10:55:09,876 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555 2025-09-12 10:55:09,881 INFO [ping_erddap] nowcast-green ERDDAP dataset flag file(s) created 2025-09-12 10:55:09,883 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker 2025-09-12 10:55:10,876 INFO [make_averaged_dataset] running in process 2876706 2025-09-12 10:55:10,877 INFO [make_averaged_dataset] running in process 2876705 2025-09-12 10:55:10,878 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-09-12 10:55:10,879 INFO [make_averaged_dataset] publishing log messages to tcp://*:5563 2025-09-12 10:55:10,880 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555 2025-09-12 10:55:10,881 INFO [make_averaged_dataset] creating day-averaged dataset for 12-Sep-2025 biology 2025-09-12 10:55:10,881 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-09-12 10:55:10,882 INFO [make_averaged_dataset] publishing log messages to tcp://*:5564 2025-09-12 10:55:10,882 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555 2025-09-12 10:55:10,883 INFO [make_averaged_dataset] creating day-averaged dataset for 12-Sep-2025 physics 2025-09-12 10:55:10,885 INFO [make_averaged_dataset] running in process 2876707 2025-09-12 10:55:10,886 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-09-12 10:55:10,887 INFO [make_averaged_dataset] publishing log messages to tcp://*:5565 2025-09-12 10:55:10,887 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555 2025-09-12 10:55:10,888 INFO [make_averaged_dataset] creating day-averaged dataset for 12-Sep-2025 chemistry 2025-09-12 10:55:11,418 INFO [make_plots] running in process 2876702 2025-09-12 10:55:11,419 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-09-12 10:55:11,420 INFO [make_plots] publishing log messages to tcp://*:5566 2025-09-12 10:55:11,421 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555 2025-09-12 10:55:11,756 INFO [make_plots] preparing render list for 2025-09-12 NEMO nowcast-green research figures 2025-09-12 10:55:13,546 INFO [make_plots] added temperature_thalweg_and_surface figures to 2025-09-12 NEMO nowcast-green research render list 2025-09-12 10:55:14,278 INFO [make_plots] added nitrate_thalweg_and_surface figures to 2025-09-12 NEMO nowcast-green research render list 2025-09-12 10:55:15,043 INFO [make_plots] added ammonium_thalweg_and_surface figures to 2025-09-12 NEMO nowcast-green research render list 2025-09-12 10:55:15,845 INFO [make_plots] added silicon_thalweg_and_surface figures to 2025-09-12 NEMO nowcast-green research render list 2025-09-12 10:55:16,677 INFO [make_plots] added dissolved_organic_nitrogen_thalweg_and_surface figures to 2025-09-12 NEMO nowcast-green research render list 2025-09-12 10:55:17,487 INFO [make_plots] added particulate_organic_nitrogen_thalweg_and_surface figures to 2025-09-12 NEMO nowcast-green research render list 2025-09-12 10:55:18,263 INFO [make_plots] added biogenic_silicon_thalweg_and_surface figures to 2025-09-12 NEMO nowcast-green research render list 2025-09-12 10:55:19,113 INFO [make_plots] added diatoms_thalweg_and_surface figures to 2025-09-12 NEMO nowcast-green research render list 2025-09-12 10:55:19,973 INFO [make_plots] added flagellates_thalweg_and_surface figures to 2025-09-12 NEMO nowcast-green research render list 2025-09-12 10:55:20,831 INFO [make_plots] added microzooplankton_thalweg_and_surface figures to 2025-09-12 NEMO nowcast-green research render list 2025-09-12 10:55:21,677 INFO [make_plots] added mesozooplankton_thalweg_and_surface figures to 2025-09-12 NEMO nowcast-green research render list 2025-09-12 10:55:22,416 INFO [make_plots] added turbidity_thalweg_and_surface figures to 2025-09-12 NEMO nowcast-green research render list 2025-09-12 10:55:23,797 INFO [make_plots] added temperature_salinity_timeseries figure to 2025-09-12 NEMO nowcast-green research render list 2025-09-12 10:55:23,797 INFO [make_plots] added nitrate_diatoms_timeseries figure to 2025-09-12 NEMO nowcast-green research render list 2025-09-12 10:55:23,798 INFO [make_plots] added diatoms_flagellates_timeseries figure to 2025-09-12 NEMO nowcast-green research render list 2025-09-12 10:55:23,798 INFO [make_plots] added z1_z2_zooplankton_timeseries figure to 2025-09-12 NEMO nowcast-green research render list 2025-09-12 10:55:23,799 INFO [make_plots] starting to render nemo nowcast-green research 12sep25 figures 2025-09-12 10:56:31,871 INFO [make_averaged_dataset] day-averaged dataset for 12-Sep-2025 physics created 2025-09-12 10:56:31,875 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker 2025-09-12 10:56:53,549 INFO [make_averaged_dataset] day-averaged dataset for 12-Sep-2025 chemistry created 2025-09-12 10:56:53,552 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker 2025-09-12 10:57:16,339 INFO [make_averaged_dataset] day-averaged dataset for 12-Sep-2025 biology created 2025-09-12 10:57:16,341 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker 2025-09-12 10:59:16,470 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker 2025-09-12 10:59:16,472 INFO [watch_ww3] forecast WWATCH3 run on arbutus.cloud-nowcast completed 2025-09-12 10:59:16,992 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2025-09-12'], host='localhost') 2025-09-12 10:59:18,306 INFO [download_wwatch3_results] running in process 2876829 2025-09-12 10:59:18,309 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-09-12 10:59:18,310 INFO [download_wwatch3_results] publishing log messages to tcp://*:5563 2025-09-12 10:59:18,311 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555 2025-09-12 10:59:56,187 INFO [download_wwatch3_results] forecast 2025-09-12 results files from arbutus.cloud-nowcast downloaded 2025-09-12 10:59:56,190 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker 2025-09-12 10:59:56,513 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['wwatch3', 'forecast', '--run-date', '2025-09-12'], host='localhost') 2025-09-12 10:59:57,642 INFO [update_forecast_datasets] running in process 2876840 2025-09-12 10:59:57,668 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-09-12 10:59:57,686 INFO [update_forecast_datasets] publishing log messages to tcp://*:5563 2025-09-12 10:59:57,704 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555 2025-09-12 10:59:57,716 INFO [update_forecast_datasets] updating wwatch3 most_recent_forecast directory from forecast/12sep25 run 2025-09-12 10:59:57,719 INFO [update_forecast_datasets] updated wwatch3 most_recent_forecast directory from forecast/12sep25 run 2025-09-12 10:59:57,720 INFO [update_forecast_datasets] updating wwatch3 forecast directory for forecast/12sep25 run 2025-09-12 10:59:57,762 INFO [update_forecast_datasets] updated wwatch3 forecast directory for forecast/12sep25 run: /results/SalishSea/rolling-forecasts/wwatch3 2025-09-12 10:59:57,763 INFO [update_forecast_datasets] wwatch3 2025-09-12 forecast rolling forecast datasets updated 2025-09-12 10:59:57,766 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker 2025-09-12 10:59:58,108 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['wwatch3-forecast'], host='localhost') 2025-09-12 10:59:59,241 INFO [ping_erddap] running in process 2876845 2025-09-12 10:59:59,242 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-09-12 10:59:59,243 INFO [ping_erddap] publishing log messages to tcp://*:5563 2025-09-12 10:59:59,244 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555 2025-09-12 10:59:59,246 INFO [ping_erddap] wwatch3-forecast ERDDAP dataset flag file(s) created 2025-09-12 10:59:59,248 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker 2025-09-12 10:59:59,583 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['wwatch3', 'forecast', 'publish', '--run-date', '2025-09-12'], host='localhost') 2025-09-12 11:00:02,533 INFO [make_plots] running in process 2876851 2025-09-12 11:00:02,535 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-09-12 11:00:02,536 INFO [make_plots] publishing log messages to tcp://*:5563 2025-09-12 11:00:02,537 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555 2025-09-12 11:00:02,538 INFO [make_plots] preparing render list for 2025-09-12 WaveWatch3 forecast publish figures 2025-09-12 11:00:02,538 INFO [make_plots] added Halibut Bank figure to 2025-09-12 WaveWatch3 forecast publish render list 2025-09-12 11:00:02,539 INFO [make_plots] added Sentry Shoal figure to 2025-09-12 WaveWatch3 forecast publish render list 2025-09-12 11:00:02,540 INFO [make_plots] starting to render wwatch3 forecast publish 12sep25 figures 2025-09-12 11:00:03,082 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46146 Halibut Bank 2025-09-12 11:00:07,780 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46131 Sentry Shoal 2025-09-12 11:00:12,070 INFO [make_plots] finished rendering wwatch3 forecast publish 12sep25 figures 2025-09-12 11:00:12,070 INFO [make_plots] wwatch3 publish plots for 2025-09-12 forecast completed 2025-09-12 11:00:12,074 INFO [manager] checklist updated with [plots] items from make_plots worker 2025-09-12 11:56:10,129 INFO [make_plots] finished rendering nemo nowcast-green research 12sep25 figures 2025-09-12 11:56:10,321 INFO [make_plots] nemo research plots for 2025-09-12 nowcast-green completed 2025-09-12 11:56:10,386 INFO [manager] checklist updated with [plots] items from make_plots worker 2025-09-12 14:56:15,562 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/18/ to /results/forcing/atmospheric/continental2.5/GRIB/20250912/18/ 2025-09-12 14:56:15,563 INFO [collect_weather] 2025-09-12 2.5km weather forecast 18 collection complete 2025-09-12 14:56:15,568 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker 2025-09-12 14:56:16,239 INFO [manager] launching NextWorker(module='nowcast.workers.download_weather', args=['00', '1km'], host='localhost') 2025-09-12 14:56:16,246 INFO [manager] launching NextWorker(module='nowcast.workers.download_weather', args=['12', '1km'], host='localhost') 2025-09-12 14:56:16,251 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['00', '2.5km'], host='localhost') 2025-09-12 14:56:16,256 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['00', '--fcst-date', '2025-09-13'], host='localhost') 2025-09-12 14:56:16,790 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20250912/18/ 2025-09-12 14:56:16,791 INFO [crop_gribs] 2025-09-12 18 GRIBs cropping complete 2025-09-12 14:56:16,793 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker 2025-09-12 14:56:17,991 INFO [collect_weather] running in process 2879235 2025-09-12 14:56:17,992 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-09-12 14:56:17,993 INFO [collect_weather] publishing log messages to tcp://*:5562 2025-09-12 14:56:17,994 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555 2025-09-12 14:56:17,995 INFO [download_weather] running in process 2879232 2025-09-12 14:56:17,996 INFO [download_weather] running in process 2879234 2025-09-12 14:56:17,997 INFO [download_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-09-12 14:56:17,998 INFO [download_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-09-12 14:56:17,999 INFO [download_weather] publishing log messages to tcp://*:5563 2025-09-12 14:56:18,001 INFO [download_weather] publishing log messages to tcp://*:5564 2025-09-12 14:56:18,001 INFO [download_weather] connected to skookum.eos.ubc.ca port 5555 2025-09-12 14:56:18,002 INFO [download_weather] connected to skookum.eos.ubc.ca port 5555 2025-09-12 14:56:18,003 INFO [download_weather] downloading 00 1 km forecast GRIB2 files for 20250912 2025-09-12 14:56:18,003 INFO [download_weather] downloading 12 1 km forecast GRIB2 files for 20250912 2025-09-12 14:56:18,045 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/00/ 2025-09-12 14:56:18,881 INFO [crop_gribs] running in process 2879236 2025-09-12 14:56:18,882 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-09-12 14:56:18,884 INFO [crop_gribs] publishing log messages to tcp://*:5565 2025-09-12 14:56:18,885 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555 2025-09-12 14:56:18,885 INFO [crop_gribs] cropping 2025-09-13 ECCC HRDPS 2.5km continental 00Z GRIB files to SalishSeaCast subdomain 2025-09-12 14:56:18,900 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20250913/00/ 2025-09-12 14:59:47,820 INFO [download_weather] 2025-09-12 1km weather forecast 00 downloads complete 2025-09-12 14:59:47,825 INFO [manager] checklist updated with [weather forecast] items from download_weather worker 2025-09-12 14:59:50,627 INFO [download_weather] 2025-09-12 1km weather forecast 12 downloads complete 2025-09-12 14:59:50,630 INFO [manager] checklist updated with [weather forecast] items from download_weather worker 2025-09-12 20:45:21,698 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/00/ to /results/forcing/atmospheric/continental2.5/GRIB/20250913/00/ 2025-09-12 20:45:21,699 INFO [collect_weather] 2025-09-13 2.5km weather forecast 00 collection complete 2025-09-12 20:45:21,702 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker 2025-09-12 20:45:22,216 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['06', '2.5km'], host='localhost') 2025-09-12 20:45:22,222 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['06', '--fcst-date', '2025-09-13'], host='localhost') 2025-09-12 20:45:22,234 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20250913/00/ 2025-09-12 20:45:22,235 INFO [crop_gribs] 2025-09-13 00 GRIBs cropping complete 2025-09-12 20:45:22,238 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker 2025-09-12 20:45:23,466 INFO [collect_weather] running in process 2882822 2025-09-12 20:45:23,467 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-09-12 20:45:23,468 INFO [collect_weather] publishing log messages to tcp://*:5562 2025-09-12 20:45:23,469 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555 2025-09-12 20:45:23,483 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/06/ 2025-09-12 20:45:24,077 INFO [crop_gribs] running in process 2882824 2025-09-12 20:45:24,078 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-09-12 20:45:24,079 INFO [crop_gribs] publishing log messages to tcp://*:5563 2025-09-12 20:45:24,080 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555 2025-09-12 20:45:24,081 INFO [crop_gribs] cropping 2025-09-13 ECCC HRDPS 2.5km continental 06Z GRIB files to SalishSeaCast subdomain 2025-09-12 20:45:24,095 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20250913/06/ 2025-09-13 02:46:47,218 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/06/ to /results/forcing/atmospheric/continental2.5/GRIB/20250913/06/ 2025-09-13 02:46:47,219 INFO [collect_weather] 2025-09-13 2.5km weather forecast 06 collection complete 2025-09-13 02:46:47,222 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker 2025-09-13 02:46:47,756 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Capilano', '--data-date', '2025-09-12'], host='localhost') 2025-09-13 02:46:47,763 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'ChilliwackVedder', '--data-date', '2025-09-12'], host='localhost') 2025-09-13 02:46:47,767 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'ClowhomClowhomLake', '--data-date', '2025-09-12'], host='localhost') 2025-09-13 02:46:47,772 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Englishman', '--data-date', '2025-09-12'], host='localhost') 2025-09-13 02:46:47,775 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Fraser', '--data-date', '2025-09-12'], host='localhost') 2025-09-13 02:46:47,778 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'HomathkoMouth', '--data-date', '2025-09-12'], host='localhost') 2025-09-13 02:46:47,783 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'NicomeklLangley', '--data-date', '2025-09-12'], host='localhost') 2025-09-13 02:46:47,790 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'RobertsCreek', '--data-date', '2025-09-12'], host='localhost') 2025-09-13 02:46:47,795 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SalmonSayward', '--data-date', '2025-09-12'], host='localhost') 2025-09-13 02:46:47,801 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SanJuanPortRenfrew', '--data-date', '2025-09-12'], host='localhost') 2025-09-13 02:46:47,809 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SquamishBrackendale', '--data-date', '2025-09-12'], host='localhost') 2025-09-13 02:46:47,818 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaScotty', '--data-date', '2025-09-12'], host='localhost') 2025-09-13 02:46:47,822 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaBypass', '--data-date', '2025-09-12'], host='localhost') 2025-09-13 02:46:47,836 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaDiversion', '--data-date', '2025-09-12'], host='localhost') 2025-09-13 02:46:47,843 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ctd', args=['SCVIP'], host='localhost') 2025-09-13 02:46:47,857 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ctd', args=['SEVIP'], host='localhost') 2025-09-13 02:46:47,866 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ferry', args=['TWDP'], host='localhost') 2025-09-13 02:46:47,882 INFO [manager] launching NextWorker(module='nowcast.workers.get_vfpa_hadcp', args=['--data-date', '2025-09-12'], host='localhost') 2025-09-13 02:46:47,890 INFO [manager] launching NextWorker(module='nowcast.workers.collect_NeahBay_ssh', args=['00'], host='localhost') 2025-09-13 02:46:47,909 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['12', '2.5km'], host='localhost') 2025-09-13 02:46:47,930 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['12'], host='localhost') 2025-09-13 02:46:50,370 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20250913/06/ 2025-09-13 02:46:50,372 INFO [crop_gribs] 2025-09-13 06 GRIBs cropping complete 2025-09-13 02:46:50,373 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker 2025-09-13 02:46:51,077 INFO [collect_NeahBay_ssh] running in process 2886542 2025-09-13 02:46:51,079 INFO [collect_NeahBay_ssh] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-09-13 02:46:51,080 INFO [collect_NeahBay_ssh] publishing log messages to tcp://*:5562 2025-09-13 02:46:51,081 INFO [collect_NeahBay_ssh] connected to skookum.eos.ubc.ca port 5555 2025-09-13 02:46:51,081 INFO [collect_NeahBay_ssh] collecting Neah Bay ssh 00Z obs/forecast for 2025-09-13 2025-09-13 02:46:51,122 INFO [collect_weather] running in process 2886543 2025-09-13 02:46:51,123 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-09-13 02:46:51,129 INFO [collect_weather] publishing log messages to tcp://*:5564 2025-09-13 02:46:51,131 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555 2025-09-13 02:46:51,164 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/12/ 2025-09-13 02:46:51,215 INFO [manager] launching NextWorker(module='nowcast.workers.grib_to_netcdf', args=['forecast2'], host='localhost') 2025-09-13 02:46:52,217 INFO [collect_river_data] running in process 2886528 2025-09-13 02:46:52,218 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-09-13 02:46:52,220 INFO [collect_river_data] publishing log messages to tcp://*:5563 2025-09-13 02:46:52,221 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2025-09-13 02:46:52,222 INFO [collect_river_data] Collecting ECCC HomathkoMouth river data for 2025-09-12 2025-09-13 02:46:52,246 INFO [collect_river_data] running in process 2886522 2025-09-13 02:46:52,247 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-09-13 02:46:52,248 INFO [collect_river_data] publishing log messages to tcp://*:5565 2025-09-13 02:46:52,249 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2025-09-13 02:46:52,249 INFO [collect_river_data] Collecting ECCC Capilano river data for 2025-09-12 2025-09-13 02:46:52,269 INFO [collect_river_data] Appended ECCC HomathkoMouth river average discharge for 2025-09-12 to: /results/forcing/rivers/observations/Homathko_Mouth_flow 2025-09-13 02:46:52,270 INFO [collect_river_data] ECCC HomathkoMouth river data collection for 2025-09-12 completed 2025-09-13 02:46:52,273 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2025-09-13 02:46:52,297 INFO [collect_river_data] Appended ECCC Capilano river average discharge for 2025-09-12 to: /opp/observations/rivers/Capilano/Caplilano_08GA010_day_avg_flow 2025-09-13 02:46:52,298 INFO [collect_river_data] ECCC Capilano river data collection for 2025-09-12 completed 2025-09-13 02:46:52,309 INFO [collect_river_data] running in process 2886530 2025-09-13 02:46:52,310 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-09-13 02:46:52,312 INFO [collect_river_data] publishing log messages to tcp://*:5566 2025-09-13 02:46:52,313 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2025-09-13 02:46:52,314 INFO [collect_river_data] Collecting ECCC RobertsCreek river data for 2025-09-12 2025-09-13 02:46:52,328 INFO [collect_river_data] running in process 2886531 2025-09-13 02:46:52,329 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-09-13 02:46:52,329 INFO [collect_river_data] publishing log messages to tcp://*:5567 2025-09-13 02:46:52,331 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2025-09-13 02:46:52,331 INFO [collect_river_data] Collecting ECCC SalmonSayward river data for 2025-09-12 2025-09-13 02:46:52,336 INFO [collect_river_data] running in process 2886526 2025-09-13 02:46:52,337 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-09-13 02:46:52,338 INFO [collect_river_data] publishing log messages to tcp://*:5568 2025-09-13 02:46:52,339 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2025-09-13 02:46:52,341 INFO [collect_river_data] Collecting ECCC Englishman river data for 2025-09-12 2025-09-13 02:46:52,351 INFO [collect_river_data] running in process 2886529 2025-09-13 02:46:52,352 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-09-13 02:46:52,352 INFO [collect_river_data] publishing log messages to tcp://*:5569 2025-09-13 02:46:52,353 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2025-09-13 02:46:52,354 INFO [collect_river_data] Collecting ECCC NicomeklLangley river data for 2025-09-12 2025-09-13 02:46:52,359 INFO [collect_river_data] Appended ECCC RobertsCreek river average discharge for 2025-09-12 to: /results/forcing/rivers/observations/RobertsCreek_flow 2025-09-13 02:46:52,360 INFO [collect_river_data] ECCC RobertsCreek river data collection for 2025-09-12 completed 2025-09-13 02:46:52,363 INFO [collect_river_data] Appended ECCC SalmonSayward river average discharge for 2025-09-12 to: /results/forcing/rivers/observations/Salmon_Sayward_flow 2025-09-13 02:46:52,364 INFO [collect_river_data] ECCC SalmonSayward river data collection for 2025-09-12 completed 2025-09-13 02:46:52,375 INFO [collect_river_data] Appended ECCC Englishman river average discharge for 2025-09-12 to: /data/dlatorne/SOG-projects/SOG-forcing/ECget/Englishman_flow 2025-09-13 02:46:52,375 INFO [collect_river_data] ECCC Englishman river data collection for 2025-09-12 completed 2025-09-13 02:46:52,382 INFO [collect_river_data] Appended ECCC NicomeklLangley river average discharge for 2025-09-12 to: /results/forcing/rivers/observations/Nicomekl_Langley_flow 2025-09-13 02:46:52,383 INFO [collect_river_data] ECCC NicomeklLangley river data collection for 2025-09-12 completed 2025-09-13 02:46:52,384 INFO [collect_river_data] running in process 2886532 2025-09-13 02:46:52,385 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-09-13 02:46:52,386 INFO [collect_river_data] publishing log messages to tcp://*:5570 2025-09-13 02:46:52,386 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2025-09-13 02:46:52,387 INFO [collect_river_data] Collecting ECCC SanJuanPortRenfrew river data for 2025-09-12 2025-09-13 02:46:52,393 INFO [collect_NeahBay_ssh] 2025-09-13T00:00:00-07:00 Neah Bay ssh 00Z obs/forecast data collection complete 2025-09-13 02:46:52,410 INFO [collect_river_data] running in process 2886527 2025-09-13 02:46:52,410 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-09-13 02:46:52,411 INFO [collect_river_data] publishing log messages to tcp://*:5571 2025-09-13 02:46:52,412 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2025-09-13 02:46:52,413 INFO [collect_river_data] Collecting ECCC Fraser river data for 2025-09-12 2025-09-13 02:46:52,415 INFO [collect_river_data] Appended ECCC SanJuanPortRenfrew river average discharge for 2025-09-12 to: /results/forcing/rivers/observations/SanJuan_PortRenfrew_flow 2025-09-13 02:46:52,416 INFO [collect_river_data] ECCC SanJuanPortRenfrew river data collection for 2025-09-12 completed 2025-09-13 02:46:52,432 INFO [collect_river_data] running in process 2886524 2025-09-13 02:46:52,433 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-09-13 02:46:52,593 INFO [collect_river_data] publishing log messages to tcp://*:5572 2025-09-13 02:46:52,596 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2025-09-13 02:46:52,597 INFO [collect_river_data] Collecting ECCC ChilliwackVedder river data for 2025-09-12 2025-09-13 02:46:52,599 INFO [collect_river_data] Appended ECCC ChilliwackVedder river average discharge for 2025-09-12 to: /results/forcing/rivers/observations/Chilliwack_Vedder_flow 2025-09-13 02:46:52,599 INFO [collect_river_data] ECCC ChilliwackVedder river data collection for 2025-09-12 completed 2025-09-13 02:46:52,601 INFO [collect_river_data] running in process 2886533 2025-09-13 02:46:52,602 INFO [collect_river_data] running in process 2886536 2025-09-13 02:46:52,602 INFO [collect_river_data] running in process 2886525 2025-09-13 02:46:52,603 INFO [collect_river_data] running in process 2886535 2025-09-13 02:46:52,603 INFO [collect_river_data] running in process 2886534 2025-09-13 02:46:52,604 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-09-13 02:46:52,605 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-09-13 02:46:52,605 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-09-13 02:46:52,605 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-09-13 02:46:52,606 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-09-13 02:46:52,606 INFO [collect_river_data] Appended ECCC Fraser river average discharge for 2025-09-12 to: /data/dlatorne/SOG-projects/SOG-forcing/ECget/Fraser_flow 2025-09-13 02:46:52,607 INFO [collect_river_data] publishing log messages to tcp://*:5573 2025-09-13 02:46:52,607 INFO [collect_river_data] publishing log messages to tcp://*:5574 2025-09-13 02:46:52,608 INFO [collect_river_data] publishing log messages to tcp://*:5575 2025-09-13 02:46:52,608 INFO [collect_river_data] publishing log messages to tcp://*:5576 2025-09-13 02:46:52,609 INFO [collect_river_data] publishing log messages to tcp://*:5577 2025-09-13 02:46:52,609 INFO [collect_river_data] ECCC Fraser river data collection for 2025-09-12 completed 2025-09-13 02:46:52,610 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2025-09-13 02:46:52,611 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2025-09-13 02:46:52,611 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2025-09-13 02:46:52,612 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2025-09-13 02:46:52,612 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2025-09-13 02:46:52,614 INFO [collect_river_data] Collecting ECCC SquamishBrackendale river data for 2025-09-12 2025-09-13 02:46:52,614 INFO [collect_river_data] Collecting ECCC TheodosiaDiversion river data for 2025-09-12 2025-09-13 02:46:52,615 INFO [collect_river_data] Collecting ECCC ClowhomClowhomLake river data for 2025-09-12 2025-09-13 02:46:52,616 INFO [collect_river_data] Collecting ECCC TheodosiaBypass river data for 2025-09-12 2025-09-13 02:46:52,617 INFO [collect_river_data] Collecting ECCC TheodosiaScotty river data for 2025-09-12 2025-09-13 02:46:52,622 INFO [collect_river_data] Appended ECCC TheodosiaScotty river average discharge for 2025-09-12 to: /results/forcing/rivers/observations/Theodosia_Scotty_flow 2025-09-13 02:46:52,622 INFO [collect_river_data] Appended ECCC SquamishBrackendale river average discharge for 2025-09-12 to: /results/forcing/rivers/observations/Squamish_Brackendale_flow 2025-09-13 02:46:52,623 INFO [collect_river_data] Appended ECCC TheodosiaDiversion river average discharge for 2025-09-12 to: /results/forcing/rivers/observations/Theodosia_Diversion_flow 2025-09-13 02:46:52,624 INFO [collect_river_data] Appended ECCC ClowhomClowhomLake river average discharge for 2025-09-12 to: /results/forcing/rivers/observations/Clowhom_ClowhomLake_flow 2025-09-13 02:46:52,624 INFO [collect_river_data] Appended ECCC TheodosiaBypass river average discharge for 2025-09-12 to: /results/forcing/rivers/observations/Theodosia_Bypass_flow 2025-09-13 02:46:52,625 INFO [collect_river_data] ECCC TheodosiaScotty river data collection for 2025-09-12 completed 2025-09-13 02:46:52,625 INFO [collect_river_data] ECCC SquamishBrackendale river data collection for 2025-09-12 completed 2025-09-13 02:46:52,626 INFO [collect_river_data] ECCC TheodosiaDiversion river data collection for 2025-09-12 completed 2025-09-13 02:46:52,627 INFO [collect_river_data] ECCC ClowhomClowhomLake river data collection for 2025-09-12 completed 2025-09-13 02:46:52,628 INFO [collect_river_data] ECCC TheodosiaBypass river data collection for 2025-09-12 completed 2025-09-13 02:46:52,631 INFO [get_vfpa_hadcp] running in process 2886541 2025-09-13 02:46:52,631 INFO [get_vfpa_hadcp] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-09-13 02:46:52,632 INFO [get_vfpa_hadcp] publishing log messages to tcp://*:5578 2025-09-13 02:46:52,632 INFO [get_vfpa_hadcp] connected to skookum.eos.ubc.ca port 5555 2025-09-13 02:46:52,633 INFO [get_vfpa_hadcp] processing VFPA HADCP data from 2nd Narrows Rail Bridge for 2025-09-12 2025-09-13 02:46:52,698 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2025-09-13 02:46:52,737 INFO [crop_gribs] running in process 2886544 2025-09-13 02:46:52,737 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-09-13 02:46:52,738 INFO [crop_gribs] publishing log messages to tcp://*:5579 2025-09-13 02:46:52,739 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555 2025-09-13 02:46:52,739 INFO [crop_gribs] cropping 2025-09-13 ECCC HRDPS 2.5km continental 12Z GRIB files to SalishSeaCast subdomain 2025-09-13 02:46:52,748 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20250913/12/ 2025-09-13 02:46:53,115 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2025-09-13 02:46:53,160 INFO [get_onc_ctd] running in process 2886538 2025-09-13 02:46:53,161 INFO [get_onc_ctd] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-09-13 02:46:53,162 INFO [get_onc_ctd] publishing log messages to tcp://*:5563 2025-09-13 02:46:53,162 INFO [get_onc_ctd] connected to skookum.eos.ubc.ca port 5555 2025-09-13 02:46:53,163 INFO [get_onc_ctd] requesting ONC SCVIP CTD T&S data for 2025-09-12 2025-09-13 02:46:53,223 INFO [get_onc_ctd] running in process 2886539 2025-09-13 02:46:53,224 INFO [get_onc_ctd] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-09-13 02:46:53,225 INFO [get_onc_ctd] publishing log messages to tcp://*:5590 2025-09-13 02:46:53,226 INFO [get_onc_ctd] connected to skookum.eos.ubc.ca port 5555 2025-09-13 02:46:53,227 INFO [get_onc_ctd] requesting ONC SEVIP CTD T&S data for 2025-09-12 2025-09-13 02:46:53,238 INFO [get_onc_ferry] running in process 2886540 2025-09-13 02:46:53,239 INFO [get_onc_ferry] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-09-13 02:46:53,240 INFO [get_onc_ferry] publishing log messages to tcp://*:5591 2025-09-13 02:46:53,241 INFO [get_onc_ferry] connected to skookum.eos.ubc.ca port 5555 2025-09-13 02:46:53,242 INFO [get_onc_ferry] requesting ONC TWDP.N1 NAV data for 2025-09-12 2025-09-13 02:46:53,482 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2025-09-13 02:46:53,519 ERROR [get_onc_ferry] request for ONC TWDP.N1 NAV data for 2025-09-12 failed: 400 Client Error: for url: https://data.oceannetworks.ca/api/scalardata?method=getByLocation&token=8efe44eb-0fb5-4413-bba3-ae3dbe0fbb1d&locationCode=TWDP.N1&deviceCategoryCode=NAV&sensorCategoryCodes=longitude%2Clatitude&dateFrom=2025-09-12T00:00:00.000Z&dateTo=2025-09-12T23:59:00.000Z&resampleType=avg&resamplePeriod=1 2025-09-13 02:46:53,749 INFO [grib_to_netcdf] running in process 2886781 2025-09-13 02:46:53,750 INFO [grib_to_netcdf] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-09-13 02:46:53,751 INFO [grib_to_netcdf] publishing log messages to tcp://*:5565 2025-09-13 02:46:53,752 INFO [grib_to_netcdf] connected to skookum.eos.ubc.ca port 5555 2025-09-13 02:46:53,753 INFO [grib_to_netcdf] creating NEMO-atmos forcing files for 2025-09-13 forecast2 run 2025-09-13 02:46:53,818 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2025-09-13 02:46:54,073 CRITICAL [get_onc_ferry] 2025-09-12 ONC TWDP ferry data file creation failed 2025-09-13 02:46:54,316 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2025-09-13 02:46:54,594 INFO [manager] checklist updated with [Neah Bay ssh data] items from collect_NeahBay_ssh worker 2025-09-13 02:46:54,609 INFO [get_vfpa_hadcp] added VFPA HADCP data from 2nd Narrows Rail Bridge for 2025-09-12 to /opp/observations/AISDATA/netcdf/VFPA_2ND_NARROWS_HADCP_2s_202509.nc 2025-09-13 02:46:54,609 INFO [get_vfpa_hadcp] VFPA HADCP observations added to 2025-09 netcdf file 2025-09-13 02:46:54,906 INFO [manager] launching NextWorker(module='nowcast.workers.make_ssh_files', args=['forecast2', '--run-date', '2025-09-13'], host='localhost') 2025-09-13 02:46:54,913 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2025-09-13 02:46:55,318 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2025-09-13 02:46:55,650 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2025-09-13 02:46:56,027 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2025-09-13 02:46:56,430 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2025-09-13 02:46:56,803 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2025-09-13 02:46:57,220 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2025-09-13 02:46:57,559 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2025-09-13 02:46:58,072 INFO [manager] checklist updated with [VFPA HADCP data] items from get_vfpa_hadcp worker 2025-09-13 02:46:58,231 INFO [make_ssh_files] running in process 2886873 2025-09-13 02:46:58,232 INFO [make_ssh_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-09-13 02:46:58,233 INFO [make_ssh_files] publishing log messages to tcp://*:5562 2025-09-13 02:46:58,233 INFO [make_ssh_files] connected to skookum.eos.ubc.ca port 5555 2025-09-13 02:46:58,234 INFO [make_ssh_files] building forecast2 sea surface height boundary conditions file(s) from 2025-09-13 NOAA Neah Bay 00Z observation and forecast values 2025-09-13 02:46:58,615 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['VFPA-HADCP'], host='localhost') 2025-09-13 02:46:59,938 INFO [ping_erddap] running in process 2886899 2025-09-13 02:46:59,939 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-09-13 02:46:59,940 INFO [ping_erddap] publishing log messages to tcp://*:5566 2025-09-13 02:46:59,940 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555 2025-09-13 02:46:59,942 INFO [ping_erddap] VFPA-HADCP ERDDAP dataset flag file(s) created 2025-09-13 02:46:59,943 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker 2025-09-13 02:47:00,616 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2025m09d08.nc 2025-09-13 02:47:00,797 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2025m09d09.nc 2025-09-13 02:47:00,980 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2025m09d10.nc 2025-09-13 02:47:01,204 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2025m09d11.nc 2025-09-13 02:47:01,388 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2025m09d12.nc 2025-09-13 02:47:01,592 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2025m09d13.nc 2025-09-13 02:47:01,796 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2025m09d14.nc 2025-09-13 02:47:02,100 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2025m09d15.nc 2025-09-13 02:47:02,311 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2025m09d16.nc 2025-09-13 02:47:02,465 INFO [make_ssh_files] sea surface height boundary file for 2025-09-13 forecast2 run created 2025-09-13 02:47:02,468 INFO [manager] checklist updated with [sea surface height forcing] items from make_ssh_files worker 2025-09-13 02:47:02,842 INFO [manager] launching NextWorker(module='nowcast.workers.make_runoff_file', args=['v202108'], host='localhost') 2025-09-13 02:47:02,846 INFO [manager] launching NextWorker(module='nowcast.workers.make_201702_runoff_file', args=[], host='localhost') 2025-09-13 02:47:03,751 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2025m09d14.nc 2025-09-13 02:47:04,705 INFO [make_201702_runoff_file] running in process 2886927 2025-09-13 02:47:04,706 INFO [make_201702_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-09-13 02:47:04,707 INFO [make_201702_runoff_file] publishing log messages to tcp://*:5562 2025-09-13 02:47:04,708 INFO [make_201702_runoff_file] connected to skookum.eos.ubc.ca port 5555 2025-09-13 02:47:04,750 INFO [make_201702_runoff_file] calculating runoff forcing for b201702 bathymetry 2025-09-13 02:47:04,857 INFO [make_201702_runoff_file] 2025-09-13 runoff file creation from Fraser at Hope and climatology elsewhere complete 2025-09-13 02:47:04,859 INFO [manager] checklist updated with [rivers forcing] items from make_201702_runoff_file worker 2025-09-13 02:47:05,377 INFO [make_runoff_file] running in process 2886925 2025-09-13 02:47:05,378 INFO [make_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-09-13 02:47:05,378 INFO [make_runoff_file] publishing log messages to tcp://*:5566 2025-09-13 02:47:05,379 INFO [make_runoff_file] connected to skookum.eos.ubc.ca port 5555 2025-09-13 02:47:05,384 INFO [make_runoff_file] calculating NEMO runoff forcing for v202108 bathymetry for 2025-09-12 2025-09-13 02:47:06,675 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2025m09d15.nc 2025-09-13 02:47:06,683 INFO [grib_to_netcdf] 2025-09-13 NEMO-atmos forcing file for forecast2 created 2025-09-13 02:47:06,685 INFO [manager] checklist updated with [weather forcing] items from grib_to_netcdf worker 2025-09-13 02:47:07,366 ERROR [make_runoff_file] no 2025-09-12 discharge obs for SkagitMountVernon: patching 2025-09-13 02:47:07,830 ERROR [make_runoff_file] no 2025-09-12 discharge obs for SnohomishMonroe: patching 2025-09-13 02:47:08,062 ERROR [make_runoff_file] no 2025-09-12 discharge obs for NisquallyMcKenna: patching 2025-09-13 02:47:08,156 ERROR [make_runoff_file] no 2025-09-12 discharge obs for GreenwaterGreenwater: patching 2025-09-13 02:47:09,395 INFO [make_runoff_file] stored NEMO runoff forcing for v202108 bathymetry for 2025-09-12: /results/forcing/rivers/R202108Dailies_y2025m09d12.nc 2025-09-13 02:47:09,396 INFO [make_runoff_file] 2025-09-12 runoff file creation completed 2025-09-13 02:47:09,398 INFO [manager] checklist updated with [rivers forcing] items from make_runoff_file worker 2025-09-13 02:47:09,757 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'forecast2'], host='localhost') 2025-09-13 02:47:09,763 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'forecast2'], host='localhost') 2025-09-13 02:47:09,766 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.graham', 'forecast2'], host='localhost') 2025-09-13 02:47:09,772 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'forecast2'], host='localhost') 2025-09-13 02:47:11,109 INFO [upload_forcing] running in process 2886966 2025-09-13 02:47:11,111 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-09-13 02:47:11,111 INFO [upload_forcing] publishing log messages to tcp://*:5562 2025-09-13 02:47:11,113 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2025-09-13 02:47:11,116 INFO [upload_forcing] running in process 2886964 2025-09-13 02:47:11,117 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-09-13 02:47:11,118 INFO [upload_forcing] publishing log messages to tcp://*:5565 2025-09-13 02:47:11,119 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2025-09-13 02:47:11,164 INFO [upload_forcing] running in process 2886965 2025-09-13 02:47:11,166 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-09-13 02:47:11,166 INFO [upload_forcing] publishing log messages to tcp://*:5566 2025-09-13 02:47:11,167 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2025-09-13 02:47:11,243 INFO [upload_forcing] running in process 2886963 2025-09-13 02:47:11,244 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-09-13 02:47:11,245 INFO [upload_forcing] publishing log messages to tcp://*:5567 2025-09-13 02:47:11,246 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2025-09-13 02:47:11,768 INFO [get_onc_ctd] 2025-09-12 ONC SCVIP CTD T&S file created 2025-09-13 02:47:11,771 INFO [manager] checklist updated with [ONC CTD data] items from get_onc_ctd worker 2025-09-13 02:47:12,088 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['SCVIP-CTD'], host='localhost') 2025-09-13 02:47:12,092 INFO [get_onc_ctd] 2025-09-12 ONC SEVIP CTD T&S file created 2025-09-13 02:47:12,094 INFO [manager] checklist updated with [ONC CTD data] items from get_onc_ctd worker 2025-09-13 02:47:12,484 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['SEVIP-CTD'], host='localhost') 2025-09-13 02:47:12,747 INFO [upload_forcing] LiveOcean boundary conditions file not found; created symlink to /results/forcing/LiveOcean/boundary_conditions/LiveOcean_v201905_y2025m09d12.nc 2025-09-13 02:47:12,797 INFO [upload_forcing] forecast2 2025-09-13 forcing files upload to optimum-hindcast completed 2025-09-13 02:47:12,800 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2025-09-13 02:47:13,287 INFO [ping_erddap] running in process 2886989 2025-09-13 02:47:13,289 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-09-13 02:47:13,290 INFO [ping_erddap] publishing log messages to tcp://*:5563 2025-09-13 02:47:13,290 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555 2025-09-13 02:47:13,291 INFO [ping_erddap] SCVIP-CTD ERDDAP dataset flag file(s) created 2025-09-13 02:47:13,293 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker 2025-09-13 02:47:13,623 INFO [ping_erddap] running in process 2886991 2025-09-13 02:47:13,625 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-09-13 02:47:13,625 INFO [ping_erddap] publishing log messages to tcp://*:5562 2025-09-13 02:47:13,626 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555 2025-09-13 02:47:13,627 INFO [ping_erddap] SEVIP-CTD ERDDAP dataset flag file(s) created 2025-09-13 02:47:13,628 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker 2025-09-13 02:47:16,413 INFO [upload_forcing] forecast2 2025-09-13 forcing files upload to arbutus.cloud-nowcast completed 2025-09-13 02:47:16,418 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2025-09-13 02:47:16,785 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2025-09-13'], host='localhost') 2025-09-13 02:47:17,942 INFO [make_forcing_links] running in process 2887002 2025-09-13 02:47:17,944 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-09-13 02:47:17,945 INFO [make_forcing_links] publishing log messages to tcp://*:5562 2025-09-13 02:47:17,945 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555 2025-09-13 02:47:19,353 INFO [make_forcing_links] forecast2 2025-09-13 forcing file links on arbutus.cloud-nowcast created 2025-09-13 02:47:19,356 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker 2025-09-13 02:47:19,742 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2025-09-13'], host='arbutus.cloud-nowcast') 2025-09-13 02:47:21,167 INFO [upload_forcing] forecast2 2025-09-13 forcing files upload to robot.graham completed 2025-09-13 02:47:21,172 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2025-09-13 02:47:22,145 INFO [run_NEMO] running in process 17040 2025-09-13 02:47:22,147 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2025-09-13 02:47:22,148 INFO [run_NEMO] publishing log messages to tcp://*:5556 2025-09-13 02:47:22,239 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555 2025-09-13 02:47:23,651 WARNING [run_NEMO] There are uncommitted changes in /nemoShare/MEOPAR/nowcast-sys/SS-run-sets 2025-09-13 02:47:23,988 INFO [run_NEMO] forecast2: launching /nemoShare/MEOPAR/nowcast-sys/runs/12sep25forecast2_2025-09-13T024722.310458-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast 2025-09-13 02:47:24,000 INFO [run_NEMO] forecast2 NEMO run for 2025-09-13 on arbutus.cloud-nowcast started 2025-09-13 02:47:24,004 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker 2025-09-13 02:47:24,332 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'forecast2'], host='arbutus.cloud-nowcast') 2025-09-13 02:47:26,365 INFO [watch_NEMO] running in process 17315 2025-09-13 02:47:26,367 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2025-09-13 02:47:26,368 INFO [watch_NEMO] publishing log messages to tcp://*:5557 2025-09-13 02:47:26,368 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555 2025-09-13 02:47:26,440 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: time.step not found; continuing to watch... 2025-09-13 02:49:21,751 CRITICAL [upload_forcing] unhandled exception: Traceback (most recent call last): File "/SalishSeaCast/NEMO_Nowcast/nemo_nowcast/worker.py", line 387, in _do_work checklist = self.worker_func( self._parsed_args, self.config, self.tell_manager ) File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/upload_forcing.py", line 94, in upload_forcing ssh_client, sftp_client = ssh_sftp.sftp(host_name, ssh_key) ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/SalishSeaNowcast/nowcast/ssh_sftp.py", line 144, in sftp ssh_client = ssh(host, key_filename, ssh_config_file) File "/SalishSeaCast/SalishSeaNowcast/nowcast/ssh_sftp.py", line 75, in ssh ssh_client.connect( ~~~~~~~~~~~~~~~~~~^ host["hostname"], ^^^^^^^^^^^^^^^^^ ...<3 lines>... look_for_keys=False, ^^^^^^^^^^^^^^^^^^^^ ) ^ File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/paramiko/client.py", line 386, in connect sock.connect(addr) ~~~~~~~~~~~~^^^^^^ TimeoutError: [Errno 110] Connection timed out 2025-09-13 02:52:26,442 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 18706064 = 2025-09-14 05:08:40 UTC, 17.2% complete 2025-09-13 02:57:26,443 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 18706604 = 2025-09-14 11:08:40 UTC, 37.2% complete 2025-09-13 03:02:26,442 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 18707098 = 2025-09-14 16:38:00 UTC, 55.5% complete 2025-09-13 03:07:26,442 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 18707608 = 2025-09-14 22:18:00 UTC, 74.4% complete 2025-09-13 03:12:26,443 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 18708130 = 2025-09-15 04:06:00 UTC, 93.7% complete 2025-09-13 03:17:26,448 INFO [watch_NEMO] forecast2 NEMO run on arbutus.cloud-nowcast completed 2025-09-13 03:17:26,452 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker 2025-09-13 03:17:26,955 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_wind_file', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2025-09-13'], host='arbutus.cloud-nowcast') 2025-09-13 03:17:26,962 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_current_file', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2025-09-13'], host='arbutus.cloud-nowcast') 2025-09-13 03:17:26,966 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2025-09-12'], host='localhost') 2025-09-13 03:17:28,111 INFO [download_results] running in process 2887411 2025-09-13 03:17:28,113 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-09-13 03:17:28,114 INFO [download_results] publishing log messages to tcp://*:5562 2025-09-13 03:17:28,115 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2025-09-13 03:17:28,116 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/forecast2/12sep25 to /results/SalishSea/forecast2.202111 2025-09-13 03:17:29,913 INFO [make_ww3_current_file] running in process 17714 2025-09-13 03:17:29,914 INFO [make_ww3_current_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2025-09-13 03:17:29,916 INFO [make_ww3_current_file] publishing log messages to tcp://*:5571 2025-09-13 03:17:29,917 INFO [make_ww3_current_file] connected to skookum.eos.ubc.ca port 5555 2025-09-13 03:17:29,918 INFO [make_ww3_current_file] Creating wwatch3 currents forcing file for 2025-09-13 forecast2 run 2025-09-13 03:17:29,920 INFO [make_ww3_wind_file] running in process 17716 2025-09-13 03:17:29,921 INFO [make_ww3_wind_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2025-09-13 03:17:29,922 INFO [make_ww3_wind_file] publishing log messages to tcp://*:5570 2025-09-13 03:17:29,924 INFO [make_ww3_wind_file] connected to skookum.eos.ubc.ca port 5555 2025-09-13 03:17:29,925 INFO [make_ww3_wind_file] Creating wwatch3 wind forcing file for 2025-09-13 forecast2 run 2025-09-13 03:17:38,735 INFO [make_ww3_wind_file] wwatch3 wind forcing file created on arbutus.cloud-nowcast for 2025-09-13 forecast2 run 2025-09-13 03:17:38,739 INFO [manager] checklist updated with [WW3 wind forcing] items from make_ww3_wind_file worker 2025-09-13 03:19:28,708 INFO [download_results] forecast2 2025-09-12 results files from arbutus.cloud-nowcast downloaded 2025-09-13 03:19:28,712 INFO [manager] checklist updated with [results files] items from download_results worker 2025-09-13 03:19:29,072 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['forecast2', '--run-date', '2025-09-12'], host='localhost') 2025-09-13 03:19:30,877 INFO [make_CHS_currents_file] running in process 2887428 2025-09-13 03:19:30,878 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-09-13 03:19:30,880 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5562 2025-09-13 03:19:30,881 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555 2025-09-13 03:20:09,654 INFO [make_CHS_currents_file] Made CHS currents file for 2025-09-12 for forecast2 2025-09-13 03:20:09,657 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker 2025-09-13 03:20:10,031 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['nemo', 'forecast2', '--run-date', '2025-09-12'], host='localhost') 2025-09-13 03:20:11,064 INFO [update_forecast_datasets] running in process 2887451 2025-09-13 03:20:11,066 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-09-13 03:20:11,067 INFO [update_forecast_datasets] publishing log messages to tcp://*:5562 2025-09-13 03:20:11,068 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555 2025-09-13 03:20:11,069 INFO [update_forecast_datasets] updating nemo forecast directory for forecast2/12sep25 run 2025-09-13 03:21:05,537 INFO [make_ww3_current_file] wwatch3 currents forcing file created on arbutus.cloud-nowcast for 2025-09-13 forecast2 run 2025-09-13 03:21:05,545 INFO [manager] checklist updated with [WW3 currents forcing] items from make_ww3_current_file worker 2025-09-13 03:21:05,875 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2025-09-13'], host='arbutus.cloud-nowcast') 2025-09-13 03:21:07,803 INFO [run_ww3] running in process 18283 2025-09-13 03:21:07,804 INFO [run_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2025-09-13 03:21:07,805 INFO [run_ww3] publishing log messages to tcp://*:5572 2025-09-13 03:21:07,806 INFO [run_ww3] connected to skookum.eos.ubc.ca port 5555 2025-09-13 03:21:07,808 INFO [run_ww3] Created run directory /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast2_2025-09-13T032107.800660-0700 2025-09-13 03:21:07,809 INFO [run_ww3] forecast2: launching /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast2_2025-09-13T032107.800660-0700/SoGWW3.sh on arbutus.cloud-nowcast 2025-09-13 03:21:07,821 INFO [run_ww3] forecast2 WaveWatch3 run for 2025-09-13 on arbutus.cloud-nowcast started 2025-09-13 03:21:07,823 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker 2025-09-13 03:21:08,173 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'forecast2'], host='arbutus.cloud-nowcast') 2025-09-13 03:21:10,081 INFO [watch_ww3] running in process 18370 2025-09-13 03:21:10,081 INFO [watch_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2025-09-13 03:21:10,082 INFO [watch_ww3] publishing log messages to tcp://*:5573 2025-09-13 03:21:10,083 INFO [watch_ww3] connected to skookum.eos.ubc.ca port 5555 2025-09-13 03:21:10,107 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch... 2025-09-13 03:24:23,712 INFO [update_forecast_datasets] updated nemo forecast directory for forecast2/12sep25 run: /results/SalishSea/rolling-forecasts/nemo 2025-09-13 03:24:23,713 INFO [update_forecast_datasets] nemo 2025-09-12 forecast2 rolling forecast datasets updated 2025-09-13 03:24:23,716 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker 2025-09-13 03:24:24,074 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nemo-forecast'], host='localhost') 2025-09-13 03:24:24,078 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'forecast2', 'publish', '--run-date', '2025-09-12'], host='localhost') 2025-09-13 03:24:24,083 INFO [manager] launching NextWorker(module='nowcast.workers.make_surface_current_tiles', args=['forecast2', '--run-date', '2025-09-12'], host='localhost') 2025-09-13 03:24:25,421 INFO [ping_erddap] running in process 2887549 2025-09-13 03:24:25,423 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-09-13 03:24:25,424 INFO [ping_erddap] publishing log messages to tcp://*:5562 2025-09-13 03:24:25,424 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555 2025-09-13 03:24:25,432 INFO [ping_erddap] nemo-forecast ERDDAP dataset flag file(s) created 2025-09-13 03:24:25,434 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker 2025-09-13 03:24:26,658 INFO [make_surface_current_tiles] running in process 2887552 2025-09-13 03:24:26,659 INFO [make_surface_current_tiles] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-09-13 03:24:26,661 INFO [make_surface_current_tiles] publishing log messages to tcp://*:5562 2025-09-13 03:24:26,662 INFO [make_surface_current_tiles] connected to skookum.eos.ubc.ca port 5555 2025-09-13 03:24:26,663 INFO [make_surface_current_tiles] starting rendering of tiles for 2025-09-12 forecast2 in 6 processes into /results/nowcast-sys/figures/surface_currents/forecast2/12sep25 2025-09-13 03:24:27,166 INFO [make_plots] running in process 2887551 2025-09-13 03:24:27,167 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-09-13 03:24:27,168 INFO [make_plots] publishing log messages to tcp://*:5563 2025-09-13 03:24:27,169 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555 2025-09-13 03:24:27,628 INFO [make_plots] preparing render list for 2025-09-12 NEMO forecast2 publish figures 2025-09-13 03:24:27,761 INFO [make_plots] added Website_thumbnail figure to 2025-09-12 NEMO forecast2 publish render list 2025-09-13 03:24:27,762 INFO [make_plots] added Threshold_website figure to 2025-09-12 NEMO forecast2 publish render list 2025-09-13 03:24:27,763 INFO [make_plots] added PA_tidal_predictions figure to 2025-09-12 NEMO forecast2 publish render list 2025-09-13 03:24:27,764 INFO [make_plots] added Boundary Bay figure to 2025-09-12 NEMO forecast2 publish render list 2025-09-13 03:24:27,765 INFO [make_plots] added Campbell River figure to 2025-09-12 NEMO forecast2 publish render list 2025-09-13 03:24:27,766 INFO [make_plots] added Cherry Point figure to 2025-09-12 NEMO forecast2 publish render list 2025-09-13 03:24:27,767 INFO [make_plots] added Friday Harbor figure to 2025-09-12 NEMO forecast2 publish render list 2025-09-13 03:24:27,768 INFO [make_plots] added Halfmoon Bay figure to 2025-09-12 NEMO forecast2 publish render list 2025-09-13 03:24:27,769 INFO [make_plots] added Nanaimo figure to 2025-09-12 NEMO forecast2 publish render list 2025-09-13 03:24:27,770 INFO [make_plots] added Neah Bay figure to 2025-09-12 NEMO forecast2 publish render list 2025-09-13 03:24:27,771 INFO [make_plots] added New Westminster figure to 2025-09-12 NEMO forecast2 publish render list 2025-09-13 03:24:27,772 INFO [make_plots] added Patricia Bay figure to 2025-09-12 NEMO forecast2 publish render list 2025-09-13 03:24:27,772 INFO [make_plots] added Point Atkinson figure to 2025-09-12 NEMO forecast2 publish render list 2025-09-13 03:24:27,773 INFO [make_plots] added Port Renfrew figure to 2025-09-12 NEMO forecast2 publish render list 2025-09-13 03:24:27,774 INFO [make_plots] added Sand Heads figure to 2025-09-12 NEMO forecast2 publish render list 2025-09-13 03:24:27,774 INFO [make_plots] added Sandy Cove figure to 2025-09-12 NEMO forecast2 publish render list 2025-09-13 03:24:27,775 INFO [make_plots] added Squamish figure to 2025-09-12 NEMO forecast2 publish render list 2025-09-13 03:24:27,776 INFO [make_plots] added Victoria figure to 2025-09-12 NEMO forecast2 publish render list 2025-09-13 03:24:27,776 INFO [make_plots] added Woodwards Landing figure to 2025-09-12 NEMO forecast2 publish render list 2025-09-13 03:24:27,777 INFO [make_plots] starting to render nemo forecast2 publish 12sep25 figures 2025-09-13 03:26:10,100 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 75, 13.9% complete 2025-09-13 03:31:10,101 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 156, 28.9% complete 2025-09-13 03:32:40,044 WARNING [make_plots] invalid station number for Boundary Bay station: None 2025-09-13 03:32:40,045 WARNING [make_plots] can't resolve a valid CHS station code for Boundary Bay 2025-09-13 03:33:03,443 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb996/data for station 08074 Campbell River from 2025-09-08 00:00:00Z to 2025-09-15 05:55:00Z 2025-09-13 03:34:15,839 WARNING [make_plots] invalid station number for Halfmoon Bay station: None 2025-09-13 03:34:15,840 WARNING [make_plots] can't resolve a valid CHS station code for Halfmoon Bay 2025-09-13 03:34:37,521 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb96d/data for station 07917 Nanaimo from 2025-09-08 00:00:00Z to 2025-09-15 05:55:00Z 2025-09-13 03:35:18,110 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e13d0f4a073c4bbf93/data for station 07654 New Westminster from 2025-09-08 00:00:00Z to 2025-09-15 05:55:00Z 2025-09-13 03:35:37,480 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd2d/data for station 07277 Patricia Bay from 2025-09-08 00:00:00Z to 2025-09-15 05:55:00Z 2025-09-13 03:35:50,376 INFO [make_surface_current_tiles] starting PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast2/12sep25 2025-09-13 03:35:55,129 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94c/data for station 07795 Point Atkinson from 2025-09-08 00:00:00Z to 2025-09-15 05:55:00Z 2025-09-13 03:36:10,103 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 246, 45.6% complete 2025-09-13 03:36:12,083 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e23d0f4a073c4bc060/data for station 08525 Port Renfrew from 2025-09-08 00:00:00Z to 2025-09-15 05:55:00Z 2025-09-13 03:36:28,703 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb93c/data for station 07594 Sand Heads from 2025-09-08 00:00:00Z to 2025-09-15 05:55:00Z 2025-09-13 03:36:45,498 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc43b/data for station 07786 Sandy Cove from 2025-09-08 00:00:00Z to 2025-09-15 05:55:00Z 2025-09-13 03:37:03,187 WARNING [make_plots] invalid station number for Squamish station: None 2025-09-13 03:37:03,188 WARNING [make_plots] can't resolve a valid CHS station code for Squamish 2025-09-13 03:37:19,265 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd1e/data for station 07120 Victoria from 2025-09-08 00:00:00Z to 2025-09-15 05:55:00Z 2025-09-13 03:37:36,441 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc42d/data for station 07610 Woodwards Landing from 2025-09-08 00:00:00Z to 2025-09-15 05:55:00Z 2025-09-13 03:37:52,802 INFO [make_plots] finished rendering nemo forecast2 publish 12sep25 figures 2025-09-13 03:37:52,803 INFO [make_plots] nemo publish plots for 2025-09-12 forecast2 completed 2025-09-13 03:37:52,810 INFO [manager] checklist updated with [plots] items from make_plots worker 2025-09-13 03:37:53,172 INFO [manager] launching NextWorker(module='nowcast.workers.make_feeds', args=['forecast2', '--run-date', '2025-09-12'], host='localhost') 2025-09-13 03:37:56,038 INFO [make_feeds] running in process 2889596 2025-09-13 03:37:56,042 INFO [make_feeds] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-09-13 03:37:56,043 INFO [make_feeds] publishing log messages to tcp://*:5563 2025-09-13 03:37:56,044 INFO [make_feeds] connected to skookum.eos.ubc.ca port 5555 2025-09-13 03:38:10,472 INFO [make_feeds] ATOM feeds for 2025-09-12 forecast2 run completed 2025-09-13 03:38:10,475 INFO [manager] checklist updated with [storm surge alert feeds] items from make_feeds worker 2025-09-13 03:38:10,846 INFO [manager] launching NextWorker(module='nemo_nowcast.workers.clear_checklist', args=[], host='localhost') 2025-09-13 03:38:11,971 INFO [clear_checklist] running in process 2889614 2025-09-13 03:38:11,972 INFO [clear_checklist] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-09-13 03:38:11,973 INFO [clear_checklist] publishing log messages to tcp://*:5563 2025-09-13 03:38:11,975 INFO [clear_checklist] connected to skookum.eos.ubc.ca port 5555 2025-09-13 03:38:11,976 INFO [clear_checklist] requesting that manager clear system state checklist 2025-09-13 03:38:11,978 INFO [manager] writing checklist to log file 2025-09-13 03:38:12,026 INFO [manager] checklist cleared 2025-09-13 03:38:12,029 INFO [clear_checklist] nowcast system checklist cleared 2025-09-13 03:38:12,032 INFO [manager] launching NextWorker(module='nemo_nowcast.workers.rotate_logs', args=[], host='localhost') 2025-09-13 03:38:13,171 INFO [rotate_logs] running in process 2889619 2025-09-13 03:38:13,173 INFO [rotate_logs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-09-13 03:38:13,174 INFO [rotate_logs] publishing log messages to tcp://*:5563 2025-09-13 03:38:13,175 INFO [rotate_logs] connected to skookum.eos.ubc.ca port 5555 2025-09-13 03:38:13,175 INFO [rotate_logs] rotating log files