2025-11-13 02:50:22,595 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/nowcast.log
2025-11-13 02:50:22,602 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/nowcast.debug.log
2025-11-13 02:50:22,606 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/checklist.log
2025-11-13 02:50:22,607 INFO [rotate_logs] log files rotated
2025-11-13 02:50:22,610 INFO [manager] checklist updated with [log rotation] items from rotate_logs worker
2025-11-13 02:51:36,127 INFO [make_surface_current_tiles] finished PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast2/12nov25
2025-11-13 02:51:36,150 INFO [make_surface_current_tiles] finished rendering of tiles for 2025-11-12 forecast2 into /results/nowcast-sys/figures/surface_currents/forecast2/12nov25
2025-11-13 02:51:36,151 INFO [make_surface_current_tiles] surface current tile figures for 2025-11-12 forecast2 completed
2025-11-13 02:51:36,415 INFO [manager] checklist updated with [surface current tiles] items from make_surface_current_tiles worker
2025-11-13 02:54:23,355 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 306, 56.7% complete
2025-11-13 02:59:23,356 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 381, 70.6% complete
2025-11-13 03:04:23,354 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 459, 85.0% complete
2025-11-13 03:09:23,346 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 534, 98.9% complete
2025-11-13 03:14:23,348 INFO [watch_ww3] forecast2 WWATCH3 run on arbutus.cloud-nowcast completed
2025-11-13 03:14:23,371 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2025-11-13 03:14:23,697 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2025-11-13'], host='localhost')
2025-11-13 03:14:24,789 INFO [download_wwatch3_results] running in process 362798
2025-11-13 03:14:24,790 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-11-13 03:14:24,791 INFO [download_wwatch3_results] publishing log messages to tcp://*:5563
2025-11-13 03:14:24,792 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2025-11-13 03:14:59,540 INFO [download_wwatch3_results] forecast2 2025-11-13 results files from arbutus.cloud-nowcast downloaded
2025-11-13 03:14:59,545 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2025-11-13 03:14:59,694 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['wwatch3', 'forecast2', '--run-date', '2025-11-13'], host='localhost')
2025-11-13 03:15:00,897 INFO [update_forecast_datasets] running in process 362813
2025-11-13 03:15:00,898 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-11-13 03:15:00,900 INFO [update_forecast_datasets] publishing log messages to tcp://*:5563
2025-11-13 03:15:00,901 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2025-11-13 03:15:00,901 INFO [update_forecast_datasets] updating wwatch3 most_recent_forecast directory from forecast2/13nov25 run
2025-11-13 03:15:00,905 INFO [update_forecast_datasets] updated wwatch3 most_recent_forecast directory from forecast2/13nov25 run
2025-11-13 03:15:00,907 INFO [update_forecast_datasets] updating wwatch3 forecast directory for forecast2/13nov25 run
2025-11-13 03:15:24,694 INFO [update_forecast_datasets] updated wwatch3 forecast directory for forecast2/13nov25 run: /results/SalishSea/rolling-forecasts/wwatch3
2025-11-13 03:15:24,695 INFO [update_forecast_datasets] wwatch3 2025-11-13 forecast2 rolling forecast datasets updated
2025-11-13 03:15:24,698 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2025-11-13 03:15:24,850 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['wwatch3-forecast'], host='localhost')
2025-11-13 03:15:25,879 INFO [ping_erddap] running in process 362830
2025-11-13 03:15:25,880 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-11-13 03:15:25,882 INFO [ping_erddap] publishing log messages to tcp://*:5563
2025-11-13 03:15:25,883 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2025-11-13 03:15:25,884 INFO [ping_erddap] wwatch3-forecast ERDDAP dataset flag file(s) created
2025-11-13 03:15:25,887 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2025-11-13 03:15:26,037 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['wwatch3', 'forecast2', 'publish', '--run-date', '2025-11-13'], host='localhost')
2025-11-13 03:15:29,421 INFO [make_plots] running in process 362835
2025-11-13 03:15:29,422 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-11-13 03:15:29,423 INFO [make_plots] publishing log messages to tcp://*:5563
2025-11-13 03:15:29,424 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2025-11-13 03:15:29,425 INFO [make_plots] preparing render list for 2025-11-13 WaveWatch3 forecast2 publish figures
2025-11-13 03:15:29,425 INFO [make_plots] added Halibut Bank figure to 2025-11-13 WaveWatch3 forecast2 publish render list
2025-11-13 03:15:29,426 INFO [make_plots] added Sentry Shoal figure to 2025-11-13 WaveWatch3 forecast2 publish render list
2025-11-13 03:15:29,427 INFO [make_plots] starting to render wwatch3 forecast2 publish 13nov25 figures
2025-11-13 03:15:30,016 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46146 Halibut Bank
2025-11-13 03:15:35,379 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46131 Sentry Shoal
2025-11-13 03:15:40,002 INFO [make_plots] finished rendering wwatch3 forecast2 publish 13nov25 figures
2025-11-13 03:15:40,003 INFO [make_plots] wwatch3 publish plots for 2025-11-13 forecast2 completed
2025-11-13 03:15:40,006 INFO [manager] checklist updated with [plots] items from make_plots worker
2025-11-13 07:56:05,058 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/12/ to /results/forcing/atmospheric/continental2.5/GRIB/20251113/12/
2025-11-13 07:56:05,059 INFO [collect_weather] 2025-11-13 2.5km weather forecast 12 collection complete
2025-11-13 07:56:05,063 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2025-11-13 07:56:05,440 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'SkagitMountVernon', '--data-date', '2025-11-12'], host='localhost')
2025-11-13 07:56:05,447 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'SnohomishMonroe', '--data-date', '2025-11-12'], host='localhost')
2025-11-13 07:56:05,452 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'NisquallyMcKenna', '--data-date', '2025-11-12'], host='localhost')
2025-11-13 07:56:05,457 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'GreenwaterGreenwater', '--data-date', '2025-11-12'], host='localhost')
2025-11-13 07:56:05,462 INFO [manager] launching NextWorker(module='nowcast.workers.make_turbidity_file', args=[], host='localhost')
2025-11-13 07:56:05,469 INFO [manager] launching NextWorker(module='nowcast.workers.collect_NeahBay_ssh', args=['06'], host='localhost')
2025-11-13 07:56:05,476 INFO [manager] launching NextWorker(module='nowcast.workers.download_live_ocean', args=[], host='localhost')
2025-11-13 07:56:05,481 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['18', '2.5km'], host='localhost')
2025-11-13 07:56:05,502 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['18'], host='localhost')
2025-11-13 07:56:06,382 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20251113/12/
2025-11-13 07:56:06,385 INFO [crop_gribs] 2025-11-13 12 GRIBs cropping complete
2025-11-13 07:56:06,386 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2025-11-13 07:56:06,832 INFO [manager] launching NextWorker(module='nowcast.workers.grib_to_netcdf', args=['nowcast+'], host='localhost')
2025-11-13 07:56:07,143 INFO [collect_NeahBay_ssh] running in process 367211
2025-11-13 07:56:07,144 INFO [collect_NeahBay_ssh] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-11-13 07:56:07,144 INFO [collect_NeahBay_ssh] publishing log messages to tcp://*:5562
2025-11-13 07:56:07,145 INFO [collect_NeahBay_ssh] connected to skookum.eos.ubc.ca port 5555
2025-11-13 07:56:07,146 INFO [collect_NeahBay_ssh] collecting Neah Bay ssh 06Z obs/forecast for 2025-11-13
2025-11-13 07:56:07,148 INFO [collect_weather] running in process 367213
2025-11-13 07:56:07,148 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-11-13 07:56:07,151 INFO [collect_weather] publishing log messages to tcp://*:5563
2025-11-13 07:56:07,152 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2025-11-13 07:56:07,164 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/18/
2025-11-13 07:56:07,207 INFO [download_live_ocean] running in process 367212
2025-11-13 07:56:07,208 INFO [download_live_ocean] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-11-13 07:56:07,209 INFO [download_live_ocean] publishing log messages to tcp://*:5564
2025-11-13 07:56:07,209 INFO [download_live_ocean] connected to skookum.eos.ubc.ca port 5555
2025-11-13 07:56:07,210 INFO [download_live_ocean] downloading Salish Sea western boundary day-averaged LiveOcean file for 2025-11-13
2025-11-13 07:56:07,714 INFO [collect_river_data] running in process 367208
2025-11-13 07:56:07,714 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-11-13 07:56:07,715 INFO [collect_river_data] publishing log messages to tcp://*:5565
2025-11-13 07:56:07,716 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-11-13 07:56:07,717 INFO [collect_river_data] Collecting USGS NisquallyMcKenna river data for 2025-11-12
2025-11-13 07:56:07,741 INFO [collect_river_data] running in process 367207
2025-11-13 07:56:07,742 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-11-13 07:56:07,744 INFO [collect_river_data] publishing log messages to tcp://*:5566
2025-11-13 07:56:07,745 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-11-13 07:56:07,746 INFO [collect_river_data] Collecting USGS SnohomishMonroe river data for 2025-11-12
2025-11-13 07:56:07,749 INFO [collect_river_data] running in process 367205
2025-11-13 07:56:07,750 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-11-13 07:56:07,751 INFO [collect_river_data] publishing log messages to tcp://*:5567
2025-11-13 07:56:07,752 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-11-13 07:56:07,753 INFO [collect_river_data] Collecting USGS SkagitMountVernon river data for 2025-11-12
2025-11-13 07:56:07,754 INFO [collect_river_data] running in process 367209
2025-11-13 07:56:07,755 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-11-13 07:56:07,755 INFO [collect_river_data] publishing log messages to tcp://*:5568
2025-11-13 07:56:07,756 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-11-13 07:56:07,758 INFO [collect_river_data] Collecting USGS GreenwaterGreenwater river data for 2025-11-12
2025-11-13 07:56:07,849 INFO [download_live_ocean] downloaded 10210445 bytes from https://s3.kopah.uw.edu/liveocean-share/f2025.11.13/ubc.nc to /results/forcing/LiveOcean/downloaded/20251113/low_passed_UBC.nc
2025-11-13 07:56:07,850 INFO [download_live_ocean] Deflating in up to 1 concurrent sub-processes
2025-11-13 07:56:07,911 INFO [make_turbidity_file] running in process 367210
2025-11-13 07:56:07,912 INFO [make_turbidity_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-11-13 07:56:07,913 INFO [make_turbidity_file] publishing log messages to tcp://*:5569
2025-11-13 07:56:07,914 INFO [make_turbidity_file] connected to skookum.eos.ubc.ca port 5555
2025-11-13 07:56:07,914 INFO [make_turbidity_file] Creating Fraser River turbidity forcing file for 2025-11-13
2025-11-13 07:56:08,261 INFO [crop_gribs] running in process 367214
2025-11-13 07:56:08,261 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-11-13 07:56:08,262 INFO [crop_gribs] publishing log messages to tcp://*:5570
2025-11-13 07:56:08,263 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2025-11-13 07:56:08,263 INFO [crop_gribs] cropping 2025-11-13 ECCC HRDPS 2.5km continental 18Z GRIB files to SalishSeaCast subdomain
2025-11-13 07:56:08,270 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20251113/18/
2025-11-13 07:56:08,354 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12097500&startDT=2025-11-12&endDT=2025-11-12 "HTTP/1.1 200 "
2025-11-13 07:56:08,355 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12089500&startDT=2025-11-12&endDT=2025-11-12 "HTTP/1.1 200 "
2025-11-13 07:56:08,363 INFO [collect_river_data] Appended USGS GreenwaterGreenwater river average discharge for 2025-11-12 to: /results/forcing/rivers/observations/Greenwater_Greenwater_flow
2025-11-13 07:56:08,364 INFO [collect_river_data] Appended USGS NisquallyMcKenna river average discharge for 2025-11-12 to: /results/forcing/rivers/observations/Nisqually_McKenna_flow
2025-11-13 07:56:08,365 INFO [collect_river_data] USGS GreenwaterGreenwater river data collection for 2025-11-12 completed
2025-11-13 07:56:08,365 INFO [collect_river_data] USGS NisquallyMcKenna river data collection for 2025-11-12 completed
2025-11-13 07:56:08,368 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12200500&startDT=2025-11-12&endDT=2025-11-12 "HTTP/1.1 200 "
2025-11-13 07:56:08,369 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12150800&startDT=2025-11-12&endDT=2025-11-12 "HTTP/1.1 200 "
2025-11-13 07:56:08,370 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-11-13 07:56:08,380 INFO [collect_river_data] Appended USGS SkagitMountVernon river average discharge for 2025-11-12 to: /results/forcing/rivers/observations/Skagit_MountVernon_flow
2025-11-13 07:56:08,381 INFO [collect_river_data] Appended USGS SnohomishMonroe river average discharge for 2025-11-12 to: /results/forcing/rivers/observations/Snohomish_Monroe_flow
2025-11-13 07:56:08,382 INFO [collect_river_data] USGS SkagitMountVernon river data collection for 2025-11-12 completed
2025-11-13 07:56:08,382 INFO [collect_river_data] USGS SnohomishMonroe river data collection for 2025-11-12 completed
2025-11-13 07:56:08,514 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-11-13 07:56:08,665 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-11-13 07:56:08,810 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-11-13 07:56:08,847 INFO [grib_to_netcdf] running in process 367295
2025-11-13 07:56:08,849 INFO [grib_to_netcdf] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-11-13 07:56:08,850 INFO [grib_to_netcdf] publishing log messages to tcp://*:5565
2025-11-13 07:56:08,851 INFO [grib_to_netcdf] connected to skookum.eos.ubc.ca port 5555
2025-11-13 07:56:08,852 INFO [grib_to_netcdf] creating NEMO-atmos forcing files for 2025-11-13 nowcast and forecast runs
2025-11-13 07:56:08,861 INFO [download_live_ocean] netCDF4 deflated /results/forcing/LiveOcean/downloaded/20251113/low_passed_UBC.nc
2025-11-13 07:56:08,862 INFO [download_live_ocean] 2025-11-13 Live Ocean file for Salish Sea western boundary downloaded
2025-11-13 07:56:08,950 INFO [manager] checklist updated with [Live Ocean products] items from download_live_ocean worker
2025-11-13 07:56:09,278 INFO [collect_NeahBay_ssh] 2025-11-13T00:00:00-08:00 Neah Bay ssh 06Z obs/forecast data collection complete
2025-11-13 07:56:09,282 INFO [manager] launching NextWorker(module='nowcast.workers.make_live_ocean_files', args=['--run-date', '2025-11-13'], host='localhost')
2025-11-13 07:56:09,289 INFO [manager] checklist updated with [Neah Bay ssh data] items from collect_NeahBay_ssh worker
2025-11-13 07:56:09,462 INFO [manager] launching NextWorker(module='nowcast.workers.make_ssh_files', args=['nowcast', '--run-date', '2025-11-13'], host='localhost')
2025-11-13 07:56:10,433 INFO [make_turbidity_file] 2025-11-13 Fraser River turbidity file creation complete
2025-11-13 07:56:10,435 INFO [manager] checklist updated with [Fraser River turbidity] items from make_turbidity_file worker
2025-11-13 07:56:11,432 INFO [make_live_ocean_files] running in process 367355
2025-11-13 07:56:11,434 INFO [make_live_ocean_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-11-13 07:56:11,435 INFO [make_live_ocean_files] publishing log messages to tcp://*:5562
2025-11-13 07:56:11,436 INFO [make_live_ocean_files] connected to skookum.eos.ubc.ca port 5555
2025-11-13 07:56:11,437 INFO [make_live_ocean_files] Creating T&S western boundary conditions file from 2025-11-13 Live Ocean run
2025-11-13 07:56:12,131 INFO [make_ssh_files] running in process 367357
2025-11-13 07:56:12,132 INFO [make_ssh_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-11-13 07:56:12,134 INFO [make_ssh_files] publishing log messages to tcp://*:5564
2025-11-13 07:56:12,134 INFO [make_ssh_files] connected to skookum.eos.ubc.ca port 5555
2025-11-13 07:56:12,135 INFO [make_ssh_files] building nowcast sea surface height boundary conditions file(s) from 2025-11-13 NOAA Neah Bay 06Z observation and forecast values
2025-11-13 07:56:13,789 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2025m11d09.nc
2025-11-13 07:56:13,966 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2025m11d10.nc
2025-11-13 07:56:14,150 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2025m11d11.nc
2025-11-13 07:56:14,334 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2025m11d12.nc
2025-11-13 07:56:14,536 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2025m11d13.nc
2025-11-13 07:56:14,716 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2025m11d14.nc
2025-11-13 07:56:14,922 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2025m11d15.nc
2025-11-13 07:56:15,116 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2025m11d16.nc
2025-11-13 07:56:15,236 INFO [make_ssh_files] sea surface height boundary file for 2025-11-13 nowcast run created
2025-11-13 07:56:15,239 INFO [manager] checklist updated with [sea surface height forcing] items from make_ssh_files worker
2025-11-13 07:56:15,390 INFO [manager] launching NextWorker(module='nowcast.workers.make_runoff_file', args=['v202108'], host='localhost')
2025-11-13 07:56:15,395 INFO [manager] launching NextWorker(module='nowcast.workers.make_201702_runoff_file', args=[], host='localhost')
2025-11-13 07:56:16,570 INFO [make_201702_runoff_file] running in process 367434
2025-11-13 07:56:16,572 INFO [make_201702_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-11-13 07:56:16,573 INFO [make_201702_runoff_file] publishing log messages to tcp://*:5564
2025-11-13 07:56:16,574 INFO [make_201702_runoff_file] connected to skookum.eos.ubc.ca port 5555
2025-11-13 07:56:16,611 INFO [make_201702_runoff_file] calculating runoff forcing for b201702 bathymetry
2025-11-13 07:56:16,704 INFO [make_201702_runoff_file] 2025-11-13 runoff file creation from Fraser at Hope and climatology elsewhere complete
2025-11-13 07:56:16,707 INFO [manager] checklist updated with [rivers forcing] items from make_201702_runoff_file worker
2025-11-13 07:56:17,224 INFO [make_runoff_file] running in process 367432
2025-11-13 07:56:17,226 INFO [make_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-11-13 07:56:17,227 INFO [make_runoff_file] publishing log messages to tcp://*:5564
2025-11-13 07:56:17,229 INFO [make_runoff_file] connected to skookum.eos.ubc.ca port 5555
2025-11-13 07:56:17,230 INFO [make_runoff_file] calculating NEMO runoff forcing for v202108 bathymetry for 2025-11-12
2025-11-13 07:56:19,703 INFO [make_runoff_file] stored NEMO runoff forcing for v202108 bathymetry for 2025-11-12: /results/forcing/rivers/R202108Dailies_y2025m11d12.nc
2025-11-13 07:56:19,704 INFO [make_runoff_file] 2025-11-12 runoff file creation completed
2025-11-13 07:56:19,706 INFO [manager] checklist updated with [rivers forcing] items from make_runoff_file worker
2025-11-13 07:56:20,116 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/hrdps_y2025m11d13.nc
2025-11-13 07:56:29,149 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2025m11d14.nc
2025-11-13 07:56:34,120 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2025m11d15.nc
2025-11-13 07:56:34,134 INFO [grib_to_netcdf] 2025-11-13 NEMO-atmos forcing file for nowcast+ created
2025-11-13 07:56:34,137 INFO [manager] checklist updated with [weather forcing] items from grib_to_netcdf worker
2025-11-13 07:56:52,527 INFO [make_live_ocean_files] Stored T&S western boundary conditions file: /results/forcing/LiveOcean/boundary_conditions/LiveOcean_v201905_y2025m11d13.nc
2025-11-13 07:56:52,528 INFO [make_live_ocean_files] 2025-11-13 Live Ocean western boundary conditions files created
2025-11-13 07:56:52,530 INFO [manager] checklist updated with [Live Ocean boundary conditions] items from make_live_ocean_files worker
2025-11-13 07:56:52,686 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['weather'], host='localhost')
2025-11-13 07:56:52,689 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'nowcast+'], host='localhost')
2025-11-13 07:56:52,695 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'nowcast+'], host='localhost')
2025-11-13 07:56:52,699 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.graham', 'nowcast+'], host='localhost')
2025-11-13 07:56:52,703 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'nowcast+'], host='localhost')
2025-11-13 07:56:53,758 INFO [ping_erddap] running in process 367508
2025-11-13 07:56:53,760 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-11-13 07:56:53,761 INFO [ping_erddap] publishing log messages to tcp://*:5562
2025-11-13 07:56:53,762 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2025-11-13 07:56:53,764 INFO [ping_erddap] weather ERDDAP dataset flag file(s) created
2025-11-13 07:56:53,766 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2025-11-13 07:56:53,863 INFO [upload_forcing] running in process 367513
2025-11-13 07:56:53,864 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-11-13 07:56:53,865 INFO [upload_forcing] publishing log messages to tcp://*:5564
2025-11-13 07:56:53,866 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-11-13 07:56:53,873 INFO [upload_forcing] running in process 367512
2025-11-13 07:56:53,875 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-11-13 07:56:53,876 INFO [upload_forcing] publishing log messages to tcp://*:5565
2025-11-13 07:56:53,877 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-11-13 07:56:53,880 INFO [upload_forcing] running in process 367511
2025-11-13 07:56:53,881 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-11-13 07:56:53,882 INFO [upload_forcing] publishing log messages to tcp://*:5566
2025-11-13 07:56:53,883 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-11-13 07:56:53,886 INFO [upload_forcing] running in process 367510
2025-11-13 07:56:53,887 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-11-13 07:56:53,888 INFO [upload_forcing] publishing log messages to tcp://*:5567
2025-11-13 07:56:53,889 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-11-13 07:56:55,685 INFO [upload_forcing] nowcast+ 2025-11-13 forcing files upload to optimum-hindcast completed
2025-11-13 07:56:55,688 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-11-13 07:56:57,063 CRITICAL [upload_forcing] unhandled exception:
Traceback (most recent call last):
  File "/SalishSeaCast/NEMO_Nowcast/nemo_nowcast/worker.py", line 387, in _do_work
    checklist = self.worker_func(
        self._parsed_args, self.config, self.tell_manager
    )
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/upload_forcing.py", line 94, in upload_forcing
    ssh_client, sftp_client = ssh_sftp.sftp(host_name, ssh_key)
                              ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/ssh_sftp.py", line 144, in sftp
    ssh_client = ssh(host, key_filename, ssh_config_file)
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/ssh_sftp.py", line 75, in ssh
    ssh_client.connect(
    ~~~~~~~~~~~~~~~~~~^
        host["hostname"],
        ^^^^^^^^^^^^^^^^^
    ...<3 lines>...
        look_for_keys=False,
        ^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/paramiko/client.py", line 384, in connect
    sock.connect(addr)
    ~~~~~~~~~~~~^^^^^^
OSError: [Errno 101] Network is unreachable
2025-11-13 07:57:01,255 INFO [upload_forcing] nowcast+ 2025-11-13 forcing files upload to orcinus-nowcast-agrif completed
2025-11-13 07:57:01,259 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-11-13 07:57:01,832 INFO [upload_forcing] nowcast+ 2025-11-13 forcing files upload to arbutus.cloud-nowcast completed
2025-11-13 07:57:01,837 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-11-13 07:57:01,977 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'nowcast+', '--run-date', '2025-11-13'], host='localhost')
2025-11-13 07:57:03,088 INFO [make_forcing_links] running in process 367549
2025-11-13 07:57:03,089 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-11-13 07:57:03,090 INFO [make_forcing_links] publishing log messages to tcp://*:5562
2025-11-13 07:57:03,091 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2025-11-13 07:57:04,247 INFO [make_forcing_links] nowcast+ 2025-11-13 forcing file links on arbutus.cloud-nowcast created
2025-11-13 07:57:04,252 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2025-11-13 07:57:04,407 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2025-11-13'], host='arbutus.cloud-nowcast')
2025-11-13 07:57:06,434 INFO [run_NEMO] running in process 339
2025-11-13 07:57:06,436 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-11-13 07:57:06,437 INFO [run_NEMO] publishing log messages to tcp://*:5556
2025-11-13 07:57:06,448 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2025-11-13 07:57:07,109 WARNING [run_NEMO] There are uncommitted changes in /nemoShare/MEOPAR/nowcast-sys/SS-run-sets
2025-11-13 07:57:07,404 INFO [run_NEMO] nowcast: launching /nemoShare/MEOPAR/nowcast-sys/runs/13nov25nowcast_2025-11-13T075706.481047-0800/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2025-11-13 07:57:07,418 INFO [run_NEMO] nowcast NEMO run for 2025-11-13 on arbutus.cloud-nowcast started
2025-11-13 07:57:07,422 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2025-11-13 07:57:07,576 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'nowcast'], host='arbutus.cloud-nowcast')
2025-11-13 07:57:09,505 INFO [watch_NEMO] running in process 635
2025-11-13 07:57:09,506 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-11-13 07:57:09,508 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2025-11-13 07:57:09,509 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2025-11-13 07:57:09,566 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2025-11-13 08:02:09,566 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 18835652 = 2025-11-13 05:00:40 UTC, 20.9% complete
2025-11-13 08:07:09,569 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 18836134 = 2025-11-13 10:22:00 UTC, 43.2% complete
2025-11-13 08:12:09,570 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 18836654 = 2025-11-13 16:08:40 UTC, 67.3% complete
2025-11-13 08:17:09,571 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 18837143 = 2025-11-13 21:34:40 UTC, 89.9% complete
2025-11-13 08:22:09,576 INFO [watch_NEMO] nowcast NEMO run on arbutus.cloud-nowcast completed
2025-11-13 08:22:09,580 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2025-11-13 08:22:09,914 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'ssh', '--run-date', '2025-11-13'], host='localhost')
2025-11-13 08:22:09,921 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2025-11-13'], host='localhost')
2025-11-13 08:22:11,033 INFO [download_results] running in process 367847
2025-11-13 08:22:11,035 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-11-13 08:22:11,036 INFO [download_results] publishing log messages to tcp://*:5562
2025-11-13 08:22:11,037 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2025-11-13 08:22:11,038 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/nowcast/13nov25 to /results/SalishSea/nowcast-blue.202111
2025-11-13 08:22:11,206 INFO [make_forcing_links] running in process 367846
2025-11-13 08:22:11,207 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-11-13 08:22:11,208 INFO [make_forcing_links] publishing log messages to tcp://*:5564
2025-11-13 08:22:11,210 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2025-11-13 08:22:12,147 INFO [make_forcing_links] ssh 2025-11-13 forcing file links on arbutus.cloud-nowcast created
2025-11-13 08:22:12,151 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2025-11-13 08:22:12,309 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2025-11-13'], host='arbutus.cloud-nowcast')
2025-11-13 08:22:14,667 INFO [run_NEMO] running in process 1019
2025-11-13 08:22:14,670 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-11-13 08:22:14,672 INFO [run_NEMO] publishing log messages to tcp://*:5556
2025-11-13 08:22:14,673 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2025-11-13 08:22:15,428 WARNING [run_NEMO] There are uncommitted changes in /nemoShare/MEOPAR/nowcast-sys/SS-run-sets
2025-11-13 08:22:15,779 INFO [run_NEMO] forecast: launching /nemoShare/MEOPAR/nowcast-sys/runs/13nov25forecast_2025-11-13T082214.747546-0800/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2025-11-13 08:22:15,794 INFO [run_NEMO] forecast NEMO run for 2025-11-13 on arbutus.cloud-nowcast started
2025-11-13 08:22:15,801 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2025-11-13 08:22:15,964 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'forecast'], host='arbutus.cloud-nowcast')
2025-11-13 08:22:17,995 INFO [watch_NEMO] running in process 1308
2025-11-13 08:22:17,997 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-11-13 08:22:17,999 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2025-11-13 08:22:18,000 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2025-11-13 08:22:18,099 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2025-11-13 08:22:57,051 INFO [download_results] nowcast 2025-11-13 results files from arbutus.cloud-nowcast downloaded
2025-11-13 08:22:57,055 INFO [manager] checklist updated with [results files] items from download_results worker
2025-11-13 08:22:57,214 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast', 'research', '--run-date', '2025-11-13'], host='localhost')
2025-11-13 08:22:57,218 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast', 'comparison', '--run-date', '2025-11-12'], host='localhost')
2025-11-13 08:22:57,223 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['nowcast', '--run-date', '2025-11-13'], host='localhost')
2025-11-13 08:22:59,050 INFO [make_CHS_currents_file] running in process 367869
2025-11-13 08:22:59,051 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-11-13 08:22:59,051 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5562
2025-11-13 08:22:59,052 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555
2025-11-13 08:23:00,107 INFO [make_plots] running in process 367866
2025-11-13 08:23:00,108 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-11-13 08:23:00,109 INFO [make_plots] publishing log messages to tcp://*:5564
2025-11-13 08:23:00,111 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2025-11-13 08:23:00,135 INFO [make_plots] running in process 367868
2025-11-13 08:23:00,140 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-11-13 08:23:00,162 INFO [make_plots] publishing log messages to tcp://*:5565
2025-11-13 08:23:00,171 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2025-11-13 08:23:00,568 INFO [make_plots] preparing render list for 2025-11-12 NEMO nowcast-blue comparison figures
2025-11-13 08:23:00,569 INFO [make_plots] preparing render list for 2025-11-13 NEMO nowcast-blue research figures
2025-11-13 08:23:00,617 INFO [make_plots] added SH_wind figure to 2025-11-12 NEMO nowcast-blue comparison render list
2025-11-13 08:23:00,618 INFO [make_plots] added Compare_VENUS_East figure to 2025-11-12 NEMO nowcast-blue comparison render list
2025-11-13 08:23:00,620 INFO [make_plots] added Compare_VENUS_Central figure to 2025-11-12 NEMO nowcast-blue comparison render list
2025-11-13 08:23:00,621 INFO [make_plots] starting to render nemo nowcast comparison 12nov25 figures
2025-11-13 08:23:02,093 INFO [make_plots] added salinity_thalweg_and_surface figures to 2025-11-13 NEMO nowcast-blue research render list
2025-11-13 08:23:03,267 INFO [make_plots] added temperature_thalweg_and_surface figures to 2025-11-13 NEMO nowcast-blue research render list
2025-11-13 08:23:03,269 INFO [make_plots] added Currents_sections_and_surface figure to 2025-11-13 NEMO nowcast-blue research render list
2025-11-13 08:23:03,270 INFO [make_plots] added Currents_at_VENUS_Central figure to 2025-11-13 NEMO nowcast-blue research render list
2025-11-13 08:23:03,271 INFO [make_plots] added Currents_at_VENUS_East figure to 2025-11-13 NEMO nowcast-blue research render list
2025-11-13 08:23:03,272 INFO [make_plots] starting to render nemo nowcast research 13nov25 figures
2025-11-13 08:23:34,269 INFO [make_CHS_currents_file] Made CHS currents file for 2025-11-13 for nowcast
2025-11-13 08:23:34,272 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker
2025-11-13 08:24:05,892 INFO [make_plots] finished rendering nemo nowcast research 13nov25 figures
2025-11-13 08:24:05,895 INFO [make_plots] nemo research plots for 2025-11-13 nowcast completed
2025-11-13 08:24:05,915 INFO [manager] checklist updated with [plots] items from make_plots worker
2025-11-13 08:24:31,414 INFO [make_plots] finished rendering nemo nowcast comparison 12nov25 figures
2025-11-13 08:24:31,416 INFO [make_plots] nemo comparison plots for 2025-11-12 nowcast completed
2025-11-13 08:24:31,421 INFO [manager] checklist updated with [plots] items from make_plots worker
2025-11-13 08:27:18,098 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 18837814 = 2025-11-14 05:02:00 UTC, 14.0% complete
2025-11-13 08:32:18,099 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 18838300 = 2025-11-14 10:26:00 UTC, 29.0% complete
2025-11-13 08:37:18,100 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 18838796 = 2025-11-14 15:56:40 UTC, 44.3% complete
2025-11-13 08:42:18,101 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 18839266 = 2025-11-14 21:10:00 UTC, 58.8% complete
2025-11-13 08:47:18,102 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 18839753 = 2025-11-15 02:34:40 UTC, 73.8% complete
2025-11-13 08:52:18,103 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 18840255 = 2025-11-15 08:09:20 UTC, 89.3% complete
2025-11-13 08:57:18,104 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 18840600 = 2025-11-15 11:59:20 UTC, 100.0% complete
2025-11-13 09:02:18,114 INFO [watch_NEMO] forecast NEMO run on arbutus.cloud-nowcast completed
2025-11-13 09:02:18,118 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2025-11-13 09:02:18,469 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_wind_file', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2025-11-13'], host='arbutus.cloud-nowcast')
2025-11-13 09:02:18,480 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_current_file', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2025-11-13'], host='arbutus.cloud-nowcast')
2025-11-13 09:02:18,485 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'turbidity', '--run-date', '2025-11-13'], host='localhost')
2025-11-13 09:02:18,493 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'turbidity', '--run-date', '2025-11-13'], host='localhost')
2025-11-13 09:02:18,498 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.graham', 'turbidity', '--run-date', '2025-11-13'], host='localhost')
2025-11-13 09:02:18,502 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'turbidity', '--run-date', '2025-11-13'], host='localhost')
2025-11-13 09:02:18,506 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2025-11-13'], host='localhost')
2025-11-13 09:02:19,626 INFO [download_results] running in process 368544
2025-11-13 09:02:19,627 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-11-13 09:02:19,628 INFO [download_results] publishing log messages to tcp://*:5562
2025-11-13 09:02:19,630 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2025-11-13 09:02:19,631 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/forecast/13nov25 to /results/SalishSea/forecast.202111
2025-11-13 09:02:19,718 INFO [upload_forcing] running in process 368543
2025-11-13 09:02:19,720 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-11-13 09:02:19,721 INFO [upload_forcing] publishing log messages to tcp://*:5564
2025-11-13 09:02:19,722 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-11-13 09:02:19,723 INFO [upload_forcing] running in process 368540
2025-11-13 09:02:19,724 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-11-13 09:02:19,725 INFO [upload_forcing] publishing log messages to tcp://*:5565
2025-11-13 09:02:19,729 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-11-13 09:02:19,731 INFO [upload_forcing] running in process 368539
2025-11-13 09:02:19,733 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-11-13 09:02:19,734 INFO [upload_forcing] publishing log messages to tcp://*:5566
2025-11-13 09:02:19,735 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-11-13 09:02:19,736 INFO [upload_forcing] running in process 368542
2025-11-13 09:02:19,737 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-11-13 09:02:19,738 INFO [upload_forcing] publishing log messages to tcp://*:5567
2025-11-13 09:02:19,739 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-11-13 09:02:20,365 INFO [upload_forcing] turbidity 2025-11-13 forcing files upload to optimum-hindcast completed
2025-11-13 09:02:20,370 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-11-13 09:02:20,787 INFO [upload_forcing] turbidity 2025-11-13 forcing files upload to arbutus.cloud-nowcast completed
2025-11-13 09:02:20,791 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-11-13 09:02:20,940 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2025-11-13'], host='localhost')
2025-11-13 09:02:21,524 INFO [make_ww3_current_file] running in process 2642
2025-11-13 09:02:21,526 INFO [make_ww3_current_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-11-13 09:02:21,527 INFO [make_ww3_current_file] publishing log messages to tcp://*:5571
2025-11-13 09:02:21,534 INFO [make_ww3_wind_file] running in process 2643
2025-11-13 09:02:21,535 INFO [make_ww3_wind_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-11-13 09:02:21,536 INFO [make_ww3_wind_file] publishing log messages to tcp://*:5570
2025-11-13 09:02:21,537 INFO [make_ww3_wind_file] connected to skookum.eos.ubc.ca port 5555
2025-11-13 09:02:21,538 INFO [make_ww3_wind_file] Creating wwatch3 wind forcing file for 2025-11-13 forecast run
2025-11-13 09:02:21,540 INFO [make_ww3_current_file] connected to skookum.eos.ubc.ca port 5555
2025-11-13 09:02:21,541 INFO [make_ww3_current_file] Creating wwatch3 currents forcing file for 2025-11-13 forecast run
2025-11-13 09:02:22,498 INFO [make_forcing_links] running in process 368571
2025-11-13 09:02:22,499 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-11-13 09:02:22,500 INFO [make_forcing_links] publishing log messages to tcp://*:5564
2025-11-13 09:02:22,502 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2025-11-13 09:02:22,925 CRITICAL [upload_forcing] unhandled exception:
Traceback (most recent call last):
  File "/SalishSeaCast/NEMO_Nowcast/nemo_nowcast/worker.py", line 387, in _do_work
    checklist = self.worker_func(
        self._parsed_args, self.config, self.tell_manager
    )
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/upload_forcing.py", line 94, in upload_forcing
    ssh_client, sftp_client = ssh_sftp.sftp(host_name, ssh_key)
                              ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/ssh_sftp.py", line 144, in sftp
    ssh_client = ssh(host, key_filename, ssh_config_file)
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/ssh_sftp.py", line 75, in ssh
    ssh_client.connect(
    ~~~~~~~~~~~~~~~~~~^
        host["hostname"],
        ^^^^^^^^^^^^^^^^^
    ...<3 lines>...
        look_for_keys=False,
        ^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/paramiko/client.py", line 384, in connect
    sock.connect(addr)
    ~~~~~~~~~~~~^^^^^^
OSError: [Errno 101] Network is unreachable
2025-11-13 09:02:23,673 INFO [make_forcing_links] nowcast-green 2025-11-13 forcing file links on arbutus.cloud-nowcast created
2025-11-13 09:02:23,677 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2025-11-13 09:02:23,853 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2025-11-13'], host='arbutus.cloud-nowcast')
2025-11-13 09:02:25,340 INFO [upload_forcing] turbidity 2025-11-13 forcing files upload to orcinus-nowcast-agrif completed
2025-11-13 09:02:25,345 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-11-13 09:02:25,511 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['orcinus-nowcast-agrif', 'nowcast-agrif', '--run-date', '2025-11-13'], host='localhost')
2025-11-13 09:02:26,432 INFO [run_NEMO] running in process 2976
2025-11-13 09:02:26,433 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-11-13 09:02:26,434 INFO [run_NEMO] publishing log messages to tcp://*:5556
2025-11-13 09:02:26,436 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2025-11-13 09:02:27,005 INFO [make_forcing_links] running in process 368587
2025-11-13 09:02:27,007 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-11-13 09:02:27,008 INFO [make_forcing_links] publishing log messages to tcp://*:5564
2025-11-13 09:02:27,009 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2025-11-13 09:02:28,779 WARNING [run_NEMO] There are uncommitted changes in /nemoShare/MEOPAR/nowcast-sys/SS-run-sets
2025-11-13 09:02:29,790 INFO [run_NEMO] nowcast-green: launching /nemoShare/MEOPAR/nowcast-sys/runs/13nov25nowcast-green_2025-11-13T090226.526477-0800/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2025-11-13 09:02:29,854 INFO [run_NEMO] nowcast-green NEMO run for 2025-11-13 on arbutus.cloud-nowcast started
2025-11-13 09:02:29,859 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2025-11-13 09:02:30,020 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'nowcast-green'], host='arbutus.cloud-nowcast')
2025-11-13 09:02:32,997 INFO [manager] checklist updated with [WW3 wind forcing] items from make_ww3_wind_file worker
2025-11-13 09:02:32,999 INFO [make_ww3_wind_file] wwatch3 wind forcing file created on arbutus.cloud-nowcast for 2025-11-13 forecast run
2025-11-13 09:02:33,014 INFO [watch_NEMO] running in process 3552
2025-11-13 09:02:33,015 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-11-13 09:02:33,016 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2025-11-13 09:02:33,017 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2025-11-13 09:02:33,084 INFO [make_forcing_links] nowcast-agrif 2025-11-13 forcing file links on orcinus-nowcast-agrif created
2025-11-13 09:02:33,165 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2025-11-13 09:02:33,230 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2025-11-13 09:02:33,321 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_agrif', args=['orcinus-nowcast-agrif', 'nowcast-agrif', '--run-date', '2025-11-13'], host='localhost')
2025-11-13 09:02:34,797 INFO [run_NEMO_agrif] running in process 368602
2025-11-13 09:02:34,798 INFO [run_NEMO_agrif] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-11-13 09:02:34,799 INFO [run_NEMO_agrif] publishing log messages to tcp://*:5564
2025-11-13 09:02:34,800 INFO [run_NEMO_agrif] connected to skookum.eos.ubc.ca port 5555
2025-11-13 09:02:40,780 CRITICAL [run_NEMO_agrif] unhandled exception:
Traceback (most recent call last):
  File "/SalishSeaCast/NEMO_Nowcast/nemo_nowcast/worker.py", line 387, in _do_work
    checklist = self.worker_func(
        self._parsed_args, self.config, self.tell_manager
    )
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/run_NEMO_agrif.py", line 109, in run_NEMO_agrif
    prev_run_namelists_info = _get_prev_run_namelists_info(
        sftp_client, host_name, run_date.shift(days=-1), config
    )
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/run_NEMO_agrif.py", line 155, in _get_prev_run_namelists_info
    sftp_client.get(os.fspath(prev_namelist_cfg), namelist_cfg.name)
    ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/paramiko/sftp_client.py", line 840, in get
    size = self.getfo(
        remotepath,
    ...<3 lines>...
        max_concurrent_prefetch_requests,
    )
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/paramiko/sftp_client.py", line 795, in getfo
    file_size = self.stat(remotepath).st_size
                ~~~~~~~~~^^^^^^^^^^^^
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/paramiko/sftp_client.py", line 493, in stat
    t, msg = self._request(CMD_STAT, path)
             ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/paramiko/sftp_client.py", line 857, in _request
    return self._read_response(num)
           ~~~~~~~~~~~~~~~~~~~^^^^^
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/paramiko/sftp_client.py", line 909, in _read_response
    self._convert_status(msg)
    ~~~~~~~~~~~~~~~~~~~~^^^^^
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/paramiko/sftp_client.py", line 938, in _convert_status
    raise IOError(errno.ENOENT, text)
FileNotFoundError: [Errno 2] No such file
2025-11-13 09:03:15,936 INFO [download_results] forecast 2025-11-13 results files from arbutus.cloud-nowcast downloaded
2025-11-13 09:03:15,940 INFO [manager] checklist updated with [results files] items from download_results worker
2025-11-13 09:03:16,103 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['forecast', '--run-date', '2025-11-13'], host='localhost')
2025-11-13 09:03:18,065 INFO [make_CHS_currents_file] running in process 368625
2025-11-13 09:03:18,067 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-11-13 09:03:18,068 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5562
2025-11-13 09:03:18,069 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555
2025-11-13 09:04:05,824 INFO [make_CHS_currents_file] Made CHS currents file for 2025-11-13 for forecast
2025-11-13 09:04:05,827 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker
2025-11-13 09:04:05,988 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['nemo', 'forecast', '--run-date', '2025-11-13'], host='localhost')
2025-11-13 09:04:07,101 INFO [update_forecast_datasets] running in process 368659
2025-11-13 09:04:07,102 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-11-13 09:04:07,103 INFO [update_forecast_datasets] publishing log messages to tcp://*:5562
2025-11-13 09:04:07,104 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2025-11-13 09:04:07,105 INFO [update_forecast_datasets] updating nemo forecast directory for forecast/13nov25 run
2025-11-13 09:04:07,385 INFO [update_forecast_datasets] updated nemo forecast directory for forecast/13nov25 run: /results/SalishSea/rolling-forecasts/nemo
2025-11-13 09:04:07,386 INFO [update_forecast_datasets] nemo 2025-11-13 forecast rolling forecast datasets updated
2025-11-13 09:04:07,389 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2025-11-13 09:04:07,548 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nemo-forecast'], host='localhost')
2025-11-13 09:04:07,553 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'forecast', 'publish', '--run-date', '2025-11-13'], host='localhost')
2025-11-13 09:04:07,558 INFO [manager] launching NextWorker(module='nowcast.workers.make_surface_current_tiles', args=['forecast', '--run-date', '2025-11-13'], host='localhost')
2025-11-13 09:04:08,674 INFO [ping_erddap] running in process 368664
2025-11-13 09:04:08,675 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-11-13 09:04:08,676 INFO [ping_erddap] publishing log messages to tcp://*:5562
2025-11-13 09:04:08,677 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2025-11-13 09:04:08,686 INFO [ping_erddap] nemo-forecast ERDDAP dataset flag file(s) created
2025-11-13 09:04:08,688 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2025-11-13 09:04:09,799 INFO [make_surface_current_tiles] running in process 368667
2025-11-13 09:04:09,800 INFO [make_surface_current_tiles] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-11-13 09:04:09,801 INFO [make_surface_current_tiles] publishing log messages to tcp://*:5562
2025-11-13 09:04:09,801 INFO [make_surface_current_tiles] connected to skookum.eos.ubc.ca port 5555
2025-11-13 09:04:09,803 INFO [make_surface_current_tiles] starting rendering of tiles for 2025-11-13 forecast in 6 processes into /results/nowcast-sys/figures/surface_currents/forecast/13nov25
2025-11-13 09:04:10,551 INFO [make_plots] running in process 368666
2025-11-13 09:04:10,552 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-11-13 09:04:10,553 INFO [make_plots] publishing log messages to tcp://*:5564
2025-11-13 09:04:10,554 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2025-11-13 09:04:10,980 INFO [make_plots] preparing render list for 2025-11-13 NEMO forecast publish figures
2025-11-13 09:04:11,250 INFO [make_plots] added Website_thumbnail figure to 2025-11-13 NEMO forecast publish render list
2025-11-13 09:04:11,251 INFO [make_plots] added Threshold_website figure to 2025-11-13 NEMO forecast publish render list
2025-11-13 09:04:11,252 INFO [make_plots] added PA_tidal_predictions figure to 2025-11-13 NEMO forecast publish render list
2025-11-13 09:04:11,253 INFO [make_plots] added Boundary Bay figure to 2025-11-13 NEMO forecast publish render list
2025-11-13 09:04:11,254 INFO [make_plots] added Campbell River figure to 2025-11-13 NEMO forecast publish render list
2025-11-13 09:04:11,255 INFO [make_plots] added Cherry Point figure to 2025-11-13 NEMO forecast publish render list
2025-11-13 09:04:11,256 INFO [make_plots] added Friday Harbor figure to 2025-11-13 NEMO forecast publish render list
2025-11-13 09:04:11,257 INFO [make_plots] added Halfmoon Bay figure to 2025-11-13 NEMO forecast publish render list
2025-11-13 09:04:11,258 INFO [make_plots] added Nanaimo figure to 2025-11-13 NEMO forecast publish render list
2025-11-13 09:04:11,259 INFO [make_plots] added Neah Bay figure to 2025-11-13 NEMO forecast publish render list
2025-11-13 09:04:11,260 INFO [make_plots] added New Westminster figure to 2025-11-13 NEMO forecast publish render list
2025-11-13 09:04:11,261 INFO [make_plots] added Patricia Bay figure to 2025-11-13 NEMO forecast publish render list
2025-11-13 09:04:11,261 INFO [make_plots] added Point Atkinson figure to 2025-11-13 NEMO forecast publish render list
2025-11-13 09:04:11,262 INFO [make_plots] added Port Renfrew figure to 2025-11-13 NEMO forecast publish render list
2025-11-13 09:04:11,271 INFO [make_plots] added Sand Heads figure to 2025-11-13 NEMO forecast publish render list
2025-11-13 09:04:11,277 INFO [make_plots] added Sandy Cove figure to 2025-11-13 NEMO forecast publish render list
2025-11-13 09:04:11,277 INFO [make_plots] added Squamish figure to 2025-11-13 NEMO forecast publish render list
2025-11-13 09:04:11,278 INFO [make_plots] added Victoria figure to 2025-11-13 NEMO forecast publish render list
2025-11-13 09:04:11,278 INFO [make_plots] added Woodwards Landing figure to 2025-11-13 NEMO forecast publish render list
2025-11-13 09:04:11,279 INFO [make_plots] starting to render nemo forecast publish 13nov25 figures
2025-11-13 09:04:59,124 INFO [make_ww3_current_file] wwatch3 currents forcing file created on arbutus.cloud-nowcast for 2025-11-13 forecast run
2025-11-13 09:04:59,127 INFO [manager] checklist updated with [WW3 currents forcing] items from make_ww3_current_file worker
2025-11-13 09:04:59,356 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2025-11-13'], host='arbutus.cloud-nowcast')
2025-11-13 09:05:01,463 INFO [run_ww3] running in process 3663
2025-11-13 09:05:01,464 INFO [run_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-11-13 09:05:01,465 INFO [run_ww3] publishing log messages to tcp://*:5572
2025-11-13 09:05:01,466 INFO [run_ww3] connected to skookum.eos.ubc.ca port 5555
2025-11-13 09:05:01,469 INFO [run_ww3] Created run directory /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/nowcast_2025-11-13T090501.455775-0800
2025-11-13 09:05:01,469 INFO [run_ww3] nowcast: launching /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/nowcast_2025-11-13T090501.455775-0800/SoGWW3.sh on arbutus.cloud-nowcast
2025-11-13 09:05:01,482 INFO [run_ww3] nowcast WaveWatch3 run for 2025-11-13 on arbutus.cloud-nowcast started
2025-11-13 09:05:01,485 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker
2025-11-13 09:05:01,685 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'nowcast'], host='arbutus.cloud-nowcast')
2025-11-13 09:05:03,797 INFO [watch_ww3] running in process 3750
2025-11-13 09:05:03,798 INFO [watch_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-11-13 09:05:03,799 INFO [watch_ww3] publishing log messages to tcp://*:5573
2025-11-13 09:05:03,800 INFO [watch_ww3] connected to skookum.eos.ubc.ca port 5555
2025-11-13 09:05:03,833 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2025-11-13 09:07:33,238 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18835368 = 2025-11-13 01:51:20 UTC, 7.7% complete
2025-11-13 09:10:03,837 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 66, 15.3% complete
2025-11-13 09:12:26,698 WARNING [make_plots] invalid station number for Boundary Bay station: None
2025-11-13 09:12:26,699 WARNING [make_plots] can't resolve a valid CHS station code for Boundary Bay
2025-11-13 09:12:33,238 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18835556 = 2025-11-13 03:56:40 UTC, 16.4% complete
2025-11-13 09:12:49,673 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb996/data for station 08074 Campbell River from 2025-11-08 00:00:00Z to 2025-11-15 11:55:00Z
2025-11-13 09:13:58,434 WARNING [make_plots] invalid station number for Halfmoon Bay station: None
2025-11-13 09:13:58,435 WARNING [make_plots] can't resolve a valid CHS station code for Halfmoon Bay
2025-11-13 09:14:20,130 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb96d/data for station 07917 Nanaimo from 2025-11-08 00:00:00Z to 2025-11-15 11:55:00Z
2025-11-13 09:14:49,569 INFO [make_surface_current_tiles] starting PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast/13nov25
2025-11-13 09:14:56,913 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e13d0f4a073c4bbf93/data for station 07654 New Westminster from 2025-11-08 00:00:00Z to 2025-11-15 11:55:00Z
2025-11-13 09:15:03,839 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 144, 33.3% complete
2025-11-13 09:15:15,258 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd2d/data for station 07277 Patricia Bay from 2025-11-08 00:00:00Z to 2025-11-15 11:55:00Z
2025-11-13 09:15:34,072 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94c/data for station 07795 Point Atkinson from 2025-11-08 00:00:00Z to 2025-11-15 11:55:00Z
2025-11-13 09:15:53,004 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e23d0f4a073c4bc060/data for station 08525 Port Renfrew from 2025-11-08 00:00:00Z to 2025-11-15 11:55:00Z
2025-11-13 09:16:10,820 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb93c/data for station 07594 Sand Heads from 2025-11-08 00:00:00Z to 2025-11-15 11:55:00Z
2025-11-13 09:16:29,530 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc43b/data for station 07786 Sandy Cove from 2025-11-08 00:00:00Z to 2025-11-15 11:55:00Z
2025-11-13 09:16:47,239 WARNING [make_plots] invalid station number for Squamish station: None
2025-11-13 09:16:47,239 WARNING [make_plots] can't resolve a valid CHS station code for Squamish
2025-11-13 09:17:05,852 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd1e/data for station 07120 Victoria from 2025-11-08 00:00:00Z to 2025-11-15 11:55:00Z
2025-11-13 09:17:24,224 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc42d/data for station 07610 Woodwards Landing from 2025-11-08 00:00:00Z to 2025-11-15 11:55:00Z
2025-11-13 09:17:33,245 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18835741 = 2025-11-13 06:00:00 UTC, 25.0% complete
2025-11-13 09:17:41,671 INFO [make_plots] finished rendering nemo forecast publish 13nov25 figures
2025-11-13 09:17:41,674 INFO [make_plots] nemo publish plots for 2025-11-13 forecast completed
2025-11-13 09:17:41,683 INFO [manager] checklist updated with [plots] items from make_plots worker
2025-11-13 09:17:41,851 INFO [manager] launching NextWorker(module='nowcast.workers.make_feeds', args=['forecast', '--run-date', '2025-11-13'], host='localhost')
2025-11-13 09:17:44,970 INFO [make_feeds] running in process 374532
2025-11-13 09:17:44,972 INFO [make_feeds] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-11-13 09:17:44,973 INFO [make_feeds] publishing log messages to tcp://*:5564
2025-11-13 09:17:44,974 INFO [make_feeds] connected to skookum.eos.ubc.ca port 5555
2025-11-13 09:18:00,081 INFO [make_feeds] ATOM feeds for 2025-11-13 forecast run completed
2025-11-13 09:18:00,084 INFO [manager] checklist updated with [storm surge alert feeds] items from make_feeds worker
2025-11-13 09:19:40,418 INFO [make_surface_current_tiles] finished PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast/13nov25
2025-11-13 09:19:40,443 INFO [make_surface_current_tiles] finished rendering of tiles for 2025-11-13 forecast into /results/nowcast-sys/figures/surface_currents/forecast/13nov25
2025-11-13 09:19:40,444 INFO [make_surface_current_tiles] surface current tile figures for 2025-11-13 forecast completed
2025-11-13 09:19:40,765 INFO [manager] checklist updated with [surface current tiles] items from make_surface_current_tiles worker
2025-11-13 09:20:03,842 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 219, 50.7% complete
2025-11-13 09:22:33,243 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18835928 = 2025-11-13 08:04:40 UTC, 33.7% complete
2025-11-13 09:25:03,844 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 300, 69.4% complete
2025-11-13 09:27:33,239 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18836113 = 2025-11-13 10:08:00 UTC, 42.2% complete
2025-11-13 09:30:03,833 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 378, 87.5% complete
2025-11-13 09:32:33,229 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18836301 = 2025-11-13 12:13:20 UTC, 50.9% complete
2025-11-13 09:35:03,827 INFO [watch_ww3] nowcast WWATCH3 run on arbutus.cloud-nowcast completed
2025-11-13 09:35:03,830 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2025-11-13 09:35:04,328 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2025-11-13'], host='localhost')
2025-11-13 09:35:04,335 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2025-11-13'], host='arbutus.cloud-nowcast')
2025-11-13 09:35:05,567 INFO [download_wwatch3_results] running in process 374997
2025-11-13 09:35:05,568 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-11-13 09:35:05,569 INFO [download_wwatch3_results] publishing log messages to tcp://*:5562
2025-11-13 09:35:05,571 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2025-11-13 09:35:06,361 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker
2025-11-13 09:35:06,362 INFO [run_ww3] running in process 4090
2025-11-13 09:35:06,364 INFO [run_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-11-13 09:35:06,365 INFO [run_ww3] publishing log messages to tcp://*:5572
2025-11-13 09:35:06,366 INFO [run_ww3] connected to skookum.eos.ubc.ca port 5555
2025-11-13 09:35:06,370 INFO [run_ww3] Created run directory /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast_2025-11-13T093506.341850-0800
2025-11-13 09:35:06,371 INFO [run_ww3] forecast: launching /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast_2025-11-13T093506.341850-0800/SoGWW3.sh on arbutus.cloud-nowcast
2025-11-13 09:35:06,373 INFO [run_ww3] forecast WaveWatch3 run for 2025-11-13 on arbutus.cloud-nowcast started
2025-11-13 09:35:06,683 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'forecast'], host='arbutus.cloud-nowcast')
2025-11-13 09:35:09,111 INFO [watch_ww3] running in process 4245
2025-11-13 09:35:09,112 INFO [watch_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-11-13 09:35:09,114 INFO [watch_ww3] publishing log messages to tcp://*:5573
2025-11-13 09:35:09,115 INFO [watch_ww3] connected to skookum.eos.ubc.ca port 5555
2025-11-13 09:35:09,149 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2025-11-13 09:35:38,539 INFO [download_wwatch3_results] nowcast 2025-11-13 results files from arbutus.cloud-nowcast downloaded
2025-11-13 09:35:38,543 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2025-11-13 09:37:33,224 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18836483 = 2025-11-13 14:14:40 UTC, 59.4% complete
2025-11-13 09:40:09,144 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 72, 11.1% complete
2025-11-13 09:42:33,229 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18836670 = 2025-11-13 16:19:20 UTC, 68.0% complete
2025-11-13 09:45:09,141 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 147, 22.7% complete
2025-11-13 09:47:33,219 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18836851 = 2025-11-13 18:20:00 UTC, 76.4% complete
2025-11-13 09:50:09,140 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 228, 35.2% complete
2025-11-13 09:52:33,218 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18837029 = 2025-11-13 20:18:40 UTC, 84.7% complete
2025-11-13 09:55:09,146 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 309, 47.7% complete
2025-11-13 09:57:33,227 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18837214 = 2025-11-13 22:22:00 UTC, 93.2% complete
2025-11-13 10:00:09,152 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 390, 60.2% complete
2025-11-13 10:02:33,233 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18837360 = 2025-11-13 23:59:20 UTC, 100.0% complete
2025-11-13 10:05:09,156 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 468, 72.2% complete
2025-11-13 10:07:33,343 INFO [watch_NEMO] nowcast-green NEMO run on arbutus.cloud-nowcast completed
2025-11-13 10:07:33,346 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2025-11-13 10:07:33,848 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2025-11-13'], host='localhost')
2025-11-13 10:07:35,137 INFO [download_results] running in process 375969
2025-11-13 10:07:35,139 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-11-13 10:07:35,139 INFO [download_results] publishing log messages to tcp://*:5562
2025-11-13 10:07:35,140 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2025-11-13 10:07:35,141 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/nowcast-green/13nov25 to /results2/SalishSea/nowcast-green.202111
2025-11-13 10:10:06,490 INFO [download_results] nowcast-green 2025-11-13 results files from arbutus.cloud-nowcast downloaded
2025-11-13 10:10:06,497 INFO [manager] checklist updated with [results files] items from download_results worker
2025-11-13 10:10:06,799 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast-green', 'research', '--run-date', '2025-11-13'], host='localhost')
2025-11-13 10:10:06,804 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nowcast-green'], host='localhost')
2025-11-13 10:10:06,808 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'biology', '--run-date', '2025-11-13'], host='localhost')
2025-11-13 10:10:06,813 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'chemistry', '--run-date', '2025-11-13'], host='localhost')
2025-11-13 10:10:06,818 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'physics', '--run-date', '2025-11-13'], host='localhost')
2025-11-13 10:10:08,155 INFO [ping_erddap] running in process 376041
2025-11-13 10:10:08,156 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-11-13 10:10:08,157 INFO [ping_erddap] publishing log messages to tcp://*:5562
2025-11-13 10:10:08,158 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2025-11-13 10:10:08,163 INFO [ping_erddap] nowcast-green ERDDAP dataset flag file(s) created
2025-11-13 10:10:08,165 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2025-11-13 10:10:09,159 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 543, 83.8% complete
2025-11-13 10:10:09,699 INFO [make_averaged_dataset] running in process 376042
2025-11-13 10:10:09,700 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-11-13 10:10:09,701 INFO [make_averaged_dataset] publishing log messages to tcp://*:5562
2025-11-13 10:10:09,702 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555
2025-11-13 10:10:09,703 INFO [make_averaged_dataset] creating day-averaged dataset for 13-Nov-2025 biology
2025-11-13 10:10:09,798 INFO [make_averaged_dataset] running in process 376044
2025-11-13 10:10:09,800 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-11-13 10:10:09,800 INFO [make_averaged_dataset] publishing log messages to tcp://*:5564
2025-11-13 10:10:09,801 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555
2025-11-13 10:10:09,802 INFO [make_averaged_dataset] creating day-averaged dataset for 13-Nov-2025 physics
2025-11-13 10:10:09,809 INFO [make_averaged_dataset] running in process 376043
2025-11-13 10:10:09,810 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-11-13 10:10:09,811 INFO [make_averaged_dataset] publishing log messages to tcp://*:5565
2025-11-13 10:10:09,812 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555
2025-11-13 10:10:09,813 INFO [make_averaged_dataset] creating day-averaged dataset for 13-Nov-2025 chemistry
2025-11-13 10:10:10,793 INFO [make_plots] running in process 376039
2025-11-13 10:10:10,794 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-11-13 10:10:10,795 INFO [make_plots] publishing log messages to tcp://*:5566
2025-11-13 10:10:10,796 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2025-11-13 10:10:11,208 INFO [make_plots] preparing render list for 2025-11-13 NEMO nowcast-green research figures
2025-11-13 10:10:13,214 INFO [make_plots] added temperature_thalweg_and_surface figures to 2025-11-13 NEMO nowcast-green research render list
2025-11-13 10:10:14,053 INFO [make_plots] added nitrate_thalweg_and_surface figures to 2025-11-13 NEMO nowcast-green research render list
2025-11-13 10:10:14,817 INFO [make_plots] added ammonium_thalweg_and_surface figures to 2025-11-13 NEMO nowcast-green research render list
2025-11-13 10:10:15,583 INFO [make_plots] added silicon_thalweg_and_surface figures to 2025-11-13 NEMO nowcast-green research render list
2025-11-13 10:10:16,417 INFO [make_plots] added dissolved_organic_nitrogen_thalweg_and_surface figures to 2025-11-13 NEMO nowcast-green research render list
2025-11-13 10:10:17,221 INFO [make_plots] added particulate_organic_nitrogen_thalweg_and_surface figures to 2025-11-13 NEMO nowcast-green research render list
2025-11-13 10:10:18,012 INFO [make_plots] added biogenic_silicon_thalweg_and_surface figures to 2025-11-13 NEMO nowcast-green research render list
2025-11-13 10:10:18,935 INFO [make_plots] added diatoms_thalweg_and_surface figures to 2025-11-13 NEMO nowcast-green research render list
2025-11-13 10:10:19,871 INFO [make_plots] added flagellates_thalweg_and_surface figures to 2025-11-13 NEMO nowcast-green research render list
2025-11-13 10:10:20,776 INFO [make_plots] added microzooplankton_thalweg_and_surface figures to 2025-11-13 NEMO nowcast-green research render list
2025-11-13 10:10:21,689 INFO [make_plots] added mesozooplankton_thalweg_and_surface figures to 2025-11-13 NEMO nowcast-green research render list
2025-11-13 10:10:22,466 INFO [make_plots] added turbidity_thalweg_and_surface figures to 2025-11-13 NEMO nowcast-green research render list
2025-11-13 10:10:23,618 INFO [make_plots] added temperature_salinity_timeseries figure to 2025-11-13 NEMO nowcast-green research render list
2025-11-13 10:10:23,619 INFO [make_plots] added nitrate_diatoms_timeseries figure to 2025-11-13 NEMO nowcast-green research render list
2025-11-13 10:10:23,620 INFO [make_plots] added diatoms_flagellates_timeseries figure to 2025-11-13 NEMO nowcast-green research render list
2025-11-13 10:10:23,621 INFO [make_plots] added z1_z2_zooplankton_timeseries figure to 2025-11-13 NEMO nowcast-green research render list
2025-11-13 10:10:23,622 INFO [make_plots] starting to render nemo nowcast-green research 13nov25 figures
2025-11-13 10:10:41,008 INFO [make_averaged_dataset] day-averaged dataset for 13-Nov-2025 physics created
2025-11-13 10:10:41,010 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker
2025-11-13 10:11:02,982 INFO [make_averaged_dataset] day-averaged dataset for 13-Nov-2025 chemistry created
2025-11-13 10:11:02,985 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker
2025-11-13 10:11:20,095 INFO [make_averaged_dataset] day-averaged dataset for 13-Nov-2025 biology created
2025-11-13 10:11:20,099 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker
2025-11-13 10:15:09,161 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 624, 96.3% complete
2025-11-13 10:20:09,163 INFO [watch_ww3] forecast WWATCH3 run on arbutus.cloud-nowcast completed
2025-11-13 10:20:09,166 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2025-11-13 10:20:09,693 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2025-11-13'], host='localhost')
2025-11-13 10:20:11,284 INFO [download_wwatch3_results] running in process 376463
2025-11-13 10:20:11,307 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-11-13 10:20:11,326 INFO [download_wwatch3_results] publishing log messages to tcp://*:5562
2025-11-13 10:20:11,329 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2025-11-13 10:20:50,613 INFO [download_wwatch3_results] forecast 2025-11-13 results files from arbutus.cloud-nowcast downloaded
2025-11-13 10:20:50,617 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2025-11-13 10:20:50,934 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['wwatch3', 'forecast', '--run-date', '2025-11-13'], host='localhost')
2025-11-13 10:20:52,309 INFO [update_forecast_datasets] running in process 376489
2025-11-13 10:20:52,323 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-11-13 10:20:52,324 INFO [update_forecast_datasets] publishing log messages to tcp://*:5562
2025-11-13 10:20:52,325 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2025-11-13 10:20:52,326 INFO [update_forecast_datasets] updating wwatch3 most_recent_forecast directory from forecast/13nov25 run
2025-11-13 10:20:52,328 INFO [update_forecast_datasets] updated wwatch3 most_recent_forecast directory from forecast/13nov25 run
2025-11-13 10:20:52,329 INFO [update_forecast_datasets] updating wwatch3 forecast directory for forecast/13nov25 run
2025-11-13 10:20:52,519 INFO [update_forecast_datasets] updated wwatch3 forecast directory for forecast/13nov25 run: /results/SalishSea/rolling-forecasts/wwatch3
2025-11-13 10:20:52,520 INFO [update_forecast_datasets] wwatch3 2025-11-13 forecast rolling forecast datasets updated
2025-11-13 10:20:52,523 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2025-11-13 10:20:52,840 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['wwatch3-forecast'], host='localhost')
2025-11-13 10:20:54,234 INFO [ping_erddap] running in process 376494
2025-11-13 10:20:54,237 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-11-13 10:20:54,238 INFO [ping_erddap] publishing log messages to tcp://*:5562
2025-11-13 10:20:54,239 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2025-11-13 10:20:54,241 INFO [ping_erddap] wwatch3-forecast ERDDAP dataset flag file(s) created
2025-11-13 10:20:54,244 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2025-11-13 10:20:54,564 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['wwatch3', 'forecast', 'publish', '--run-date', '2025-11-13'], host='localhost')
2025-11-13 10:20:59,004 INFO [make_plots] running in process 376499
2025-11-13 10:20:59,026 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-11-13 10:20:59,045 INFO [make_plots] publishing log messages to tcp://*:5562
2025-11-13 10:20:59,055 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2025-11-13 10:20:59,056 INFO [make_plots] preparing render list for 2025-11-13 WaveWatch3 forecast publish figures
2025-11-13 10:20:59,057 INFO [make_plots] added Halibut Bank figure to 2025-11-13 WaveWatch3 forecast publish render list
2025-11-13 10:20:59,058 INFO [make_plots] added Sentry Shoal figure to 2025-11-13 WaveWatch3 forecast publish render list
2025-11-13 10:20:59,059 INFO [make_plots] starting to render wwatch3 forecast publish 13nov25 figures
2025-11-13 10:20:59,649 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46146 Halibut Bank
2025-11-13 10:21:03,888 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46131 Sentry Shoal
2025-11-13 10:21:08,120 INFO [make_plots] finished rendering wwatch3 forecast publish 13nov25 figures
2025-11-13 10:21:08,121 INFO [make_plots] wwatch3 publish plots for 2025-11-13 forecast completed
2025-11-13 10:21:08,124 INFO [manager] checklist updated with [plots] items from make_plots worker
2025-11-13 11:09:56,391 INFO [make_plots] finished rendering nemo nowcast-green research 13nov25 figures
2025-11-13 11:09:56,400 INFO [make_plots] nemo research plots for 2025-11-13 nowcast-green completed
2025-11-13 11:09:56,464 INFO [manager] checklist updated with [plots] items from make_plots worker
2025-11-13 13:47:36,806 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/18/ to /results/forcing/atmospheric/continental2.5/GRIB/20251113/18/
2025-11-13 13:47:36,811 INFO [collect_weather] 2025-11-13 2.5km weather forecast 18 collection complete
2025-11-13 13:47:36,822 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2025-11-13 13:47:37,158 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20251113/18/
2025-11-13 13:47:37,159 INFO [crop_gribs] 2025-11-13 18 GRIBs cropping complete
2025-11-13 13:47:37,324 INFO [manager] launching NextWorker(module='nowcast.workers.download_weather', args=['00', '1km'], host='localhost')
2025-11-13 13:47:37,331 INFO [manager] launching NextWorker(module='nowcast.workers.download_weather', args=['12', '1km'], host='localhost')
2025-11-13 13:47:37,336 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['00', '2.5km'], host='localhost')
2025-11-13 13:47:37,341 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['00', '--fcst-date', '2025-11-14'], host='localhost')
2025-11-13 13:47:37,347 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2025-11-13 13:47:38,923 INFO [collect_weather] running in process 383366
2025-11-13 13:47:38,924 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-11-13 13:47:38,925 INFO [collect_weather] publishing log messages to tcp://*:5562
2025-11-13 13:47:38,926 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2025-11-13 13:47:38,927 INFO [download_weather] running in process 383363
2025-11-13 13:47:38,928 INFO [download_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-11-13 13:47:38,929 INFO [download_weather] publishing log messages to tcp://*:5563
2025-11-13 13:47:38,930 INFO [download_weather] connected to skookum.eos.ubc.ca port 5555
2025-11-13 13:47:38,931 INFO [download_weather] downloading 00 1 km forecast GRIB2 files for 20251113
2025-11-13 13:47:38,931 INFO [download_weather] running in process 383365
2025-11-13 13:47:38,932 INFO [download_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-11-13 13:47:38,933 INFO [download_weather] publishing log messages to tcp://*:5564
2025-11-13 13:47:38,935 INFO [download_weather] connected to skookum.eos.ubc.ca port 5555
2025-11-13 13:47:38,936 INFO [download_weather] downloading 12 1 km forecast GRIB2 files for 20251113
2025-11-13 13:47:38,961 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/00/
2025-11-13 13:47:40,340 INFO [crop_gribs] running in process 383367
2025-11-13 13:47:40,341 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-11-13 13:47:40,343 INFO [crop_gribs] publishing log messages to tcp://*:5565
2025-11-13 13:47:40,344 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2025-11-13 13:47:40,345 INFO [crop_gribs] cropping 2025-11-14 ECCC HRDPS 2.5km continental 00Z GRIB files to SalishSeaCast subdomain
2025-11-13 13:47:40,358 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20251114/00/
2025-11-13 13:51:25,239 INFO [download_weather] 2025-11-13 1km weather forecast 12 downloads complete
2025-11-13 13:51:25,242 INFO [download_weather] 2025-11-13 1km weather forecast 00 downloads complete
2025-11-13 13:51:25,244 INFO [manager] checklist updated with [weather forecast] items from download_weather worker
2025-11-13 13:51:25,733 INFO [manager] checklist updated with [weather forecast] items from download_weather worker
2025-11-13 19:55:39,846 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/00/ to /results/forcing/atmospheric/continental2.5/GRIB/20251114/00/
2025-11-13 19:55:39,847 INFO [collect_weather] 2025-11-14 2.5km weather forecast 00 collection complete
2025-11-13 19:55:39,850 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2025-11-13 19:55:40,361 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['06', '2.5km'], host='localhost')
2025-11-13 19:55:40,368 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['06', '--fcst-date', '2025-11-14'], host='localhost')
2025-11-13 19:55:41,118 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20251114/00/
2025-11-13 19:55:41,119 INFO [crop_gribs] 2025-11-14 00 GRIBs cropping complete
2025-11-13 19:55:41,122 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2025-11-13 19:55:42,298 INFO [collect_weather] running in process 392736
2025-11-13 19:55:42,300 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-11-13 19:55:42,300 INFO [collect_weather] publishing log messages to tcp://*:5562
2025-11-13 19:55:42,301 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2025-11-13 19:55:42,350 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/06/
2025-11-13 19:55:43,498 INFO [crop_gribs] running in process 392738
2025-11-13 19:55:43,499 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-11-13 19:55:43,501 INFO [crop_gribs] publishing log messages to tcp://*:5563
2025-11-13 19:55:43,502 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2025-11-13 19:55:43,503 INFO [crop_gribs] cropping 2025-11-14 ECCC HRDPS 2.5km continental 06Z GRIB files to SalishSeaCast subdomain
2025-11-13 19:55:43,517 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20251114/06/
2025-11-14 02:30:00,322 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/06/ to /results/forcing/atmospheric/continental2.5/GRIB/20251114/06/
2025-11-14 02:30:00,324 INFO [collect_weather] 2025-11-14 2.5km weather forecast 06 collection complete
2025-11-14 02:30:00,326 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2025-11-14 02:30:00,866 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Capilano', '--data-date', '2025-11-13'], host='localhost')
2025-11-14 02:30:00,874 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'ChilliwackVedder', '--data-date', '2025-11-13'], host='localhost')
2025-11-14 02:30:00,879 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'ClowhomClowhomLake', '--data-date', '2025-11-13'], host='localhost')
2025-11-14 02:30:00,883 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Englishman', '--data-date', '2025-11-13'], host='localhost')
2025-11-14 02:30:00,888 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Fraser', '--data-date', '2025-11-13'], host='localhost')
2025-11-14 02:30:00,893 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'HomathkoMouth', '--data-date', '2025-11-13'], host='localhost')
2025-11-14 02:30:00,897 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'NicomeklLangley', '--data-date', '2025-11-13'], host='localhost')
2025-11-14 02:30:00,903 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'RobertsCreek', '--data-date', '2025-11-13'], host='localhost')
2025-11-14 02:30:00,909 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SalmonSayward', '--data-date', '2025-11-13'], host='localhost')
2025-11-14 02:30:00,913 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SanJuanPortRenfrew', '--data-date', '2025-11-13'], host='localhost')
2025-11-14 02:30:00,918 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SquamishBrackendale', '--data-date', '2025-11-13'], host='localhost')
2025-11-14 02:30:00,928 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaScotty', '--data-date', '2025-11-13'], host='localhost')
2025-11-14 02:30:00,949 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaBypass', '--data-date', '2025-11-13'], host='localhost')
2025-11-14 02:30:00,960 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaDiversion', '--data-date', '2025-11-13'], host='localhost')
2025-11-14 02:30:00,969 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ctd', args=['SCVIP'], host='localhost')
2025-11-14 02:30:00,978 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ctd', args=['SEVIP'], host='localhost')
2025-11-14 02:30:00,992 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ferry', args=['TWDP'], host='localhost')
2025-11-14 02:30:01,020 INFO [manager] launching NextWorker(module='nowcast.workers.get_vfpa_hadcp', args=['--data-date', '2025-11-13'], host='localhost')
2025-11-14 02:30:01,037 INFO [manager] launching NextWorker(module='nowcast.workers.collect_NeahBay_ssh', args=['00'], host='localhost')
2025-11-14 02:30:01,062 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['12', '2.5km'], host='localhost')
2025-11-14 02:30:01,077 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['12'], host='localhost')
2025-11-14 02:30:02,112 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20251114/06/
2025-11-14 02:30:02,113 INFO [crop_gribs] 2025-11-14 06 GRIBs cropping complete
2025-11-14 02:30:02,116 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2025-11-14 02:30:03,273 INFO [manager] launching NextWorker(module='nowcast.workers.grib_to_netcdf', args=['forecast2'], host='localhost')
2025-11-14 02:30:04,079 INFO [collect_weather] running in process 398229
2025-11-14 02:30:04,080 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-11-14 02:30:04,089 INFO [collect_weather] publishing log messages to tcp://*:5562
2025-11-14 02:30:04,091 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2025-11-14 02:30:04,114 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/12/
2025-11-14 02:30:04,148 INFO [collect_NeahBay_ssh] running in process 398228
2025-11-14 02:30:04,149 INFO [collect_NeahBay_ssh] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-11-14 02:30:04,150 INFO [collect_NeahBay_ssh] publishing log messages to tcp://*:5563
2025-11-14 02:30:04,153 INFO [collect_NeahBay_ssh] connected to skookum.eos.ubc.ca port 5555
2025-11-14 02:30:04,154 INFO [collect_NeahBay_ssh] collecting Neah Bay ssh 00Z obs/forecast for 2025-11-14
2025-11-14 02:30:05,053 INFO [collect_river_data] running in process 398216
2025-11-14 02:30:05,054 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-11-14 02:30:05,055 INFO [collect_river_data] publishing log messages to tcp://*:5564
2025-11-14 02:30:05,056 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-11-14 02:30:05,056 INFO [collect_river_data] Collecting ECCC NicomeklLangley river data for 2025-11-13
2025-11-14 02:30:05,069 INFO [collect_river_data] running in process 398213
2025-11-14 02:30:05,070 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-11-14 02:30:05,071 INFO [collect_river_data] publishing log messages to tcp://*:5565
2025-11-14 02:30:05,072 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-11-14 02:30:05,072 INFO [collect_river_data] Collecting ECCC Fraser river data for 2025-11-13
2025-11-14 02:30:05,092 INFO [collect_river_data] running in process 398217
2025-11-14 02:30:05,093 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-11-14 02:30:05,094 INFO [collect_river_data] publishing log messages to tcp://*:5566
2025-11-14 02:30:05,094 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-11-14 02:30:05,095 INFO [collect_river_data] Collecting ECCC RobertsCreek river data for 2025-11-13
2025-11-14 02:30:05,102 INFO [collect_river_data] Appended ECCC Fraser river average discharge for 2025-11-13 to: /data/dlatorne/SOG-projects/SOG-forcing/ECget/Fraser_flow
2025-11-14 02:30:05,102 INFO [collect_river_data] ECCC Fraser river data collection for 2025-11-13 completed
2025-11-14 02:30:05,104 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-11-14 02:30:05,126 INFO [collect_river_data] running in process 398210
2025-11-14 02:30:05,127 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-11-14 02:30:05,128 INFO [collect_river_data] publishing log messages to tcp://*:5567
2025-11-14 02:30:05,130 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-11-14 02:30:05,131 INFO [collect_river_data] Collecting ECCC ChilliwackVedder river data for 2025-11-13
2025-11-14 02:30:05,143 INFO [collect_river_data] running in process 398208
2025-11-14 02:30:05,144 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-11-14 02:30:05,146 INFO [collect_river_data] publishing log messages to tcp://*:5568
2025-11-14 02:30:05,147 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-11-14 02:30:05,149 INFO [collect_river_data] Collecting ECCC Capilano river data for 2025-11-13
2025-11-14 02:30:05,156 INFO [collect_river_data] running in process 398219
2025-11-14 02:30:05,158 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-11-14 02:30:05,164 INFO [collect_river_data] Appended ECCC NicomeklLangley river average discharge for 2025-11-13 to: /results/forcing/rivers/observations/Nicomekl_Langley_flow
2025-11-14 02:30:05,166 INFO [collect_river_data] ECCC NicomeklLangley river data collection for 2025-11-13 completed
2025-11-14 02:30:05,167 INFO [collect_river_data] Appended ECCC ChilliwackVedder river average discharge for 2025-11-13 to: /results/forcing/rivers/observations/Chilliwack_Vedder_flow
2025-11-14 02:30:05,168 INFO [collect_river_data] ECCC ChilliwackVedder river data collection for 2025-11-13 completed
2025-11-14 02:30:05,169 INFO [collect_river_data] publishing log messages to tcp://*:5569
2025-11-14 02:30:05,171 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-11-14 02:30:05,171 INFO [collect_river_data] Collecting ECCC SanJuanPortRenfrew river data for 2025-11-13
2025-11-14 02:30:05,172 INFO [collect_river_data] running in process 398212
2025-11-14 02:30:05,173 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-11-14 02:30:05,174 INFO [collect_river_data] publishing log messages to tcp://*:5570
2025-11-14 02:30:05,176 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-11-14 02:30:05,177 INFO [collect_river_data] Collecting ECCC Englishman river data for 2025-11-13
2025-11-14 02:30:05,178 INFO [collect_river_data] Appended ECCC RobertsCreek river average discharge for 2025-11-13 to: /results/forcing/rivers/observations/RobertsCreek_flow
2025-11-14 02:30:05,179 INFO [collect_river_data] ECCC RobertsCreek river data collection for 2025-11-13 completed
2025-11-14 02:30:05,187 INFO [collect_river_data] running in process 398218
2025-11-14 02:30:05,188 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-11-14 02:30:05,189 INFO [collect_river_data] publishing log messages to tcp://*:5571
2025-11-14 02:30:05,189 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-11-14 02:30:05,190 INFO [collect_river_data] Collecting ECCC SalmonSayward river data for 2025-11-13
2025-11-14 02:30:05,192 INFO [collect_river_data] running in process 398214
2025-11-14 02:30:05,193 INFO [collect_river_data] running in process 398211
2025-11-14 02:30:05,194 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-11-14 02:30:05,195 INFO [collect_river_data] publishing log messages to tcp://*:5573
2025-11-14 02:30:05,196 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-11-14 02:30:05,197 INFO [collect_river_data] publishing log messages to tcp://*:5572
2025-11-14 02:30:05,198 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-11-14 02:30:05,199 INFO [collect_river_data] Collecting ECCC HomathkoMouth river data for 2025-11-13
2025-11-14 02:30:05,200 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-11-14 02:30:05,201 INFO [collect_river_data] Collecting ECCC ClowhomClowhomLake river data for 2025-11-13
2025-11-14 02:30:05,202 INFO [collect_river_data] Appended ECCC Englishman river average discharge for 2025-11-13 to: /data/dlatorne/SOG-projects/SOG-forcing/ECget/Englishman_flow
2025-11-14 02:30:05,202 INFO [collect_river_data] ECCC Englishman river data collection for 2025-11-13 completed
2025-11-14 02:30:05,205 INFO [collect_river_data] Appended ECCC SanJuanPortRenfrew river average discharge for 2025-11-13 to: /results/forcing/rivers/observations/SanJuan_PortRenfrew_flow
2025-11-14 02:30:05,206 INFO [collect_river_data] ECCC SanJuanPortRenfrew river data collection for 2025-11-13 completed
2025-11-14 02:30:05,211 INFO [collect_river_data] running in process 398220
2025-11-14 02:30:05,212 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-11-14 02:30:05,212 INFO [collect_river_data] publishing log messages to tcp://*:5574
2025-11-14 02:30:05,213 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-11-14 02:30:05,214 INFO [collect_river_data] Collecting ECCC SquamishBrackendale river data for 2025-11-13
2025-11-14 02:30:05,228 INFO [collect_river_data] Appended ECCC SalmonSayward river average discharge for 2025-11-13 to: /results/forcing/rivers/observations/Salmon_Sayward_flow
2025-11-14 02:30:05,230 INFO [collect_river_data] ECCC SalmonSayward river data collection for 2025-11-13 completed
2025-11-14 02:30:05,238 INFO [collect_river_data] Appended ECCC ClowhomClowhomLake river average discharge for 2025-11-13 to: /results/forcing/rivers/observations/Clowhom_ClowhomLake_flow
2025-11-14 02:30:05,239 INFO [collect_river_data] ECCC ClowhomClowhomLake river data collection for 2025-11-13 completed
2025-11-14 02:30:05,240 INFO [collect_river_data] Appended ECCC SquamishBrackendale river average discharge for 2025-11-13 to: /results/forcing/rivers/observations/Squamish_Brackendale_flow
2025-11-14 02:30:05,241 INFO [collect_river_data] ECCC SquamishBrackendale river data collection for 2025-11-13 completed
2025-11-14 02:30:05,242 INFO [collect_river_data] Appended ECCC HomathkoMouth river average discharge for 2025-11-13 to: /results/forcing/rivers/observations/Homathko_Mouth_flow
2025-11-14 02:30:05,243 INFO [collect_river_data] ECCC HomathkoMouth river data collection for 2025-11-13 completed
2025-11-14 02:30:05,268 INFO [collect_river_data] Appended ECCC Capilano river average discharge for 2025-11-13 to: /opp/observations/rivers/Capilano/Caplilano_08GA010_day_avg_flow
2025-11-14 02:30:05,269 INFO [collect_river_data] ECCC Capilano river data collection for 2025-11-13 completed
2025-11-14 02:30:05,274 INFO [collect_river_data] running in process 398223
2025-11-14 02:30:05,274 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-11-14 02:30:05,275 INFO [collect_river_data] publishing log messages to tcp://*:5575
2025-11-14 02:30:05,276 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-11-14 02:30:05,277 INFO [collect_river_data] Collecting ECCC TheodosiaDiversion river data for 2025-11-13
2025-11-14 02:30:05,278 INFO [collect_river_data] running in process 398221
2025-11-14 02:30:05,279 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-11-14 02:30:05,280 INFO [collect_river_data] publishing log messages to tcp://*:5576
2025-11-14 02:30:05,281 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-11-14 02:30:05,282 INFO [collect_river_data] Collecting ECCC TheodosiaScotty river data for 2025-11-13
2025-11-14 02:30:05,297 INFO [collect_river_data] Appended ECCC TheodosiaDiversion river average discharge for 2025-11-13 to: /results/forcing/rivers/observations/Theodosia_Diversion_flow
2025-11-14 02:30:05,298 INFO [collect_river_data] ECCC TheodosiaDiversion river data collection for 2025-11-13 completed
2025-11-14 02:30:05,303 INFO [collect_river_data] Appended ECCC TheodosiaScotty river average discharge for 2025-11-13 to: /results/forcing/rivers/observations/Theodosia_Scotty_flow
2025-11-14 02:30:05,304 INFO [collect_river_data] ECCC TheodosiaScotty river data collection for 2025-11-13 completed
2025-11-14 02:30:05,305 INFO [collect_river_data] running in process 398222
2025-11-14 02:30:05,306 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-11-14 02:30:05,306 INFO [collect_river_data] publishing log messages to tcp://*:5577
2025-11-14 02:30:05,307 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-11-14 02:30:05,308 INFO [collect_river_data] Collecting ECCC TheodosiaBypass river data for 2025-11-13
2025-11-14 02:30:05,338 INFO [collect_river_data] Appended ECCC TheodosiaBypass river average discharge for 2025-11-13 to: /results/forcing/rivers/observations/Theodosia_Bypass_flow
2025-11-14 02:30:05,339 INFO [collect_river_data] ECCC TheodosiaBypass river data collection for 2025-11-13 completed
2025-11-14 02:30:05,493 INFO [get_vfpa_hadcp] running in process 398227
2025-11-14 02:30:05,495 INFO [get_vfpa_hadcp] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-11-14 02:30:05,497 INFO [get_vfpa_hadcp] publishing log messages to tcp://*:5578
2025-11-14 02:30:05,497 INFO [get_vfpa_hadcp] connected to skookum.eos.ubc.ca port 5555
2025-11-14 02:30:05,499 INFO [get_vfpa_hadcp] processing VFPA HADCP data from 2nd Narrows Rail Bridge for 2025-11-13
2025-11-14 02:30:05,539 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-11-14 02:30:05,603 INFO [collect_NeahBay_ssh] 2025-11-14T00:00:00-08:00 Neah Bay ssh 00Z obs/forecast data collection complete
2025-11-14 02:30:05,714 INFO [crop_gribs] running in process 398230
2025-11-14 02:30:05,715 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-11-14 02:30:05,716 INFO [crop_gribs] publishing log messages to tcp://*:5579
2025-11-14 02:30:05,716 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2025-11-14 02:30:05,717 INFO [crop_gribs] cropping 2025-11-14 ECCC HRDPS 2.5km continental 12Z GRIB files to SalishSeaCast subdomain
2025-11-14 02:30:05,730 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20251114/12/
2025-11-14 02:30:05,963 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-11-14 02:30:06,314 INFO [get_onc_ctd] running in process 398225
2025-11-14 02:30:06,315 INFO [get_onc_ctd] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-11-14 02:30:06,318 INFO [get_onc_ctd] publishing log messages to tcp://*:5564
2025-11-14 02:30:06,319 INFO [get_onc_ctd] connected to skookum.eos.ubc.ca port 5555
2025-11-14 02:30:06,320 INFO [get_onc_ctd] requesting ONC SEVIP CTD T&S data for 2025-11-13
2025-11-14 02:30:06,332 INFO [get_onc_ferry] running in process 398226
2025-11-14 02:30:06,333 INFO [get_onc_ferry] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-11-14 02:30:06,334 INFO [get_onc_ferry] publishing log messages to tcp://*:5565
2025-11-14 02:30:06,335 INFO [get_onc_ferry] connected to skookum.eos.ubc.ca port 5555
2025-11-14 02:30:06,336 INFO [get_onc_ferry] requesting ONC TWDP.N1 NAV data for 2025-11-13
2025-11-14 02:30:06,351 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-11-14 02:30:06,382 INFO [get_onc_ctd] running in process 398224
2025-11-14 02:30:06,383 INFO [get_onc_ctd] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-11-14 02:30:06,385 INFO [get_onc_ctd] publishing log messages to tcp://*:5590
2025-11-14 02:30:06,386 INFO [get_onc_ctd] connected to skookum.eos.ubc.ca port 5555
2025-11-14 02:30:06,386 INFO [get_onc_ctd] requesting ONC SCVIP CTD T&S data for 2025-11-13
2025-11-14 02:30:06,411 INFO [grib_to_netcdf] running in process 398465
2025-11-14 02:30:06,413 INFO [grib_to_netcdf] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-11-14 02:30:06,414 INFO [grib_to_netcdf] publishing log messages to tcp://*:5591
2025-11-14 02:30:06,415 INFO [grib_to_netcdf] connected to skookum.eos.ubc.ca port 5555
2025-11-14 02:30:06,416 INFO [grib_to_netcdf] creating NEMO-atmos forcing files for 2025-11-14 forecast2 run
2025-11-14 02:30:06,681 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-11-14 02:30:07,005 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-11-14 02:30:07,075 INFO [get_onc_ferry] requesting ONC TWDP.N2 NAV data for 2025-11-13
2025-11-14 02:30:07,346 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-11-14 02:30:07,357 ERROR [get_onc_ferry] request for ONC TWDP.N2 NAV data for 2025-11-13 failed: 400 Client Error:  for url: https://data.oceannetworks.ca/api/scalardata?method=getByLocation&token=8efe44eb-0fb5-4413-bba3-ae3dbe0fbb1d&locationCode=TWDP.N2&deviceCategoryCode=NAV&sensorCategoryCodes=longitude%2Clatitude&dateFrom=2025-11-13T00:00:00.000Z&dateTo=2025-11-13T23:59:00.000Z&resampleType=avg&resamplePeriod=1
2025-11-14 02:30:07,685 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-11-14 02:30:07,854 CRITICAL [get_onc_ferry] 2025-11-13 ONC TWDP ferry data file creation failed
2025-11-14 02:30:08,121 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-11-14 02:30:08,398 INFO [get_vfpa_hadcp] added VFPA HADCP data from 2nd Narrows Rail Bridge for 2025-11-13 to /opp/observations/AISDATA/netcdf/VFPA_2ND_NARROWS_HADCP_2s_202511.nc
2025-11-14 02:30:08,399 INFO [get_vfpa_hadcp] VFPA HADCP observations added to 2025-11 netcdf file
2025-11-14 02:30:08,469 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-11-14 02:30:08,767 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-11-14 02:30:09,070 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-11-14 02:30:09,410 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-11-14 02:30:09,767 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-11-14 02:30:10,090 INFO [manager] checklist updated with [Neah Bay ssh data] items from collect_NeahBay_ssh worker
2025-11-14 02:30:10,426 INFO [manager] launching NextWorker(module='nowcast.workers.make_ssh_files', args=['forecast2', '--run-date', '2025-11-14'], host='localhost')
2025-11-14 02:30:10,434 INFO [manager] checklist updated with [VFPA HADCP data] items from get_vfpa_hadcp worker
2025-11-14 02:30:10,826 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['VFPA-HADCP'], host='localhost')
2025-11-14 02:30:12,083 INFO [ping_erddap] running in process 398739
2025-11-14 02:30:12,084 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-11-14 02:30:12,085 INFO [ping_erddap] publishing log messages to tcp://*:5563
2025-11-14 02:30:12,086 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2025-11-14 02:30:12,088 INFO [ping_erddap] VFPA-HADCP ERDDAP dataset flag file(s) created
2025-11-14 02:30:12,090 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2025-11-14 02:30:14,200 INFO [make_ssh_files] running in process 398726
2025-11-14 02:30:14,201 INFO [make_ssh_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-11-14 02:30:14,202 INFO [make_ssh_files] publishing log messages to tcp://*:5563
2025-11-14 02:30:14,203 INFO [make_ssh_files] connected to skookum.eos.ubc.ca port 5555
2025-11-14 02:30:14,204 INFO [make_ssh_files] building forecast2 sea surface height boundary conditions file(s) from 2025-11-14 NOAA Neah Bay 00Z observation and forecast values
2025-11-14 02:30:16,516 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2025m11d09.nc
2025-11-14 02:30:16,772 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2025m11d10.nc
2025-11-14 02:30:17,045 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2025m11d11.nc
2025-11-14 02:30:17,250 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2025m11d12.nc
2025-11-14 02:30:17,430 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2025m11d13.nc
2025-11-14 02:30:17,642 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2025m11d14.nc
2025-11-14 02:30:17,846 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2025m11d15.nc
2025-11-14 02:30:18,062 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2025m11d16.nc
2025-11-14 02:30:18,243 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2025m11d17.nc
2025-11-14 02:30:18,371 INFO [make_ssh_files] sea surface height boundary file for 2025-11-14 forecast2 run created
2025-11-14 02:30:18,375 INFO [manager] checklist updated with [sea surface height forcing] items from make_ssh_files worker
2025-11-14 02:30:18,592 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2025m11d15.nc
2025-11-14 02:30:18,725 INFO [manager] launching NextWorker(module='nowcast.workers.make_runoff_file', args=['v202108'], host='localhost')
2025-11-14 02:30:18,731 INFO [manager] launching NextWorker(module='nowcast.workers.make_201702_runoff_file', args=[], host='localhost')
2025-11-14 02:30:20,040 INFO [make_201702_runoff_file] running in process 398795
2025-11-14 02:30:20,042 INFO [make_201702_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-11-14 02:30:20,043 INFO [make_201702_runoff_file] publishing log messages to tcp://*:5563
2025-11-14 02:30:20,044 INFO [make_201702_runoff_file] connected to skookum.eos.ubc.ca port 5555
2025-11-14 02:30:20,079 INFO [make_201702_runoff_file] calculating runoff forcing for b201702 bathymetry
2025-11-14 02:30:20,181 INFO [make_201702_runoff_file] 2025-11-14 runoff file creation from Fraser at Hope and climatology elsewhere complete
2025-11-14 02:30:20,183 INFO [manager] checklist updated with [rivers forcing] items from make_201702_runoff_file worker
2025-11-14 02:30:20,580 INFO [make_runoff_file] running in process 398793
2025-11-14 02:30:20,581 INFO [make_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-11-14 02:30:20,583 INFO [make_runoff_file] publishing log messages to tcp://*:5565
2025-11-14 02:30:20,584 INFO [make_runoff_file] connected to skookum.eos.ubc.ca port 5555
2025-11-14 02:30:20,585 INFO [make_runoff_file] calculating NEMO runoff forcing for v202108 bathymetry for 2025-11-13
2025-11-14 02:30:21,599 ERROR [make_runoff_file] no 2025-11-13 discharge obs for SkagitMountVernon: patching
2025-11-14 02:30:21,837 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2025m11d16.nc
2025-11-14 02:30:21,844 INFO [grib_to_netcdf] 2025-11-14 NEMO-atmos forcing file for forecast2 created
2025-11-14 02:30:21,846 INFO [manager] checklist updated with [weather forcing] items from grib_to_netcdf worker
2025-11-14 02:30:22,127 ERROR [make_runoff_file] no 2025-11-13 discharge obs for SnohomishMonroe: patching
2025-11-14 02:30:22,483 ERROR [make_runoff_file] no 2025-11-13 discharge obs for NisquallyMcKenna: patching
2025-11-14 02:30:22,589 ERROR [make_runoff_file] no 2025-11-13 discharge obs for GreenwaterGreenwater: patching
2025-11-14 02:30:23,837 INFO [make_runoff_file] stored NEMO runoff forcing for v202108 bathymetry for 2025-11-13: /results/forcing/rivers/R202108Dailies_y2025m11d13.nc
2025-11-14 02:30:23,839 INFO [make_runoff_file] 2025-11-13 runoff file creation completed
2025-11-14 02:30:23,841 INFO [manager] checklist updated with [rivers forcing] items from make_runoff_file worker
2025-11-14 02:30:24,185 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'forecast2'], host='localhost')
2025-11-14 02:30:24,191 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'forecast2'], host='localhost')
2025-11-14 02:30:24,195 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.graham', 'forecast2'], host='localhost')
2025-11-14 02:30:24,199 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'forecast2'], host='localhost')
2025-11-14 02:30:25,001 INFO [get_onc_ctd] 2025-11-13 ONC SCVIP CTD T&S file created
2025-11-14 02:30:25,003 INFO [manager] checklist updated with [ONC CTD data] items from get_onc_ctd worker
2025-11-14 02:30:25,361 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['SCVIP-CTD'], host='localhost')
2025-11-14 02:30:25,396 INFO [get_onc_ctd] 2025-11-13 ONC SEVIP CTD T&S file created
2025-11-14 02:30:25,398 INFO [manager] checklist updated with [ONC CTD data] items from get_onc_ctd worker
2025-11-14 02:30:25,613 INFO [upload_forcing] running in process 398837
2025-11-14 02:30:25,614 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-11-14 02:30:25,615 INFO [upload_forcing] publishing log messages to tcp://*:5563
2025-11-14 02:30:25,616 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-11-14 02:30:25,622 INFO [upload_forcing] running in process 398835
2025-11-14 02:30:25,623 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-11-14 02:30:25,624 INFO [upload_forcing] publishing log messages to tcp://*:5565
2025-11-14 02:30:25,625 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-11-14 02:30:25,647 INFO [upload_forcing] running in process 398836
2025-11-14 02:30:25,648 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-11-14 02:30:25,649 INFO [upload_forcing] publishing log messages to tcp://*:5566
2025-11-14 02:30:25,650 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-11-14 02:30:25,674 INFO [upload_forcing] running in process 398838
2025-11-14 02:30:25,675 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-11-14 02:30:25,676 INFO [upload_forcing] publishing log messages to tcp://*:5567
2025-11-14 02:30:25,677 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-11-14 02:30:25,769 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['SEVIP-CTD'], host='localhost')
2025-11-14 02:30:26,434 INFO [ping_erddap] running in process 398847
2025-11-14 02:30:26,435 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-11-14 02:30:26,436 INFO [ping_erddap] publishing log messages to tcp://*:5564
2025-11-14 02:30:26,437 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2025-11-14 02:30:26,439 INFO [ping_erddap] SCVIP-CTD ERDDAP dataset flag file(s) created
2025-11-14 02:30:26,441 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2025-11-14 02:30:26,758 INFO [ping_erddap] running in process 398861
2025-11-14 02:30:26,760 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-11-14 02:30:26,761 INFO [ping_erddap] publishing log messages to tcp://*:5568
2025-11-14 02:30:26,762 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2025-11-14 02:30:26,763 INFO [ping_erddap] SEVIP-CTD ERDDAP dataset flag file(s) created
2025-11-14 02:30:26,820 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2025-11-14 02:30:27,132 INFO [upload_forcing] LiveOcean boundary conditions file not found; created symlink to /results/forcing/LiveOcean/boundary_conditions/LiveOcean_v201905_y2025m11d13.nc
2025-11-14 02:30:27,191 INFO [upload_forcing] forecast2 2025-11-14 forcing files upload to optimum-hindcast completed
2025-11-14 02:30:27,194 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-11-14 02:30:28,786 CRITICAL [upload_forcing] unhandled exception:
Traceback (most recent call last):
  File "/SalishSeaCast/NEMO_Nowcast/nemo_nowcast/worker.py", line 387, in _do_work
    checklist = self.worker_func(
        self._parsed_args, self.config, self.tell_manager
    )
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/upload_forcing.py", line 94, in upload_forcing
    ssh_client, sftp_client = ssh_sftp.sftp(host_name, ssh_key)
                              ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/ssh_sftp.py", line 144, in sftp
    ssh_client = ssh(host, key_filename, ssh_config_file)
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/ssh_sftp.py", line 75, in ssh
    ssh_client.connect(
    ~~~~~~~~~~~~~~~~~~^
        host["hostname"],
        ^^^^^^^^^^^^^^^^^
    ...<3 lines>...
        look_for_keys=False,
        ^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/paramiko/client.py", line 384, in connect
    sock.connect(addr)
    ~~~~~~~~~~~~^^^^^^
OSError: [Errno 101] Network is unreachable
2025-11-14 02:30:29,303 INFO [upload_forcing] forecast2 2025-11-14 forcing files upload to arbutus.cloud-nowcast completed
2025-11-14 02:30:29,306 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-11-14 02:30:29,647 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2025-11-14'], host='localhost')
2025-11-14 02:30:30,949 INFO [make_forcing_links] running in process 398883
2025-11-14 02:30:30,950 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-11-14 02:30:30,951 INFO [make_forcing_links] publishing log messages to tcp://*:5563
2025-11-14 02:30:30,953 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2025-11-14 02:30:32,190 INFO [make_forcing_links] forecast2 2025-11-14 forcing file links on arbutus.cloud-nowcast created
2025-11-14 02:30:32,194 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2025-11-14 02:30:32,299 INFO [upload_forcing] forecast2 2025-11-14 forcing files upload to orcinus-nowcast-agrif completed
2025-11-14 02:30:32,547 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2025-11-14'], host='arbutus.cloud-nowcast')
2025-11-14 02:30:32,555 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-11-14 02:30:34,669 INFO [run_NEMO] running in process 13809
2025-11-14 02:30:34,671 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-11-14 02:30:34,672 INFO [run_NEMO] publishing log messages to tcp://*:5556
2025-11-14 02:30:34,734 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2025-11-14 02:30:35,791 WARNING [run_NEMO] There are uncommitted changes in /nemoShare/MEOPAR/nowcast-sys/SS-run-sets
2025-11-14 02:30:36,098 INFO [run_NEMO] forecast2: launching /nemoShare/MEOPAR/nowcast-sys/runs/13nov25forecast2_2025-11-14T023034.818583-0800/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2025-11-14 02:30:36,110 INFO [run_NEMO] forecast2 NEMO run for 2025-11-14 on arbutus.cloud-nowcast started
2025-11-14 02:30:36,121 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2025-11-14 02:30:36,466 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'forecast2'], host='arbutus.cloud-nowcast')
2025-11-14 02:30:38,461 INFO [watch_NEMO] running in process 14082
2025-11-14 02:30:38,463 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-11-14 02:30:38,464 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2025-11-14 02:30:38,465 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2025-11-14 02:30:38,533 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2025-11-14 02:35:38,532 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 18839969 = 2025-11-15 04:58:40 UTC, 16.6% complete
2025-11-14 02:40:38,534 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 18840447 = 2025-11-15 10:17:20 UTC, 34.3% complete
2025-11-14 02:45:38,535 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 18840930 = 2025-11-15 15:39:20 UTC, 52.2% complete
2025-11-14 02:50:38,535 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 18841413 = 2025-11-15 21:01:20 UTC, 70.1% complete
2025-11-14 02:55:38,537 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 18841883 = 2025-11-16 02:14:40 UTC, 87.5% complete
2025-11-14 03:00:38,541 INFO [watch_NEMO] forecast2 NEMO run on arbutus.cloud-nowcast completed
2025-11-14 03:00:38,546 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2025-11-14 03:00:39,056 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_wind_file', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2025-11-14'], host='arbutus.cloud-nowcast')
2025-11-14 03:00:39,067 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_current_file', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2025-11-14'], host='arbutus.cloud-nowcast')
2025-11-14 03:00:39,071 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2025-11-13'], host='localhost')
2025-11-14 03:00:40,309 INFO [download_results] running in process 399219
2025-11-14 03:00:40,310 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-11-14 03:00:40,311 INFO [download_results] publishing log messages to tcp://*:5563
2025-11-14 03:00:40,313 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2025-11-14 03:00:40,314 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/forecast2/13nov25 to /results/SalishSea/forecast2.202111
2025-11-14 03:00:42,229 INFO [make_ww3_current_file] running in process 14603
2025-11-14 03:00:42,230 INFO [make_ww3_current_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-11-14 03:00:42,232 INFO [make_ww3_current_file] publishing log messages to tcp://*:5571
2025-11-14 03:00:42,233 INFO [make_ww3_current_file] connected to skookum.eos.ubc.ca port 5555
2025-11-14 03:00:42,235 INFO [make_ww3_current_file] Creating wwatch3 currents forcing file for 2025-11-14 forecast2 run
2025-11-14 03:00:51,891 INFO [make_ww3_wind_file] wwatch3 wind forcing file created on arbutus.cloud-nowcast for 2025-11-14 forecast2 run
2025-11-14 03:00:51,895 INFO [manager] checklist updated with [WW3 wind forcing] items from make_ww3_wind_file worker
2025-11-14 03:02:40,551 INFO [download_results] forecast2 2025-11-13 results files from arbutus.cloud-nowcast downloaded
2025-11-14 03:02:40,556 INFO [manager] checklist updated with [results files] items from download_results worker
2025-11-14 03:02:40,879 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['forecast2', '--run-date', '2025-11-13'], host='localhost')
2025-11-14 03:02:42,709 INFO [make_CHS_currents_file] running in process 399250
2025-11-14 03:02:42,710 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-11-14 03:02:42,711 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5563
2025-11-14 03:02:42,712 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555
2025-11-14 03:03:23,356 INFO [make_CHS_currents_file] Made CHS currents file for 2025-11-13 for forecast2
2025-11-14 03:03:23,359 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker
2025-11-14 03:03:23,695 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['nemo', 'forecast2', '--run-date', '2025-11-13'], host='localhost')
2025-11-14 03:03:24,865 INFO [update_forecast_datasets] running in process 399287
2025-11-14 03:03:24,867 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-11-14 03:03:24,868 INFO [update_forecast_datasets] publishing log messages to tcp://*:5563
2025-11-14 03:03:24,869 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2025-11-14 03:03:24,871 INFO [update_forecast_datasets] updating nemo forecast directory for forecast2/13nov25 run
2025-11-14 03:04:19,208 INFO [make_ww3_current_file] wwatch3 currents forcing file created on arbutus.cloud-nowcast for 2025-11-14 forecast2 run
2025-11-14 03:04:19,213 INFO [manager] checklist updated with [WW3 currents forcing] items from make_ww3_current_file worker
2025-11-14 03:04:19,554 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2025-11-14'], host='arbutus.cloud-nowcast')
2025-11-14 03:04:21,588 INFO [run_ww3] running in process 15168
2025-11-14 03:04:21,589 INFO [run_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-11-14 03:04:21,590 INFO [run_ww3] publishing log messages to tcp://*:5572
2025-11-14 03:04:21,591 INFO [run_ww3] connected to skookum.eos.ubc.ca port 5555
2025-11-14 03:04:21,596 INFO [run_ww3] Created run directory /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast2_2025-11-14T030421.574430-0800
2025-11-14 03:04:21,597 INFO [run_ww3] forecast2: launching /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast2_2025-11-14T030421.574430-0800/SoGWW3.sh on arbutus.cloud-nowcast
2025-11-14 03:04:21,606 INFO [run_ww3] forecast2 WaveWatch3 run for 2025-11-14 on arbutus.cloud-nowcast started
2025-11-14 03:04:21,610 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker
2025-11-14 03:04:21,947 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'forecast2'], host='arbutus.cloud-nowcast')
2025-11-14 03:04:23,923 INFO [watch_ww3] running in process 15255
2025-11-14 03:04:23,925 INFO [watch_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-11-14 03:04:23,926 INFO [watch_ww3] publishing log messages to tcp://*:5573
2025-11-14 03:04:23,927 INFO [watch_ww3] connected to skookum.eos.ubc.ca port 5555
2025-11-14 03:04:23,962 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2025-11-14 03:07:37,127 INFO [update_forecast_datasets] updated nemo forecast directory for forecast2/13nov25 run: /results/SalishSea/rolling-forecasts/nemo
2025-11-14 03:07:37,128 INFO [update_forecast_datasets] nemo 2025-11-13 forecast2 rolling forecast datasets updated
2025-11-14 03:07:37,131 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2025-11-14 03:07:37,483 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nemo-forecast'], host='localhost')
2025-11-14 03:07:37,489 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'forecast2', 'publish', '--run-date', '2025-11-13'], host='localhost')
2025-11-14 03:07:37,496 INFO [manager] launching NextWorker(module='nowcast.workers.make_surface_current_tiles', args=['forecast2', '--run-date', '2025-11-13'], host='localhost')
2025-11-14 03:07:38,815 INFO [ping_erddap] running in process 399426
2025-11-14 03:07:38,816 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-11-14 03:07:38,817 INFO [ping_erddap] publishing log messages to tcp://*:5563
2025-11-14 03:07:38,818 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2025-11-14 03:07:38,825 INFO [ping_erddap] nemo-forecast ERDDAP dataset flag file(s) created
2025-11-14 03:07:38,827 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2025-11-14 03:07:40,064 INFO [make_surface_current_tiles] running in process 399429
2025-11-14 03:07:40,065 INFO [make_surface_current_tiles] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-11-14 03:07:40,066 INFO [make_surface_current_tiles] publishing log messages to tcp://*:5563
2025-11-14 03:07:40,067 INFO [make_surface_current_tiles] connected to skookum.eos.ubc.ca port 5555
2025-11-14 03:07:40,098 INFO [make_surface_current_tiles] starting rendering of tiles for 2025-11-13 forecast2 in 6 processes into /results/nowcast-sys/figures/surface_currents/forecast2/13nov25
2025-11-14 03:07:41,010 INFO [make_plots] running in process 399428
2025-11-14 03:07:41,011 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-11-14 03:07:41,012 INFO [make_plots] publishing log messages to tcp://*:5564
2025-11-14 03:07:41,013 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2025-11-14 03:07:41,687 INFO [make_plots] preparing render list for 2025-11-13 NEMO forecast2 publish figures
2025-11-14 03:07:41,934 INFO [make_plots] added Website_thumbnail figure to 2025-11-13 NEMO forecast2 publish render list
2025-11-14 03:07:41,935 INFO [make_plots] added Threshold_website figure to 2025-11-13 NEMO forecast2 publish render list
2025-11-14 03:07:41,936 INFO [make_plots] added PA_tidal_predictions figure to 2025-11-13 NEMO forecast2 publish render list
2025-11-14 03:07:41,936 INFO [make_plots] added Boundary Bay figure to 2025-11-13 NEMO forecast2 publish render list
2025-11-14 03:07:41,937 INFO [make_plots] added Campbell River figure to 2025-11-13 NEMO forecast2 publish render list
2025-11-14 03:07:41,938 INFO [make_plots] added Cherry Point figure to 2025-11-13 NEMO forecast2 publish render list
2025-11-14 03:07:41,938 INFO [make_plots] added Friday Harbor figure to 2025-11-13 NEMO forecast2 publish render list
2025-11-14 03:07:41,939 INFO [make_plots] added Halfmoon Bay figure to 2025-11-13 NEMO forecast2 publish render list
2025-11-14 03:07:41,940 INFO [make_plots] added Nanaimo figure to 2025-11-13 NEMO forecast2 publish render list
2025-11-14 03:07:41,940 INFO [make_plots] added Neah Bay figure to 2025-11-13 NEMO forecast2 publish render list
2025-11-14 03:07:41,941 INFO [make_plots] added New Westminster figure to 2025-11-13 NEMO forecast2 publish render list
2025-11-14 03:07:41,942 INFO [make_plots] added Patricia Bay figure to 2025-11-13 NEMO forecast2 publish render list
2025-11-14 03:07:41,942 INFO [make_plots] added Point Atkinson figure to 2025-11-13 NEMO forecast2 publish render list
2025-11-14 03:07:41,943 INFO [make_plots] added Port Renfrew figure to 2025-11-13 NEMO forecast2 publish render list
2025-11-14 03:07:41,944 INFO [make_plots] added Sand Heads figure to 2025-11-13 NEMO forecast2 publish render list
2025-11-14 03:07:41,944 INFO [make_plots] added Sandy Cove figure to 2025-11-13 NEMO forecast2 publish render list
2025-11-14 03:07:41,945 INFO [make_plots] added Squamish figure to 2025-11-13 NEMO forecast2 publish render list
2025-11-14 03:07:41,945 INFO [make_plots] added Victoria figure to 2025-11-13 NEMO forecast2 publish render list
2025-11-14 03:07:41,946 INFO [make_plots] added Woodwards Landing figure to 2025-11-13 NEMO forecast2 publish render list
2025-11-14 03:07:41,947 INFO [make_plots] starting to render nemo forecast2 publish 13nov25 figures
2025-11-14 03:09:23,958 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 66, 12.2% complete
2025-11-14 03:14:23,954 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 138, 25.6% complete
2025-11-14 03:15:15,947 WARNING [make_plots] invalid station number for Boundary Bay station: None
2025-11-14 03:15:15,947 WARNING [make_plots] can't resolve a valid CHS station code for Boundary Bay
2025-11-14 03:15:40,628 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb996/data for station 08074 Campbell River from 2025-11-09 00:00:00Z to 2025-11-16 05:55:00Z
2025-11-14 03:16:49,698 WARNING [make_plots] invalid station number for Halfmoon Bay station: None
2025-11-14 03:16:49,699 WARNING [make_plots] can't resolve a valid CHS station code for Halfmoon Bay
2025-11-14 03:17:11,229 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb96d/data for station 07917 Nanaimo from 2025-11-09 00:00:00Z to 2025-11-16 05:55:00Z
2025-11-14 03:17:56,199 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e13d0f4a073c4bbf93/data for station 07654 New Westminster from 2025-11-09 00:00:00Z to 2025-11-16 05:55:00Z
2025-11-14 03:18:19,170 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd2d/data for station 07277 Patricia Bay from 2025-11-09 00:00:00Z to 2025-11-16 05:55:00Z
2025-11-14 03:18:38,119 INFO [make_surface_current_tiles] starting PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast2/13nov25
2025-11-14 03:18:39,136 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94c/data for station 07795 Point Atkinson from 2025-11-09 00:00:00Z to 2025-11-16 05:55:00Z
2025-11-14 03:18:56,748 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e23d0f4a073c4bc060/data for station 08525 Port Renfrew from 2025-11-09 00:00:00Z to 2025-11-16 05:55:00Z
2025-11-14 03:19:13,107 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb93c/data for station 07594 Sand Heads from 2025-11-09 00:00:00Z to 2025-11-16 05:55:00Z
2025-11-14 03:19:23,953 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 225, 41.7% complete
2025-11-14 03:19:31,398 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc43b/data for station 07786 Sandy Cove from 2025-11-09 00:00:00Z to 2025-11-16 05:55:00Z
2025-11-14 03:19:47,970 WARNING [make_plots] invalid station number for Squamish station: None
2025-11-14 03:19:47,971 WARNING [make_plots] can't resolve a valid CHS station code for Squamish
2025-11-14 03:20:05,827 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd1e/data for station 07120 Victoria from 2025-11-09 00:00:00Z to 2025-11-16 05:55:00Z
2025-11-14 03:20:23,343 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc42d/data for station 07610 Woodwards Landing from 2025-11-09 00:00:00Z to 2025-11-16 05:55:00Z
2025-11-14 03:20:40,014 INFO [make_plots] finished rendering nemo forecast2 publish 13nov25 figures
2025-11-14 03:20:40,017 INFO [make_plots] nemo publish plots for 2025-11-13 forecast2 completed
2025-11-14 03:20:40,024 INFO [manager] checklist updated with [plots] items from make_plots worker
2025-11-14 03:20:40,316 INFO [manager] launching NextWorker(module='nowcast.workers.make_feeds', args=['forecast2', '--run-date', '2025-11-13'], host='localhost')
2025-11-14 03:20:43,434 INFO [make_feeds] running in process 404383
2025-11-14 03:20:43,435 INFO [make_feeds] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-11-14 03:20:43,436 INFO [make_feeds] publishing log messages to tcp://*:5564
2025-11-14 03:20:43,437 INFO [make_feeds] connected to skookum.eos.ubc.ca port 5555
2025-11-14 03:20:57,219 INFO [make_feeds] ATOM feeds for 2025-11-13 forecast2 run completed
2025-11-14 03:20:57,223 INFO [manager] checklist updated with [storm surge alert feeds] items from make_feeds worker
2025-11-14 03:20:57,560 INFO [manager] launching NextWorker(module='nemo_nowcast.workers.clear_checklist', args=[], host='localhost')
2025-11-14 03:20:58,546 INFO [clear_checklist] running in process 404404
2025-11-14 03:20:58,548 INFO [clear_checklist] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-11-14 03:20:58,549 INFO [clear_checklist] publishing log messages to tcp://*:5564
2025-11-14 03:20:58,550 INFO [clear_checklist] connected to skookum.eos.ubc.ca port 5555
2025-11-14 03:20:58,552 INFO [clear_checklist] requesting that manager clear system state checklist
2025-11-14 03:20:58,554 INFO [manager] writing checklist to log file
2025-11-14 03:20:58,594 INFO [manager] checklist cleared
2025-11-14 03:20:58,596 INFO [clear_checklist] nowcast system checklist cleared
2025-11-14 03:20:58,598 INFO [manager] launching NextWorker(module='nemo_nowcast.workers.rotate_logs', args=[], host='localhost')
2025-11-14 03:20:59,738 INFO [rotate_logs] running in process 404409
2025-11-14 03:20:59,740 INFO [rotate_logs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-11-14 03:20:59,741 INFO [rotate_logs] publishing log messages to tcp://*:5564
2025-11-14 03:20:59,742 INFO [rotate_logs] connected to skookum.eos.ubc.ca port 5555
2025-11-14 03:20:59,742 INFO [rotate_logs] rotating log files