2025-12-13 02:44:15,103 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/nowcast.log
2025-12-13 02:44:15,108 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/nowcast.debug.log
2025-12-13 02:44:15,113 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/checklist.log
2025-12-13 02:44:15,114 INFO [rotate_logs] log files rotated
2025-12-13 02:44:15,118 INFO [manager] checklist updated with [log rotation] items from rotate_logs worker
2025-12-13 02:45:47,083 INFO [make_surface_current_tiles] finished PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast2/12dec25
2025-12-13 02:45:47,103 INFO [make_surface_current_tiles] finished rendering of tiles for 2025-12-12 forecast2 into /results/nowcast-sys/figures/surface_currents/forecast2/12dec25
2025-12-13 02:45:47,104 INFO [make_surface_current_tiles] surface current tile figures for 2025-12-12 forecast2 completed
2025-12-13 02:45:47,357 INFO [manager] checklist updated with [surface current tiles] items from make_surface_current_tiles worker
2025-12-13 02:46:06,328 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 399, 73.9% complete
2025-12-13 02:51:06,328 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 498, 92.2% complete
2025-12-13 02:56:06,329 INFO [watch_ww3] forecast2 WWATCH3 run on arbutus.cloud-nowcast completed
2025-12-13 02:56:06,378 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2025-12-13 02:56:06,780 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2025-12-13'], host='localhost')
2025-12-13 02:56:08,361 INFO [download_wwatch3_results] running in process 1342321
2025-12-13 02:56:08,363 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-13 02:56:08,364 INFO [download_wwatch3_results] publishing log messages to tcp://*:5564
2025-12-13 02:56:08,365 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2025-12-13 02:56:45,602 INFO [download_wwatch3_results] forecast2 2025-12-13 results files from arbutus.cloud-nowcast downloaded
2025-12-13 02:56:45,605 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2025-12-13 02:56:45,756 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['wwatch3', 'forecast2', '--run-date', '2025-12-13'], host='localhost')
2025-12-13 02:56:47,211 INFO [update_forecast_datasets] running in process 1342334
2025-12-13 02:56:47,212 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-13 02:56:47,214 INFO [update_forecast_datasets] publishing log messages to tcp://*:5564
2025-12-13 02:56:47,215 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2025-12-13 02:56:47,216 INFO [update_forecast_datasets] updating wwatch3 most_recent_forecast directory from forecast2/13dec25 run
2025-12-13 02:56:47,219 INFO [update_forecast_datasets] updated wwatch3 most_recent_forecast directory from forecast2/13dec25 run
2025-12-13 02:56:47,220 INFO [update_forecast_datasets] updating wwatch3 forecast directory for forecast2/13dec25 run
2025-12-13 02:57:19,740 INFO [update_forecast_datasets] updated wwatch3 forecast directory for forecast2/13dec25 run: /results/SalishSea/rolling-forecasts/wwatch3
2025-12-13 02:57:19,741 INFO [update_forecast_datasets] wwatch3 2025-12-13 forecast2 rolling forecast datasets updated
2025-12-13 02:57:19,745 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2025-12-13 02:57:19,895 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['wwatch3-forecast'], host='localhost')
2025-12-13 02:57:21,390 INFO [ping_erddap] running in process 1342347
2025-12-13 02:57:21,414 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-13 02:57:21,432 INFO [ping_erddap] publishing log messages to tcp://*:5564
2025-12-13 02:57:21,453 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2025-12-13 02:57:21,458 INFO [ping_erddap] wwatch3-forecast ERDDAP dataset flag file(s) created
2025-12-13 02:57:21,461 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2025-12-13 02:57:21,569 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['wwatch3', 'forecast2', 'publish', '--run-date', '2025-12-13'], host='localhost')
2025-12-13 02:57:26,354 INFO [make_plots] running in process 1342352
2025-12-13 02:57:26,355 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-13 02:57:26,357 INFO [make_plots] publishing log messages to tcp://*:5564
2025-12-13 02:57:26,358 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2025-12-13 02:57:26,359 INFO [make_plots] preparing render list for 2025-12-13 WaveWatch3 forecast2 publish figures
2025-12-13 02:57:26,360 INFO [make_plots] added Halibut Bank figure to 2025-12-13 WaveWatch3 forecast2 publish render list
2025-12-13 02:57:26,360 INFO [make_plots] added Sentry Shoal figure to 2025-12-13 WaveWatch3 forecast2 publish render list
2025-12-13 02:57:26,361 INFO [make_plots] starting to render wwatch3 forecast2 publish 13dec25 figures
2025-12-13 02:57:27,333 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46146 Halibut Bank
2025-12-13 02:57:35,388 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46131 Sentry Shoal
2025-12-13 02:57:40,216 INFO [make_plots] finished rendering wwatch3 forecast2 publish 13dec25 figures
2025-12-13 02:57:40,216 INFO [make_plots] wwatch3 publish plots for 2025-12-13 forecast2 completed
2025-12-13 02:57:40,220 INFO [manager] checklist updated with [plots] items from make_plots worker
2025-12-13 07:47:22,706 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/12/ to /results/forcing/atmospheric/continental2.5/GRIB/20251213/12/
2025-12-13 07:47:22,708 INFO [collect_weather] 2025-12-13 2.5km weather forecast 12 collection complete
2025-12-13 07:47:22,712 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2025-12-13 07:47:23,045 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'SkagitMountVernon', '--data-date', '2025-12-12'], host='localhost')
2025-12-13 07:47:23,055 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'SnohomishMonroe', '--data-date', '2025-12-12'], host='localhost')
2025-12-13 07:47:23,061 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'NisquallyMcKenna', '--data-date', '2025-12-12'], host='localhost')
2025-12-13 07:47:23,067 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'GreenwaterGreenwater', '--data-date', '2025-12-12'], host='localhost')
2025-12-13 07:47:23,074 INFO [manager] launching NextWorker(module='nowcast.workers.make_turbidity_file', args=[], host='localhost')
2025-12-13 07:47:23,083 INFO [manager] launching NextWorker(module='nowcast.workers.collect_NeahBay_ssh', args=['06'], host='localhost')
2025-12-13 07:47:23,090 INFO [manager] launching NextWorker(module='nowcast.workers.download_live_ocean', args=[], host='localhost')
2025-12-13 07:47:23,095 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['18', '2.5km'], host='localhost')
2025-12-13 07:47:23,101 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['18'], host='localhost')
2025-12-13 07:47:25,331 INFO [collect_weather] running in process 1345294
2025-12-13 07:47:25,332 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-13 07:47:25,333 INFO [collect_weather] publishing log messages to tcp://*:5563
2025-12-13 07:47:25,334 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2025-12-13 07:47:25,335 INFO [collect_NeahBay_ssh] running in process 1345292
2025-12-13 07:47:25,337 INFO [collect_NeahBay_ssh] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-13 07:47:25,337 INFO [collect_NeahBay_ssh] publishing log messages to tcp://*:5564
2025-12-13 07:47:25,338 INFO [collect_NeahBay_ssh] connected to skookum.eos.ubc.ca port 5555
2025-12-13 07:47:25,339 INFO [collect_NeahBay_ssh] collecting Neah Bay ssh 06Z obs/forecast for 2025-12-13
2025-12-13 07:47:25,356 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/18/
2025-12-13 07:47:25,370 INFO [download_live_ocean] running in process 1345293
2025-12-13 07:47:25,371 INFO [download_live_ocean] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-13 07:47:25,372 INFO [download_live_ocean] publishing log messages to tcp://*:5565
2025-12-13 07:47:25,373 INFO [download_live_ocean] connected to skookum.eos.ubc.ca port 5555
2025-12-13 07:47:25,374 INFO [download_live_ocean] downloading Salish Sea western boundary day-averaged LiveOcean file for 2025-12-13
2025-12-13 07:47:25,886 INFO [collect_river_data] running in process 1345289
2025-12-13 07:47:25,895 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-13 07:47:25,896 INFO [collect_river_data] publishing log messages to tcp://*:5566
2025-12-13 07:47:25,897 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-12-13 07:47:25,898 INFO [collect_river_data] Collecting USGS NisquallyMcKenna river data for 2025-12-12
2025-12-13 07:47:25,899 INFO [collect_river_data] running in process 1345286
2025-12-13 07:47:25,900 INFO [collect_river_data] running in process 1345290
2025-12-13 07:47:25,901 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-13 07:47:25,902 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-13 07:47:25,902 INFO [collect_river_data] publishing log messages to tcp://*:5567
2025-12-13 07:47:25,903 INFO [collect_river_data] publishing log messages to tcp://*:5568
2025-12-13 07:47:25,904 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-12-13 07:47:25,904 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-12-13 07:47:25,905 INFO [collect_river_data] Collecting USGS SkagitMountVernon river data for 2025-12-12
2025-12-13 07:47:25,906 INFO [collect_river_data] Collecting USGS GreenwaterGreenwater river data for 2025-12-12
2025-12-13 07:47:25,928 INFO [collect_river_data] running in process 1345288
2025-12-13 07:47:25,929 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-13 07:47:25,931 INFO [collect_river_data] publishing log messages to tcp://*:5569
2025-12-13 07:47:25,936 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-12-13 07:47:25,939 INFO [collect_river_data] Collecting USGS SnohomishMonroe river data for 2025-12-12
2025-12-13 07:47:26,011 INFO [download_live_ocean] downloaded 10197485 bytes from https://s3.kopah.uw.edu/liveocean-share/f2025.12.13/ubc.nc to /results/forcing/LiveOcean/downloaded/20251213/low_passed_UBC.nc
2025-12-13 07:47:26,013 INFO [download_live_ocean] Deflating in up to 1 concurrent sub-processes
2025-12-13 07:47:26,099 INFO [make_turbidity_file] running in process 1345291
2025-12-13 07:47:26,100 INFO [make_turbidity_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-13 07:47:26,101 INFO [make_turbidity_file] publishing log messages to tcp://*:5570
2025-12-13 07:47:26,102 INFO [make_turbidity_file] connected to skookum.eos.ubc.ca port 5555
2025-12-13 07:47:26,103 INFO [make_turbidity_file] Creating Fraser River turbidity forcing file for 2025-12-13
2025-12-13 07:47:26,457 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12150800&startDT=2025-12-12&endDT=2025-12-12 "HTTP/1.1 200 "
2025-12-13 07:47:26,458 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12089500&startDT=2025-12-12&endDT=2025-12-12 "HTTP/1.1 200 "
2025-12-13 07:47:26,464 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12200500&startDT=2025-12-12&endDT=2025-12-12 "HTTP/1.1 200 "
2025-12-13 07:47:26,464 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12097500&startDT=2025-12-12&endDT=2025-12-12 "HTTP/1.1 200 "
2025-12-13 07:47:26,483 INFO [collect_river_data] Appended USGS NisquallyMcKenna river average discharge for 2025-12-12 to: /results/forcing/rivers/observations/Nisqually_McKenna_flow
2025-12-13 07:47:26,484 INFO [collect_river_data] USGS NisquallyMcKenna river data collection for 2025-12-12 completed
2025-12-13 07:47:26,486 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-12-13 07:47:26,488 INFO [collect_river_data] Appended USGS SnohomishMonroe river average discharge for 2025-12-12 to: /results/forcing/rivers/observations/Snohomish_Monroe_flow
2025-12-13 07:47:26,488 INFO [collect_river_data] USGS SnohomishMonroe river data collection for 2025-12-12 completed
2025-12-13 07:47:26,493 INFO [collect_river_data] Appended USGS SkagitMountVernon river average discharge for 2025-12-12 to: /results/forcing/rivers/observations/Skagit_MountVernon_flow
2025-12-13 07:47:26,493 INFO [collect_river_data] USGS SkagitMountVernon river data collection for 2025-12-12 completed
2025-12-13 07:47:26,514 INFO [crop_gribs] running in process 1345295
2025-12-13 07:47:26,519 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-13 07:47:26,520 INFO [crop_gribs] publishing log messages to tcp://*:5571
2025-12-13 07:47:26,520 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2025-12-13 07:47:26,521 INFO [crop_gribs] cropping 2025-12-13 ECCC HRDPS 2.5km continental 18Z GRIB files to SalishSeaCast subdomain
2025-12-13 07:47:26,530 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20251213/18/
2025-12-13 07:47:26,532 INFO [collect_river_data] Appended USGS GreenwaterGreenwater river average discharge for 2025-12-12 to: /results/forcing/rivers/observations/Greenwater_Greenwater_flow
2025-12-13 07:47:26,533 INFO [collect_river_data] USGS GreenwaterGreenwater river data collection for 2025-12-12 completed
2025-12-13 07:47:26,640 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-12-13 07:47:26,794 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-12-13 07:47:26,934 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-12-13 07:47:27,031 INFO [download_live_ocean] netCDF4 deflated /results/forcing/LiveOcean/downloaded/20251213/low_passed_UBC.nc
2025-12-13 07:47:27,032 INFO [download_live_ocean] 2025-12-13 Live Ocean file for Salish Sea western boundary downloaded
2025-12-13 07:47:27,088 INFO [manager] checklist updated with [Live Ocean products] items from download_live_ocean worker
2025-12-13 07:47:27,192 INFO [collect_NeahBay_ssh] 2025-12-13T00:00:00-08:00 Neah Bay ssh 06Z obs/forecast data collection complete
2025-12-13 07:47:27,398 INFO [manager] launching NextWorker(module='nowcast.workers.make_live_ocean_files', args=['--run-date', '2025-12-13'], host='localhost')
2025-12-13 07:47:27,405 INFO [manager] checklist updated with [Neah Bay ssh data] items from collect_NeahBay_ssh worker
2025-12-13 07:47:27,551 INFO [manager] launching NextWorker(module='nowcast.workers.make_ssh_files', args=['nowcast', '--run-date', '2025-12-13'], host='localhost')
2025-12-13 07:47:28,868 INFO [make_turbidity_file] 2025-12-13 Fraser River turbidity file creation complete
2025-12-13 07:47:28,869 INFO [manager] checklist updated with [Fraser River turbidity] items from make_turbidity_file worker
2025-12-13 07:47:30,869 INFO [make_ssh_files] running in process 1345423
2025-12-13 07:47:30,870 INFO [make_ssh_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-13 07:47:30,872 INFO [make_ssh_files] publishing log messages to tcp://*:5564
2025-12-13 07:47:30,873 INFO [make_ssh_files] connected to skookum.eos.ubc.ca port 5555
2025-12-13 07:47:30,874 INFO [make_ssh_files] building nowcast sea surface height boundary conditions file(s) from 2025-12-13 NOAA Neah Bay 06Z observation and forecast values
2025-12-13 07:47:30,876 INFO [make_live_ocean_files] running in process 1345421
2025-12-13 07:47:30,877 INFO [make_live_ocean_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-13 07:47:30,878 INFO [make_live_ocean_files] publishing log messages to tcp://*:5565
2025-12-13 07:47:30,879 INFO [make_live_ocean_files] connected to skookum.eos.ubc.ca port 5555
2025-12-13 07:47:30,880 INFO [make_live_ocean_files] Creating T&S western boundary conditions file from 2025-12-13 Live Ocean run
2025-12-13 07:47:32,584 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2025m12d09.nc
2025-12-13 07:47:32,750 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2025m12d10.nc
2025-12-13 07:47:32,915 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2025m12d11.nc
2025-12-13 07:47:33,296 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2025m12d12.nc
2025-12-13 07:47:33,520 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2025m12d13.nc
2025-12-13 07:47:33,718 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2025m12d14.nc
2025-12-13 07:47:33,915 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2025m12d15.nc
2025-12-13 07:47:34,119 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2025m12d16.nc
2025-12-13 07:47:34,274 INFO [make_ssh_files] sea surface height boundary file for 2025-12-13 nowcast run created
2025-12-13 07:47:34,278 INFO [manager] checklist updated with [sea surface height forcing] items from make_ssh_files worker
2025-12-13 07:47:34,405 INFO [manager] launching NextWorker(module='nowcast.workers.make_runoff_file', args=['v202108'], host='localhost')
2025-12-13 07:47:34,410 INFO [manager] launching NextWorker(module='nowcast.workers.make_201702_runoff_file', args=[], host='localhost')
2025-12-13 07:47:35,698 INFO [make_201702_runoff_file] running in process 1345494
2025-12-13 07:47:35,699 INFO [make_201702_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-13 07:47:35,701 INFO [make_201702_runoff_file] publishing log messages to tcp://*:5564
2025-12-13 07:47:35,702 INFO [make_201702_runoff_file] connected to skookum.eos.ubc.ca port 5555
2025-12-13 07:47:35,808 INFO [make_201702_runoff_file] calculating runoff forcing for b201702 bathymetry
2025-12-13 07:47:35,971 INFO [make_201702_runoff_file] 2025-12-13 runoff file creation from Fraser at Hope and climatology elsewhere complete
2025-12-13 07:47:35,973 INFO [manager] checklist updated with [rivers forcing] items from make_201702_runoff_file worker
2025-12-13 07:47:36,377 INFO [make_runoff_file] running in process 1345492
2025-12-13 07:47:36,378 INFO [make_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-13 07:47:36,379 INFO [make_runoff_file] publishing log messages to tcp://*:5566
2025-12-13 07:47:36,380 INFO [make_runoff_file] connected to skookum.eos.ubc.ca port 5555
2025-12-13 07:47:36,383 INFO [make_runoff_file] calculating NEMO runoff forcing for v202108 bathymetry for 2025-12-12
2025-12-13 07:47:39,304 INFO [make_runoff_file] stored NEMO runoff forcing for v202108 bathymetry for 2025-12-12: /results/forcing/rivers/R202108Dailies_y2025m12d12.nc
2025-12-13 07:47:39,305 INFO [make_runoff_file] 2025-12-12 runoff file creation completed
2025-12-13 07:47:39,307 INFO [manager] checklist updated with [rivers forcing] items from make_runoff_file worker
2025-12-13 07:48:13,472 INFO [make_live_ocean_files] Stored T&S western boundary conditions file: /results/forcing/LiveOcean/boundary_conditions/LiveOcean_v201905_y2025m12d13.nc
2025-12-13 07:48:13,473 INFO [make_live_ocean_files] 2025-12-13 Live Ocean western boundary conditions files created
2025-12-13 07:48:13,475 INFO [manager] checklist updated with [Live Ocean boundary conditions] items from make_live_ocean_files worker
2025-12-13 09:51:43,844 INFO [crop_gribs] crop_gribs 12 has watched for 8h; retrying remaining 1 file(s)
2025-12-13 09:51:44,750 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20251213/12/
2025-12-13 09:51:44,751 INFO [crop_gribs] 2025-12-13 12 GRIBs cropping complete
2025-12-13 09:51:44,754 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2025-12-13 09:51:45,082 INFO [manager] launching NextWorker(module='nowcast.workers.grib_to_netcdf', args=['nowcast+'], host='localhost')
2025-12-13 09:51:47,635 INFO [grib_to_netcdf] running in process 1347841
2025-12-13 09:51:47,636 INFO [grib_to_netcdf] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-13 09:51:47,638 INFO [grib_to_netcdf] publishing log messages to tcp://*:5562
2025-12-13 09:51:47,639 INFO [grib_to_netcdf] connected to skookum.eos.ubc.ca port 5555
2025-12-13 09:51:47,640 INFO [grib_to_netcdf] creating NEMO-atmos forcing files for 2025-12-13 nowcast and forecast runs
2025-12-13 09:51:59,529 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/hrdps_y2025m12d13.nc
2025-12-13 09:52:08,332 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2025m12d14.nc
2025-12-13 09:52:13,220 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2025m12d15.nc
2025-12-13 09:52:13,234 INFO [grib_to_netcdf] 2025-12-13 NEMO-atmos forcing file for nowcast+ created
2025-12-13 09:52:13,237 INFO [manager] checklist updated with [weather forcing] items from grib_to_netcdf worker
2025-12-13 09:52:13,396 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'nowcast+'], host='localhost')
2025-12-13 09:52:13,401 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'nowcast+'], host='localhost')
2025-12-13 09:52:13,406 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.nibi', 'nowcast+'], host='localhost')
2025-12-13 09:52:13,411 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'nowcast+'], host='localhost')
2025-12-13 09:52:13,415 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['weather'], host='localhost')
2025-12-13 09:52:14,601 INFO [ping_erddap] running in process 1347901
2025-12-13 09:52:14,602 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-13 09:52:14,603 INFO [ping_erddap] publishing log messages to tcp://*:5562
2025-12-13 09:52:14,604 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2025-12-13 09:52:14,605 INFO [ping_erddap] weather ERDDAP dataset flag file(s) created
2025-12-13 09:52:14,607 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2025-12-13 09:52:14,904 INFO [upload_forcing] running in process 1347900
2025-12-13 09:52:14,905 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-13 09:52:14,906 INFO [upload_forcing] publishing log messages to tcp://*:5564
2025-12-13 09:52:14,909 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-12-13 09:52:14,910 INFO [upload_forcing] running in process 1347899
2025-12-13 09:52:14,911 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-13 09:52:14,912 INFO [upload_forcing] publishing log messages to tcp://*:5565
2025-12-13 09:52:14,913 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-12-13 09:52:14,940 INFO [upload_forcing] running in process 1347896
2025-12-13 09:52:14,941 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-13 09:52:14,942 INFO [upload_forcing] publishing log messages to tcp://*:5566
2025-12-13 09:52:14,943 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-12-13 09:52:14,961 INFO [upload_forcing] running in process 1347898
2025-12-13 09:52:14,962 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-13 09:52:14,963 INFO [upload_forcing] publishing log messages to tcp://*:5567
2025-12-13 09:52:14,964 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-12-13 09:52:16,841 INFO [upload_forcing] nowcast+ 2025-12-13 forcing files upload to optimum-hindcast completed
2025-12-13 09:52:16,844 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-12-13 09:52:21,987 INFO [upload_forcing] nowcast+ 2025-12-13 forcing files upload to arbutus.cloud-nowcast completed
2025-12-13 09:52:21,992 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-12-13 09:52:22,143 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'nowcast+', '--run-date', '2025-12-13'], host='localhost')
2025-12-13 09:52:22,218 INFO [upload_forcing] nowcast+ 2025-12-13 forcing files upload to orcinus-nowcast-agrif completed
2025-12-13 09:52:22,221 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-12-13 09:52:23,461 INFO [make_forcing_links] running in process 1347936
2025-12-13 09:52:23,462 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-13 09:52:23,463 INFO [make_forcing_links] publishing log messages to tcp://*:5562
2025-12-13 09:52:23,465 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2025-12-13 09:52:24,644 INFO [make_forcing_links] nowcast+ 2025-12-13 forcing file links on arbutus.cloud-nowcast created
2025-12-13 09:52:24,648 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2025-12-13 09:52:24,800 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2025-12-13'], host='arbutus.cloud-nowcast')
2025-12-13 09:52:26,876 INFO [run_NEMO] running in process 31391
2025-12-13 09:52:26,879 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-12-13 09:52:26,881 INFO [run_NEMO] publishing log messages to tcp://*:5556
2025-12-13 09:52:26,994 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2025-12-13 09:52:27,587 WARNING [run_NEMO] There are uncommitted changes in /nemoShare/MEOPAR/nowcast-sys/SS-run-sets
2025-12-13 09:52:28,207 INFO [run_NEMO] nowcast: launching /nemoShare/MEOPAR/nowcast-sys/runs/13dec25nowcast_2025-12-13T095226.931788-0800/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2025-12-13 09:52:28,220 INFO [run_NEMO] nowcast NEMO run for 2025-12-13 on arbutus.cloud-nowcast started
2025-12-13 09:52:28,224 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2025-12-13 09:52:28,378 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'nowcast'], host='arbutus.cloud-nowcast')
2025-12-13 09:52:30,306 INFO [watch_NEMO] running in process 31665
2025-12-13 09:52:30,309 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-12-13 09:52:30,310 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2025-12-13 09:52:30,311 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2025-12-13 09:52:30,374 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2025-12-13 09:53:16,484 INFO [upload_forcing] nowcast+ 2025-12-13 forcing files upload to robot.nibi completed
2025-12-13 09:53:16,489 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-12-13 09:57:30,374 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 18900464 = 2025-12-13 05:08:40 UTC, 21.4% complete
2025-12-13 10:02:30,381 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 18900956 = 2025-12-13 10:36:40 UTC, 44.2% complete
2025-12-13 10:07:30,380 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 18901454 = 2025-12-13 16:08:40 UTC, 67.3% complete
2025-12-13 10:12:30,382 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 18901960 = 2025-12-13 21:46:00 UTC, 90.7% complete
2025-12-13 10:17:30,385 INFO [watch_NEMO] nowcast NEMO run on arbutus.cloud-nowcast completed
2025-12-13 10:17:30,391 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2025-12-13 10:17:30,724 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'ssh', '--run-date', '2025-12-13'], host='localhost')
2025-12-13 10:17:30,731 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2025-12-13'], host='localhost')
2025-12-13 10:17:31,950 INFO [download_results] running in process 1348171
2025-12-13 10:17:31,951 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-13 10:17:31,953 INFO [download_results] publishing log messages to tcp://*:5562
2025-12-13 10:17:31,954 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2025-12-13 10:17:31,956 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/nowcast/13dec25 to /results/SalishSea/nowcast-blue.202111
2025-12-13 10:17:32,101 INFO [make_forcing_links] running in process 1348170
2025-12-13 10:17:32,102 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-13 10:17:32,103 INFO [make_forcing_links] publishing log messages to tcp://*:5564
2025-12-13 10:17:32,105 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2025-12-13 10:17:33,123 INFO [make_forcing_links] ssh 2025-12-13 forcing file links on arbutus.cloud-nowcast created
2025-12-13 10:17:33,127 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2025-12-13 10:17:33,307 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2025-12-13'], host='arbutus.cloud-nowcast')
2025-12-13 10:17:35,412 INFO [run_NEMO] running in process 32166
2025-12-13 10:17:35,415 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-12-13 10:17:35,416 INFO [run_NEMO] publishing log messages to tcp://*:5556
2025-12-13 10:17:35,417 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2025-12-13 10:17:36,145 WARNING [run_NEMO] There are uncommitted changes in /nemoShare/MEOPAR/nowcast-sys/SS-run-sets
2025-12-13 10:17:36,467 INFO [run_NEMO] forecast: launching /nemoShare/MEOPAR/nowcast-sys/runs/13dec25forecast_2025-12-13T101735.481778-0800/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2025-12-13 10:17:36,477 INFO [run_NEMO] forecast NEMO run for 2025-12-13 on arbutus.cloud-nowcast started
2025-12-13 10:17:36,480 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2025-12-13 10:17:36,643 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'forecast'], host='arbutus.cloud-nowcast')
2025-12-13 10:17:38,782 INFO [watch_NEMO] running in process 32441
2025-12-13 10:17:38,783 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-12-13 10:17:38,784 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2025-12-13 10:17:38,786 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2025-12-13 10:17:38,871 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2025-12-13 10:19:03,642 INFO [download_results] nowcast 2025-12-13 results files from arbutus.cloud-nowcast downloaded
2025-12-13 10:19:03,646 INFO [manager] checklist updated with [results files] items from download_results worker
2025-12-13 10:19:03,798 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast', 'research', '--run-date', '2025-12-13'], host='localhost')
2025-12-13 10:19:03,806 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast', 'comparison', '--run-date', '2025-12-12'], host='localhost')
2025-12-13 10:19:03,812 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['nowcast', '--run-date', '2025-12-13'], host='localhost')
2025-12-13 10:19:05,789 INFO [make_CHS_currents_file] running in process 1348198
2025-12-13 10:19:05,790 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-13 10:19:05,790 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5562
2025-12-13 10:19:05,791 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555
2025-12-13 10:19:07,496 INFO [make_plots] running in process 1348197
2025-12-13 10:19:07,497 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-13 10:19:07,499 INFO [make_plots] publishing log messages to tcp://*:5564
2025-12-13 10:19:07,500 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2025-12-13 10:19:07,509 INFO [make_plots] running in process 1348195
2025-12-13 10:19:07,511 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-13 10:19:07,512 INFO [make_plots] publishing log messages to tcp://*:5565
2025-12-13 10:19:07,513 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2025-12-13 10:19:08,029 INFO [make_plots] preparing render list for 2025-12-12 NEMO nowcast-blue comparison figures
2025-12-13 10:19:08,031 INFO [make_plots] preparing render list for 2025-12-13 NEMO nowcast-blue research figures
2025-12-13 10:19:08,097 INFO [make_plots] added SH_wind figure to 2025-12-12 NEMO nowcast-blue comparison render list
2025-12-13 10:19:08,099 INFO [make_plots] added Compare_VENUS_East figure to 2025-12-12 NEMO nowcast-blue comparison render list
2025-12-13 10:19:08,100 INFO [make_plots] added Compare_VENUS_Central figure to 2025-12-12 NEMO nowcast-blue comparison render list
2025-12-13 10:19:08,101 INFO [make_plots] starting to render nemo nowcast comparison 12dec25 figures
2025-12-13 10:19:09,409 INFO [make_plots] added salinity_thalweg_and_surface figures to 2025-12-13 NEMO nowcast-blue research render list
2025-12-13 10:19:10,406 INFO [make_plots] added temperature_thalweg_and_surface figures to 2025-12-13 NEMO nowcast-blue research render list
2025-12-13 10:19:10,407 INFO [make_plots] added Currents_sections_and_surface figure to 2025-12-13 NEMO nowcast-blue research render list
2025-12-13 10:19:10,408 INFO [make_plots] added Currents_at_VENUS_Central figure to 2025-12-13 NEMO nowcast-blue research render list
2025-12-13 10:19:10,409 INFO [make_plots] added Currents_at_VENUS_East figure to 2025-12-13 NEMO nowcast-blue research render list
2025-12-13 10:19:10,411 INFO [make_plots] starting to render nemo nowcast research 13dec25 figures
2025-12-13 10:19:40,959 INFO [make_CHS_currents_file] Made CHS currents file for 2025-12-13 for nowcast
2025-12-13 10:19:40,963 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker
2025-12-13 10:20:16,668 INFO [make_plots] finished rendering nemo nowcast research 13dec25 figures
2025-12-13 10:20:16,670 INFO [make_plots] nemo research plots for 2025-12-13 nowcast completed
2025-12-13 10:20:16,682 INFO [manager] checklist updated with [plots] items from make_plots worker
2025-12-13 10:20:37,244 INFO [make_plots] finished rendering nemo nowcast comparison 12dec25 figures
2025-12-13 10:20:37,246 INFO [make_plots] nemo comparison plots for 2025-12-12 nowcast completed
2025-12-13 10:20:37,251 INFO [manager] checklist updated with [plots] items from make_plots worker
2025-12-13 10:22:38,874 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 18902614 = 2025-12-14 05:02:00 UTC, 14.0% complete
2025-12-13 10:27:38,880 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 18903122 = 2025-12-14 10:40:40 UTC, 29.7% complete
2025-12-13 10:32:38,886 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 18903628 = 2025-12-14 16:18:00 UTC, 45.3% complete
2025-12-13 10:37:38,890 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 18904138 = 2025-12-14 21:58:00 UTC, 61.0% complete
2025-12-13 10:42:38,893 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 18904647 = 2025-12-15 03:37:20 UTC, 76.8% complete
2025-12-13 10:47:38,895 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 18905153 = 2025-12-15 09:14:40 UTC, 92.4% complete
2025-12-13 10:52:38,895 INFO [watch_NEMO] forecast NEMO run on arbutus.cloud-nowcast completed
2025-12-13 10:52:38,899 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2025-12-13 10:52:39,221 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_wind_file', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2025-12-13'], host='arbutus.cloud-nowcast')
2025-12-13 10:52:39,234 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_current_file', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2025-12-13'], host='arbutus.cloud-nowcast')
2025-12-13 10:52:39,238 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'turbidity', '--run-date', '2025-12-13'], host='localhost')
2025-12-13 10:52:39,245 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'turbidity', '--run-date', '2025-12-13'], host='localhost')
2025-12-13 10:52:39,252 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.nibi', 'turbidity', '--run-date', '2025-12-13'], host='localhost')
2025-12-13 10:52:39,258 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'turbidity', '--run-date', '2025-12-13'], host='localhost')
2025-12-13 10:52:39,264 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2025-12-13'], host='localhost')
2025-12-13 10:52:40,526 INFO [download_results] running in process 1348533
2025-12-13 10:52:40,529 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-13 10:52:40,530 INFO [download_results] publishing log messages to tcp://*:5562
2025-12-13 10:52:40,531 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2025-12-13 10:52:40,532 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/forecast/13dec25 to /results/SalishSea/forecast.202111
2025-12-13 10:52:40,627 INFO [upload_forcing] running in process 1348532
2025-12-13 10:52:40,628 INFO [upload_forcing] running in process 1348531
2025-12-13 10:52:40,630 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-13 10:52:40,631 INFO [upload_forcing] publishing log messages to tcp://*:5565
2025-12-13 10:52:40,632 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-12-13 10:52:40,633 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-13 10:52:40,634 INFO [upload_forcing] publishing log messages to tcp://*:5564
2025-12-13 10:52:40,635 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-12-13 10:52:40,636 INFO [upload_forcing] running in process 1348529
2025-12-13 10:52:40,638 INFO [upload_forcing] running in process 1348530
2025-12-13 10:52:40,639 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-13 10:52:40,640 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-13 10:52:40,641 INFO [upload_forcing] publishing log messages to tcp://*:5566
2025-12-13 10:52:40,641 INFO [upload_forcing] publishing log messages to tcp://*:5567
2025-12-13 10:52:40,642 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-12-13 10:52:40,643 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-12-13 10:52:41,305 INFO [upload_forcing] turbidity 2025-12-13 forcing files upload to optimum-hindcast completed
2025-12-13 10:52:41,310 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-12-13 10:52:41,812 INFO [upload_forcing] turbidity 2025-12-13 forcing files upload to arbutus.cloud-nowcast completed
2025-12-13 10:52:41,816 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-12-13 10:52:41,982 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2025-12-13'], host='localhost')
2025-12-13 10:52:42,550 INFO [make_ww3_wind_file] running in process 666
2025-12-13 10:52:42,552 INFO [make_ww3_wind_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-12-13 10:52:42,553 INFO [make_ww3_wind_file] publishing log messages to tcp://*:5570
2025-12-13 10:52:42,619 INFO [make_ww3_wind_file] connected to skookum.eos.ubc.ca port 5555
2025-12-13 10:52:42,620 INFO [make_ww3_wind_file] Creating wwatch3 wind forcing file for 2025-12-13 forecast run
2025-12-13 10:52:42,663 INFO [make_ww3_current_file] running in process 664
2025-12-13 10:52:42,664 INFO [make_ww3_current_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-12-13 10:52:42,665 INFO [make_ww3_current_file] publishing log messages to tcp://*:5571
2025-12-13 10:52:42,666 INFO [make_ww3_current_file] connected to skookum.eos.ubc.ca port 5555
2025-12-13 10:52:42,667 INFO [make_ww3_current_file] Creating wwatch3 currents forcing file for 2025-12-13 forecast run
2025-12-13 10:52:43,465 INFO [make_forcing_links] running in process 1348563
2025-12-13 10:52:43,466 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-13 10:52:43,467 INFO [make_forcing_links] publishing log messages to tcp://*:5564
2025-12-13 10:52:43,469 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2025-12-13 10:52:44,618 INFO [make_forcing_links] nowcast-green 2025-12-13 forcing file links on arbutus.cloud-nowcast created
2025-12-13 10:52:44,623 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2025-12-13 10:52:44,814 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2025-12-13'], host='arbutus.cloud-nowcast')
2025-12-13 10:52:46,757 INFO [upload_forcing] turbidity 2025-12-13 forcing files upload to orcinus-nowcast-agrif completed
2025-12-13 10:52:46,762 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-12-13 10:52:46,926 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['orcinus-nowcast-agrif', 'nowcast-agrif', '--run-date', '2025-12-13'], host='localhost')
2025-12-13 10:52:47,675 INFO [run_NEMO] running in process 1011
2025-12-13 10:52:47,676 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-12-13 10:52:47,677 INFO [run_NEMO] publishing log messages to tcp://*:5556
2025-12-13 10:52:47,678 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2025-12-13 10:52:48,317 INFO [make_forcing_links] running in process 1348572
2025-12-13 10:52:48,318 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-13 10:52:48,319 INFO [make_forcing_links] publishing log messages to tcp://*:5564
2025-12-13 10:52:48,321 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2025-12-13 10:52:49,897 WARNING [run_NEMO] There are uncommitted changes in /nemoShare/MEOPAR/nowcast-sys/SS-run-sets
2025-12-13 10:52:50,490 INFO [make_forcing_links] nowcast-agrif 2025-12-13 forcing file links on orcinus-nowcast-agrif created
2025-12-13 10:52:50,495 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2025-12-13 10:52:50,689 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_agrif', args=['orcinus-nowcast-agrif', 'nowcast-agrif', '--run-date', '2025-12-13'], host='localhost')
2025-12-13 10:52:50,940 INFO [run_NEMO] nowcast-green: launching /nemoShare/MEOPAR/nowcast-sys/runs/13dec25nowcast-green_2025-12-13T105247.761836-0800/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2025-12-13 10:52:51,013 INFO [run_NEMO] nowcast-green NEMO run for 2025-12-13 on arbutus.cloud-nowcast started
2025-12-13 10:52:51,016 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2025-12-13 10:52:51,181 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'nowcast-green'], host='arbutus.cloud-nowcast')
2025-12-13 10:52:52,148 INFO [run_NEMO_agrif] running in process 1348579
2025-12-13 10:52:52,149 INFO [run_NEMO_agrif] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-13 10:52:52,150 INFO [run_NEMO_agrif] publishing log messages to tcp://*:5564
2025-12-13 10:52:52,151 INFO [run_NEMO_agrif] connected to skookum.eos.ubc.ca port 5555
2025-12-13 10:52:53,042 INFO [make_ww3_wind_file] wwatch3 wind forcing file created on arbutus.cloud-nowcast for 2025-12-13 forecast run
2025-12-13 10:52:53,045 INFO [manager] checklist updated with [WW3 wind forcing] items from make_ww3_wind_file worker
2025-12-13 10:52:53,955 INFO [watch_NEMO] running in process 1620
2025-12-13 10:52:53,959 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-12-13 10:52:53,960 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2025-12-13 10:52:53,961 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2025-12-13 10:52:54,035 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2025-12-13 10:52:57,895 CRITICAL [run_NEMO_agrif] unhandled exception:
Traceback (most recent call last):
  File "/SalishSeaCast/NEMO_Nowcast/nemo_nowcast/worker.py", line 387, in _do_work
    checklist = self.worker_func(
        self._parsed_args, self.config, self.tell_manager
    )
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/run_NEMO_agrif.py", line 109, in run_NEMO_agrif
    prev_run_namelists_info = _get_prev_run_namelists_info(
        sftp_client, host_name, run_date.shift(days=-1), config
    )
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/run_NEMO_agrif.py", line 155, in _get_prev_run_namelists_info
    sftp_client.get(os.fspath(prev_namelist_cfg), namelist_cfg.name)
    ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/paramiko/sftp_client.py", line 840, in get
    size = self.getfo(
        remotepath,
    ...<3 lines>...
        max_concurrent_prefetch_requests,
    )
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/paramiko/sftp_client.py", line 795, in getfo
    file_size = self.stat(remotepath).st_size
                ~~~~~~~~~^^^^^^^^^^^^
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/paramiko/sftp_client.py", line 493, in stat
    t, msg = self._request(CMD_STAT, path)
             ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/paramiko/sftp_client.py", line 857, in _request
    return self._read_response(num)
           ~~~~~~~~~~~~~~~~~~~^^^^^
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/paramiko/sftp_client.py", line 909, in _read_response
    self._convert_status(msg)
    ~~~~~~~~~~~~~~~~~~~~^^^^^
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/paramiko/sftp_client.py", line 938, in _convert_status
    raise IOError(errno.ENOENT, text)
FileNotFoundError: [Errno 2] No such file
2025-12-13 10:53:07,171 INFO [upload_forcing] turbidity 2025-12-13 forcing files upload to robot.nibi completed
2025-12-13 10:53:07,175 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-12-13 10:54:52,730 INFO [download_results] forecast 2025-12-13 results files from arbutus.cloud-nowcast downloaded
2025-12-13 10:54:52,735 INFO [manager] checklist updated with [results files] items from download_results worker
2025-12-13 10:54:52,895 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['forecast', '--run-date', '2025-12-13'], host='localhost')
2025-12-13 10:54:55,313 INFO [make_CHS_currents_file] running in process 1348605
2025-12-13 10:54:55,314 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-13 10:54:55,315 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5562
2025-12-13 10:54:55,316 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555
2025-12-13 10:55:17,598 INFO [make_ww3_current_file] wwatch3 currents forcing file created on arbutus.cloud-nowcast for 2025-12-13 forecast run
2025-12-13 10:55:17,602 INFO [manager] checklist updated with [WW3 currents forcing] items from make_ww3_current_file worker
2025-12-13 10:55:17,757 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2025-12-13'], host='arbutus.cloud-nowcast')
2025-12-13 10:55:19,741 INFO [run_ww3] running in process 1764
2025-12-13 10:55:19,742 INFO [run_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-12-13 10:55:19,743 INFO [run_ww3] publishing log messages to tcp://*:5572
2025-12-13 10:55:19,744 INFO [run_ww3] connected to skookum.eos.ubc.ca port 5555
2025-12-13 10:55:19,749 INFO [run_ww3] Created run directory /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/nowcast_2025-12-13T105519.734914-0800
2025-12-13 10:55:19,750 INFO [run_ww3] nowcast: launching /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/nowcast_2025-12-13T105519.734914-0800/SoGWW3.sh on arbutus.cloud-nowcast
2025-12-13 10:55:19,759 INFO [run_ww3] nowcast WaveWatch3 run for 2025-12-13 on arbutus.cloud-nowcast started
2025-12-13 10:55:19,763 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker
2025-12-13 10:55:19,915 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'nowcast'], host='arbutus.cloud-nowcast')
2025-12-13 10:55:21,913 INFO [watch_ww3] running in process 1854
2025-12-13 10:55:21,913 INFO [watch_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-12-13 10:55:21,914 INFO [watch_ww3] publishing log messages to tcp://*:5573
2025-12-13 10:55:21,916 INFO [watch_ww3] connected to skookum.eos.ubc.ca port 5555
2025-12-13 10:55:21,947 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2025-12-13 10:55:43,967 INFO [make_CHS_currents_file] Made CHS currents file for 2025-12-13 for forecast
2025-12-13 10:55:43,970 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker
2025-12-13 10:55:44,133 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['nemo', 'forecast', '--run-date', '2025-12-13'], host='localhost')
2025-12-13 10:55:45,329 INFO [update_forecast_datasets] running in process 1348643
2025-12-13 10:55:45,331 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-13 10:55:45,332 INFO [update_forecast_datasets] publishing log messages to tcp://*:5562
2025-12-13 10:55:45,333 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2025-12-13 10:55:45,335 INFO [update_forecast_datasets] updating nemo forecast directory for forecast/13dec25 run
2025-12-13 10:55:45,753 INFO [update_forecast_datasets] updated nemo forecast directory for forecast/13dec25 run: /results/SalishSea/rolling-forecasts/nemo
2025-12-13 10:55:45,755 INFO [update_forecast_datasets] nemo 2025-12-13 forecast rolling forecast datasets updated
2025-12-13 10:55:45,757 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2025-12-13 10:55:45,918 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nemo-forecast'], host='localhost')
2025-12-13 10:55:45,923 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'forecast', 'publish', '--run-date', '2025-12-13'], host='localhost')
2025-12-13 10:55:45,928 INFO [manager] launching NextWorker(module='nowcast.workers.make_surface_current_tiles', args=['forecast', '--run-date', '2025-12-13'], host='localhost')
2025-12-13 10:55:47,052 INFO [ping_erddap] running in process 1348648
2025-12-13 10:55:47,053 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-13 10:55:47,054 INFO [ping_erddap] publishing log messages to tcp://*:5562
2025-12-13 10:55:47,055 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2025-12-13 10:55:47,065 INFO [ping_erddap] nemo-forecast ERDDAP dataset flag file(s) created
2025-12-13 10:55:47,066 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2025-12-13 10:55:48,337 INFO [make_surface_current_tiles] running in process 1348651
2025-12-13 10:55:48,338 INFO [make_surface_current_tiles] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-13 10:55:48,339 INFO [make_surface_current_tiles] publishing log messages to tcp://*:5562
2025-12-13 10:55:48,340 INFO [make_surface_current_tiles] connected to skookum.eos.ubc.ca port 5555
2025-12-13 10:55:48,342 INFO [make_surface_current_tiles] starting rendering of tiles for 2025-12-13 forecast in 6 processes into /results/nowcast-sys/figures/surface_currents/forecast/13dec25
2025-12-13 10:55:49,108 INFO [make_plots] running in process 1348650
2025-12-13 10:55:49,111 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-13 10:55:49,113 INFO [make_plots] publishing log messages to tcp://*:5564
2025-12-13 10:55:49,114 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2025-12-13 10:55:49,522 INFO [make_plots] preparing render list for 2025-12-13 NEMO forecast publish figures
2025-12-13 10:55:49,820 INFO [make_plots] added Website_thumbnail figure to 2025-12-13 NEMO forecast publish render list
2025-12-13 10:55:49,822 INFO [make_plots] added Threshold_website figure to 2025-12-13 NEMO forecast publish render list
2025-12-13 10:55:49,823 INFO [make_plots] added PA_tidal_predictions figure to 2025-12-13 NEMO forecast publish render list
2025-12-13 10:55:49,824 INFO [make_plots] added Boundary Bay figure to 2025-12-13 NEMO forecast publish render list
2025-12-13 10:55:49,825 INFO [make_plots] added Campbell River figure to 2025-12-13 NEMO forecast publish render list
2025-12-13 10:55:49,826 INFO [make_plots] added Cherry Point figure to 2025-12-13 NEMO forecast publish render list
2025-12-13 10:55:49,827 INFO [make_plots] added Friday Harbor figure to 2025-12-13 NEMO forecast publish render list
2025-12-13 10:55:49,828 INFO [make_plots] added Halfmoon Bay figure to 2025-12-13 NEMO forecast publish render list
2025-12-13 10:55:49,828 INFO [make_plots] added Nanaimo figure to 2025-12-13 NEMO forecast publish render list
2025-12-13 10:55:49,829 INFO [make_plots] added Neah Bay figure to 2025-12-13 NEMO forecast publish render list
2025-12-13 10:55:49,830 INFO [make_plots] added New Westminster figure to 2025-12-13 NEMO forecast publish render list
2025-12-13 10:55:49,831 INFO [make_plots] added Patricia Bay figure to 2025-12-13 NEMO forecast publish render list
2025-12-13 10:55:49,833 INFO [make_plots] added Point Atkinson figure to 2025-12-13 NEMO forecast publish render list
2025-12-13 10:55:49,835 INFO [make_plots] added Port Renfrew figure to 2025-12-13 NEMO forecast publish render list
2025-12-13 10:55:49,836 INFO [make_plots] added Sand Heads figure to 2025-12-13 NEMO forecast publish render list
2025-12-13 10:55:49,837 INFO [make_plots] added Sandy Cove figure to 2025-12-13 NEMO forecast publish render list
2025-12-13 10:55:49,838 INFO [make_plots] added Squamish figure to 2025-12-13 NEMO forecast publish render list
2025-12-13 10:55:49,839 INFO [make_plots] added Victoria figure to 2025-12-13 NEMO forecast publish render list
2025-12-13 10:55:49,840 INFO [make_plots] added Woodwards Landing figure to 2025-12-13 NEMO forecast publish render list
2025-12-13 10:55:49,841 INFO [make_plots] starting to render nemo forecast publish 13dec25 figures
2025-12-13 10:57:54,031 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18900169 = 2025-12-13 01:52:00 UTC, 7.8% complete
2025-12-13 11:00:21,945 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 93, 21.5% complete
2025-12-13 11:02:54,031 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18900358 = 2025-12-13 03:58:00 UTC, 16.5% complete
2025-12-13 11:03:30,087 WARNING [make_plots] invalid station number for Boundary Bay station: None
2025-12-13 11:03:30,088 WARNING [make_plots] can't resolve a valid CHS station code for Boundary Bay
2025-12-13 11:03:53,435 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb996/data for station 08074 Campbell River from 2025-12-08 00:00:00Z to 2025-12-15 11:55:00Z
2025-12-13 11:04:59,677 WARNING [make_plots] invalid station number for Halfmoon Bay station: None
2025-12-13 11:04:59,678 WARNING [make_plots] can't resolve a valid CHS station code for Halfmoon Bay
2025-12-13 11:05:21,944 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 201, 46.5% complete
2025-12-13 11:05:23,341 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb96d/data for station 07917 Nanaimo from 2025-12-08 00:00:00Z to 2025-12-15 11:55:00Z
2025-12-13 11:06:06,543 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e13d0f4a073c4bbf93/data for station 07654 New Westminster from 2025-12-08 00:00:00Z to 2025-12-15 11:55:00Z
2025-12-13 11:06:24,661 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd2d/data for station 07277 Patricia Bay from 2025-12-08 00:00:00Z to 2025-12-15 11:55:00Z
2025-12-13 11:06:25,531 INFO [make_surface_current_tiles] starting PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast/13dec25
2025-12-13 11:06:43,279 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94c/data for station 07795 Point Atkinson from 2025-12-08 00:00:00Z to 2025-12-15 11:55:00Z
2025-12-13 11:07:02,440 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e23d0f4a073c4bc060/data for station 08525 Port Renfrew from 2025-12-08 00:00:00Z to 2025-12-15 11:55:00Z
2025-12-13 11:07:19,721 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb93c/data for station 07594 Sand Heads from 2025-12-08 00:00:00Z to 2025-12-15 11:55:00Z
2025-12-13 11:07:37,676 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc43b/data for station 07786 Sandy Cove from 2025-12-08 00:00:00Z to 2025-12-15 11:55:00Z
2025-12-13 11:07:54,029 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18900548 = 2025-12-13 06:04:40 UTC, 25.3% complete
2025-12-13 11:07:55,029 WARNING [make_plots] invalid station number for Squamish station: None
2025-12-13 11:07:55,031 WARNING [make_plots] can't resolve a valid CHS station code for Squamish
2025-12-13 11:08:12,414 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd1e/data for station 07120 Victoria from 2025-12-08 00:00:00Z to 2025-12-15 11:55:00Z
2025-12-13 11:08:30,464 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc42d/data for station 07610 Woodwards Landing from 2025-12-08 00:00:00Z to 2025-12-15 11:55:00Z
2025-12-13 11:08:47,067 INFO [make_plots] finished rendering nemo forecast publish 13dec25 figures
2025-12-13 11:08:47,069 INFO [make_plots] nemo publish plots for 2025-12-13 forecast completed
2025-12-13 11:08:47,076 INFO [manager] checklist updated with [plots] items from make_plots worker
2025-12-13 11:08:47,237 INFO [manager] launching NextWorker(module='nowcast.workers.make_feeds', args=['forecast', '--run-date', '2025-12-13'], host='localhost')
2025-12-13 11:08:50,861 INFO [make_feeds] running in process 1353321
2025-12-13 11:08:50,863 INFO [make_feeds] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-13 11:08:50,864 INFO [make_feeds] publishing log messages to tcp://*:5564
2025-12-13 11:08:50,865 INFO [make_feeds] connected to skookum.eos.ubc.ca port 5555
2025-12-13 11:09:05,775 INFO [make_feeds] ATOM feeds for 2025-12-13 forecast run completed
2025-12-13 11:09:05,778 INFO [manager] checklist updated with [storm surge alert feeds] items from make_feeds worker
2025-12-13 11:10:21,943 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 309, 71.5% complete
2025-12-13 11:11:10,333 INFO [make_surface_current_tiles] finished PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast/13dec25
2025-12-13 11:11:10,357 INFO [make_surface_current_tiles] finished rendering of tiles for 2025-12-13 forecast into /results/nowcast-sys/figures/surface_currents/forecast/13dec25
2025-12-13 11:11:10,358 INFO [make_surface_current_tiles] surface current tile figures for 2025-12-13 forecast completed
2025-12-13 11:11:10,831 INFO [manager] checklist updated with [surface current tiles] items from make_surface_current_tiles worker
2025-12-13 11:12:54,030 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18900737 = 2025-12-13 08:10:40 UTC, 34.1% complete
2025-12-13 11:15:21,944 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 408, 94.4% complete
2025-12-13 11:17:54,033 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18900920 = 2025-12-13 10:12:40 UTC, 42.6% complete
2025-12-13 11:20:21,947 INFO [watch_ww3] nowcast WWATCH3 run on arbutus.cloud-nowcast completed
2025-12-13 11:20:21,950 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2025-12-13 11:20:22,417 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2025-12-13'], host='localhost')
2025-12-13 11:20:22,425 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2025-12-13'], host='arbutus.cloud-nowcast')
2025-12-13 11:20:23,641 INFO [download_wwatch3_results] running in process 1353472
2025-12-13 11:20:23,642 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-13 11:20:23,643 INFO [download_wwatch3_results] publishing log messages to tcp://*:5562
2025-12-13 11:20:23,644 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2025-12-13 11:20:24,431 INFO [run_ww3] running in process 2314
2025-12-13 11:20:24,433 INFO [run_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-12-13 11:20:24,434 INFO [run_ww3] publishing log messages to tcp://*:5572
2025-12-13 11:20:24,435 INFO [run_ww3] connected to skookum.eos.ubc.ca port 5555
2025-12-13 11:20:24,439 INFO [run_ww3] Created run directory /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast_2025-12-13T112024.432648-0800
2025-12-13 11:20:24,440 INFO [run_ww3] forecast: launching /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast_2025-12-13T112024.432648-0800/SoGWW3.sh on arbutus.cloud-nowcast
2025-12-13 11:20:24,451 INFO [run_ww3] forecast WaveWatch3 run for 2025-12-13 on arbutus.cloud-nowcast started
2025-12-13 11:20:24,456 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker
2025-12-13 11:20:24,747 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'forecast'], host='arbutus.cloud-nowcast')
2025-12-13 11:20:26,822 INFO [watch_ww3] running in process 2469
2025-12-13 11:20:26,824 INFO [watch_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-12-13 11:20:26,825 INFO [watch_ww3] publishing log messages to tcp://*:5573
2025-12-13 11:20:26,826 INFO [watch_ww3] connected to skookum.eos.ubc.ca port 5555
2025-12-13 11:20:26,860 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2025-12-13 11:20:56,212 INFO [download_wwatch3_results] nowcast 2025-12-13 results files from arbutus.cloud-nowcast downloaded
2025-12-13 11:20:56,215 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2025-12-13 11:22:54,033 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18901104 = 2025-12-13 12:15:20 UTC, 51.1% complete
2025-12-13 11:25:26,861 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 93, 14.4% complete
2025-12-13 11:27:54,034 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18901288 = 2025-12-13 14:18:00 UTC, 59.6% complete
2025-12-13 11:30:26,863 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 198, 30.6% complete
2025-12-13 11:32:54,035 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18901473 = 2025-12-13 16:21:20 UTC, 68.2% complete
2025-12-13 11:35:26,864 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 300, 46.3% complete
2025-12-13 11:37:54,044 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18901659 = 2025-12-13 18:25:20 UTC, 76.8% complete
2025-12-13 11:40:26,867 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 399, 61.6% complete
2025-12-13 11:42:54,037 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18901845 = 2025-12-13 20:29:20 UTC, 85.4% complete
2025-12-13 11:45:26,866 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 498, 76.9% complete
2025-12-13 11:47:54,039 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18902030 = 2025-12-13 22:32:40 UTC, 94.0% complete
2025-12-13 11:50:26,868 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 594, 91.7% complete
2025-12-13 11:52:54,043 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18902160 = 2025-12-13 23:59:20 UTC, 100.0% complete
2025-12-13 11:55:26,869 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2025-12-13 11:57:54,049 INFO [watch_NEMO] nowcast-green NEMO run on arbutus.cloud-nowcast completed
2025-12-13 11:57:54,052 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2025-12-13 11:57:54,531 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2025-12-13'], host='localhost')
2025-12-13 11:57:55,827 INFO [download_results] running in process 1353739
2025-12-13 11:57:55,828 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-13 11:57:55,829 INFO [download_results] publishing log messages to tcp://*:5562
2025-12-13 11:57:55,830 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2025-12-13 11:57:55,831 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/nowcast-green/13dec25 to /results2/SalishSea/nowcast-green.202111
2025-12-13 12:00:26,871 INFO [watch_ww3] forecast WWATCH3 run on arbutus.cloud-nowcast completed
2025-12-13 12:00:26,874 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2025-12-13 12:00:27,356 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2025-12-13'], host='localhost')
2025-12-13 12:00:28,812 INFO [download_wwatch3_results] running in process 1353771
2025-12-13 12:00:28,814 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-13 12:00:28,815 INFO [download_wwatch3_results] publishing log messages to tcp://*:5564
2025-12-13 12:00:28,816 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2025-12-13 12:01:00,829 INFO [download_results] nowcast-green 2025-12-13 results files from arbutus.cloud-nowcast downloaded
2025-12-13 12:01:00,836 INFO [manager] checklist updated with [results files] items from download_results worker
2025-12-13 12:01:01,150 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast-green', 'research', '--run-date', '2025-12-13'], host='localhost')
2025-12-13 12:01:01,156 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nowcast-green'], host='localhost')
2025-12-13 12:01:01,160 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'physics', '--run-date', '2025-12-13'], host='localhost')
2025-12-13 12:01:01,165 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'chemistry', '--run-date', '2025-12-13'], host='localhost')
2025-12-13 12:01:01,171 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'biology', '--run-date', '2025-12-13'], host='localhost')
2025-12-13 12:01:02,387 INFO [ping_erddap] running in process 1353807
2025-12-13 12:01:02,388 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-13 12:01:02,389 INFO [ping_erddap] publishing log messages to tcp://*:5562
2025-12-13 12:01:02,390 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2025-12-13 12:01:02,394 INFO [ping_erddap] nowcast-green ERDDAP dataset flag file(s) created
2025-12-13 12:01:02,396 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2025-12-13 12:01:03,664 INFO [make_averaged_dataset] running in process 1353808
2025-12-13 12:01:03,665 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-13 12:01:03,666 INFO [make_averaged_dataset] publishing log messages to tcp://*:5562
2025-12-13 12:01:03,668 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555
2025-12-13 12:01:03,669 INFO [make_averaged_dataset] creating day-averaged dataset for 13-Dec-2025 physics
2025-12-13 12:01:03,683 INFO [make_averaged_dataset] running in process 1353809
2025-12-13 12:01:03,684 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-13 12:01:03,685 INFO [make_averaged_dataset] publishing log messages to tcp://*:5565
2025-12-13 12:01:03,686 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555
2025-12-13 12:01:03,687 INFO [make_averaged_dataset] creating day-averaged dataset for 13-Dec-2025 chemistry
2025-12-13 12:01:03,697 INFO [make_averaged_dataset] running in process 1353810
2025-12-13 12:01:03,698 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-13 12:01:03,699 INFO [make_averaged_dataset] publishing log messages to tcp://*:5566
2025-12-13 12:01:03,703 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555
2025-12-13 12:01:03,703 INFO [make_averaged_dataset] creating day-averaged dataset for 13-Dec-2025 biology
2025-12-13 12:01:06,474 INFO [download_wwatch3_results] forecast 2025-12-13 results files from arbutus.cloud-nowcast downloaded
2025-12-13 12:01:06,477 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2025-12-13 12:01:06,788 INFO [make_plots] running in process 1353805
2025-12-13 12:01:06,790 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-13 12:01:06,791 INFO [make_plots] publishing log messages to tcp://*:5567
2025-12-13 12:01:06,792 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2025-12-13 12:01:06,792 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['wwatch3', 'forecast', '--run-date', '2025-12-13'], host='localhost')
2025-12-13 12:01:07,152 INFO [make_plots] preparing render list for 2025-12-13 NEMO nowcast-green research figures
2025-12-13 12:01:08,200 INFO [update_forecast_datasets] running in process 1353889
2025-12-13 12:01:08,201 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-13 12:01:08,202 INFO [update_forecast_datasets] publishing log messages to tcp://*:5564
2025-12-13 12:01:08,203 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2025-12-13 12:01:08,204 INFO [update_forecast_datasets] updating wwatch3 most_recent_forecast directory from forecast/13dec25 run
2025-12-13 12:01:08,269 INFO [update_forecast_datasets] updated wwatch3 most_recent_forecast directory from forecast/13dec25 run
2025-12-13 12:01:08,270 INFO [update_forecast_datasets] updating wwatch3 forecast directory for forecast/13dec25 run
2025-12-13 12:01:08,315 INFO [update_forecast_datasets] updated wwatch3 forecast directory for forecast/13dec25 run: /results/SalishSea/rolling-forecasts/wwatch3
2025-12-13 12:01:08,316 INFO [update_forecast_datasets] wwatch3 2025-12-13 forecast rolling forecast datasets updated
2025-12-13 12:01:08,319 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2025-12-13 12:01:08,646 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['wwatch3-forecast'], host='localhost')
2025-12-13 12:01:08,932 INFO [make_plots] added temperature_thalweg_and_surface figures to 2025-12-13 NEMO nowcast-green research render list
2025-12-13 12:01:09,706 INFO [make_plots] added nitrate_thalweg_and_surface figures to 2025-12-13 NEMO nowcast-green research render list
2025-12-13 12:01:09,797 INFO [ping_erddap] running in process 1353912
2025-12-13 12:01:09,798 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-13 12:01:09,799 INFO [ping_erddap] publishing log messages to tcp://*:5564
2025-12-13 12:01:09,800 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2025-12-13 12:01:09,801 INFO [ping_erddap] wwatch3-forecast ERDDAP dataset flag file(s) created
2025-12-13 12:01:09,804 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2025-12-13 12:01:10,078 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['wwatch3', 'forecast', 'publish', '--run-date', '2025-12-13'], host='localhost')
2025-12-13 12:01:10,466 INFO [make_plots] added ammonium_thalweg_and_surface figures to 2025-12-13 NEMO nowcast-green research render list
2025-12-13 12:01:11,195 INFO [make_plots] added silicon_thalweg_and_surface figures to 2025-12-13 NEMO nowcast-green research render list
2025-12-13 12:01:11,973 INFO [make_plots] added dissolved_organic_nitrogen_thalweg_and_surface figures to 2025-12-13 NEMO nowcast-green research render list
2025-12-13 12:01:12,711 INFO [make_plots] added particulate_organic_nitrogen_thalweg_and_surface figures to 2025-12-13 NEMO nowcast-green research render list
2025-12-13 12:01:13,058 INFO [make_plots] running in process 1353947
2025-12-13 12:01:13,059 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-13 12:01:13,060 INFO [make_plots] publishing log messages to tcp://*:5564
2025-12-13 12:01:13,061 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2025-12-13 12:01:13,062 INFO [make_plots] preparing render list for 2025-12-13 WaveWatch3 forecast publish figures
2025-12-13 12:01:13,063 INFO [make_plots] added Halibut Bank figure to 2025-12-13 WaveWatch3 forecast publish render list
2025-12-13 12:01:13,063 INFO [make_plots] added Sentry Shoal figure to 2025-12-13 WaveWatch3 forecast publish render list
2025-12-13 12:01:13,064 INFO [make_plots] starting to render wwatch3 forecast publish 13dec25 figures
2025-12-13 12:01:13,457 INFO [make_plots] added biogenic_silicon_thalweg_and_surface figures to 2025-12-13 NEMO nowcast-green research render list
2025-12-13 12:01:13,724 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46146 Halibut Bank
2025-12-13 12:01:14,303 INFO [make_plots] added diatoms_thalweg_and_surface figures to 2025-12-13 NEMO nowcast-green research render list
2025-12-13 12:01:15,133 INFO [make_plots] added flagellates_thalweg_and_surface figures to 2025-12-13 NEMO nowcast-green research render list
2025-12-13 12:01:15,955 INFO [make_plots] added microzooplankton_thalweg_and_surface figures to 2025-12-13 NEMO nowcast-green research render list
2025-12-13 12:01:16,808 INFO [make_plots] added mesozooplankton_thalweg_and_surface figures to 2025-12-13 NEMO nowcast-green research render list
2025-12-13 12:01:17,601 INFO [make_plots] added turbidity_thalweg_and_surface figures to 2025-12-13 NEMO nowcast-green research render list
2025-12-13 12:01:18,604 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46131 Sentry Shoal
2025-12-13 12:01:18,822 INFO [make_plots] added temperature_salinity_timeseries figure to 2025-12-13 NEMO nowcast-green research render list
2025-12-13 12:01:18,823 INFO [make_plots] added nitrate_diatoms_timeseries figure to 2025-12-13 NEMO nowcast-green research render list
2025-12-13 12:01:18,823 INFO [make_plots] added diatoms_flagellates_timeseries figure to 2025-12-13 NEMO nowcast-green research render list
2025-12-13 12:01:18,824 INFO [make_plots] added z1_z2_zooplankton_timeseries figure to 2025-12-13 NEMO nowcast-green research render list
2025-12-13 12:01:18,825 INFO [make_plots] starting to render nemo nowcast-green research 13dec25 figures
2025-12-13 12:01:22,644 INFO [make_plots] finished rendering wwatch3 forecast publish 13dec25 figures
2025-12-13 12:01:22,644 INFO [make_plots] wwatch3 publish plots for 2025-12-13 forecast completed
2025-12-13 12:01:22,648 INFO [manager] checklist updated with [plots] items from make_plots worker
2025-12-13 12:01:38,954 INFO [make_averaged_dataset] day-averaged dataset for 13-Dec-2025 chemistry created
2025-12-13 12:01:38,957 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker
2025-12-13 12:02:03,270 INFO [make_averaged_dataset] day-averaged dataset for 13-Dec-2025 physics created
2025-12-13 12:02:03,273 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker
2025-12-13 12:02:16,412 INFO [make_averaged_dataset] day-averaged dataset for 13-Dec-2025 biology created
2025-12-13 12:02:16,415 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker
2025-12-13 13:02:36,156 INFO [make_plots] finished rendering nemo nowcast-green research 13dec25 figures
2025-12-13 13:02:36,327 INFO [make_plots] nemo research plots for 2025-12-13 nowcast-green completed
2025-12-13 13:02:36,392 INFO [manager] checklist updated with [plots] items from make_plots worker
2025-12-13 13:48:50,130 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/18/ to /results/forcing/atmospheric/continental2.5/GRIB/20251213/18/
2025-12-13 13:48:50,132 INFO [collect_weather] 2025-12-13 2.5km weather forecast 18 collection complete
2025-12-13 13:48:50,134 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2025-12-13 13:48:50,615 INFO [manager] launching NextWorker(module='nowcast.workers.download_weather', args=['00', '1km'], host='localhost')
2025-12-13 13:48:50,623 INFO [manager] launching NextWorker(module='nowcast.workers.download_weather', args=['12', '1km'], host='localhost')
2025-12-13 13:48:50,629 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['00', '2.5km'], host='localhost')
2025-12-13 13:48:50,634 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['00', '--fcst-date', '2025-12-14'], host='localhost')
2025-12-13 13:48:50,782 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20251213/18/
2025-12-13 13:48:50,783 INFO [crop_gribs] 2025-12-13 18 GRIBs cropping complete
2025-12-13 13:48:50,785 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2025-12-13 13:48:52,070 INFO [download_weather] running in process 1354979
2025-12-13 13:48:52,072 INFO [download_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-13 13:48:52,073 INFO [collect_weather] running in process 1354982
2025-12-13 13:48:52,074 INFO [download_weather] publishing log messages to tcp://*:5562
2025-12-13 13:48:52,075 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-13 13:48:52,076 INFO [download_weather] connected to skookum.eos.ubc.ca port 5555
2025-12-13 13:48:52,077 INFO [collect_weather] publishing log messages to tcp://*:5563
2025-12-13 13:48:52,078 INFO [download_weather] downloading 00 1 km forecast GRIB2 files for 20251213
2025-12-13 13:48:52,078 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2025-12-13 13:48:52,079 INFO [download_weather] running in process 1354981
2025-12-13 13:48:52,080 INFO [download_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-13 13:48:52,081 INFO [download_weather] publishing log messages to tcp://*:5564
2025-12-13 13:48:52,082 INFO [download_weather] connected to skookum.eos.ubc.ca port 5555
2025-12-13 13:48:52,083 INFO [download_weather] downloading 12 1 km forecast GRIB2 files for 20251213
2025-12-13 13:48:52,147 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/00/
2025-12-13 13:48:53,332 INFO [crop_gribs] running in process 1354983
2025-12-13 13:48:53,333 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-13 13:48:53,335 INFO [crop_gribs] publishing log messages to tcp://*:5565
2025-12-13 13:48:53,336 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2025-12-13 13:48:53,337 INFO [crop_gribs] cropping 2025-12-14 ECCC HRDPS 2.5km continental 00Z GRIB files to SalishSeaCast subdomain
2025-12-13 13:48:53,350 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20251214/00/
2025-12-13 13:52:45,698 INFO [download_weather] 2025-12-13 1km weather forecast 12 downloads complete
2025-12-13 13:52:45,701 INFO [manager] checklist updated with [weather forecast] items from download_weather worker
2025-12-13 13:52:54,511 INFO [download_weather] 2025-12-13 1km weather forecast 00 downloads complete
2025-12-13 13:52:54,515 INFO [manager] checklist updated with [weather forecast] items from download_weather worker
2025-12-13 19:54:27,002 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/00/ to /results/forcing/atmospheric/continental2.5/GRIB/20251214/00/
2025-12-13 19:54:27,003 INFO [collect_weather] 2025-12-14 2.5km weather forecast 00 collection complete
2025-12-13 19:54:27,006 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2025-12-13 19:54:27,514 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['06', '2.5km'], host='localhost')
2025-12-13 19:54:27,521 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['06', '--fcst-date', '2025-12-14'], host='localhost')
2025-12-13 19:54:29,362 INFO [collect_weather] running in process 1359157
2025-12-13 19:54:29,363 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-13 19:54:29,364 INFO [collect_weather] publishing log messages to tcp://*:5562
2025-12-13 19:54:29,366 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2025-12-13 19:54:29,380 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/06/
2025-12-13 19:54:30,758 INFO [crop_gribs] running in process 1359159
2025-12-13 19:54:30,762 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-13 19:54:30,763 INFO [crop_gribs] publishing log messages to tcp://*:5563
2025-12-13 19:54:30,764 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2025-12-13 19:54:30,766 INFO [crop_gribs] cropping 2025-12-14 ECCC HRDPS 2.5km continental 06Z GRIB files to SalishSeaCast subdomain
2025-12-13 19:54:30,778 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20251214/06/
2025-12-13 21:48:54,657 INFO [crop_gribs] crop_gribs 00 has watched for 8h; retrying remaining 1 file(s)
2025-12-13 21:48:55,966 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20251214/00/
2025-12-13 21:48:55,968 INFO [crop_gribs] 2025-12-14 00 GRIBs cropping complete
2025-12-13 21:48:55,970 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2025-12-14 01:49:14,174 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/06/ to /results/forcing/atmospheric/continental2.5/GRIB/20251214/06/
2025-12-14 01:49:14,176 INFO [collect_weather] 2025-12-14 2.5km weather forecast 06 collection complete
2025-12-14 01:49:14,179 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2025-12-14 01:49:14,668 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Capilano', '--data-date', '2025-12-13'], host='localhost')
2025-12-14 01:49:14,676 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'ChilliwackVedder', '--data-date', '2025-12-13'], host='localhost')
2025-12-14 01:49:14,681 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'ClowhomClowhomLake', '--data-date', '2025-12-13'], host='localhost')
2025-12-14 01:49:14,686 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Englishman', '--data-date', '2025-12-13'], host='localhost')
2025-12-14 01:49:14,692 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Fraser', '--data-date', '2025-12-13'], host='localhost')
2025-12-14 01:49:14,696 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'HomathkoMouth', '--data-date', '2025-12-13'], host='localhost')
2025-12-14 01:49:14,699 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'NicomeklLangley', '--data-date', '2025-12-13'], host='localhost')
2025-12-14 01:49:14,703 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'RobertsCreek', '--data-date', '2025-12-13'], host='localhost')
2025-12-14 01:49:14,708 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SalmonSayward', '--data-date', '2025-12-13'], host='localhost')
2025-12-14 01:49:14,713 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SanJuanPortRenfrew', '--data-date', '2025-12-13'], host='localhost')
2025-12-14 01:49:14,723 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SquamishBrackendale', '--data-date', '2025-12-13'], host='localhost')
2025-12-14 01:49:14,729 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaScotty', '--data-date', '2025-12-13'], host='localhost')
2025-12-14 01:49:14,749 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaBypass', '--data-date', '2025-12-13'], host='localhost')
2025-12-14 01:49:14,777 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaDiversion', '--data-date', '2025-12-13'], host='localhost')
2025-12-14 01:49:14,800 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ctd', args=['SCVIP'], host='localhost')
2025-12-14 01:49:14,811 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ctd', args=['SEVIP'], host='localhost')
2025-12-14 01:49:14,824 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ferry', args=['TWDP'], host='localhost')
2025-12-14 01:49:14,832 INFO [manager] launching NextWorker(module='nowcast.workers.get_vfpa_hadcp', args=['--data-date', '2025-12-13'], host='localhost')
2025-12-14 01:49:14,847 INFO [manager] launching NextWorker(module='nowcast.workers.collect_NeahBay_ssh', args=['00'], host='localhost')
2025-12-14 01:49:14,862 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['12', '2.5km'], host='localhost')
2025-12-14 01:49:14,873 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['12'], host='localhost')
2025-12-14 01:49:14,958 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20251214/06/
2025-12-14 01:49:14,961 INFO [crop_gribs] 2025-12-14 06 GRIBs cropping complete
2025-12-14 01:49:14,963 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2025-12-14 01:49:15,741 INFO [manager] launching NextWorker(module='nowcast.workers.grib_to_netcdf', args=['forecast2'], host='localhost')
2025-12-14 01:49:17,821 INFO [collect_weather] running in process 1363080
2025-12-14 01:49:17,824 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-14 01:49:17,825 INFO [collect_weather] publishing log messages to tcp://*:5562
2025-12-14 01:49:17,826 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2025-12-14 01:49:17,831 INFO [collect_NeahBay_ssh] running in process 1363079
2025-12-14 01:49:17,834 INFO [collect_NeahBay_ssh] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-14 01:49:17,839 INFO [collect_NeahBay_ssh] publishing log messages to tcp://*:5563
2025-12-14 01:49:17,840 INFO [collect_NeahBay_ssh] connected to skookum.eos.ubc.ca port 5555
2025-12-14 01:49:17,840 INFO [collect_NeahBay_ssh] collecting Neah Bay ssh 00Z obs/forecast for 2025-12-14
2025-12-14 01:49:17,848 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/12/
2025-12-14 01:49:18,869 INFO [collect_river_data] running in process 1363069
2025-12-14 01:49:18,872 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-14 01:49:18,873 INFO [collect_river_data] publishing log messages to tcp://*:5564
2025-12-14 01:49:18,873 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-12-14 01:49:18,874 INFO [collect_river_data] Collecting ECCC SalmonSayward river data for 2025-12-13
2025-12-14 01:49:18,888 INFO [collect_river_data] running in process 1363064
2025-12-14 01:49:18,889 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-14 01:49:18,890 INFO [collect_river_data] publishing log messages to tcp://*:5565
2025-12-14 01:49:18,892 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-12-14 01:49:18,892 INFO [collect_river_data] Collecting ECCC Englishman river data for 2025-12-13
2025-12-14 01:49:18,911 INFO [collect_river_data] running in process 1363067
2025-12-14 01:49:18,912 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-14 01:49:18,913 INFO [collect_river_data] publishing log messages to tcp://*:5566
2025-12-14 01:49:18,914 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-12-14 01:49:18,914 INFO [collect_river_data] Collecting ECCC NicomeklLangley river data for 2025-12-13
2025-12-14 01:49:18,919 INFO [collect_river_data] Appended ECCC Englishman river average discharge for 2025-12-13 to: /data/dlatorne/SOG-projects/SOG-forcing/ECget/Englishman_flow
2025-12-14 01:49:18,920 INFO [collect_river_data] ECCC Englishman river data collection for 2025-12-13 completed
2025-12-14 01:49:18,921 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-12-14 01:49:18,929 INFO [collect_river_data] running in process 1363060
2025-12-14 01:49:18,930 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-14 01:49:18,931 INFO [collect_river_data] publishing log messages to tcp://*:5567
2025-12-14 01:49:18,931 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-12-14 01:49:18,933 INFO [collect_river_data] Collecting ECCC Capilano river data for 2025-12-13
2025-12-14 01:49:18,938 INFO [collect_river_data] Appended ECCC SalmonSayward river average discharge for 2025-12-13 to: /results/forcing/rivers/observations/Salmon_Sayward_flow
2025-12-14 01:49:18,939 INFO [collect_river_data] ECCC SalmonSayward river data collection for 2025-12-13 completed
2025-12-14 01:49:18,964 INFO [collect_river_data] Appended ECCC NicomeklLangley river average discharge for 2025-12-13 to: /results/forcing/rivers/observations/Nicomekl_Langley_flow
2025-12-14 01:49:18,964 INFO [collect_river_data] ECCC NicomeklLangley river data collection for 2025-12-13 completed
2025-12-14 01:49:18,971 INFO [collect_river_data] running in process 1363070
2025-12-14 01:49:18,971 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-14 01:49:18,972 INFO [collect_river_data] publishing log messages to tcp://*:5568
2025-12-14 01:49:18,974 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-12-14 01:49:18,974 INFO [collect_river_data] Collecting ECCC SanJuanPortRenfrew river data for 2025-12-13
2025-12-14 01:49:18,976 INFO [collect_river_data] Appended ECCC Capilano river average discharge for 2025-12-13 to: /opp/observations/rivers/Capilano/Caplilano_08GA010_day_avg_flow
2025-12-14 01:49:18,977 INFO [collect_river_data] ECCC Capilano river data collection for 2025-12-13 completed
2025-12-14 01:49:18,989 INFO [collect_river_data] running in process 1363071
2025-12-14 01:49:18,991 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-14 01:49:18,992 INFO [collect_river_data] publishing log messages to tcp://*:5569
2025-12-14 01:49:18,993 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-12-14 01:49:18,994 INFO [collect_river_data] Collecting ECCC SquamishBrackendale river data for 2025-12-13
2025-12-14 01:49:19,001 INFO [collect_river_data] Appended ECCC SanJuanPortRenfrew river average discharge for 2025-12-13 to: /results/forcing/rivers/observations/SanJuan_PortRenfrew_flow
2025-12-14 01:49:19,001 INFO [collect_river_data] ECCC SanJuanPortRenfrew river data collection for 2025-12-13 completed
2025-12-14 01:49:19,024 INFO [collect_river_data] running in process 1363063
2025-12-14 01:49:19,026 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-14 01:49:19,027 INFO [collect_river_data] publishing log messages to tcp://*:5570
2025-12-14 01:49:19,029 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-12-14 01:49:19,031 INFO [collect_river_data] Collecting ECCC ClowhomClowhomLake river data for 2025-12-13
2025-12-14 01:49:19,033 INFO [collect_river_data] running in process 1363066
2025-12-14 01:49:19,035 INFO [collect_river_data] Appended ECCC SquamishBrackendale river average discharge for 2025-12-13 to: /results/forcing/rivers/observations/Squamish_Brackendale_flow
2025-12-14 01:49:19,036 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-14 01:49:19,037 INFO [collect_river_data] ECCC SquamishBrackendale river data collection for 2025-12-13 completed
2025-12-14 01:49:19,038 INFO [collect_river_data] publishing log messages to tcp://*:5571
2025-12-14 01:49:19,041 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-12-14 01:49:19,043 INFO [collect_river_data] Collecting ECCC HomathkoMouth river data for 2025-12-13
2025-12-14 01:49:19,058 INFO [collect_river_data] Appended ECCC HomathkoMouth river average discharge for 2025-12-13 to: /results/forcing/rivers/observations/Homathko_Mouth_flow
2025-12-14 01:49:19,059 INFO [collect_river_data] ECCC HomathkoMouth river data collection for 2025-12-13 completed
2025-12-14 01:49:19,061 INFO [collect_river_data] Appended ECCC ClowhomClowhomLake river average discharge for 2025-12-13 to: /results/forcing/rivers/observations/Clowhom_ClowhomLake_flow
2025-12-14 01:49:19,062 INFO [collect_river_data] ECCC ClowhomClowhomLake river data collection for 2025-12-13 completed
2025-12-14 01:49:19,080 INFO [collect_river_data] running in process 1363073
2025-12-14 01:49:19,081 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-14 01:49:19,081 INFO [collect_river_data] publishing log messages to tcp://*:5572
2025-12-14 01:49:19,082 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-12-14 01:49:19,083 INFO [collect_river_data] Collecting ECCC TheodosiaBypass river data for 2025-12-13
2025-12-14 01:49:19,084 INFO [collect_river_data] running in process 1363074
2025-12-14 01:49:19,085 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-14 01:49:19,086 INFO [collect_river_data] publishing log messages to tcp://*:5573
2025-12-14 01:49:19,087 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-12-14 01:49:19,088 INFO [collect_river_data] Collecting ECCC TheodosiaDiversion river data for 2025-12-13
2025-12-14 01:49:19,089 INFO [collect_river_data] running in process 1363062
2025-12-14 01:49:19,090 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-14 01:49:19,090 INFO [collect_river_data] publishing log messages to tcp://*:5574
2025-12-14 01:49:19,091 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-12-14 01:49:19,092 INFO [collect_river_data] Collecting ECCC ChilliwackVedder river data for 2025-12-13
2025-12-14 01:49:19,093 INFO [collect_river_data] running in process 1363068
2025-12-14 01:49:19,095 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-14 01:49:19,096 INFO [collect_river_data] publishing log messages to tcp://*:5575
2025-12-14 01:49:19,097 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-12-14 01:49:19,097 INFO [collect_river_data] Collecting ECCC RobertsCreek river data for 2025-12-13
2025-12-14 01:49:19,098 INFO [collect_river_data] running in process 1363065
2025-12-14 01:49:19,100 INFO [collect_river_data] running in process 1363072
2025-12-14 01:49:19,100 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-14 01:49:19,101 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-14 01:49:19,102 INFO [collect_river_data] publishing log messages to tcp://*:5576
2025-12-14 01:49:19,102 INFO [collect_river_data] publishing log messages to tcp://*:5577
2025-12-14 01:49:19,103 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-12-14 01:49:19,104 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-12-14 01:49:19,105 INFO [collect_river_data] Collecting ECCC Fraser river data for 2025-12-13
2025-12-14 01:49:19,106 INFO [collect_river_data] Collecting ECCC TheodosiaScotty river data for 2025-12-13
2025-12-14 01:49:19,120 INFO [collect_river_data] Appended ECCC TheodosiaDiversion river average discharge for 2025-12-13 to: /results/forcing/rivers/observations/Theodosia_Diversion_flow
2025-12-14 01:49:19,121 INFO [collect_river_data] Appended ECCC TheodosiaBypass river average discharge for 2025-12-13 to: /results/forcing/rivers/observations/Theodosia_Bypass_flow
2025-12-14 01:49:19,122 INFO [collect_river_data] ECCC TheodosiaDiversion river data collection for 2025-12-13 completed
2025-12-14 01:49:19,123 INFO [collect_river_data] ECCC TheodosiaBypass river data collection for 2025-12-13 completed
2025-12-14 01:49:19,126 INFO [collect_river_data] Appended ECCC TheodosiaScotty river average discharge for 2025-12-13 to: /results/forcing/rivers/observations/Theodosia_Scotty_flow
2025-12-14 01:49:19,127 INFO [collect_river_data] ECCC TheodosiaScotty river data collection for 2025-12-13 completed
2025-12-14 01:49:19,128 INFO [collect_river_data] Appended ECCC Fraser river average discharge for 2025-12-13 to: /data/dlatorne/SOG-projects/SOG-forcing/ECget/Fraser_flow
2025-12-14 01:49:19,129 INFO [collect_river_data] ECCC Fraser river data collection for 2025-12-13 completed
2025-12-14 01:49:19,130 INFO [collect_river_data] Appended ECCC RobertsCreek river average discharge for 2025-12-13 to: /results/forcing/rivers/observations/RobertsCreek_flow
2025-12-14 01:49:19,131 INFO [collect_river_data] ECCC RobertsCreek river data collection for 2025-12-13 completed
2025-12-14 01:49:19,137 INFO [collect_river_data] Appended ECCC ChilliwackVedder river average discharge for 2025-12-13 to: /results/forcing/rivers/observations/Chilliwack_Vedder_flow
2025-12-14 01:49:19,138 INFO [collect_river_data] ECCC ChilliwackVedder river data collection for 2025-12-13 completed
2025-12-14 01:49:19,343 INFO [collect_NeahBay_ssh] 2025-12-14T00:00:00-08:00 Neah Bay ssh 00Z obs/forecast data collection complete
2025-12-14 01:49:19,364 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-12-14 01:49:19,509 INFO [get_vfpa_hadcp] running in process 1363078
2025-12-14 01:49:19,510 INFO [get_vfpa_hadcp] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-14 01:49:19,510 INFO [get_vfpa_hadcp] publishing log messages to tcp://*:5578
2025-12-14 01:49:19,511 INFO [get_vfpa_hadcp] connected to skookum.eos.ubc.ca port 5555
2025-12-14 01:49:19,512 INFO [get_vfpa_hadcp] processing VFPA HADCP data from 2nd Narrows Rail Bridge for 2025-12-13
2025-12-14 01:49:19,515 INFO [grib_to_netcdf] running in process 1363138
2025-12-14 01:49:19,516 INFO [grib_to_netcdf] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-14 01:49:19,518 INFO [grib_to_netcdf] publishing log messages to tcp://*:5579
2025-12-14 01:49:19,520 INFO [grib_to_netcdf] connected to skookum.eos.ubc.ca port 5555
2025-12-14 01:49:19,521 INFO [grib_to_netcdf] creating NEMO-atmos forcing files for 2025-12-14 forecast2 run
2025-12-14 01:49:19,631 INFO [crop_gribs] running in process 1363081
2025-12-14 01:49:19,632 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-14 01:49:19,633 INFO [crop_gribs] publishing log messages to tcp://*:5565
2025-12-14 01:49:19,635 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2025-12-14 01:49:19,636 INFO [crop_gribs] cropping 2025-12-14 ECCC HRDPS 2.5km continental 12Z GRIB files to SalishSeaCast subdomain
2025-12-14 01:49:19,654 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20251214/12/
2025-12-14 01:49:19,723 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-12-14 01:49:20,191 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-12-14 01:49:20,435 INFO [get_onc_ferry] running in process 1363077
2025-12-14 01:49:20,435 INFO [get_onc_ferry] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-14 01:49:20,436 INFO [get_onc_ferry] publishing log messages to tcp://*:5564
2025-12-14 01:49:20,437 INFO [get_onc_ferry] connected to skookum.eos.ubc.ca port 5555
2025-12-14 01:49:20,438 INFO [get_onc_ferry] requesting ONC TWDP.N1 NAV data for 2025-12-13
2025-12-14 01:49:20,448 INFO [get_onc_ctd] running in process 1363075
2025-12-14 01:49:20,449 INFO [get_onc_ctd] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-14 01:49:20,450 INFO [get_onc_ctd] publishing log messages to tcp://*:5566
2025-12-14 01:49:20,451 INFO [get_onc_ctd] connected to skookum.eos.ubc.ca port 5555
2025-12-14 01:49:20,452 INFO [get_onc_ctd] requesting ONC SCVIP CTD T&S data for 2025-12-13
2025-12-14 01:49:20,474 INFO [get_onc_ctd] running in process 1363076
2025-12-14 01:49:20,475 INFO [get_onc_ctd] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-14 01:49:20,476 INFO [get_onc_ctd] publishing log messages to tcp://*:5590
2025-12-14 01:49:20,477 INFO [get_onc_ctd] connected to skookum.eos.ubc.ca port 5555
2025-12-14 01:49:20,478 INFO [get_onc_ctd] requesting ONC SEVIP CTD T&S data for 2025-12-13
2025-12-14 01:49:20,572 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-12-14 01:49:20,748 INFO [get_onc_ferry] requesting ONC TWDP.N2 NAV data for 2025-12-13
2025-12-14 01:49:20,832 ERROR [get_onc_ferry] request for ONC TWDP.N2 NAV data for 2025-12-13 failed: 400 Client Error:  for url: https://data.oceannetworks.ca/api/scalardata?method=getByLocation&token=8efe44eb-0fb5-4413-bba3-ae3dbe0fbb1d&locationCode=TWDP.N2&deviceCategoryCode=NAV&sensorCategoryCodes=longitude%2Clatitude&dateFrom=2025-12-13T00:00:00.000Z&dateTo=2025-12-13T23:59:00.000Z&resampleType=avg&resamplePeriod=1
2025-12-14 01:49:20,905 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-12-14 01:49:21,296 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-12-14 01:49:21,310 CRITICAL [get_onc_ferry] 2025-12-13 ONC TWDP ferry data file creation failed
2025-12-14 01:49:21,680 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-12-14 01:49:22,045 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-12-14 01:49:22,245 INFO [get_vfpa_hadcp] added VFPA HADCP data from 2nd Narrows Rail Bridge for 2025-12-13 to /opp/observations/AISDATA/netcdf/VFPA_2ND_NARROWS_HADCP_2s_202512.nc
2025-12-14 01:49:22,246 INFO [get_vfpa_hadcp] VFPA HADCP observations added to 2025-12 netcdf file
2025-12-14 01:49:22,413 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-12-14 01:49:22,706 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-12-14 01:49:23,026 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-12-14 01:49:23,341 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-12-14 01:49:23,704 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-12-14 01:49:24,022 INFO [manager] checklist updated with [Neah Bay ssh data] items from collect_NeahBay_ssh worker
2025-12-14 01:49:24,349 INFO [manager] launching NextWorker(module='nowcast.workers.make_ssh_files', args=['forecast2', '--run-date', '2025-12-14'], host='localhost')
2025-12-14 01:49:24,359 INFO [manager] checklist updated with [VFPA HADCP data] items from get_vfpa_hadcp worker
2025-12-14 01:49:24,863 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['VFPA-HADCP'], host='localhost')
2025-12-14 01:49:26,005 INFO [ping_erddap] running in process 1363586
2025-12-14 01:49:26,006 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-14 01:49:26,007 INFO [ping_erddap] publishing log messages to tcp://*:5563
2025-12-14 01:49:26,008 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2025-12-14 01:49:26,009 INFO [ping_erddap] VFPA-HADCP ERDDAP dataset flag file(s) created
2025-12-14 01:49:26,011 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2025-12-14 01:49:27,689 INFO [make_ssh_files] running in process 1363573
2025-12-14 01:49:27,690 INFO [make_ssh_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-14 01:49:27,691 INFO [make_ssh_files] publishing log messages to tcp://*:5563
2025-12-14 01:49:27,692 INFO [make_ssh_files] connected to skookum.eos.ubc.ca port 5555
2025-12-14 01:49:27,693 INFO [make_ssh_files] building forecast2 sea surface height boundary conditions file(s) from 2025-12-14 NOAA Neah Bay 00Z observation and forecast values
2025-12-14 01:49:30,428 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2025m12d09.nc
2025-12-14 01:49:30,695 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2025m12d10.nc
2025-12-14 01:49:30,918 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2025m12d11.nc
2025-12-14 01:49:31,297 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2025m12d12.nc
2025-12-14 01:49:31,487 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2025m12d13.nc
2025-12-14 01:49:31,634 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2025m12d15.nc
2025-12-14 01:49:31,702 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2025m12d14.nc
2025-12-14 01:49:31,904 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2025m12d15.nc
2025-12-14 01:49:32,088 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2025m12d16.nc
2025-12-14 01:49:32,335 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2025m12d17.nc
2025-12-14 01:49:32,511 INFO [make_ssh_files] sea surface height boundary file for 2025-12-14 forecast2 run created
2025-12-14 01:49:32,516 INFO [manager] checklist updated with [sea surface height forcing] items from make_ssh_files worker
2025-12-14 01:49:32,844 INFO [manager] launching NextWorker(module='nowcast.workers.make_runoff_file', args=['v202108'], host='localhost')
2025-12-14 01:49:32,848 INFO [manager] launching NextWorker(module='nowcast.workers.make_201702_runoff_file', args=[], host='localhost')
2025-12-14 01:49:34,209 INFO [make_201702_runoff_file] running in process 1363649
2025-12-14 01:49:34,210 INFO [make_201702_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-14 01:49:34,211 INFO [make_201702_runoff_file] publishing log messages to tcp://*:5563
2025-12-14 01:49:34,212 INFO [make_201702_runoff_file] connected to skookum.eos.ubc.ca port 5555
2025-12-14 01:49:34,241 INFO [make_201702_runoff_file] calculating runoff forcing for b201702 bathymetry
2025-12-14 01:49:34,364 INFO [make_201702_runoff_file] 2025-12-14 runoff file creation from Fraser at Hope and climatology elsewhere complete
2025-12-14 01:49:34,366 INFO [manager] checklist updated with [rivers forcing] items from make_201702_runoff_file worker
2025-12-14 01:49:34,886 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2025m12d16.nc
2025-12-14 01:49:34,892 INFO [grib_to_netcdf] 2025-12-14 NEMO-atmos forcing file for forecast2 created
2025-12-14 01:49:34,895 INFO [manager] checklist updated with [weather forcing] items from grib_to_netcdf worker
2025-12-14 01:49:35,033 INFO [make_runoff_file] running in process 1363647
2025-12-14 01:49:35,034 INFO [make_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-14 01:49:35,036 INFO [make_runoff_file] publishing log messages to tcp://*:5563
2025-12-14 01:49:35,037 INFO [make_runoff_file] connected to skookum.eos.ubc.ca port 5555
2025-12-14 01:49:35,040 INFO [make_runoff_file] calculating NEMO runoff forcing for v202108 bathymetry for 2025-12-13
2025-12-14 01:49:36,096 ERROR [make_runoff_file] no 2025-12-13 discharge obs for SkagitMountVernon: patching
2025-12-14 01:49:36,547 ERROR [make_runoff_file] no 2025-12-13 discharge obs for SnohomishMonroe: patching
2025-12-14 01:49:36,746 ERROR [make_runoff_file] no 2025-12-13 discharge obs for NisquallyMcKenna: patching
2025-12-14 01:49:36,842 ERROR [make_runoff_file] no 2025-12-13 discharge obs for GreenwaterGreenwater: patching
2025-12-14 01:49:38,264 INFO [make_runoff_file] stored NEMO runoff forcing for v202108 bathymetry for 2025-12-13: /results/forcing/rivers/R202108Dailies_y2025m12d13.nc
2025-12-14 01:49:38,265 INFO [make_runoff_file] 2025-12-13 runoff file creation completed
2025-12-14 01:49:38,267 INFO [manager] checklist updated with [rivers forcing] items from make_runoff_file worker
2025-12-14 01:49:38,460 INFO [get_onc_ctd] 2025-12-13 ONC SCVIP CTD T&S file created
2025-12-14 01:49:38,623 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'forecast2'], host='localhost')
2025-12-14 01:49:38,627 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'forecast2'], host='localhost')
2025-12-14 01:49:38,632 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.nibi', 'forecast2'], host='localhost')
2025-12-14 01:49:38,636 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'forecast2'], host='localhost')
2025-12-14 01:49:38,642 INFO [manager] checklist updated with [ONC CTD data] items from get_onc_ctd worker
2025-12-14 01:49:38,764 INFO [get_onc_ctd] 2025-12-13 ONC SEVIP CTD T&S file created
2025-12-14 01:49:38,970 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['SCVIP-CTD'], host='localhost')
2025-12-14 01:49:38,975 INFO [manager] checklist updated with [ONC CTD data] items from get_onc_ctd worker
2025-12-14 01:49:39,314 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['SEVIP-CTD'], host='localhost')
2025-12-14 01:49:40,108 INFO [ping_erddap] running in process 1363694
2025-12-14 01:49:40,110 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-14 01:49:40,111 INFO [ping_erddap] publishing log messages to tcp://*:5563
2025-12-14 01:49:40,112 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2025-12-14 01:49:40,113 INFO [ping_erddap] SCVIP-CTD ERDDAP dataset flag file(s) created
2025-12-14 01:49:40,115 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2025-12-14 01:49:40,120 INFO [upload_forcing] running in process 1363691
2025-12-14 01:49:40,121 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-14 01:49:40,122 INFO [upload_forcing] publishing log messages to tcp://*:5564
2025-12-14 01:49:40,122 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-12-14 01:49:40,137 INFO [upload_forcing] running in process 1363693
2025-12-14 01:49:40,139 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-14 01:49:40,139 INFO [upload_forcing] publishing log messages to tcp://*:5566
2025-12-14 01:49:40,140 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-12-14 01:49:40,160 INFO [upload_forcing] running in process 1363690
2025-12-14 01:49:40,161 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-14 01:49:40,163 INFO [upload_forcing] publishing log messages to tcp://*:5567
2025-12-14 01:49:40,166 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-12-14 01:49:40,167 INFO [upload_forcing] running in process 1363692
2025-12-14 01:49:40,168 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-14 01:49:40,169 INFO [upload_forcing] publishing log messages to tcp://*:5568
2025-12-14 01:49:40,170 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-12-14 01:49:40,519 INFO [ping_erddap] running in process 1363696
2025-12-14 01:49:40,521 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-14 01:49:40,521 INFO [ping_erddap] publishing log messages to tcp://*:5569
2025-12-14 01:49:40,523 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2025-12-14 01:49:40,524 INFO [ping_erddap] SEVIP-CTD ERDDAP dataset flag file(s) created
2025-12-14 01:49:40,526 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2025-12-14 01:49:41,549 INFO [upload_forcing] LiveOcean boundary conditions file not found; created symlink to /results/forcing/LiveOcean/boundary_conditions/LiveOcean_v201905_y2025m12d13.nc
2025-12-14 01:49:41,616 INFO [upload_forcing] forecast2 2025-12-14 forcing files upload to optimum-hindcast completed
2025-12-14 01:49:41,620 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-12-14 01:49:44,467 INFO [upload_forcing] forecast2 2025-12-14 forcing files upload to arbutus.cloud-nowcast completed
2025-12-14 01:49:44,472 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-12-14 01:49:44,827 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2025-12-14'], host='localhost')
2025-12-14 01:49:45,670 INFO [upload_forcing] forecast2 2025-12-14 forcing files upload to orcinus-nowcast-agrif completed
2025-12-14 01:49:45,673 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-12-14 01:49:46,337 INFO [make_forcing_links] running in process 1363732
2025-12-14 01:49:46,338 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-14 01:49:46,339 INFO [make_forcing_links] publishing log messages to tcp://*:5563
2025-12-14 01:49:46,341 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2025-12-14 01:49:47,471 INFO [make_forcing_links] forecast2 2025-12-14 forcing file links on arbutus.cloud-nowcast created
2025-12-14 01:49:47,475 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2025-12-14 01:49:47,809 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2025-12-14'], host='arbutus.cloud-nowcast')
2025-12-14 01:49:49,886 INFO [run_NEMO] running in process 13782
2025-12-14 01:49:49,887 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-12-14 01:49:49,888 INFO [run_NEMO] publishing log messages to tcp://*:5556
2025-12-14 01:49:49,936 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2025-12-14 01:49:50,719 WARNING [run_NEMO] There are uncommitted changes in /nemoShare/MEOPAR/nowcast-sys/SS-run-sets
2025-12-14 01:49:51,018 INFO [run_NEMO] forecast2: launching /nemoShare/MEOPAR/nowcast-sys/runs/13dec25forecast2_2025-12-14T014950.000998-0800/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2025-12-14 01:49:51,030 INFO [run_NEMO] forecast2 NEMO run for 2025-12-14 on arbutus.cloud-nowcast started
2025-12-14 01:49:51,034 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2025-12-14 01:49:51,364 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'forecast2'], host='arbutus.cloud-nowcast')
2025-12-14 01:49:53,325 INFO [watch_NEMO] running in process 14055
2025-12-14 01:49:53,327 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-12-14 01:49:53,328 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2025-12-14 01:49:53,330 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2025-12-14 01:49:53,393 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2025-12-14 01:50:11,592 INFO [upload_forcing] forecast2 2025-12-14 forcing files upload to robot.nibi completed
2025-12-14 01:50:11,597 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-12-14 01:54:53,394 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 18904770 = 2025-12-15 04:59:20 UTC, 16.6% complete
2025-12-14 01:59:53,395 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 18905247 = 2025-12-15 10:17:20 UTC, 34.3% complete
2025-12-14 02:04:53,396 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 18905720 = 2025-12-15 15:32:40 UTC, 51.8% complete
2025-12-14 02:09:53,398 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 18906208 = 2025-12-15 20:58:00 UTC, 69.9% complete
2025-12-14 02:14:53,398 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 18906674 = 2025-12-16 02:08:40 UTC, 87.2% complete
2025-12-14 02:19:53,398 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 18907020 = 2025-12-16 05:59:20 UTC, 100.0% complete
2025-12-14 02:24:53,406 INFO [watch_NEMO] forecast2 NEMO run on arbutus.cloud-nowcast completed
2025-12-14 02:24:53,407 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2025-12-14 02:24:53,965 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_wind_file', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2025-12-14'], host='arbutus.cloud-nowcast')
2025-12-14 02:24:53,975 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_current_file', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2025-12-14'], host='arbutus.cloud-nowcast')
2025-12-14 02:24:53,979 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2025-12-13'], host='localhost')
2025-12-14 02:24:55,539 INFO [download_results] running in process 1364163
2025-12-14 02:24:55,541 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-14 02:24:55,542 INFO [download_results] publishing log messages to tcp://*:5563
2025-12-14 02:24:55,543 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2025-12-14 02:24:55,544 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/forecast2/13dec25 to /results/SalishSea/forecast2.202111
2025-12-14 02:24:56,657 INFO [make_ww3_current_file] running in process 14671
2025-12-14 02:24:56,658 INFO [make_ww3_current_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-12-14 02:24:56,659 INFO [make_ww3_current_file] publishing log messages to tcp://*:5571
2025-12-14 02:24:56,661 INFO [make_ww3_current_file] connected to skookum.eos.ubc.ca port 5555
2025-12-14 02:24:56,663 INFO [make_ww3_current_file] Creating wwatch3 currents forcing file for 2025-12-14 forecast2 run
2025-12-14 02:24:56,808 INFO [make_ww3_wind_file] running in process 14669
2025-12-14 02:24:56,812 INFO [make_ww3_wind_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-12-14 02:24:56,813 INFO [make_ww3_wind_file] publishing log messages to tcp://*:5570
2025-12-14 02:24:56,815 INFO [make_ww3_wind_file] connected to skookum.eos.ubc.ca port 5555
2025-12-14 02:24:56,816 INFO [make_ww3_wind_file] Creating wwatch3 wind forcing file for 2025-12-14 forecast2 run
2025-12-14 02:25:05,664 INFO [make_ww3_wind_file] wwatch3 wind forcing file created on arbutus.cloud-nowcast for 2025-12-14 forecast2 run
2025-12-14 02:25:05,668 INFO [manager] checklist updated with [WW3 wind forcing] items from make_ww3_wind_file worker
2025-12-14 02:26:44,820 INFO [download_results] forecast2 2025-12-13 results files from arbutus.cloud-nowcast downloaded
2025-12-14 02:26:44,824 INFO [manager] checklist updated with [results files] items from download_results worker
2025-12-14 02:26:45,166 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['forecast2', '--run-date', '2025-12-13'], host='localhost')
2025-12-14 02:26:47,461 INFO [make_CHS_currents_file] running in process 1364244
2025-12-14 02:26:47,463 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-14 02:26:47,464 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5563
2025-12-14 02:26:47,465 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555
2025-12-14 02:27:27,463 INFO [make_CHS_currents_file] Made CHS currents file for 2025-12-13 for forecast2
2025-12-14 02:27:27,466 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker
2025-12-14 02:27:27,803 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['nemo', 'forecast2', '--run-date', '2025-12-13'], host='localhost')
2025-12-14 02:27:29,012 INFO [update_forecast_datasets] running in process 1364282
2025-12-14 02:27:29,014 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-14 02:27:29,015 INFO [update_forecast_datasets] publishing log messages to tcp://*:5563
2025-12-14 02:27:29,016 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2025-12-14 02:27:29,017 INFO [update_forecast_datasets] updating nemo forecast directory for forecast2/13dec25 run
2025-12-14 02:28:30,220 INFO [make_ww3_current_file] wwatch3 currents forcing file created on arbutus.cloud-nowcast for 2025-12-14 forecast2 run
2025-12-14 02:28:30,225 INFO [manager] checklist updated with [WW3 currents forcing] items from make_ww3_current_file worker
2025-12-14 02:28:30,559 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2025-12-14'], host='arbutus.cloud-nowcast')
2025-12-14 02:28:32,614 INFO [run_ww3] running in process 15275
2025-12-14 02:28:32,616 INFO [run_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-12-14 02:28:32,617 INFO [run_ww3] publishing log messages to tcp://*:5572
2025-12-14 02:28:32,618 INFO [run_ww3] connected to skookum.eos.ubc.ca port 5555
2025-12-14 02:28:32,622 INFO [run_ww3] Created run directory /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast2_2025-12-14T022832.614558-0800
2025-12-14 02:28:32,623 INFO [run_ww3] forecast2: launching /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast2_2025-12-14T022832.614558-0800/SoGWW3.sh on arbutus.cloud-nowcast
2025-12-14 02:28:32,634 INFO [run_ww3] forecast2 WaveWatch3 run for 2025-12-14 on arbutus.cloud-nowcast started
2025-12-14 02:28:32,638 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker
2025-12-14 02:28:32,958 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'forecast2'], host='arbutus.cloud-nowcast')
2025-12-14 02:28:34,909 INFO [watch_ww3] running in process 15364
2025-12-14 02:28:34,910 INFO [watch_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-12-14 02:28:34,911 INFO [watch_ww3] publishing log messages to tcp://*:5573
2025-12-14 02:28:34,913 INFO [watch_ww3] connected to skookum.eos.ubc.ca port 5555
2025-12-14 02:28:34,946 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2025-12-14 02:31:47,989 INFO [update_forecast_datasets] updated nemo forecast directory for forecast2/13dec25 run: /results/SalishSea/rolling-forecasts/nemo
2025-12-14 02:31:47,990 INFO [update_forecast_datasets] nemo 2025-12-13 forecast2 rolling forecast datasets updated
2025-12-14 02:31:47,993 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2025-12-14 02:31:48,346 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nemo-forecast'], host='localhost')
2025-12-14 02:31:48,351 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'forecast2', 'publish', '--run-date', '2025-12-13'], host='localhost')
2025-12-14 02:31:48,356 INFO [manager] launching NextWorker(module='nowcast.workers.make_surface_current_tiles', args=['forecast2', '--run-date', '2025-12-13'], host='localhost')
2025-12-14 02:31:49,659 INFO [ping_erddap] running in process 1364400
2025-12-14 02:31:49,660 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-14 02:31:49,661 INFO [ping_erddap] publishing log messages to tcp://*:5563
2025-12-14 02:31:49,662 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2025-12-14 02:31:49,671 INFO [ping_erddap] nemo-forecast ERDDAP dataset flag file(s) created
2025-12-14 02:31:49,673 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2025-12-14 02:31:50,940 INFO [make_surface_current_tiles] running in process 1364403
2025-12-14 02:31:50,941 INFO [make_surface_current_tiles] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-14 02:31:50,941 INFO [make_surface_current_tiles] publishing log messages to tcp://*:5563
2025-12-14 02:31:50,942 INFO [make_surface_current_tiles] connected to skookum.eos.ubc.ca port 5555
2025-12-14 02:31:50,980 INFO [make_surface_current_tiles] starting rendering of tiles for 2025-12-13 forecast2 in 6 processes into /results/nowcast-sys/figures/surface_currents/forecast2/13dec25
2025-12-14 02:31:51,591 INFO [make_plots] running in process 1364402
2025-12-14 02:31:51,592 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-14 02:31:51,593 INFO [make_plots] publishing log messages to tcp://*:5564
2025-12-14 02:31:51,594 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2025-12-14 02:31:52,069 INFO [make_plots] preparing render list for 2025-12-13 NEMO forecast2 publish figures
2025-12-14 02:31:52,320 INFO [make_plots] added Website_thumbnail figure to 2025-12-13 NEMO forecast2 publish render list
2025-12-14 02:31:52,326 INFO [make_plots] added Threshold_website figure to 2025-12-13 NEMO forecast2 publish render list
2025-12-14 02:31:52,341 INFO [make_plots] added PA_tidal_predictions figure to 2025-12-13 NEMO forecast2 publish render list
2025-12-14 02:31:52,342 INFO [make_plots] added Boundary Bay figure to 2025-12-13 NEMO forecast2 publish render list
2025-12-14 02:31:52,343 INFO [make_plots] added Campbell River figure to 2025-12-13 NEMO forecast2 publish render list
2025-12-14 02:31:52,344 INFO [make_plots] added Cherry Point figure to 2025-12-13 NEMO forecast2 publish render list
2025-12-14 02:31:52,345 INFO [make_plots] added Friday Harbor figure to 2025-12-13 NEMO forecast2 publish render list
2025-12-14 02:31:52,346 INFO [make_plots] added Halfmoon Bay figure to 2025-12-13 NEMO forecast2 publish render list
2025-12-14 02:31:52,347 INFO [make_plots] added Nanaimo figure to 2025-12-13 NEMO forecast2 publish render list
2025-12-14 02:31:52,348 INFO [make_plots] added Neah Bay figure to 2025-12-13 NEMO forecast2 publish render list
2025-12-14 02:31:52,349 INFO [make_plots] added New Westminster figure to 2025-12-13 NEMO forecast2 publish render list
2025-12-14 02:31:52,351 INFO [make_plots] added Patricia Bay figure to 2025-12-13 NEMO forecast2 publish render list
2025-12-14 02:31:52,351 INFO [make_plots] added Point Atkinson figure to 2025-12-13 NEMO forecast2 publish render list
2025-12-14 02:31:52,352 INFO [make_plots] added Port Renfrew figure to 2025-12-13 NEMO forecast2 publish render list
2025-12-14 02:31:52,353 INFO [make_plots] added Sand Heads figure to 2025-12-13 NEMO forecast2 publish render list
2025-12-14 02:31:52,354 INFO [make_plots] added Sandy Cove figure to 2025-12-13 NEMO forecast2 publish render list
2025-12-14 02:31:52,354 INFO [make_plots] added Squamish figure to 2025-12-13 NEMO forecast2 publish render list
2025-12-14 02:31:52,355 INFO [make_plots] added Victoria figure to 2025-12-13 NEMO forecast2 publish render list
2025-12-14 02:31:52,356 INFO [make_plots] added Woodwards Landing figure to 2025-12-13 NEMO forecast2 publish render list
2025-12-14 02:31:52,356 INFO [make_plots] starting to render nemo forecast2 publish 13dec25 figures
2025-12-14 02:33:34,946 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 90, 16.7% complete
2025-12-14 02:38:34,947 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 183, 33.9% complete
2025-12-14 02:39:26,356 WARNING [make_plots] invalid station number for Boundary Bay station: None
2025-12-14 02:39:26,357 WARNING [make_plots] can't resolve a valid CHS station code for Boundary Bay
2025-12-14 02:39:52,333 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb996/data for station 08074 Campbell River from 2025-12-09 00:00:00Z to 2025-12-16 05:55:00Z
2025-12-14 02:41:12,302 WARNING [make_plots] invalid station number for Halfmoon Bay station: None
2025-12-14 02:41:12,303 WARNING [make_plots] can't resolve a valid CHS station code for Halfmoon Bay
2025-12-14 02:41:34,192 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb96d/data for station 07917 Nanaimo from 2025-12-09 00:00:00Z to 2025-12-16 05:55:00Z
2025-12-14 02:42:24,782 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e13d0f4a073c4bbf93/data for station 07654 New Westminster from 2025-12-09 00:00:00Z to 2025-12-16 05:55:00Z
2025-12-14 02:42:47,883 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd2d/data for station 07277 Patricia Bay from 2025-12-09 00:00:00Z to 2025-12-16 05:55:00Z
2025-12-14 02:43:01,324 INFO [make_surface_current_tiles] starting PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast2/13dec25
2025-12-14 02:43:09,430 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94c/data for station 07795 Point Atkinson from 2025-12-09 00:00:00Z to 2025-12-16 05:55:00Z
2025-12-14 02:43:28,521 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e23d0f4a073c4bc060/data for station 08525 Port Renfrew from 2025-12-09 00:00:00Z to 2025-12-16 05:55:00Z
2025-12-14 02:43:34,948 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 270, 50.0% complete
2025-12-14 02:43:46,489 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb93c/data for station 07594 Sand Heads from 2025-12-09 00:00:00Z to 2025-12-16 05:55:00Z
2025-12-14 02:44:05,123 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc43b/data for station 07786 Sandy Cove from 2025-12-09 00:00:00Z to 2025-12-16 05:55:00Z
2025-12-14 02:44:22,544 WARNING [make_plots] invalid station number for Squamish station: None
2025-12-14 02:44:22,545 WARNING [make_plots] can't resolve a valid CHS station code for Squamish
2025-12-14 02:44:41,696 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd1e/data for station 07120 Victoria from 2025-12-09 00:00:00Z to 2025-12-16 05:55:00Z
2025-12-14 02:45:01,718 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc42d/data for station 07610 Woodwards Landing from 2025-12-09 00:00:00Z to 2025-12-16 05:55:00Z
2025-12-14 02:45:20,579 INFO [make_plots] finished rendering nemo forecast2 publish 13dec25 figures
2025-12-14 02:45:20,583 INFO [make_plots] nemo publish plots for 2025-12-13 forecast2 completed
2025-12-14 02:45:20,591 INFO [manager] checklist updated with [plots] items from make_plots worker
2025-12-14 02:45:20,943 INFO [manager] launching NextWorker(module='nowcast.workers.make_feeds', args=['forecast2', '--run-date', '2025-12-13'], host='localhost')
2025-12-14 02:45:24,381 INFO [make_feeds] running in process 1369449
2025-12-14 02:45:24,382 INFO [make_feeds] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-14 02:45:24,383 INFO [make_feeds] publishing log messages to tcp://*:5564
2025-12-14 02:45:24,384 INFO [make_feeds] connected to skookum.eos.ubc.ca port 5555
2025-12-14 02:45:38,829 INFO [make_feeds] ATOM feeds for 2025-12-13 forecast2 run completed
2025-12-14 02:45:38,832 INFO [manager] checklist updated with [storm surge alert feeds] items from make_feeds worker
2025-12-14 02:45:39,166 INFO [manager] launching NextWorker(module='nemo_nowcast.workers.clear_checklist', args=[], host='localhost')
2025-12-14 02:45:40,251 INFO [clear_checklist] running in process 1369476
2025-12-14 02:45:40,252 INFO [clear_checklist] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-14 02:45:40,254 INFO [clear_checklist] publishing log messages to tcp://*:5564
2025-12-14 02:45:40,255 INFO [clear_checklist] connected to skookum.eos.ubc.ca port 5555
2025-12-14 02:45:40,256 INFO [clear_checklist] requesting that manager clear system state checklist
2025-12-14 02:45:40,258 INFO [manager] writing checklist to log file
2025-12-14 02:45:40,305 INFO [manager] checklist cleared
2025-12-14 02:45:40,308 INFO [clear_checklist] nowcast system checklist cleared
2025-12-14 02:45:40,310 INFO [manager] launching NextWorker(module='nemo_nowcast.workers.rotate_logs', args=[], host='localhost')
2025-12-14 02:45:41,405 INFO [rotate_logs] running in process 1369481
2025-12-14 02:45:41,407 INFO [rotate_logs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-14 02:45:41,408 INFO [rotate_logs] publishing log messages to tcp://*:5564
2025-12-14 02:45:41,409 INFO [rotate_logs] connected to skookum.eos.ubc.ca port 5555
2025-12-14 02:45:41,410 INFO [rotate_logs] rotating log files