2025-12-17 02:43:21,030 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/nowcast.log
2025-12-17 02:43:21,038 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/nowcast.debug.log
2025-12-17 02:43:21,043 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/checklist.log
2025-12-17 02:43:21,044 INFO [rotate_logs] log files rotated
2025-12-17 02:43:21,049 INFO [manager] checklist updated with [log rotation] items from rotate_logs worker
2025-12-17 02:45:03,578 INFO [make_surface_current_tiles] finished PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast2/16dec25
2025-12-17 02:45:03,600 INFO [make_surface_current_tiles] finished rendering of tiles for 2025-12-16 forecast2 into /results/nowcast-sys/figures/surface_currents/forecast2/16dec25
2025-12-17 02:45:03,601 INFO [make_surface_current_tiles] surface current tile figures for 2025-12-16 forecast2 completed
2025-12-17 02:45:03,866 INFO [manager] checklist updated with [surface current tiles] items from make_surface_current_tiles worker
2025-12-17 02:46:30,536 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 360, 66.7% complete
2025-12-17 02:51:30,536 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 450, 83.3% complete
2025-12-17 02:56:30,538 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2025-12-17 03:01:30,541 INFO [watch_ww3] forecast2 WWATCH3 run on arbutus.cloud-nowcast completed
2025-12-17 03:01:30,544 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2025-12-17 03:01:30,870 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2025-12-17'], host='localhost')
2025-12-17 03:01:32,199 INFO [download_wwatch3_results] running in process 1464847
2025-12-17 03:01:32,201 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-17 03:01:32,202 INFO [download_wwatch3_results] publishing log messages to tcp://*:5563
2025-12-17 03:01:32,203 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2025-12-17 03:02:09,147 INFO [download_wwatch3_results] forecast2 2025-12-17 results files from arbutus.cloud-nowcast downloaded
2025-12-17 03:02:09,151 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2025-12-17 03:02:09,299 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['wwatch3', 'forecast2', '--run-date', '2025-12-17'], host='localhost')
2025-12-17 03:02:10,510 INFO [update_forecast_datasets] running in process 1464886
2025-12-17 03:02:10,513 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-17 03:02:10,515 INFO [update_forecast_datasets] publishing log messages to tcp://*:5563
2025-12-17 03:02:10,516 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2025-12-17 03:02:10,517 INFO [update_forecast_datasets] updating wwatch3 most_recent_forecast directory from forecast2/17dec25 run
2025-12-17 03:02:10,568 INFO [update_forecast_datasets] updated wwatch3 most_recent_forecast directory from forecast2/17dec25 run
2025-12-17 03:02:10,569 INFO [update_forecast_datasets] updating wwatch3 forecast directory for forecast2/17dec25 run
2025-12-17 03:02:34,653 INFO [update_forecast_datasets] updated wwatch3 forecast directory for forecast2/17dec25 run: /results/SalishSea/rolling-forecasts/wwatch3
2025-12-17 03:02:34,654 INFO [update_forecast_datasets] wwatch3 2025-12-17 forecast2 rolling forecast datasets updated
2025-12-17 03:02:34,657 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2025-12-17 03:02:34,804 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['wwatch3-forecast'], host='localhost')
2025-12-17 03:02:36,120 INFO [ping_erddap] running in process 1464901
2025-12-17 03:02:36,122 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-17 03:02:36,123 INFO [ping_erddap] publishing log messages to tcp://*:5563
2025-12-17 03:02:36,124 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2025-12-17 03:02:36,126 INFO [ping_erddap] wwatch3-forecast ERDDAP dataset flag file(s) created
2025-12-17 03:02:36,128 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2025-12-17 03:02:36,281 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['wwatch3', 'forecast2', 'publish', '--run-date', '2025-12-17'], host='localhost')
2025-12-17 03:02:39,327 INFO [make_plots] running in process 1464906
2025-12-17 03:02:39,328 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-17 03:02:39,329 INFO [make_plots] publishing log messages to tcp://*:5563
2025-12-17 03:02:39,330 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2025-12-17 03:02:39,331 INFO [make_plots] preparing render list for 2025-12-17 WaveWatch3 forecast2 publish figures
2025-12-17 03:02:39,332 INFO [make_plots] added Halibut Bank figure to 2025-12-17 WaveWatch3 forecast2 publish render list
2025-12-17 03:02:39,333 INFO [make_plots] added Sentry Shoal figure to 2025-12-17 WaveWatch3 forecast2 publish render list
2025-12-17 03:02:39,334 INFO [make_plots] starting to render wwatch3 forecast2 publish 17dec25 figures
2025-12-17 03:02:39,897 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46146 Halibut Bank
2025-12-17 03:02:44,673 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46131 Sentry Shoal
2025-12-17 03:02:49,752 INFO [make_plots] finished rendering wwatch3 forecast2 publish 17dec25 figures
2025-12-17 03:02:49,752 INFO [make_plots] wwatch3 publish plots for 2025-12-17 forecast2 completed
2025-12-17 03:02:49,756 INFO [manager] checklist updated with [plots] items from make_plots worker
2025-12-17 07:49:02,010 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/12/ to /results/forcing/atmospheric/continental2.5/GRIB/20251217/12/
2025-12-17 07:49:02,011 INFO [collect_weather] 2025-12-17 2.5km weather forecast 12 collection complete
2025-12-17 07:49:02,014 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2025-12-17 07:49:02,338 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'SkagitMountVernon', '--data-date', '2025-12-16'], host='localhost')
2025-12-17 07:49:02,345 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'SnohomishMonroe', '--data-date', '2025-12-16'], host='localhost')
2025-12-17 07:49:02,351 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'NisquallyMcKenna', '--data-date', '2025-12-16'], host='localhost')
2025-12-17 07:49:02,355 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'GreenwaterGreenwater', '--data-date', '2025-12-16'], host='localhost')
2025-12-17 07:49:02,359 INFO [manager] launching NextWorker(module='nowcast.workers.make_turbidity_file', args=[], host='localhost')
2025-12-17 07:49:02,364 INFO [manager] launching NextWorker(module='nowcast.workers.collect_NeahBay_ssh', args=['06'], host='localhost')
2025-12-17 07:49:02,371 INFO [manager] launching NextWorker(module='nowcast.workers.download_live_ocean', args=[], host='localhost')
2025-12-17 07:49:02,376 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['18', '2.5km'], host='localhost')
2025-12-17 07:49:02,381 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['18'], host='localhost')
2025-12-17 07:49:02,874 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20251217/12/
2025-12-17 07:49:02,875 INFO [crop_gribs] 2025-12-17 12 GRIBs cropping complete
2025-12-17 07:49:02,877 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2025-12-17 07:49:03,496 INFO [manager] launching NextWorker(module='nowcast.workers.grib_to_netcdf', args=['nowcast+'], host='localhost')
2025-12-17 07:49:04,021 INFO [collect_weather] running in process 1469236
2025-12-17 07:49:04,022 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-17 07:49:04,023 INFO [collect_weather] publishing log messages to tcp://*:5562
2025-12-17 07:49:04,024 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2025-12-17 07:49:04,036 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/18/
2025-12-17 07:49:04,049 INFO [collect_NeahBay_ssh] running in process 1469234
2025-12-17 07:49:04,050 INFO [collect_NeahBay_ssh] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-17 07:49:04,051 INFO [collect_NeahBay_ssh] publishing log messages to tcp://*:5563
2025-12-17 07:49:04,052 INFO [collect_NeahBay_ssh] connected to skookum.eos.ubc.ca port 5555
2025-12-17 07:49:04,053 INFO [collect_NeahBay_ssh] collecting Neah Bay ssh 06Z obs/forecast for 2025-12-17
2025-12-17 07:49:04,153 INFO [download_live_ocean] running in process 1469235
2025-12-17 07:49:04,154 INFO [download_live_ocean] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-17 07:49:04,155 INFO [download_live_ocean] publishing log messages to tcp://*:5564
2025-12-17 07:49:04,156 INFO [download_live_ocean] connected to skookum.eos.ubc.ca port 5555
2025-12-17 07:49:04,156 INFO [download_live_ocean] downloading Salish Sea western boundary day-averaged LiveOcean file for 2025-12-17
2025-12-17 07:49:04,550 INFO [collect_river_data] running in process 1469231
2025-12-17 07:49:04,551 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-17 07:49:04,552 INFO [collect_river_data] publishing log messages to tcp://*:5565
2025-12-17 07:49:04,553 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-12-17 07:49:04,554 INFO [collect_river_data] Collecting USGS NisquallyMcKenna river data for 2025-12-16
2025-12-17 07:49:04,566 INFO [collect_river_data] running in process 1469230
2025-12-17 07:49:04,567 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-17 07:49:04,568 INFO [collect_river_data] publishing log messages to tcp://*:5566
2025-12-17 07:49:04,569 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-12-17 07:49:04,569 INFO [collect_river_data] Collecting USGS SnohomishMonroe river data for 2025-12-16
2025-12-17 07:49:04,588 INFO [collect_river_data] running in process 1469232
2025-12-17 07:49:04,589 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-17 07:49:04,590 INFO [collect_river_data] publishing log messages to tcp://*:5567
2025-12-17 07:49:04,593 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-12-17 07:49:04,606 INFO [collect_river_data] Collecting USGS GreenwaterGreenwater river data for 2025-12-16
2025-12-17 07:49:04,608 INFO [collect_river_data] running in process 1469228
2025-12-17 07:49:04,609 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-17 07:49:04,610 INFO [collect_river_data] publishing log messages to tcp://*:5568
2025-12-17 07:49:04,611 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-12-17 07:49:04,612 INFO [collect_river_data] Collecting USGS SkagitMountVernon river data for 2025-12-16
2025-12-17 07:49:04,670 INFO [make_turbidity_file] running in process 1469233
2025-12-17 07:49:04,672 INFO [make_turbidity_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-17 07:49:04,673 INFO [make_turbidity_file] publishing log messages to tcp://*:5569
2025-12-17 07:49:04,673 INFO [make_turbidity_file] connected to skookum.eos.ubc.ca port 5555
2025-12-17 07:49:04,674 INFO [make_turbidity_file] Creating Fraser River turbidity forcing file for 2025-12-17
2025-12-17 07:49:04,957 INFO [crop_gribs] running in process 1469237
2025-12-17 07:49:04,958 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-17 07:49:04,959 INFO [crop_gribs] publishing log messages to tcp://*:5570
2025-12-17 07:49:04,960 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2025-12-17 07:49:04,961 INFO [crop_gribs] cropping 2025-12-17 ECCC HRDPS 2.5km continental 18Z GRIB files to SalishSeaCast subdomain
2025-12-17 07:49:04,974 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20251217/18/
2025-12-17 07:49:05,185 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12089500&startDT=2025-12-16&endDT=2025-12-16 "HTTP/1.1 200 "
2025-12-17 07:49:05,186 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12200500&startDT=2025-12-16&endDT=2025-12-16 "HTTP/1.1 200 "
2025-12-17 07:49:05,196 INFO [collect_river_data] Appended USGS NisquallyMcKenna river average discharge for 2025-12-16 to: /results/forcing/rivers/observations/Nisqually_McKenna_flow
2025-12-17 07:49:05,197 INFO [collect_river_data] Appended USGS SkagitMountVernon river average discharge for 2025-12-16 to: /results/forcing/rivers/observations/Skagit_MountVernon_flow
2025-12-17 07:49:05,198 INFO [collect_river_data] USGS NisquallyMcKenna river data collection for 2025-12-16 completed
2025-12-17 07:49:05,198 INFO [collect_river_data] USGS SkagitMountVernon river data collection for 2025-12-16 completed
2025-12-17 07:49:05,199 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12150800&startDT=2025-12-16&endDT=2025-12-16 "HTTP/1.1 200 "
2025-12-17 07:49:05,201 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-12-17 07:49:05,202 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12097500&startDT=2025-12-16&endDT=2025-12-16 "HTTP/1.1 200 "
2025-12-17 07:49:05,210 INFO [collect_river_data] Appended USGS SnohomishMonroe river average discharge for 2025-12-16 to: /results/forcing/rivers/observations/Snohomish_Monroe_flow
2025-12-17 07:49:05,211 INFO [collect_river_data] USGS SnohomishMonroe river data collection for 2025-12-16 completed
2025-12-17 07:49:05,212 INFO [collect_river_data] Appended USGS GreenwaterGreenwater river average discharge for 2025-12-16 to: /results/forcing/rivers/observations/Greenwater_Greenwater_flow
2025-12-17 07:49:05,213 INFO [collect_river_data] USGS GreenwaterGreenwater river data collection for 2025-12-16 completed
2025-12-17 07:49:05,367 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-12-17 07:49:05,497 INFO [grib_to_netcdf] running in process 1469316
2025-12-17 07:49:05,498 INFO [grib_to_netcdf] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-17 07:49:05,499 INFO [grib_to_netcdf] publishing log messages to tcp://*:5571
2025-12-17 07:49:05,500 INFO [grib_to_netcdf] connected to skookum.eos.ubc.ca port 5555
2025-12-17 07:49:05,501 INFO [grib_to_netcdf] creating NEMO-atmos forcing files for 2025-12-17 nowcast and forecast runs
2025-12-17 07:49:05,521 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-12-17 07:49:05,569 INFO [collect_NeahBay_ssh] 2025-12-17T00:00:00-08:00 Neah Bay ssh 06Z obs/forecast data collection complete
2025-12-17 07:49:05,681 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-12-17 07:49:05,817 INFO [manager] checklist updated with [Neah Bay ssh data] items from collect_NeahBay_ssh worker
2025-12-17 07:49:05,980 INFO [manager] launching NextWorker(module='nowcast.workers.make_ssh_files', args=['nowcast', '--run-date', '2025-12-17'], host='localhost')
2025-12-17 07:49:06,992 INFO [make_turbidity_file] 2025-12-17 Fraser River turbidity file creation complete
2025-12-17 07:49:06,994 INFO [manager] checklist updated with [Fraser River turbidity] items from make_turbidity_file worker
2025-12-17 07:49:08,685 INFO [make_ssh_files] running in process 1469372
2025-12-17 07:49:08,686 INFO [make_ssh_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-17 07:49:08,687 INFO [make_ssh_files] publishing log messages to tcp://*:5563
2025-12-17 07:49:08,688 INFO [make_ssh_files] connected to skookum.eos.ubc.ca port 5555
2025-12-17 07:49:08,689 INFO [make_ssh_files] building nowcast sea surface height boundary conditions file(s) from 2025-12-17 NOAA Neah Bay 06Z observation and forecast values
2025-12-17 07:49:10,423 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2025m12d13.nc
2025-12-17 07:49:10,580 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2025m12d14.nc
2025-12-17 07:49:10,747 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2025m12d15.nc
2025-12-17 07:49:10,912 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2025m12d16.nc
2025-12-17 07:49:11,114 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2025m12d17.nc
2025-12-17 07:49:11,300 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2025m12d18.nc
2025-12-17 07:49:11,492 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2025m12d19.nc
2025-12-17 07:49:11,690 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2025m12d20.nc
2025-12-17 07:49:11,813 INFO [make_ssh_files] sea surface height boundary file for 2025-12-17 nowcast run created
2025-12-17 07:49:11,816 INFO [manager] checklist updated with [sea surface height forcing] items from make_ssh_files worker
2025-12-17 07:49:11,965 INFO [manager] launching NextWorker(module='nowcast.workers.make_runoff_file', args=['v202108'], host='localhost')
2025-12-17 07:49:11,970 INFO [manager] launching NextWorker(module='nowcast.workers.make_201702_runoff_file', args=[], host='localhost')
2025-12-17 07:49:13,229 INFO [make_201702_runoff_file] running in process 1469428
2025-12-17 07:49:13,230 INFO [make_201702_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-17 07:49:13,231 INFO [make_201702_runoff_file] publishing log messages to tcp://*:5563
2025-12-17 07:49:13,232 INFO [make_201702_runoff_file] connected to skookum.eos.ubc.ca port 5555
2025-12-17 07:49:13,259 INFO [make_201702_runoff_file] calculating runoff forcing for b201702 bathymetry
2025-12-17 07:49:13,362 INFO [make_201702_runoff_file] 2025-12-17 runoff file creation from Fraser at Hope and climatology elsewhere complete
2025-12-17 07:49:13,365 INFO [manager] checklist updated with [rivers forcing] items from make_201702_runoff_file worker
2025-12-17 07:49:13,930 INFO [make_runoff_file] running in process 1469426
2025-12-17 07:49:13,932 INFO [make_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-17 07:49:13,933 INFO [make_runoff_file] publishing log messages to tcp://*:5563
2025-12-17 07:49:13,934 INFO [make_runoff_file] connected to skookum.eos.ubc.ca port 5555
2025-12-17 07:49:13,935 INFO [make_runoff_file] calculating NEMO runoff forcing for v202108 bathymetry for 2025-12-16
2025-12-17 07:49:16,329 INFO [make_runoff_file] stored NEMO runoff forcing for v202108 bathymetry for 2025-12-16: /results/forcing/rivers/R202108Dailies_y2025m12d16.nc
2025-12-17 07:49:16,330 INFO [make_runoff_file] 2025-12-16 runoff file creation completed
2025-12-17 07:49:16,332 INFO [manager] checklist updated with [rivers forcing] items from make_runoff_file worker
2025-12-17 07:49:16,429 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/hrdps_y2025m12d17.nc
2025-12-17 07:49:25,228 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2025m12d18.nc
2025-12-17 07:49:30,103 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2025m12d19.nc
2025-12-17 07:49:30,116 INFO [grib_to_netcdf] 2025-12-17 NEMO-atmos forcing file for nowcast+ created
2025-12-17 07:49:30,119 INFO [manager] checklist updated with [weather forcing] items from grib_to_netcdf worker
2025-12-17 10:49:07,812 ERROR [download_live_ocean] giving up after 37 attempts: 403 Client Error: Forbidden for url: https://s3.kopah.uw.edu/liveocean-share/f2025.12.17/ubc_done.txt for https://s3.kopah.uw.edu/liveocean-share/f2025.12.17/ubc_done.txt
2025-12-17 10:49:08,233 CRITICAL [download_live_ocean] 2025-12-17 Live Ocean file for Salish Sea western boundary download failed
2025-12-17 10:51:59,171 INFO [download_live_ocean] running in process 1472859
2025-12-17 10:51:59,172 INFO [download_live_ocean] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-17 10:51:59,173 INFO [download_live_ocean] publishing log messages to tcp://*:5563
2025-12-17 10:51:59,174 INFO [download_live_ocean] connected to skookum.eos.ubc.ca port 5555
2025-12-17 10:51:59,175 INFO [download_live_ocean] downloading Salish Sea western boundary day-averaged LiveOcean file for 2025-12-17
2025-12-17 14:02:05,842 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/18/ to /results/forcing/atmospheric/continental2.5/GRIB/20251217/18/
2025-12-17 14:02:05,844 INFO [collect_weather] 2025-12-17 2.5km weather forecast 18 collection complete
2025-12-17 14:02:05,847 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2025-12-17 14:02:06,162 INFO [manager] launching NextWorker(module='nowcast.workers.download_weather', args=['00', '1km'], host='localhost')
2025-12-17 14:02:06,169 INFO [manager] launching NextWorker(module='nowcast.workers.download_weather', args=['12', '1km'], host='localhost')
2025-12-17 14:02:06,174 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['00', '2.5km'], host='localhost')
2025-12-17 14:02:06,180 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['00', '--fcst-date', '2025-12-18'], host='localhost')
2025-12-17 14:02:06,926 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20251217/18/
2025-12-17 14:02:06,927 INFO [crop_gribs] 2025-12-17 18 GRIBs cropping complete
2025-12-17 14:02:06,929 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2025-12-17 14:02:07,510 INFO [download_weather] running in process 1476097
2025-12-17 14:02:07,512 INFO [download_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-17 14:02:07,513 INFO [download_weather] publishing log messages to tcp://*:5562
2025-12-17 14:02:07,513 INFO [download_weather] connected to skookum.eos.ubc.ca port 5555
2025-12-17 14:02:07,514 INFO [download_weather] downloading 00 1 km forecast GRIB2 files for 20251217
2025-12-17 14:02:07,515 INFO [download_weather] running in process 1476099
2025-12-17 14:02:07,516 INFO [collect_weather] running in process 1476100
2025-12-17 14:02:07,518 INFO [download_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-17 14:02:07,519 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-17 14:02:07,520 INFO [download_weather] publishing log messages to tcp://*:5563
2025-12-17 14:02:07,521 INFO [collect_weather] publishing log messages to tcp://*:5564
2025-12-17 14:02:07,522 INFO [download_weather] connected to skookum.eos.ubc.ca port 5555
2025-12-17 14:02:07,523 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2025-12-17 14:02:07,524 INFO [download_weather] downloading 12 1 km forecast GRIB2 files for 20251217
2025-12-17 14:02:07,532 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/00/
2025-12-17 14:02:08,806 INFO [crop_gribs] running in process 1476101
2025-12-17 14:02:08,808 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-17 14:02:08,809 INFO [crop_gribs] publishing log messages to tcp://*:5565
2025-12-17 14:02:08,810 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2025-12-17 14:02:08,811 INFO [crop_gribs] cropping 2025-12-18 ECCC HRDPS 2.5km continental 00Z GRIB files to SalishSeaCast subdomain
2025-12-17 14:02:08,826 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20251218/00/
2025-12-17 14:06:01,194 INFO [download_weather] 2025-12-17 1km weather forecast 12 downloads complete
2025-12-17 14:06:01,197 INFO [manager] checklist updated with [weather forecast] items from download_weather worker
2025-12-17 14:11:37,353 INFO [download_weather] 2025-12-17 1km weather forecast 00 downloads complete
2025-12-17 14:11:37,356 INFO [manager] checklist updated with [weather forecast] items from download_weather worker
2025-12-17 15:04:16,740 INFO [download_live_ocean] running in process 1477878
2025-12-17 15:04:16,742 INFO [download_live_ocean] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-17 15:04:16,743 INFO [download_live_ocean] publishing log messages to tcp://*:5562
2025-12-17 15:04:16,744 INFO [download_live_ocean] connected to skookum.eos.ubc.ca port 5555
2025-12-17 15:04:16,745 INFO [download_live_ocean] downloading Salish Sea western boundary day-averaged LiveOcean file for 2025-12-17
2025-12-17 16:44:19,312 INFO [download_live_ocean] downloaded 10128422 bytes from https://s3.kopah.uw.edu/liveocean-share/f2025.12.17/ubc.nc to /results/forcing/LiveOcean/downloaded/20251217/low_passed_UBC.nc
2025-12-17 16:44:19,313 INFO [download_live_ocean] Deflating in up to 1 concurrent sub-processes
2025-12-17 16:44:20,342 INFO [download_live_ocean] netCDF4 deflated /results/forcing/LiveOcean/downloaded/20251217/low_passed_UBC.nc
2025-12-17 16:44:20,344 INFO [download_live_ocean] 2025-12-17 Live Ocean file for Salish Sea western boundary downloaded
2025-12-17 16:44:20,346 INFO [manager] checklist updated with [Live Ocean products] items from download_live_ocean worker
2025-12-17 16:44:20,668 INFO [manager] launching NextWorker(module='nowcast.workers.make_live_ocean_files', args=['--run-date', '2025-12-17'], host='localhost')
2025-12-17 16:44:23,619 INFO [make_live_ocean_files] running in process 1479443
2025-12-17 16:44:23,620 INFO [make_live_ocean_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-17 16:44:23,621 INFO [make_live_ocean_files] publishing log messages to tcp://*:5562
2025-12-17 16:44:23,622 INFO [make_live_ocean_files] connected to skookum.eos.ubc.ca port 5555
2025-12-17 16:44:23,624 INFO [make_live_ocean_files] Creating T&S western boundary conditions file from 2025-12-17 Live Ocean run
2025-12-17 16:45:02,522 INFO [make_live_ocean_files] Stored T&S western boundary conditions file: /results/forcing/LiveOcean/boundary_conditions/LiveOcean_v201905_y2025m12d17.nc
2025-12-17 16:45:02,523 INFO [make_live_ocean_files] 2025-12-17 Live Ocean western boundary conditions files created
2025-12-17 16:45:02,526 INFO [manager] checklist updated with [Live Ocean boundary conditions] items from make_live_ocean_files worker
2025-12-17 16:45:02,680 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['weather'], host='localhost')
2025-12-17 16:45:02,685 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'nowcast+'], host='localhost')
2025-12-17 16:45:02,690 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'nowcast+'], host='localhost')
2025-12-17 16:45:02,696 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.nibi', 'nowcast+'], host='localhost')
2025-12-17 16:45:02,701 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'nowcast+'], host='localhost')
2025-12-17 16:45:03,799 INFO [ping_erddap] running in process 1479489
2025-12-17 16:45:03,800 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-17 16:45:03,801 INFO [ping_erddap] publishing log messages to tcp://*:5562
2025-12-17 16:45:03,802 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2025-12-17 16:45:03,804 INFO [ping_erddap] weather ERDDAP dataset flag file(s) created
2025-12-17 16:45:03,805 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2025-12-17 16:45:03,885 INFO [upload_forcing] running in process 1479494
2025-12-17 16:45:03,886 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-17 16:45:03,887 INFO [upload_forcing] publishing log messages to tcp://*:5563
2025-12-17 16:45:03,888 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-12-17 16:45:03,891 INFO [upload_forcing] running in process 1479491
2025-12-17 16:45:03,892 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-17 16:45:03,893 INFO [upload_forcing] publishing log messages to tcp://*:5566
2025-12-17 16:45:03,894 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-12-17 16:45:03,895 INFO [upload_forcing] running in process 1479493
2025-12-17 16:45:03,896 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-17 16:45:03,897 INFO [upload_forcing] publishing log messages to tcp://*:5567
2025-12-17 16:45:03,898 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-12-17 16:45:03,936 INFO [upload_forcing] running in process 1479492
2025-12-17 16:45:03,937 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-17 16:45:03,938 INFO [upload_forcing] publishing log messages to tcp://*:5568
2025-12-17 16:45:03,939 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-12-17 16:45:05,911 INFO [upload_forcing] nowcast+ 2025-12-17 forcing files upload to optimum-hindcast completed
2025-12-17 16:45:05,915 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-12-17 16:45:09,856 INFO [upload_forcing] nowcast+ 2025-12-17 forcing files upload to arbutus.cloud-nowcast completed
2025-12-17 16:45:09,860 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-12-17 16:45:10,011 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'nowcast+', '--run-date', '2025-12-17'], host='localhost')
2025-12-17 16:45:11,539 INFO [upload_forcing] nowcast+ 2025-12-17 forcing files upload to orcinus-nowcast-agrif completed
2025-12-17 16:45:11,543 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-12-17 16:45:11,728 INFO [make_forcing_links] running in process 1479530
2025-12-17 16:45:11,729 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-17 16:45:11,730 INFO [make_forcing_links] publishing log messages to tcp://*:5562
2025-12-17 16:45:11,731 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2025-12-17 16:45:13,183 INFO [make_forcing_links] nowcast+ 2025-12-17 forcing file links on arbutus.cloud-nowcast created
2025-12-17 16:45:13,188 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2025-12-17 16:45:13,350 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2025-12-17'], host='arbutus.cloud-nowcast')
2025-12-17 16:45:15,769 INFO [run_NEMO] running in process 5313
2025-12-17 16:45:15,773 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-12-17 16:45:15,774 INFO [run_NEMO] publishing log messages to tcp://*:5556
2025-12-17 16:45:15,839 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2025-12-17 16:45:16,620 WARNING [run_NEMO] There are uncommitted changes in /nemoShare/MEOPAR/nowcast-sys/SS-run-sets
2025-12-17 16:45:16,883 INFO [upload_forcing] nowcast+ 2025-12-17 forcing files upload to robot.nibi completed
2025-12-17 16:45:16,888 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-12-17 16:45:16,930 INFO [run_NEMO] nowcast: launching /nemoShare/MEOPAR/nowcast-sys/runs/17dec25nowcast_2025-12-17T164515.815803-0800/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2025-12-17 16:45:16,942 INFO [run_NEMO] nowcast NEMO run for 2025-12-17 on arbutus.cloud-nowcast started
2025-12-17 16:45:17,045 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2025-12-17 16:45:17,193 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'nowcast'], host='arbutus.cloud-nowcast')
2025-12-17 16:45:19,193 INFO [watch_NEMO] running in process 5586
2025-12-17 16:45:19,195 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-12-17 16:45:19,196 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2025-12-17 16:45:19,197 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2025-12-17 16:45:19,262 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2025-12-17 16:50:19,264 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 18909104 = 2025-12-17 05:08:40 UTC, 21.4% complete
2025-12-17 16:55:19,267 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 18909586 = 2025-12-17 10:30:00 UTC, 43.8% complete
2025-12-17 17:00:19,267 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 18910094 = 2025-12-17 16:08:40 UTC, 67.3% complete
2025-12-17 17:05:19,269 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 18910616 = 2025-12-17 21:56:40 UTC, 91.5% complete
2025-12-17 17:10:19,274 INFO [watch_NEMO] nowcast NEMO run on arbutus.cloud-nowcast completed
2025-12-17 17:10:19,276 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2025-12-17 17:10:19,608 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'ssh', '--run-date', '2025-12-17'], host='localhost')
2025-12-17 17:10:19,614 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2025-12-17'], host='localhost')
2025-12-17 17:10:20,702 INFO [download_results] running in process 1479896
2025-12-17 17:10:20,703 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-17 17:10:20,705 INFO [download_results] publishing log messages to tcp://*:5562
2025-12-17 17:10:20,706 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2025-12-17 17:10:20,707 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/nowcast/17dec25 to /results/SalishSea/nowcast-blue.202111
2025-12-17 17:10:20,868 INFO [make_forcing_links] running in process 1479895
2025-12-17 17:10:20,869 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-17 17:10:20,871 INFO [make_forcing_links] publishing log messages to tcp://*:5563
2025-12-17 17:10:20,872 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2025-12-17 17:10:21,836 INFO [make_forcing_links] ssh 2025-12-17 forcing file links on arbutus.cloud-nowcast created
2025-12-17 17:10:21,840 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2025-12-17 17:10:21,995 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2025-12-17'], host='arbutus.cloud-nowcast')
2025-12-17 17:10:24,160 INFO [run_NEMO] running in process 5952
2025-12-17 17:10:24,162 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-12-17 17:10:24,163 INFO [run_NEMO] publishing log messages to tcp://*:5556
2025-12-17 17:10:24,165 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2025-12-17 17:10:24,937 WARNING [run_NEMO] There are uncommitted changes in /nemoShare/MEOPAR/nowcast-sys/SS-run-sets
2025-12-17 17:10:25,258 INFO [run_NEMO] forecast: launching /nemoShare/MEOPAR/nowcast-sys/runs/17dec25forecast_2025-12-17T171024.230099-0800/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2025-12-17 17:10:25,275 INFO [run_NEMO] forecast NEMO run for 2025-12-17 on arbutus.cloud-nowcast started
2025-12-17 17:10:25,278 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2025-12-17 17:10:25,433 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'forecast'], host='arbutus.cloud-nowcast')
2025-12-17 17:10:27,443 INFO [watch_NEMO] running in process 6226
2025-12-17 17:10:27,444 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-12-17 17:10:27,445 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2025-12-17 17:10:27,446 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2025-12-17 17:10:27,515 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2025-12-17 17:11:51,251 INFO [download_results] nowcast 2025-12-17 results files from arbutus.cloud-nowcast downloaded
2025-12-17 17:11:51,255 INFO [manager] checklist updated with [results files] items from download_results worker
2025-12-17 17:11:51,406 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast', 'research', '--run-date', '2025-12-17'], host='localhost')
2025-12-17 17:11:51,412 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast', 'comparison', '--run-date', '2025-12-16'], host='localhost')
2025-12-17 17:11:51,417 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['nowcast', '--run-date', '2025-12-17'], host='localhost')
2025-12-17 17:11:53,418 INFO [make_CHS_currents_file] running in process 1479930
2025-12-17 17:11:53,419 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-17 17:11:53,419 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5562
2025-12-17 17:11:53,420 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555
2025-12-17 17:11:54,673 INFO [make_plots] running in process 1479929
2025-12-17 17:11:54,675 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-17 17:11:54,676 INFO [make_plots] publishing log messages to tcp://*:5563
2025-12-17 17:11:54,677 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2025-12-17 17:11:54,703 INFO [make_plots] running in process 1479927
2025-12-17 17:11:54,726 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-17 17:11:54,735 INFO [make_plots] publishing log messages to tcp://*:5566
2025-12-17 17:11:54,737 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2025-12-17 17:11:55,105 INFO [make_plots] preparing render list for 2025-12-17 NEMO nowcast-blue research figures
2025-12-17 17:11:55,128 INFO [make_plots] preparing render list for 2025-12-16 NEMO nowcast-blue comparison figures
2025-12-17 17:11:55,178 INFO [make_plots] added SH_wind figure to 2025-12-16 NEMO nowcast-blue comparison render list
2025-12-17 17:11:55,179 INFO [make_plots] added Compare_VENUS_East figure to 2025-12-16 NEMO nowcast-blue comparison render list
2025-12-17 17:11:55,180 INFO [make_plots] added Compare_VENUS_Central figure to 2025-12-16 NEMO nowcast-blue comparison render list
2025-12-17 17:11:55,181 INFO [make_plots] starting to render nemo nowcast comparison 16dec25 figures
2025-12-17 17:11:56,732 INFO [make_plots] added salinity_thalweg_and_surface figures to 2025-12-17 NEMO nowcast-blue research render list
2025-12-17 17:11:57,881 INFO [make_plots] added temperature_thalweg_and_surface figures to 2025-12-17 NEMO nowcast-blue research render list
2025-12-17 17:11:57,882 INFO [make_plots] added Currents_sections_and_surface figure to 2025-12-17 NEMO nowcast-blue research render list
2025-12-17 17:11:57,883 INFO [make_plots] added Currents_at_VENUS_Central figure to 2025-12-17 NEMO nowcast-blue research render list
2025-12-17 17:11:57,884 INFO [make_plots] added Currents_at_VENUS_East figure to 2025-12-17 NEMO nowcast-blue research render list
2025-12-17 17:11:57,884 INFO [make_plots] starting to render nemo nowcast research 17dec25 figures
2025-12-17 17:12:28,178 INFO [make_CHS_currents_file] Made CHS currents file for 2025-12-17 for nowcast
2025-12-17 17:12:28,181 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker
2025-12-17 17:12:58,150 INFO [make_plots] finished rendering nemo nowcast research 17dec25 figures
2025-12-17 17:12:58,152 INFO [make_plots] nemo research plots for 2025-12-17 nowcast completed
2025-12-17 17:12:58,164 INFO [manager] checklist updated with [plots] items from make_plots worker
2025-12-17 17:13:17,742 INFO [make_plots] finished rendering nemo nowcast comparison 16dec25 figures
2025-12-17 17:13:17,744 INFO [make_plots] nemo comparison plots for 2025-12-16 nowcast completed
2025-12-17 17:13:17,748 INFO [manager] checklist updated with [plots] items from make_plots worker
2025-12-17 17:15:27,512 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 18911279 = 2025-12-18 05:18:40 UTC, 14.8% complete
2025-12-17 17:20:27,512 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 18911794 = 2025-12-18 11:02:00 UTC, 30.7% complete
2025-12-17 17:25:27,511 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 18912289 = 2025-12-18 16:32:00 UTC, 45.9% complete
2025-12-17 17:30:27,511 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 18912807 = 2025-12-18 22:17:20 UTC, 61.9% complete
2025-12-17 17:35:27,511 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 18913312 = 2025-12-19 03:54:00 UTC, 77.5% complete
2025-12-17 17:40:27,512 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 18913793 = 2025-12-19 09:14:40 UTC, 92.4% complete
2025-12-17 17:45:27,520 INFO [watch_NEMO] forecast NEMO run on arbutus.cloud-nowcast completed
2025-12-17 17:45:27,524 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2025-12-17 17:45:27,852 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_wind_file', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2025-12-17'], host='arbutus.cloud-nowcast')
2025-12-17 17:45:27,862 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_current_file', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2025-12-17'], host='arbutus.cloud-nowcast')
2025-12-17 17:45:27,866 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'turbidity', '--run-date', '2025-12-17'], host='localhost')
2025-12-17 17:45:27,873 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'turbidity', '--run-date', '2025-12-17'], host='localhost')
2025-12-17 17:45:27,878 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.nibi', 'turbidity', '--run-date', '2025-12-17'], host='localhost')
2025-12-17 17:45:27,882 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'turbidity', '--run-date', '2025-12-17'], host='localhost')
2025-12-17 17:45:27,887 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2025-12-17'], host='localhost')
2025-12-17 17:45:29,088 INFO [download_results] running in process 1480457
2025-12-17 17:45:29,089 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-17 17:45:29,090 INFO [download_results] publishing log messages to tcp://*:5562
2025-12-17 17:45:29,091 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2025-12-17 17:45:29,092 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/forecast/17dec25 to /results/SalishSea/forecast.202111
2025-12-17 17:45:29,161 INFO [upload_forcing] running in process 1480451
2025-12-17 17:45:29,163 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-17 17:45:29,164 INFO [upload_forcing] publishing log messages to tcp://*:5563
2025-12-17 17:45:29,165 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-12-17 17:45:29,174 INFO [upload_forcing] running in process 1480455
2025-12-17 17:45:29,175 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-17 17:45:29,176 INFO [upload_forcing] publishing log messages to tcp://*:5566
2025-12-17 17:45:29,177 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-12-17 17:45:29,188 INFO [upload_forcing] running in process 1480453
2025-12-17 17:45:29,189 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-17 17:45:29,190 INFO [upload_forcing] publishing log messages to tcp://*:5567
2025-12-17 17:45:29,191 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-12-17 17:45:29,194 INFO [upload_forcing] running in process 1480454
2025-12-17 17:45:29,195 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-17 17:45:29,196 INFO [upload_forcing] publishing log messages to tcp://*:5568
2025-12-17 17:45:29,197 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-12-17 17:45:29,821 INFO [upload_forcing] turbidity 2025-12-17 forcing files upload to optimum-hindcast completed
2025-12-17 17:45:29,825 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-12-17 17:45:30,217 INFO [upload_forcing] turbidity 2025-12-17 forcing files upload to arbutus.cloud-nowcast completed
2025-12-17 17:45:30,222 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-12-17 17:45:30,398 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2025-12-17'], host='localhost')
2025-12-17 17:45:30,740 INFO [make_ww3_current_file] running in process 6749
2025-12-17 17:45:30,742 INFO [make_ww3_current_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-12-17 17:45:30,743 INFO [make_ww3_current_file] publishing log messages to tcp://*:5571
2025-12-17 17:45:30,804 INFO [make_ww3_current_file] connected to skookum.eos.ubc.ca port 5555
2025-12-17 17:45:30,805 INFO [make_ww3_current_file] Creating wwatch3 currents forcing file for 2025-12-17 forecast run
2025-12-17 17:45:31,876 INFO [make_forcing_links] running in process 1480487
2025-12-17 17:45:31,877 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-17 17:45:31,879 INFO [make_forcing_links] publishing log messages to tcp://*:5563
2025-12-17 17:45:31,880 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2025-12-17 17:45:33,229 INFO [make_forcing_links] nowcast-green 2025-12-17 forcing file links on arbutus.cloud-nowcast created
2025-12-17 17:45:33,233 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2025-12-17 17:45:33,392 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2025-12-17'], host='arbutus.cloud-nowcast')
2025-12-17 17:45:33,505 INFO [upload_forcing] turbidity 2025-12-17 forcing files upload to robot.nibi completed
2025-12-17 17:45:33,511 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-12-17 17:45:35,263 INFO [upload_forcing] turbidity 2025-12-17 forcing files upload to orcinus-nowcast-agrif completed
2025-12-17 17:45:35,268 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-12-17 17:45:35,424 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['orcinus-nowcast-agrif', 'nowcast-agrif', '--run-date', '2025-12-17'], host='localhost')
2025-12-17 17:45:36,769 INFO [run_NEMO] running in process 7366
2025-12-17 17:45:36,770 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-12-17 17:45:36,771 INFO [run_NEMO] publishing log messages to tcp://*:5556
2025-12-17 17:45:36,773 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2025-12-17 17:45:36,878 INFO [make_forcing_links] running in process 1480497
2025-12-17 17:45:36,879 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-17 17:45:36,880 INFO [make_forcing_links] publishing log messages to tcp://*:5563
2025-12-17 17:45:36,882 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2025-12-17 17:45:39,076 INFO [make_forcing_links] nowcast-agrif 2025-12-17 forcing file links on orcinus-nowcast-agrif created
2025-12-17 17:45:39,080 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2025-12-17 17:45:39,125 WARNING [run_NEMO] There are uncommitted changes in /nemoShare/MEOPAR/nowcast-sys/SS-run-sets
2025-12-17 17:45:39,261 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_agrif', args=['orcinus-nowcast-agrif', 'nowcast-agrif', '--run-date', '2025-12-17'], host='localhost')
2025-12-17 17:45:40,010 INFO [run_NEMO] nowcast-green: launching /nemoShare/MEOPAR/nowcast-sys/runs/17dec25nowcast-green_2025-12-17T174536.840474-0800/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2025-12-17 17:45:40,076 INFO [run_NEMO] nowcast-green NEMO run for 2025-12-17 on arbutus.cloud-nowcast started
2025-12-17 17:45:40,080 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2025-12-17 17:45:40,242 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'nowcast-green'], host='arbutus.cloud-nowcast')
2025-12-17 17:45:40,570 INFO [run_NEMO_agrif] running in process 1480507
2025-12-17 17:45:40,571 INFO [run_NEMO_agrif] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-17 17:45:40,572 INFO [run_NEMO_agrif] publishing log messages to tcp://*:5563
2025-12-17 17:45:40,573 INFO [run_NEMO_agrif] connected to skookum.eos.ubc.ca port 5555
2025-12-17 17:45:42,524 INFO [make_ww3_wind_file] wwatch3 wind forcing file created on arbutus.cloud-nowcast for 2025-12-17 forecast run
2025-12-17 17:45:42,528 INFO [manager] checklist updated with [WW3 wind forcing] items from make_ww3_wind_file worker
2025-12-17 17:45:42,629 INFO [watch_NEMO] running in process 7668
2025-12-17 17:45:42,631 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-12-17 17:45:42,632 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2025-12-17 17:45:42,634 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2025-12-17 17:45:42,748 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2025-12-17 17:45:46,182 CRITICAL [run_NEMO_agrif] unhandled exception:
Traceback (most recent call last):
  File "/SalishSeaCast/NEMO_Nowcast/nemo_nowcast/worker.py", line 387, in _do_work
    checklist = self.worker_func(
        self._parsed_args, self.config, self.tell_manager
    )
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/run_NEMO_agrif.py", line 109, in run_NEMO_agrif
    prev_run_namelists_info = _get_prev_run_namelists_info(
        sftp_client, host_name, run_date.shift(days=-1), config
    )
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/run_NEMO_agrif.py", line 155, in _get_prev_run_namelists_info
    sftp_client.get(os.fspath(prev_namelist_cfg), namelist_cfg.name)
    ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/paramiko/sftp_client.py", line 840, in get
    size = self.getfo(
        remotepath,
    ...<3 lines>...
        max_concurrent_prefetch_requests,
    )
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/paramiko/sftp_client.py", line 795, in getfo
    file_size = self.stat(remotepath).st_size
                ~~~~~~~~~^^^^^^^^^^^^
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/paramiko/sftp_client.py", line 493, in stat
    t, msg = self._request(CMD_STAT, path)
             ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/paramiko/sftp_client.py", line 857, in _request
    return self._read_response(num)
           ~~~~~~~~~~~~~~~~~~~^^^^^
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/paramiko/sftp_client.py", line 909, in _read_response
    self._convert_status(msg)
    ~~~~~~~~~~~~~~~~~~~~^^^^^
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/paramiko/sftp_client.py", line 938, in _convert_status
    raise IOError(errno.ENOENT, text)
FileNotFoundError: [Errno 2] No such file
2025-12-17 17:47:18,487 INFO [download_results] forecast 2025-12-17 results files from arbutus.cloud-nowcast downloaded
2025-12-17 17:47:18,492 INFO [manager] checklist updated with [results files] items from download_results worker
2025-12-17 17:47:18,650 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['forecast', '--run-date', '2025-12-17'], host='localhost')
2025-12-17 17:47:20,929 INFO [make_CHS_currents_file] running in process 1480528
2025-12-17 17:47:20,930 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-17 17:47:20,931 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5562
2025-12-17 17:47:20,933 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555
2025-12-17 17:47:57,912 INFO [make_ww3_current_file] wwatch3 currents forcing file created on arbutus.cloud-nowcast for 2025-12-17 forecast run
2025-12-17 17:47:57,916 INFO [manager] checklist updated with [WW3 currents forcing] items from make_ww3_current_file worker
2025-12-17 17:47:58,076 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2025-12-17'], host='arbutus.cloud-nowcast')
2025-12-17 17:48:05,169 INFO [run_ww3] Created run directory /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/nowcast_2025-12-17T174805.089472-0800
2025-12-17 17:48:05,171 INFO [run_ww3] nowcast: launching /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/nowcast_2025-12-17T174805.089472-0800/SoGWW3.sh on arbutus.cloud-nowcast
2025-12-17 17:48:05,187 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker
2025-12-17 17:48:05,241 INFO [run_ww3] nowcast WaveWatch3 run for 2025-12-17 on arbutus.cloud-nowcast started
2025-12-17 17:48:05,347 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'nowcast'], host='arbutus.cloud-nowcast')
2025-12-17 17:48:05,807 INFO [make_CHS_currents_file] Made CHS currents file for 2025-12-17 for forecast
2025-12-17 17:48:05,810 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker
2025-12-17 17:48:05,941 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['nemo', 'forecast', '--run-date', '2025-12-17'], host='localhost')
2025-12-17 17:48:07,065 INFO [update_forecast_datasets] running in process 1480567
2025-12-17 17:48:07,066 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-17 17:48:07,068 INFO [update_forecast_datasets] publishing log messages to tcp://*:5562
2025-12-17 17:48:07,069 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2025-12-17 17:48:07,070 INFO [update_forecast_datasets] updating nemo forecast directory for forecast/17dec25 run
2025-12-17 17:48:07,232 INFO [watch_ww3] running in process 7866
2025-12-17 17:48:07,233 INFO [watch_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-12-17 17:48:07,235 INFO [watch_ww3] publishing log messages to tcp://*:5573
2025-12-17 17:48:07,236 INFO [watch_ww3] connected to skookum.eos.ubc.ca port 5555
2025-12-17 17:48:07,260 INFO [update_forecast_datasets] updated nemo forecast directory for forecast/17dec25 run: /results/SalishSea/rolling-forecasts/nemo
2025-12-17 17:48:07,261 INFO [update_forecast_datasets] nemo 2025-12-17 forecast rolling forecast datasets updated
2025-12-17 17:48:07,264 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2025-12-17 17:48:07,429 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nemo-forecast'], host='localhost')
2025-12-17 17:48:07,434 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'forecast', 'publish', '--run-date', '2025-12-17'], host='localhost')
2025-12-17 17:48:07,440 INFO [manager] launching NextWorker(module='nowcast.workers.make_surface_current_tiles', args=['forecast', '--run-date', '2025-12-17'], host='localhost')
2025-12-17 17:48:07,548 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2025-12-17 17:48:08,504 INFO [ping_erddap] running in process 1480574
2025-12-17 17:48:08,505 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-17 17:48:08,507 INFO [ping_erddap] publishing log messages to tcp://*:5562
2025-12-17 17:48:08,508 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2025-12-17 17:48:08,519 INFO [ping_erddap] nemo-forecast ERDDAP dataset flag file(s) created
2025-12-17 17:48:08,521 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2025-12-17 17:48:09,719 INFO [make_surface_current_tiles] running in process 1480577
2025-12-17 17:48:09,721 INFO [make_surface_current_tiles] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-17 17:48:09,722 INFO [make_surface_current_tiles] publishing log messages to tcp://*:5562
2025-12-17 17:48:09,723 INFO [make_surface_current_tiles] connected to skookum.eos.ubc.ca port 5555
2025-12-17 17:48:09,724 INFO [make_surface_current_tiles] starting rendering of tiles for 2025-12-17 forecast in 6 processes into /results/nowcast-sys/figures/surface_currents/forecast/17dec25
2025-12-17 17:48:10,488 INFO [make_plots] running in process 1480576
2025-12-17 17:48:10,494 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-17 17:48:10,496 INFO [make_plots] publishing log messages to tcp://*:5563
2025-12-17 17:48:10,499 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2025-12-17 17:48:11,004 INFO [make_plots] preparing render list for 2025-12-17 NEMO forecast publish figures
2025-12-17 17:48:11,422 INFO [make_plots] added Website_thumbnail figure to 2025-12-17 NEMO forecast publish render list
2025-12-17 17:48:11,426 INFO [make_plots] added Threshold_website figure to 2025-12-17 NEMO forecast publish render list
2025-12-17 17:48:11,430 INFO [make_plots] added PA_tidal_predictions figure to 2025-12-17 NEMO forecast publish render list
2025-12-17 17:48:11,430 INFO [make_plots] added Boundary Bay figure to 2025-12-17 NEMO forecast publish render list
2025-12-17 17:48:11,431 INFO [make_plots] added Campbell River figure to 2025-12-17 NEMO forecast publish render list
2025-12-17 17:48:11,432 INFO [make_plots] added Cherry Point figure to 2025-12-17 NEMO forecast publish render list
2025-12-17 17:48:11,433 INFO [make_plots] added Friday Harbor figure to 2025-12-17 NEMO forecast publish render list
2025-12-17 17:48:11,434 INFO [make_plots] added Halfmoon Bay figure to 2025-12-17 NEMO forecast publish render list
2025-12-17 17:48:11,435 INFO [make_plots] added Nanaimo figure to 2025-12-17 NEMO forecast publish render list
2025-12-17 17:48:11,437 INFO [make_plots] added Neah Bay figure to 2025-12-17 NEMO forecast publish render list
2025-12-17 17:48:11,438 INFO [make_plots] added New Westminster figure to 2025-12-17 NEMO forecast publish render list
2025-12-17 17:48:11,439 INFO [make_plots] added Patricia Bay figure to 2025-12-17 NEMO forecast publish render list
2025-12-17 17:48:11,442 INFO [make_plots] added Point Atkinson figure to 2025-12-17 NEMO forecast publish render list
2025-12-17 17:48:11,442 INFO [make_plots] added Port Renfrew figure to 2025-12-17 NEMO forecast publish render list
2025-12-17 17:48:11,443 INFO [make_plots] added Sand Heads figure to 2025-12-17 NEMO forecast publish render list
2025-12-17 17:48:11,445 INFO [make_plots] added Sandy Cove figure to 2025-12-17 NEMO forecast publish render list
2025-12-17 17:48:11,446 INFO [make_plots] added Squamish figure to 2025-12-17 NEMO forecast publish render list
2025-12-17 17:48:11,447 INFO [make_plots] added Victoria figure to 2025-12-17 NEMO forecast publish render list
2025-12-17 17:48:11,448 INFO [make_plots] added Woodwards Landing figure to 2025-12-17 NEMO forecast publish render list
2025-12-17 17:48:11,458 INFO [make_plots] starting to render nemo forecast publish 17dec25 figures
2025-12-17 17:50:42,754 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18908796 = 2025-12-17 01:43:20 UTC, 7.2% complete
2025-12-17 17:53:07,552 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 84, 19.4% complete
2025-12-17 17:55:28,449 WARNING [make_plots] invalid station number for Boundary Bay station: None
2025-12-17 17:55:28,450 WARNING [make_plots] can't resolve a valid CHS station code for Boundary Bay
2025-12-17 17:55:42,760 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18908973 = 2025-12-17 03:41:20 UTC, 15.4% complete
2025-12-17 17:55:53,466 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb996/data for station 08074 Campbell River from 2025-12-12 00:00:00Z to 2025-12-19 11:55:00Z
2025-12-17 17:57:05,238 WARNING [make_plots] invalid station number for Halfmoon Bay station: None
2025-12-17 17:57:05,242 WARNING [make_plots] can't resolve a valid CHS station code for Halfmoon Bay
2025-12-17 17:57:28,201 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb96d/data for station 07917 Nanaimo from 2025-12-12 00:00:00Z to 2025-12-19 11:55:00Z
2025-12-17 17:58:07,552 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 171, 39.6% complete
2025-12-17 17:58:15,994 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e13d0f4a073c4bbf93/data for station 07654 New Westminster from 2025-12-12 00:00:00Z to 2025-12-19 11:55:00Z
2025-12-17 17:58:28,446 INFO [make_surface_current_tiles] starting PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast/17dec25
2025-12-17 17:58:35,189 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd2d/data for station 07277 Patricia Bay from 2025-12-12 00:00:00Z to 2025-12-19 11:55:00Z
2025-12-17 17:58:54,419 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94c/data for station 07795 Point Atkinson from 2025-12-12 00:00:00Z to 2025-12-19 11:55:00Z
2025-12-17 17:59:13,200 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e23d0f4a073c4bc060/data for station 08525 Port Renfrew from 2025-12-12 00:00:00Z to 2025-12-19 11:55:00Z
2025-12-17 17:59:31,389 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb93c/data for station 07594 Sand Heads from 2025-12-12 00:00:00Z to 2025-12-19 11:55:00Z
2025-12-17 17:59:49,929 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc43b/data for station 07786 Sandy Cove from 2025-12-12 00:00:00Z to 2025-12-19 11:55:00Z
2025-12-17 18:00:07,879 WARNING [make_plots] invalid station number for Squamish station: None
2025-12-17 18:00:07,880 WARNING [make_plots] can't resolve a valid CHS station code for Squamish
2025-12-17 18:00:26,181 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd1e/data for station 07120 Victoria from 2025-12-12 00:00:00Z to 2025-12-19 11:55:00Z
2025-12-17 18:00:42,756 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18909151 = 2025-12-17 05:40:00 UTC, 23.6% complete
2025-12-17 18:00:44,303 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc42d/data for station 07610 Woodwards Landing from 2025-12-12 00:00:00Z to 2025-12-19 11:55:00Z
2025-12-17 18:01:02,018 INFO [make_plots] finished rendering nemo forecast publish 17dec25 figures
2025-12-17 18:01:02,021 INFO [make_plots] nemo publish plots for 2025-12-17 forecast completed
2025-12-17 18:01:02,030 INFO [manager] checklist updated with [plots] items from make_plots worker
2025-12-17 18:01:02,194 INFO [manager] launching NextWorker(module='nowcast.workers.make_feeds', args=['forecast', '--run-date', '2025-12-17'], host='localhost')
2025-12-17 18:01:05,338 INFO [make_feeds] running in process 1485210
2025-12-17 18:01:05,339 INFO [make_feeds] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-17 18:01:05,340 INFO [make_feeds] publishing log messages to tcp://*:5563
2025-12-17 18:01:05,341 INFO [make_feeds] connected to skookum.eos.ubc.ca port 5555
2025-12-17 18:01:19,533 INFO [make_feeds] ATOM feeds for 2025-12-17 forecast run completed
2025-12-17 18:01:19,536 INFO [manager] checklist updated with [storm surge alert feeds] items from make_feeds worker
2025-12-17 18:03:07,554 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 255, 59.0% complete
2025-12-17 18:03:10,804 INFO [make_surface_current_tiles] finished PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast/17dec25
2025-12-17 18:03:10,829 INFO [make_surface_current_tiles] finished rendering of tiles for 2025-12-17 forecast into /results/nowcast-sys/figures/surface_currents/forecast/17dec25
2025-12-17 18:03:10,830 INFO [make_surface_current_tiles] surface current tile figures for 2025-12-17 forecast completed
2025-12-17 18:03:11,143 INFO [manager] checklist updated with [surface current tiles] items from make_surface_current_tiles worker
2025-12-17 18:05:42,758 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18909328 = 2025-12-17 07:38:00 UTC, 31.8% complete
2025-12-17 18:08:07,557 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 354, 81.9% complete
2025-12-17 18:10:42,759 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18909506 = 2025-12-17 09:36:40 UTC, 40.1% complete
2025-12-17 18:13:07,563 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2025-12-17 18:15:42,760 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18909684 = 2025-12-17 11:35:20 UTC, 48.3% complete
2025-12-17 18:18:07,556 INFO [watch_ww3] nowcast WWATCH3 run on arbutus.cloud-nowcast completed
2025-12-17 18:18:07,646 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2025-12-17 18:18:08,127 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2025-12-17'], host='localhost')
2025-12-17 18:18:08,134 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2025-12-17'], host='arbutus.cloud-nowcast')
2025-12-17 18:18:09,415 INFO [download_wwatch3_results] running in process 1485467
2025-12-17 18:18:09,416 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-17 18:18:09,417 INFO [download_wwatch3_results] publishing log messages to tcp://*:5562
2025-12-17 18:18:09,418 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2025-12-17 18:18:10,708 INFO [run_ww3] running in process 8229
2025-12-17 18:18:10,709 INFO [run_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-12-17 18:18:10,711 INFO [run_ww3] publishing log messages to tcp://*:5572
2025-12-17 18:18:10,712 INFO [run_ww3] connected to skookum.eos.ubc.ca port 5555
2025-12-17 18:18:10,716 INFO [run_ww3] Created run directory /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast_2025-12-17T181810.706697-0800
2025-12-17 18:18:10,718 INFO [run_ww3] forecast: launching /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast_2025-12-17T181810.706697-0800/SoGWW3.sh on arbutus.cloud-nowcast
2025-12-17 18:18:10,727 INFO [run_ww3] forecast WaveWatch3 run for 2025-12-17 on arbutus.cloud-nowcast started
2025-12-17 18:18:10,730 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker
2025-12-17 18:18:11,058 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'forecast'], host='arbutus.cloud-nowcast')
2025-12-17 18:18:13,075 INFO [watch_ww3] running in process 8377
2025-12-17 18:18:13,076 INFO [watch_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-12-17 18:18:13,078 INFO [watch_ww3] publishing log messages to tcp://*:5573
2025-12-17 18:18:13,079 INFO [watch_ww3] connected to skookum.eos.ubc.ca port 5555
2025-12-17 18:18:13,138 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2025-12-17 18:18:41,683 INFO [download_wwatch3_results] nowcast 2025-12-17 results files from arbutus.cloud-nowcast downloaded
2025-12-17 18:18:41,687 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2025-12-17 18:20:42,761 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18909860 = 2025-12-17 13:32:40 UTC, 56.5% complete
2025-12-17 18:23:13,137 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 81, 12.5% complete
2025-12-17 18:25:42,762 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18910037 = 2025-12-17 15:30:40 UTC, 64.7% complete
2025-12-17 18:28:13,138 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 171, 26.4% complete
2025-12-17 18:30:42,764 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18910214 = 2025-12-17 17:28:40 UTC, 72.9% complete
2025-12-17 18:33:13,139 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 261, 40.3% complete
2025-12-17 18:35:42,765 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18910391 = 2025-12-17 19:26:40 UTC, 81.1% complete
2025-12-17 18:38:13,140 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 345, 53.2% complete
2025-12-17 18:40:42,766 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18910568 = 2025-12-17 21:24:40 UTC, 89.3% complete
2025-12-17 18:43:13,141 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 432, 66.7% complete
2025-12-17 18:45:42,768 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18910745 = 2025-12-17 23:22:40 UTC, 97.5% complete
2025-12-17 18:48:13,143 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 516, 79.6% complete
2025-12-17 18:50:42,770 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18910800 = 2025-12-17 23:59:20 UTC, 100.0% complete
2025-12-17 18:53:13,144 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 612, 94.4% complete
2025-12-17 18:55:42,814 INFO [watch_NEMO] nowcast-green NEMO run on arbutus.cloud-nowcast completed
2025-12-17 18:55:42,817 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2025-12-17 18:55:43,290 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2025-12-17'], host='localhost')
2025-12-17 18:55:44,566 INFO [download_results] running in process 1485857
2025-12-17 18:55:44,568 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-17 18:55:44,569 INFO [download_results] publishing log messages to tcp://*:5562
2025-12-17 18:55:44,570 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2025-12-17 18:55:44,571 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/nowcast-green/17dec25 to /results2/SalishSea/nowcast-green.202111
2025-12-17 18:58:13,146 INFO [watch_ww3] forecast WWATCH3 run on arbutus.cloud-nowcast completed
2025-12-17 18:58:13,149 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2025-12-17 18:58:13,656 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2025-12-17'], host='localhost')
2025-12-17 18:58:15,247 INFO [download_wwatch3_results] running in process 1485888
2025-12-17 18:58:15,249 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-17 18:58:15,250 INFO [download_wwatch3_results] publishing log messages to tcp://*:5563
2025-12-17 18:58:15,252 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2025-12-17 18:58:55,170 INFO [download_wwatch3_results] forecast 2025-12-17 results files from arbutus.cloud-nowcast downloaded
2025-12-17 18:58:55,174 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2025-12-17 18:58:55,500 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['wwatch3', 'forecast', '--run-date', '2025-12-17'], host='localhost')
2025-12-17 18:58:56,718 INFO [update_forecast_datasets] running in process 1485900
2025-12-17 18:58:56,719 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-17 18:58:56,720 INFO [update_forecast_datasets] publishing log messages to tcp://*:5563
2025-12-17 18:58:56,721 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2025-12-17 18:58:56,723 INFO [update_forecast_datasets] updating wwatch3 most_recent_forecast directory from forecast/17dec25 run
2025-12-17 18:58:56,727 INFO [update_forecast_datasets] updated wwatch3 most_recent_forecast directory from forecast/17dec25 run
2025-12-17 18:58:56,728 INFO [update_forecast_datasets] updating wwatch3 forecast directory for forecast/17dec25 run
2025-12-17 18:58:56,745 INFO [update_forecast_datasets] updated wwatch3 forecast directory for forecast/17dec25 run: /results/SalishSea/rolling-forecasts/wwatch3
2025-12-17 18:58:56,746 INFO [update_forecast_datasets] wwatch3 2025-12-17 forecast rolling forecast datasets updated
2025-12-17 18:58:56,749 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2025-12-17 18:58:57,067 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['wwatch3-forecast'], host='localhost')
2025-12-17 18:58:58,263 INFO [ping_erddap] running in process 1485905
2025-12-17 18:58:58,264 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-17 18:58:58,265 INFO [ping_erddap] publishing log messages to tcp://*:5563
2025-12-17 18:58:58,267 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2025-12-17 18:58:58,269 INFO [ping_erddap] wwatch3-forecast ERDDAP dataset flag file(s) created
2025-12-17 18:58:58,271 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2025-12-17 18:58:58,580 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['wwatch3', 'forecast', 'publish', '--run-date', '2025-12-17'], host='localhost')
2025-12-17 18:59:05,590 INFO [make_plots] running in process 1485910
2025-12-17 18:59:05,591 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-17 18:59:05,593 INFO [make_plots] publishing log messages to tcp://*:5563
2025-12-17 18:59:05,594 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2025-12-17 18:59:05,595 INFO [make_plots] preparing render list for 2025-12-17 WaveWatch3 forecast publish figures
2025-12-17 18:59:05,596 INFO [make_plots] added Halibut Bank figure to 2025-12-17 WaveWatch3 forecast publish render list
2025-12-17 18:59:05,597 INFO [make_plots] added Sentry Shoal figure to 2025-12-17 WaveWatch3 forecast publish render list
2025-12-17 18:59:05,598 INFO [make_plots] starting to render wwatch3 forecast publish 17dec25 figures
2025-12-17 18:59:06,178 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46146 Halibut Bank
2025-12-17 18:59:11,768 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46131 Sentry Shoal
2025-12-17 18:59:18,282 INFO [make_plots] finished rendering wwatch3 forecast publish 17dec25 figures
2025-12-17 18:59:18,283 INFO [make_plots] wwatch3 publish plots for 2025-12-17 forecast completed
2025-12-17 18:59:18,287 INFO [manager] checklist updated with [plots] items from make_plots worker
2025-12-17 18:59:44,033 INFO [download_results] nowcast-green 2025-12-17 results files from arbutus.cloud-nowcast downloaded
2025-12-17 18:59:44,039 INFO [manager] checklist updated with [results files] items from download_results worker
2025-12-17 18:59:44,358 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast-green', 'research', '--run-date', '2025-12-17'], host='localhost')
2025-12-17 18:59:44,364 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nowcast-green'], host='localhost')
2025-12-17 18:59:44,370 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'physics', '--run-date', '2025-12-17'], host='localhost')
2025-12-17 18:59:44,376 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'chemistry', '--run-date', '2025-12-17'], host='localhost')
2025-12-17 18:59:44,381 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'biology', '--run-date', '2025-12-17'], host='localhost')
2025-12-17 18:59:45,841 INFO [ping_erddap] running in process 1485956
2025-12-17 18:59:45,842 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-17 18:59:45,843 INFO [ping_erddap] publishing log messages to tcp://*:5562
2025-12-17 18:59:45,844 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2025-12-17 18:59:45,848 INFO [ping_erddap] nowcast-green ERDDAP dataset flag file(s) created
2025-12-17 18:59:45,850 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2025-12-17 18:59:46,671 INFO [make_averaged_dataset] running in process 1485959
2025-12-17 18:59:46,672 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-17 18:59:46,673 INFO [make_averaged_dataset] publishing log messages to tcp://*:5562
2025-12-17 18:59:46,674 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555
2025-12-17 18:59:46,675 INFO [make_averaged_dataset] creating day-averaged dataset for 17-Dec-2025 biology
2025-12-17 18:59:46,679 INFO [make_averaged_dataset] running in process 1485958
2025-12-17 18:59:46,680 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-17 18:59:46,681 INFO [make_averaged_dataset] publishing log messages to tcp://*:5563
2025-12-17 18:59:46,681 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555
2025-12-17 18:59:46,682 INFO [make_averaged_dataset] creating day-averaged dataset for 17-Dec-2025 chemistry
2025-12-17 18:59:46,747 INFO [make_averaged_dataset] running in process 1485957
2025-12-17 18:59:46,748 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-17 18:59:46,749 INFO [make_averaged_dataset] publishing log messages to tcp://*:5566
2025-12-17 18:59:46,750 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555
2025-12-17 18:59:46,751 INFO [make_averaged_dataset] creating day-averaged dataset for 17-Dec-2025 physics
2025-12-17 18:59:47,386 INFO [make_plots] running in process 1485954
2025-12-17 18:59:47,387 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-17 18:59:47,388 INFO [make_plots] publishing log messages to tcp://*:5567
2025-12-17 18:59:47,388 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2025-12-17 18:59:47,757 INFO [make_plots] preparing render list for 2025-12-17 NEMO nowcast-green research figures
2025-12-17 18:59:49,494 INFO [make_plots] added temperature_thalweg_and_surface figures to 2025-12-17 NEMO nowcast-green research render list
2025-12-17 18:59:50,227 INFO [make_plots] added nitrate_thalweg_and_surface figures to 2025-12-17 NEMO nowcast-green research render list
2025-12-17 18:59:50,928 INFO [make_plots] added ammonium_thalweg_and_surface figures to 2025-12-17 NEMO nowcast-green research render list
2025-12-17 18:59:51,626 INFO [make_plots] added silicon_thalweg_and_surface figures to 2025-12-17 NEMO nowcast-green research render list
2025-12-17 18:59:52,326 INFO [make_plots] added dissolved_organic_nitrogen_thalweg_and_surface figures to 2025-12-17 NEMO nowcast-green research render list
2025-12-17 18:59:53,021 INFO [make_plots] added particulate_organic_nitrogen_thalweg_and_surface figures to 2025-12-17 NEMO nowcast-green research render list
2025-12-17 18:59:53,711 INFO [make_plots] added biogenic_silicon_thalweg_and_surface figures to 2025-12-17 NEMO nowcast-green research render list
2025-12-17 18:59:54,547 INFO [make_plots] added diatoms_thalweg_and_surface figures to 2025-12-17 NEMO nowcast-green research render list
2025-12-17 18:59:55,396 INFO [make_plots] added flagellates_thalweg_and_surface figures to 2025-12-17 NEMO nowcast-green research render list
2025-12-17 18:59:56,175 INFO [make_plots] added microzooplankton_thalweg_and_surface figures to 2025-12-17 NEMO nowcast-green research render list
2025-12-17 18:59:56,975 INFO [make_plots] added mesozooplankton_thalweg_and_surface figures to 2025-12-17 NEMO nowcast-green research render list
2025-12-17 18:59:57,670 INFO [make_plots] added turbidity_thalweg_and_surface figures to 2025-12-17 NEMO nowcast-green research render list
2025-12-17 18:59:58,803 INFO [make_plots] added temperature_salinity_timeseries figure to 2025-12-17 NEMO nowcast-green research render list
2025-12-17 18:59:58,804 INFO [make_plots] added nitrate_diatoms_timeseries figure to 2025-12-17 NEMO nowcast-green research render list
2025-12-17 18:59:58,805 INFO [make_plots] added diatoms_flagellates_timeseries figure to 2025-12-17 NEMO nowcast-green research render list
2025-12-17 18:59:58,806 INFO [make_plots] added z1_z2_zooplankton_timeseries figure to 2025-12-17 NEMO nowcast-green research render list
2025-12-17 18:59:58,807 INFO [make_plots] starting to render nemo nowcast-green research 17dec25 figures
2025-12-17 19:00:33,264 INFO [make_averaged_dataset] day-averaged dataset for 17-Dec-2025 physics created
2025-12-17 19:00:33,267 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker
2025-12-17 19:00:44,861 INFO [make_averaged_dataset] day-averaged dataset for 17-Dec-2025 chemistry created
2025-12-17 19:00:44,864 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker
2025-12-17 19:00:57,234 INFO [make_averaged_dataset] day-averaged dataset for 17-Dec-2025 biology created
2025-12-17 19:00:57,237 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker
2025-12-17 19:52:37,218 INFO [make_plots] finished rendering nemo nowcast-green research 17dec25 figures
2025-12-17 19:52:37,230 INFO [make_plots] nemo research plots for 2025-12-17 nowcast-green completed
2025-12-17 19:52:37,289 INFO [manager] checklist updated with [plots] items from make_plots worker
2025-12-17 19:56:00,162 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/00/ to /results/forcing/atmospheric/continental2.5/GRIB/20251218/00/
2025-12-17 19:56:00,163 INFO [collect_weather] 2025-12-18 2.5km weather forecast 00 collection complete
2025-12-17 19:56:00,166 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2025-12-17 19:56:00,653 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['06', '2.5km'], host='localhost')
2025-12-17 19:56:00,660 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['06', '--fcst-date', '2025-12-18'], host='localhost')
2025-12-17 19:56:00,662 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20251218/00/
2025-12-17 19:56:00,663 INFO [crop_gribs] 2025-12-18 00 GRIBs cropping complete
2025-12-17 19:56:00,667 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2025-12-17 19:56:01,994 INFO [collect_weather] running in process 1486744
2025-12-17 19:56:01,995 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-17 19:56:01,996 INFO [collect_weather] publishing log messages to tcp://*:5562
2025-12-17 19:56:01,997 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2025-12-17 19:56:02,014 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/06/
2025-12-17 19:56:02,967 INFO [crop_gribs] running in process 1486746
2025-12-17 19:56:02,969 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-17 19:56:02,970 INFO [crop_gribs] publishing log messages to tcp://*:5563
2025-12-17 19:56:02,971 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2025-12-17 19:56:02,972 INFO [crop_gribs] cropping 2025-12-18 ECCC HRDPS 2.5km continental 06Z GRIB files to SalishSeaCast subdomain
2025-12-17 19:56:02,985 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20251218/06/
2025-12-18 01:52:01,690 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/06/ to /results/forcing/atmospheric/continental2.5/GRIB/20251218/06/
2025-12-18 01:52:01,691 INFO [collect_weather] 2025-12-18 2.5km weather forecast 06 collection complete
2025-12-18 01:52:01,695 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2025-12-18 01:52:02,243 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Capilano', '--data-date', '2025-12-17'], host='localhost')
2025-12-18 01:52:02,251 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'ChilliwackVedder', '--data-date', '2025-12-17'], host='localhost')
2025-12-18 01:52:02,255 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'ClowhomClowhomLake', '--data-date', '2025-12-17'], host='localhost')
2025-12-18 01:52:02,261 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Englishman', '--data-date', '2025-12-17'], host='localhost')
2025-12-18 01:52:02,270 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Fraser', '--data-date', '2025-12-17'], host='localhost')
2025-12-18 01:52:02,278 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'HomathkoMouth', '--data-date', '2025-12-17'], host='localhost')
2025-12-18 01:52:02,283 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'NicomeklLangley', '--data-date', '2025-12-17'], host='localhost')
2025-12-18 01:52:02,289 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'RobertsCreek', '--data-date', '2025-12-17'], host='localhost')
2025-12-18 01:52:02,294 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SalmonSayward', '--data-date', '2025-12-17'], host='localhost')
2025-12-18 01:52:02,301 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SanJuanPortRenfrew', '--data-date', '2025-12-17'], host='localhost')
2025-12-18 01:52:02,307 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SquamishBrackendale', '--data-date', '2025-12-17'], host='localhost')
2025-12-18 01:52:02,321 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaScotty', '--data-date', '2025-12-17'], host='localhost')
2025-12-18 01:52:02,344 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaBypass', '--data-date', '2025-12-17'], host='localhost')
2025-12-18 01:52:02,356 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaDiversion', '--data-date', '2025-12-17'], host='localhost')
2025-12-18 01:52:02,370 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ctd', args=['SCVIP'], host='localhost')
2025-12-18 01:52:02,386 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ctd', args=['SEVIP'], host='localhost')
2025-12-18 01:52:02,394 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ferry', args=['TWDP'], host='localhost')
2025-12-18 01:52:02,431 INFO [manager] launching NextWorker(module='nowcast.workers.get_vfpa_hadcp', args=['--data-date', '2025-12-17'], host='localhost')
2025-12-18 01:52:02,437 INFO [manager] launching NextWorker(module='nowcast.workers.collect_NeahBay_ssh', args=['00'], host='localhost')
2025-12-18 01:52:02,454 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['12', '2.5km'], host='localhost')
2025-12-18 01:52:02,470 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['12'], host='localhost')
2025-12-18 01:52:05,054 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20251218/06/
2025-12-18 01:52:05,057 INFO [crop_gribs] 2025-12-18 06 GRIBs cropping complete
2025-12-18 01:52:05,060 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2025-12-18 01:52:05,751 INFO [collect_weather] running in process 1491150
2025-12-18 01:52:05,754 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-18 01:52:05,755 INFO [collect_weather] publishing log messages to tcp://*:5562
2025-12-18 01:52:05,755 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2025-12-18 01:52:05,779 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/12/
2025-12-18 01:52:05,786 INFO [collect_NeahBay_ssh] running in process 1491149
2025-12-18 01:52:05,787 INFO [collect_NeahBay_ssh] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-18 01:52:05,789 INFO [collect_NeahBay_ssh] publishing log messages to tcp://*:5564
2025-12-18 01:52:05,790 INFO [collect_NeahBay_ssh] connected to skookum.eos.ubc.ca port 5555
2025-12-18 01:52:05,791 INFO [collect_NeahBay_ssh] collecting Neah Bay ssh 00Z obs/forecast for 2025-12-18
2025-12-18 01:52:05,938 INFO [manager] launching NextWorker(module='nowcast.workers.grib_to_netcdf', args=['forecast2'], host='localhost')
2025-12-18 01:52:06,677 INFO [collect_river_data] running in process 1491130
2025-12-18 01:52:06,678 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-18 01:52:06,679 INFO [collect_river_data] publishing log messages to tcp://*:5563
2025-12-18 01:52:06,680 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-12-18 01:52:06,682 INFO [collect_river_data] Collecting ECCC Capilano river data for 2025-12-17
2025-12-18 01:52:06,717 INFO [collect_river_data] Appended ECCC Capilano river average discharge for 2025-12-17 to: /opp/observations/rivers/Capilano/Caplilano_08GA010_day_avg_flow
2025-12-18 01:52:06,718 INFO [collect_river_data] ECCC Capilano river data collection for 2025-12-17 completed
2025-12-18 01:52:06,720 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-12-18 01:52:06,781 INFO [collect_river_data] running in process 1491138
2025-12-18 01:52:06,782 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-18 01:52:06,783 INFO [collect_river_data] publishing log messages to tcp://*:5565
2025-12-18 01:52:06,783 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-12-18 01:52:06,784 INFO [collect_river_data] Collecting ECCC RobertsCreek river data for 2025-12-17
2025-12-18 01:52:06,829 INFO [collect_river_data] running in process 1491132
2025-12-18 01:52:06,830 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-18 01:52:06,832 INFO [collect_river_data] publishing log messages to tcp://*:5566
2025-12-18 01:52:06,833 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-12-18 01:52:06,833 INFO [collect_river_data] Collecting ECCC ChilliwackVedder river data for 2025-12-17
2025-12-18 01:52:06,848 INFO [collect_river_data] Appended ECCC RobertsCreek river average discharge for 2025-12-17 to: /results/forcing/rivers/observations/RobertsCreek_flow
2025-12-18 01:52:06,849 INFO [collect_river_data] ECCC RobertsCreek river data collection for 2025-12-17 completed
2025-12-18 01:52:06,887 INFO [collect_river_data] Appended ECCC ChilliwackVedder river average discharge for 2025-12-17 to: /results/forcing/rivers/observations/Chilliwack_Vedder_flow
2025-12-18 01:52:06,888 INFO [collect_river_data] ECCC ChilliwackVedder river data collection for 2025-12-17 completed
2025-12-18 01:52:06,982 INFO [collect_river_data] running in process 1491139
2025-12-18 01:52:06,985 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-18 01:52:06,986 INFO [collect_river_data] publishing log messages to tcp://*:5567
2025-12-18 01:52:06,987 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-12-18 01:52:06,988 INFO [collect_river_data] Collecting ECCC SalmonSayward river data for 2025-12-17
2025-12-18 01:52:07,004 INFO [collect_river_data] running in process 1491141
2025-12-18 01:52:07,007 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-18 01:52:07,007 INFO [collect_river_data] publishing log messages to tcp://*:5568
2025-12-18 01:52:07,008 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-12-18 01:52:07,009 INFO [collect_river_data] Collecting ECCC SquamishBrackendale river data for 2025-12-17
2025-12-18 01:52:07,023 INFO [collect_river_data] Appended ECCC SalmonSayward river average discharge for 2025-12-17 to: /results/forcing/rivers/observations/Salmon_Sayward_flow
2025-12-18 01:52:07,024 INFO [collect_river_data] ECCC SalmonSayward river data collection for 2025-12-17 completed
2025-12-18 01:52:07,036 INFO [collect_river_data] Appended ECCC SquamishBrackendale river average discharge for 2025-12-17 to: /results/forcing/rivers/observations/Squamish_Brackendale_flow
2025-12-18 01:52:07,038 INFO [collect_river_data] ECCC SquamishBrackendale river data collection for 2025-12-17 completed
2025-12-18 01:52:07,040 INFO [collect_river_data] running in process 1491135
2025-12-18 01:52:07,040 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-18 01:52:07,041 INFO [collect_river_data] publishing log messages to tcp://*:5569
2025-12-18 01:52:07,042 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-12-18 01:52:07,043 INFO [collect_river_data] Collecting ECCC Fraser river data for 2025-12-17
2025-12-18 01:52:07,051 INFO [collect_river_data] running in process 1491144
2025-12-18 01:52:07,052 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-18 01:52:07,053 INFO [collect_river_data] publishing log messages to tcp://*:5570
2025-12-18 01:52:07,054 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-12-18 01:52:07,055 INFO [collect_river_data] Collecting ECCC TheodosiaDiversion river data for 2025-12-17
2025-12-18 01:52:07,057 INFO [collect_river_data] running in process 1491133
2025-12-18 01:52:07,058 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-18 01:52:07,059 INFO [collect_river_data] publishing log messages to tcp://*:5571
2025-12-18 01:52:07,060 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-12-18 01:52:07,062 INFO [collect_river_data] Collecting ECCC ClowhomClowhomLake river data for 2025-12-17
2025-12-18 01:52:07,064 INFO [collect_river_data] running in process 1491134
2025-12-18 01:52:07,065 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-18 01:52:07,066 INFO [collect_river_data] publishing log messages to tcp://*:5572
2025-12-18 01:52:07,067 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-12-18 01:52:07,067 INFO [collect_river_data] Collecting ECCC Englishman river data for 2025-12-17
2025-12-18 01:52:07,074 INFO [collect_river_data] Appended ECCC Fraser river average discharge for 2025-12-17 to: /data/dlatorne/SOG-projects/SOG-forcing/ECget/Fraser_flow
2025-12-18 01:52:07,075 INFO [collect_river_data] ECCC Fraser river data collection for 2025-12-17 completed
2025-12-18 01:52:07,076 INFO [collect_river_data] running in process 1491143
2025-12-18 01:52:07,077 INFO [collect_river_data] running in process 1491140
2025-12-18 01:52:07,078 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-18 01:52:07,079 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-18 01:52:07,080 INFO [collect_river_data] publishing log messages to tcp://*:5573
2025-12-18 01:52:07,081 INFO [collect_river_data] publishing log messages to tcp://*:5574
2025-12-18 01:52:07,082 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-12-18 01:52:07,083 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-12-18 01:52:07,084 INFO [collect_river_data] Collecting ECCC TheodosiaBypass river data for 2025-12-17
2025-12-18 01:52:07,084 INFO [collect_river_data] Collecting ECCC SanJuanPortRenfrew river data for 2025-12-17
2025-12-18 01:52:07,086 INFO [collect_river_data] running in process 1491142
2025-12-18 01:52:07,087 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-18 01:52:07,088 INFO [collect_river_data] Appended ECCC TheodosiaDiversion river average discharge for 2025-12-17 to: /results/forcing/rivers/observations/Theodosia_Diversion_flow
2025-12-18 01:52:07,090 INFO [collect_river_data] publishing log messages to tcp://*:5575
2025-12-18 01:52:07,090 INFO [collect_river_data] ECCC TheodosiaDiversion river data collection for 2025-12-17 completed
2025-12-18 01:52:07,091 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-12-18 01:52:07,092 INFO [collect_river_data] Collecting ECCC TheodosiaScotty river data for 2025-12-17
2025-12-18 01:52:07,095 INFO [collect_river_data] Appended ECCC Englishman river average discharge for 2025-12-17 to: /data/dlatorne/SOG-projects/SOG-forcing/ECget/Englishman_flow
2025-12-18 01:52:07,096 INFO [collect_river_data] ECCC Englishman river data collection for 2025-12-17 completed
2025-12-18 01:52:07,098 INFO [collect_river_data] running in process 1491136
2025-12-18 01:52:07,099 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-18 01:52:07,100 INFO [collect_river_data] publishing log messages to tcp://*:5576
2025-12-18 01:52:07,101 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-12-18 01:52:07,102 INFO [collect_river_data] Collecting ECCC HomathkoMouth river data for 2025-12-17
2025-12-18 01:52:07,104 INFO [collect_river_data] Appended ECCC ClowhomClowhomLake river average discharge for 2025-12-17 to: /results/forcing/rivers/observations/Clowhom_ClowhomLake_flow
2025-12-18 01:52:07,105 INFO [collect_river_data] ECCC ClowhomClowhomLake river data collection for 2025-12-17 completed
2025-12-18 01:52:07,108 INFO [collect_river_data] Appended ECCC SanJuanPortRenfrew river average discharge for 2025-12-17 to: /results/forcing/rivers/observations/SanJuan_PortRenfrew_flow
2025-12-18 01:52:07,109 INFO [collect_river_data] ECCC SanJuanPortRenfrew river data collection for 2025-12-17 completed
2025-12-18 01:52:07,111 INFO [collect_river_data] Appended ECCC TheodosiaScotty river average discharge for 2025-12-17 to: /results/forcing/rivers/observations/Theodosia_Scotty_flow
2025-12-18 01:52:07,111 INFO [collect_river_data] ECCC TheodosiaScotty river data collection for 2025-12-17 completed
2025-12-18 01:52:07,113 INFO [collect_river_data] running in process 1491137
2025-12-18 01:52:07,114 INFO [collect_river_data] Appended ECCC TheodosiaBypass river average discharge for 2025-12-17 to: /results/forcing/rivers/observations/Theodosia_Bypass_flow
2025-12-18 01:52:07,115 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-18 01:52:07,117 INFO [collect_river_data] ECCC TheodosiaBypass river data collection for 2025-12-17 completed
2025-12-18 01:52:07,118 INFO [collect_river_data] publishing log messages to tcp://*:5577
2025-12-18 01:52:07,119 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-12-18 01:52:07,119 INFO [collect_river_data] Collecting ECCC NicomeklLangley river data for 2025-12-17
2025-12-18 01:52:07,131 INFO [collect_river_data] Appended ECCC HomathkoMouth river average discharge for 2025-12-17 to: /results/forcing/rivers/observations/Homathko_Mouth_flow
2025-12-18 01:52:07,132 INFO [collect_river_data] ECCC HomathkoMouth river data collection for 2025-12-17 completed
2025-12-18 01:52:07,207 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-12-18 01:52:07,222 INFO [collect_river_data] Appended ECCC NicomeklLangley river average discharge for 2025-12-17 to: /results/forcing/rivers/observations/Nicomekl_Langley_flow
2025-12-18 01:52:07,223 INFO [collect_river_data] ECCC NicomeklLangley river data collection for 2025-12-17 completed
2025-12-18 01:52:07,286 INFO [get_vfpa_hadcp] running in process 1491148
2025-12-18 01:52:07,287 INFO [get_vfpa_hadcp] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-18 01:52:07,287 INFO [get_vfpa_hadcp] publishing log messages to tcp://*:5578
2025-12-18 01:52:07,288 INFO [get_vfpa_hadcp] connected to skookum.eos.ubc.ca port 5555
2025-12-18 01:52:07,289 INFO [get_vfpa_hadcp] processing VFPA HADCP data from 2nd Narrows Rail Bridge for 2025-12-17
2025-12-18 01:52:07,399 INFO [collect_NeahBay_ssh] 2025-12-18T00:00:00-08:00 Neah Bay ssh 00Z obs/forecast data collection complete
2025-12-18 01:52:07,427 INFO [crop_gribs] running in process 1491151
2025-12-18 01:52:07,428 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-18 01:52:07,429 INFO [crop_gribs] publishing log messages to tcp://*:5579
2025-12-18 01:52:07,430 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2025-12-18 01:52:07,431 INFO [crop_gribs] cropping 2025-12-18 ECCC HRDPS 2.5km continental 12Z GRIB files to SalishSeaCast subdomain
2025-12-18 01:52:07,448 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20251218/12/
2025-12-18 01:52:07,661 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-12-18 01:52:07,990 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-12-18 01:52:07,996 INFO [get_onc_ctd] running in process 1491145
2025-12-18 01:52:07,997 INFO [get_onc_ctd] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-18 01:52:07,998 INFO [get_onc_ctd] publishing log messages to tcp://*:5563
2025-12-18 01:52:07,999 INFO [get_onc_ctd] connected to skookum.eos.ubc.ca port 5555
2025-12-18 01:52:08,000 INFO [get_onc_ctd] requesting ONC SCVIP CTD T&S data for 2025-12-17
2025-12-18 01:52:08,023 INFO [get_onc_ctd] running in process 1491146
2025-12-18 01:52:08,024 INFO [get_onc_ctd] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-18 01:52:08,026 INFO [get_onc_ctd] publishing log messages to tcp://*:5565
2025-12-18 01:52:08,026 INFO [get_onc_ctd] connected to skookum.eos.ubc.ca port 5555
2025-12-18 01:52:08,027 INFO [get_onc_ctd] requesting ONC SEVIP CTD T&S data for 2025-12-17
2025-12-18 01:52:08,054 INFO [get_onc_ferry] running in process 1491147
2025-12-18 01:52:08,056 INFO [get_onc_ferry] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-18 01:52:08,057 INFO [get_onc_ferry] publishing log messages to tcp://*:5590
2025-12-18 01:52:08,057 INFO [get_onc_ferry] connected to skookum.eos.ubc.ca port 5555
2025-12-18 01:52:08,058 INFO [get_onc_ferry] requesting ONC TWDP.N1 NAV data for 2025-12-17
2025-12-18 01:52:08,348 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-12-18 01:52:08,447 INFO [grib_to_netcdf] running in process 1491392
2025-12-18 01:52:08,448 INFO [grib_to_netcdf] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-18 01:52:08,449 INFO [grib_to_netcdf] publishing log messages to tcp://*:5566
2025-12-18 01:52:08,450 INFO [grib_to_netcdf] connected to skookum.eos.ubc.ca port 5555
2025-12-18 01:52:08,451 INFO [grib_to_netcdf] creating NEMO-atmos forcing files for 2025-12-18 forecast2 run
2025-12-18 01:52:08,673 INFO [get_onc_ferry] requesting ONC TWDP.N2 NAV data for 2025-12-17
2025-12-18 01:52:08,729 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-12-18 01:52:08,734 ERROR [get_onc_ferry] request for ONC TWDP.N2 NAV data for 2025-12-17 failed: 400 Client Error:  for url: https://data.oceannetworks.ca/api/scalardata?method=getByLocation&token=8efe44eb-0fb5-4413-bba3-ae3dbe0fbb1d&locationCode=TWDP.N2&deviceCategoryCode=NAV&sensorCategoryCodes=longitude%2Clatitude&dateFrom=2025-12-17T00:00:00.000Z&dateTo=2025-12-17T23:59:00.000Z&resampleType=avg&resamplePeriod=1
2025-12-18 01:52:09,100 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-12-18 01:52:09,235 CRITICAL [get_onc_ferry] 2025-12-17 ONC TWDP ferry data file creation failed
2025-12-18 01:52:09,555 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-12-18 01:52:09,695 INFO [get_vfpa_hadcp] added VFPA HADCP data from 2nd Narrows Rail Bridge for 2025-12-17 to /opp/observations/AISDATA/netcdf/VFPA_2ND_NARROWS_HADCP_2s_202512.nc
2025-12-18 01:52:09,696 INFO [get_vfpa_hadcp] VFPA HADCP observations added to 2025-12 netcdf file
2025-12-18 01:52:09,881 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-12-18 01:52:10,191 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-12-18 01:52:10,506 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-12-18 01:52:10,821 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-12-18 01:52:10,863 WARNING [get_onc_ctd] no SCVIP salinity data
2025-12-18 01:52:11,143 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-12-18 01:52:11,155 INFO [get_onc_ctd] 2025-12-17 ONC SCVIP CTD T&S file created
2025-12-18 01:52:11,453 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-12-18 01:52:11,760 INFO [manager] checklist updated with [Neah Bay ssh data] items from collect_NeahBay_ssh worker
2025-12-18 01:52:12,095 INFO [manager] launching NextWorker(module='nowcast.workers.make_ssh_files', args=['forecast2', '--run-date', '2025-12-18'], host='localhost')
2025-12-18 01:52:12,104 INFO [manager] checklist updated with [VFPA HADCP data] items from get_vfpa_hadcp worker
2025-12-18 01:52:12,552 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['VFPA-HADCP'], host='localhost')
2025-12-18 01:52:12,558 INFO [manager] checklist updated with [ONC CTD data] items from get_onc_ctd worker
2025-12-18 01:52:12,970 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['SCVIP-CTD'], host='localhost')
2025-12-18 01:52:13,780 INFO [ping_erddap] running in process 1491651
2025-12-18 01:52:13,781 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-18 01:52:13,782 INFO [ping_erddap] publishing log messages to tcp://*:5563
2025-12-18 01:52:13,783 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2025-12-18 01:52:13,784 INFO [ping_erddap] VFPA-HADCP ERDDAP dataset flag file(s) created
2025-12-18 01:52:13,786 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2025-12-18 01:52:14,157 INFO [ping_erddap] running in process 1491653
2025-12-18 01:52:14,159 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-18 01:52:14,160 INFO [ping_erddap] publishing log messages to tcp://*:5564
2025-12-18 01:52:14,161 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2025-12-18 01:52:14,163 INFO [ping_erddap] SCVIP-CTD ERDDAP dataset flag file(s) created
2025-12-18 01:52:14,166 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2025-12-18 01:52:15,433 INFO [make_ssh_files] running in process 1491638
2025-12-18 01:52:15,435 INFO [make_ssh_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-18 01:52:15,436 INFO [make_ssh_files] publishing log messages to tcp://*:5563
2025-12-18 01:52:15,437 INFO [make_ssh_files] connected to skookum.eos.ubc.ca port 5555
2025-12-18 01:52:15,438 INFO [make_ssh_files] building forecast2 sea surface height boundary conditions file(s) from 2025-12-18 NOAA Neah Bay 00Z observation and forecast values
2025-12-18 01:52:17,452 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2025m12d13.nc
2025-12-18 01:52:17,718 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2025m12d14.nc
2025-12-18 01:52:17,992 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2025m12d15.nc
2025-12-18 01:52:18,266 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2025m12d16.nc
2025-12-18 01:52:18,481 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2025m12d17.nc
2025-12-18 01:52:18,696 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2025m12d18.nc
2025-12-18 01:52:18,871 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2025m12d19.nc
2025-12-18 01:52:19,106 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2025m12d20.nc
2025-12-18 01:52:19,283 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2025m12d21.nc
2025-12-18 01:52:19,416 INFO [make_ssh_files] sea surface height boundary file for 2025-12-18 forecast2 run created
2025-12-18 01:52:19,420 INFO [manager] checklist updated with [sea surface height forcing] items from make_ssh_files worker
2025-12-18 01:52:19,719 INFO [manager] launching NextWorker(module='nowcast.workers.make_runoff_file', args=['v202108'], host='localhost')
2025-12-18 01:52:19,725 INFO [manager] launching NextWorker(module='nowcast.workers.make_201702_runoff_file', args=[], host='localhost')
2025-12-18 01:52:20,337 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2025m12d19.nc
2025-12-18 01:52:20,978 INFO [make_201702_runoff_file] running in process 1491710
2025-12-18 01:52:20,979 INFO [make_201702_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-18 01:52:20,980 INFO [make_201702_runoff_file] publishing log messages to tcp://*:5563
2025-12-18 01:52:20,981 INFO [make_201702_runoff_file] connected to skookum.eos.ubc.ca port 5555
2025-12-18 01:52:21,005 INFO [make_201702_runoff_file] calculating runoff forcing for b201702 bathymetry
2025-12-18 01:52:21,141 INFO [make_201702_runoff_file] 2025-12-18 runoff file creation from Fraser at Hope and climatology elsewhere complete
2025-12-18 01:52:21,143 INFO [manager] checklist updated with [rivers forcing] items from make_201702_runoff_file worker
2025-12-18 01:52:21,619 INFO [make_runoff_file] running in process 1491708
2025-12-18 01:52:21,621 INFO [make_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-18 01:52:21,622 INFO [make_runoff_file] publishing log messages to tcp://*:5564
2025-12-18 01:52:21,623 INFO [make_runoff_file] connected to skookum.eos.ubc.ca port 5555
2025-12-18 01:52:21,624 INFO [make_runoff_file] calculating NEMO runoff forcing for v202108 bathymetry for 2025-12-17
2025-12-18 01:52:22,575 ERROR [make_runoff_file] no 2025-12-17 discharge obs for SkagitMountVernon: patching
2025-12-18 01:52:23,084 ERROR [make_runoff_file] no 2025-12-17 discharge obs for SnohomishMonroe: patching
2025-12-18 01:52:23,322 ERROR [make_runoff_file] no 2025-12-17 discharge obs for NisquallyMcKenna: patching
2025-12-18 01:52:23,417 ERROR [make_runoff_file] no 2025-12-17 discharge obs for GreenwaterGreenwater: patching
2025-12-18 01:52:23,475 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2025m12d20.nc
2025-12-18 01:52:23,483 INFO [grib_to_netcdf] 2025-12-18 NEMO-atmos forcing file for forecast2 created
2025-12-18 01:52:23,485 INFO [manager] checklist updated with [weather forcing] items from grib_to_netcdf worker
2025-12-18 01:52:24,813 INFO [make_runoff_file] stored NEMO runoff forcing for v202108 bathymetry for 2025-12-17: /results/forcing/rivers/R202108Dailies_y2025m12d17.nc
2025-12-18 01:52:24,814 INFO [make_runoff_file] 2025-12-17 runoff file creation completed
2025-12-18 01:52:24,816 INFO [manager] checklist updated with [rivers forcing] items from make_runoff_file worker
2025-12-18 01:52:25,098 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'forecast2'], host='localhost')
2025-12-18 01:52:25,103 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'forecast2'], host='localhost')
2025-12-18 01:52:25,108 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.nibi', 'forecast2'], host='localhost')
2025-12-18 01:52:25,112 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'forecast2'], host='localhost')
2025-12-18 01:52:26,414 INFO [upload_forcing] running in process 1491753
2025-12-18 01:52:26,415 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-18 01:52:26,416 INFO [upload_forcing] publishing log messages to tcp://*:5563
2025-12-18 01:52:26,417 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-12-18 01:52:26,435 INFO [upload_forcing] running in process 1491756
2025-12-18 01:52:26,436 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-18 01:52:26,437 INFO [upload_forcing] publishing log messages to tcp://*:5564
2025-12-18 01:52:26,438 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-12-18 01:52:26,440 INFO [upload_forcing] running in process 1491755
2025-12-18 01:52:26,441 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-18 01:52:26,442 INFO [upload_forcing] publishing log messages to tcp://*:5566
2025-12-18 01:52:26,443 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-12-18 01:52:26,467 INFO [upload_forcing] running in process 1491754
2025-12-18 01:52:26,468 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-18 01:52:26,469 INFO [upload_forcing] publishing log messages to tcp://*:5567
2025-12-18 01:52:26,470 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-12-18 01:52:26,560 INFO [get_onc_ctd] 2025-12-17 ONC SEVIP CTD T&S file created
2025-12-18 01:52:26,563 INFO [manager] checklist updated with [ONC CTD data] items from get_onc_ctd worker
2025-12-18 01:52:26,921 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['SEVIP-CTD'], host='localhost')
2025-12-18 01:52:27,891 INFO [upload_forcing] LiveOcean boundary conditions file not found; created symlink to /results/forcing/LiveOcean/boundary_conditions/LiveOcean_v201905_y2025m12d17.nc
2025-12-18 01:52:27,958 INFO [upload_forcing] forecast2 2025-12-18 forcing files upload to optimum-hindcast completed
2025-12-18 01:52:27,962 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-12-18 01:52:28,182 INFO [ping_erddap] running in process 1491784
2025-12-18 01:52:28,186 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-18 01:52:28,187 INFO [ping_erddap] publishing log messages to tcp://*:5565
2025-12-18 01:52:28,188 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2025-12-18 01:52:28,190 INFO [ping_erddap] SEVIP-CTD ERDDAP dataset flag file(s) created
2025-12-18 01:52:28,297 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2025-12-18 01:52:31,026 INFO [upload_forcing] forecast2 2025-12-18 forcing files upload to arbutus.cloud-nowcast completed
2025-12-18 01:52:31,031 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-12-18 01:52:31,380 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2025-12-18'], host='localhost')
2025-12-18 01:52:32,695 INFO [upload_forcing] forecast2 2025-12-18 forcing files upload to orcinus-nowcast-agrif completed
2025-12-18 01:52:32,697 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-12-18 01:52:32,904 INFO [make_forcing_links] running in process 1491792
2025-12-18 01:52:32,906 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-18 01:52:32,907 INFO [make_forcing_links] publishing log messages to tcp://*:5563
2025-12-18 01:52:32,908 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2025-12-18 01:52:34,328 INFO [make_forcing_links] forecast2 2025-12-18 forcing file links on arbutus.cloud-nowcast created
2025-12-18 01:52:34,332 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2025-12-18 01:52:34,653 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2025-12-18'], host='arbutus.cloud-nowcast')
2025-12-18 01:52:35,880 INFO [upload_forcing] forecast2 2025-12-18 forcing files upload to robot.nibi completed
2025-12-18 01:52:35,885 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-12-18 01:52:36,920 INFO [run_NEMO] running in process 12212
2025-12-18 01:52:36,921 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-12-18 01:52:36,923 INFO [run_NEMO] publishing log messages to tcp://*:5556
2025-12-18 01:52:36,982 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2025-12-18 01:52:37,697 WARNING [run_NEMO] There are uncommitted changes in /nemoShare/MEOPAR/nowcast-sys/SS-run-sets
2025-12-18 01:52:38,006 INFO [run_NEMO] forecast2: launching /nemoShare/MEOPAR/nowcast-sys/runs/17dec25forecast2_2025-12-18T015237.045028-0800/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2025-12-18 01:52:38,018 INFO [run_NEMO] forecast2 NEMO run for 2025-12-18 on arbutus.cloud-nowcast started
2025-12-18 01:52:38,022 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2025-12-18 01:52:38,337 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'forecast2'], host='arbutus.cloud-nowcast')
2025-12-18 01:52:40,340 INFO [watch_NEMO] running in process 12486
2025-12-18 01:52:40,342 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-12-18 01:52:40,343 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2025-12-18 01:52:40,344 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2025-12-18 01:52:40,409 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2025-12-18 01:57:40,414 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 18913424 = 2025-12-19 05:08:40 UTC, 17.2% complete
2025-12-18 02:02:40,422 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 18913956 = 2025-12-19 11:03:20 UTC, 36.9% complete
2025-12-18 02:07:40,421 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 18914472 = 2025-12-19 16:47:20 UTC, 56.0% complete
2025-12-18 02:12:40,423 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 18914988 = 2025-12-19 22:31:20 UTC, 75.1% complete
2025-12-18 02:17:40,425 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 18915509 = 2025-12-20 04:18:40 UTC, 94.4% complete
2025-12-18 02:22:40,435 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2025-12-18 02:22:40,436 INFO [watch_NEMO] forecast2 NEMO run on arbutus.cloud-nowcast completed
2025-12-18 02:22:40,946 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_wind_file', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2025-12-18'], host='arbutus.cloud-nowcast')
2025-12-18 02:22:40,955 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_current_file', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2025-12-18'], host='arbutus.cloud-nowcast')
2025-12-18 02:22:40,960 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2025-12-17'], host='localhost')
2025-12-18 02:22:42,231 INFO [download_results] running in process 1492177
2025-12-18 02:22:42,232 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-18 02:22:42,233 INFO [download_results] publishing log messages to tcp://*:5563
2025-12-18 02:22:42,235 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2025-12-18 02:22:42,236 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/forecast2/17dec25 to /results/SalishSea/forecast2.202111
2025-12-18 02:22:43,732 INFO [make_ww3_wind_file] running in process 12841
2025-12-18 02:22:43,733 INFO [make_ww3_wind_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-12-18 02:22:43,734 INFO [make_ww3_wind_file] publishing log messages to tcp://*:5570
2025-12-18 02:22:43,735 INFO [make_ww3_wind_file] connected to skookum.eos.ubc.ca port 5555
2025-12-18 02:22:43,736 INFO [make_ww3_wind_file] Creating wwatch3 wind forcing file for 2025-12-18 forecast2 run
2025-12-18 02:22:43,760 INFO [make_ww3_current_file] running in process 12840
2025-12-18 02:22:43,761 INFO [make_ww3_current_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-12-18 02:22:43,763 INFO [make_ww3_current_file] publishing log messages to tcp://*:5571
2025-12-18 02:22:43,764 INFO [make_ww3_current_file] connected to skookum.eos.ubc.ca port 5555
2025-12-18 02:22:43,765 INFO [make_ww3_current_file] Creating wwatch3 currents forcing file for 2025-12-18 forecast2 run
2025-12-18 02:22:52,481 INFO [make_ww3_wind_file] wwatch3 wind forcing file created on arbutus.cloud-nowcast for 2025-12-18 forecast2 run
2025-12-18 02:22:52,484 INFO [manager] checklist updated with [WW3 wind forcing] items from make_ww3_wind_file worker
2025-12-18 02:26:21,493 INFO [make_ww3_current_file] wwatch3 currents forcing file created on arbutus.cloud-nowcast for 2025-12-18 forecast2 run
2025-12-18 02:26:21,497 INFO [manager] checklist updated with [WW3 currents forcing] items from make_ww3_current_file worker
2025-12-18 02:26:21,849 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2025-12-18'], host='arbutus.cloud-nowcast')
2025-12-18 02:26:23,884 INFO [run_ww3] running in process 13396
2025-12-18 02:26:23,885 INFO [run_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-12-18 02:26:23,886 INFO [run_ww3] publishing log messages to tcp://*:5572
2025-12-18 02:26:23,888 INFO [run_ww3] connected to skookum.eos.ubc.ca port 5555
2025-12-18 02:26:23,892 INFO [run_ww3] Created run directory /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast2_2025-12-18T022623.866869-0800
2025-12-18 02:26:23,894 INFO [run_ww3] forecast2: launching /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast2_2025-12-18T022623.866869-0800/SoGWW3.sh on arbutus.cloud-nowcast
2025-12-18 02:26:23,903 INFO [run_ww3] forecast2 WaveWatch3 run for 2025-12-18 on arbutus.cloud-nowcast started
2025-12-18 02:26:23,908 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker
2025-12-18 02:26:24,256 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'forecast2'], host='arbutus.cloud-nowcast')
2025-12-18 02:26:26,226 INFO [watch_ww3] running in process 13483
2025-12-18 02:26:26,227 INFO [watch_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-12-18 02:26:26,228 INFO [watch_ww3] publishing log messages to tcp://*:5573
2025-12-18 02:26:26,229 INFO [watch_ww3] connected to skookum.eos.ubc.ca port 5555
2025-12-18 02:26:26,266 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2025-12-18 02:26:36,670 INFO [download_results] forecast2 2025-12-17 results files from arbutus.cloud-nowcast downloaded
2025-12-18 02:26:36,674 INFO [manager] checklist updated with [results files] items from download_results worker
2025-12-18 02:26:37,016 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['forecast2', '--run-date', '2025-12-17'], host='localhost')
2025-12-18 02:26:39,064 INFO [make_CHS_currents_file] running in process 1492233
2025-12-18 02:26:39,066 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-18 02:26:39,067 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5563
2025-12-18 02:26:39,068 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555
2025-12-18 02:27:19,430 INFO [make_CHS_currents_file] Made CHS currents file for 2025-12-17 for forecast2
2025-12-18 02:27:19,433 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker
2025-12-18 02:27:19,762 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['nemo', 'forecast2', '--run-date', '2025-12-17'], host='localhost')
2025-12-18 02:27:21,053 INFO [update_forecast_datasets] running in process 1492275
2025-12-18 02:27:21,054 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-18 02:27:21,056 INFO [update_forecast_datasets] publishing log messages to tcp://*:5563
2025-12-18 02:27:21,057 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2025-12-18 02:27:21,058 INFO [update_forecast_datasets] updating nemo forecast directory for forecast2/17dec25 run
2025-12-18 02:31:26,263 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 81, 15.0% complete
2025-12-18 02:31:39,864 INFO [update_forecast_datasets] updated nemo forecast directory for forecast2/17dec25 run: /results/SalishSea/rolling-forecasts/nemo
2025-12-18 02:31:39,865 INFO [update_forecast_datasets] nemo 2025-12-17 forecast2 rolling forecast datasets updated
2025-12-18 02:31:39,867 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2025-12-18 02:31:40,212 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nemo-forecast'], host='localhost')
2025-12-18 02:31:40,218 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'forecast2', 'publish', '--run-date', '2025-12-17'], host='localhost')
2025-12-18 02:31:40,222 INFO [manager] launching NextWorker(module='nowcast.workers.make_surface_current_tiles', args=['forecast2', '--run-date', '2025-12-17'], host='localhost')
2025-12-18 02:31:41,391 INFO [ping_erddap] running in process 1492449
2025-12-18 02:31:41,393 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-18 02:31:41,394 INFO [ping_erddap] publishing log messages to tcp://*:5563
2025-12-18 02:31:41,394 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2025-12-18 02:31:41,401 INFO [ping_erddap] nemo-forecast ERDDAP dataset flag file(s) created
2025-12-18 02:31:41,406 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2025-12-18 02:31:42,843 INFO [make_surface_current_tiles] running in process 1492452
2025-12-18 02:31:42,844 INFO [make_surface_current_tiles] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-18 02:31:42,845 INFO [make_surface_current_tiles] publishing log messages to tcp://*:5563
2025-12-18 02:31:42,846 INFO [make_surface_current_tiles] connected to skookum.eos.ubc.ca port 5555
2025-12-18 02:31:42,847 INFO [make_surface_current_tiles] starting rendering of tiles for 2025-12-17 forecast2 in 6 processes into /results/nowcast-sys/figures/surface_currents/forecast2/17dec25
2025-12-18 02:31:43,971 INFO [make_plots] running in process 1492451
2025-12-18 02:31:43,985 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-18 02:31:43,986 INFO [make_plots] publishing log messages to tcp://*:5564
2025-12-18 02:31:43,988 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2025-12-18 02:31:44,545 INFO [make_plots] preparing render list for 2025-12-17 NEMO forecast2 publish figures
2025-12-18 02:31:44,856 INFO [make_plots] added Website_thumbnail figure to 2025-12-17 NEMO forecast2 publish render list
2025-12-18 02:31:44,857 INFO [make_plots] added Threshold_website figure to 2025-12-17 NEMO forecast2 publish render list
2025-12-18 02:31:44,857 INFO [make_plots] added PA_tidal_predictions figure to 2025-12-17 NEMO forecast2 publish render list
2025-12-18 02:31:44,858 INFO [make_plots] added Boundary Bay figure to 2025-12-17 NEMO forecast2 publish render list
2025-12-18 02:31:44,859 INFO [make_plots] added Campbell River figure to 2025-12-17 NEMO forecast2 publish render list
2025-12-18 02:31:44,860 INFO [make_plots] added Cherry Point figure to 2025-12-17 NEMO forecast2 publish render list
2025-12-18 02:31:44,861 INFO [make_plots] added Friday Harbor figure to 2025-12-17 NEMO forecast2 publish render list
2025-12-18 02:31:44,861 INFO [make_plots] added Halfmoon Bay figure to 2025-12-17 NEMO forecast2 publish render list
2025-12-18 02:31:44,862 INFO [make_plots] added Nanaimo figure to 2025-12-17 NEMO forecast2 publish render list
2025-12-18 02:31:44,863 INFO [make_plots] added Neah Bay figure to 2025-12-17 NEMO forecast2 publish render list
2025-12-18 02:31:44,864 INFO [make_plots] added New Westminster figure to 2025-12-17 NEMO forecast2 publish render list
2025-12-18 02:31:44,864 INFO [make_plots] added Patricia Bay figure to 2025-12-17 NEMO forecast2 publish render list
2025-12-18 02:31:44,865 INFO [make_plots] added Point Atkinson figure to 2025-12-17 NEMO forecast2 publish render list
2025-12-18 02:31:44,866 INFO [make_plots] added Port Renfrew figure to 2025-12-17 NEMO forecast2 publish render list
2025-12-18 02:31:44,867 INFO [make_plots] added Sand Heads figure to 2025-12-17 NEMO forecast2 publish render list
2025-12-18 02:31:44,867 INFO [make_plots] added Sandy Cove figure to 2025-12-17 NEMO forecast2 publish render list
2025-12-18 02:31:44,868 INFO [make_plots] added Squamish figure to 2025-12-17 NEMO forecast2 publish render list
2025-12-18 02:31:44,869 INFO [make_plots] added Victoria figure to 2025-12-17 NEMO forecast2 publish render list
2025-12-18 02:31:44,870 INFO [make_plots] added Woodwards Landing figure to 2025-12-17 NEMO forecast2 publish render list
2025-12-18 02:31:44,870 INFO [make_plots] starting to render nemo forecast2 publish 17dec25 figures
2025-12-18 02:36:26,265 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 174, 32.2% complete
2025-12-18 02:39:30,145 WARNING [make_plots] invalid station number for Boundary Bay station: None
2025-12-18 02:39:30,146 WARNING [make_plots] can't resolve a valid CHS station code for Boundary Bay
2025-12-18 02:40:06,613 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb996/data for station 08074 Campbell River from 2025-12-13 00:00:00Z to 2025-12-20 05:55:00Z
2025-12-18 02:41:26,266 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 264, 48.9% complete
2025-12-18 02:41:34,816 WARNING [make_plots] invalid station number for Halfmoon Bay station: None
2025-12-18 02:41:34,816 WARNING [make_plots] can't resolve a valid CHS station code for Halfmoon Bay
2025-12-18 02:41:59,428 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb96d/data for station 07917 Nanaimo from 2025-12-13 00:00:00Z to 2025-12-20 05:55:00Z
2025-12-18 02:42:47,637 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e13d0f4a073c4bbf93/data for station 07654 New Westminster from 2025-12-13 00:00:00Z to 2025-12-20 05:55:00Z
2025-12-18 02:42:54,460 INFO [make_surface_current_tiles] starting PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast2/17dec25
2025-12-18 02:43:08,199 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd2d/data for station 07277 Patricia Bay from 2025-12-13 00:00:00Z to 2025-12-20 05:55:00Z
2025-12-18 02:43:29,551 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94c/data for station 07795 Point Atkinson from 2025-12-13 00:00:00Z to 2025-12-20 05:55:00Z
2025-12-18 02:43:50,244 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e23d0f4a073c4bc060/data for station 08525 Port Renfrew from 2025-12-13 00:00:00Z to 2025-12-20 05:55:00Z
2025-12-18 02:44:08,456 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb93c/data for station 07594 Sand Heads from 2025-12-13 00:00:00Z to 2025-12-20 05:55:00Z
2025-12-18 02:44:30,416 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc43b/data for station 07786 Sandy Cove from 2025-12-13 00:00:00Z to 2025-12-20 05:55:00Z
2025-12-18 02:44:49,974 WARNING [make_plots] invalid station number for Squamish station: None
2025-12-18 02:44:49,975 WARNING [make_plots] can't resolve a valid CHS station code for Squamish
2025-12-18 02:45:10,814 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd1e/data for station 07120 Victoria from 2025-12-13 00:00:00Z to 2025-12-20 05:55:00Z
2025-12-18 02:45:28,900 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc42d/data for station 07610 Woodwards Landing from 2025-12-13 00:00:00Z to 2025-12-20 05:55:00Z
2025-12-18 02:45:49,382 INFO [make_plots] finished rendering nemo forecast2 publish 17dec25 figures
2025-12-18 02:45:49,385 INFO [make_plots] nemo publish plots for 2025-12-17 forecast2 completed
2025-12-18 02:45:49,391 INFO [manager] checklist updated with [plots] items from make_plots worker
2025-12-18 02:45:49,756 INFO [manager] launching NextWorker(module='nowcast.workers.make_feeds', args=['forecast2', '--run-date', '2025-12-17'], host='localhost')
2025-12-18 02:45:53,369 INFO [make_feeds] running in process 1497436
2025-12-18 02:45:53,370 INFO [make_feeds] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-18 02:45:53,371 INFO [make_feeds] publishing log messages to tcp://*:5564
2025-12-18 02:45:53,372 INFO [make_feeds] connected to skookum.eos.ubc.ca port 5555
2025-12-18 02:46:07,734 INFO [make_feeds] ATOM feeds for 2025-12-17 forecast2 run completed
2025-12-18 02:46:07,737 INFO [manager] checklist updated with [storm surge alert feeds] items from make_feeds worker
2025-12-18 02:46:08,067 INFO [manager] launching NextWorker(module='nemo_nowcast.workers.clear_checklist', args=[], host='localhost')
2025-12-18 02:46:09,226 INFO [clear_checklist] running in process 1497468
2025-12-18 02:46:09,227 INFO [clear_checklist] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-18 02:46:09,228 INFO [clear_checklist] publishing log messages to tcp://*:5564
2025-12-18 02:46:09,229 INFO [clear_checklist] connected to skookum.eos.ubc.ca port 5555
2025-12-18 02:46:09,230 INFO [clear_checklist] requesting that manager clear system state checklist
2025-12-18 02:46:09,233 INFO [manager] writing checklist to log file
2025-12-18 02:46:09,274 INFO [manager] checklist cleared
2025-12-18 02:46:09,276 INFO [clear_checklist] nowcast system checklist cleared
2025-12-18 02:46:09,278 INFO [manager] launching NextWorker(module='nemo_nowcast.workers.rotate_logs', args=[], host='localhost')
2025-12-18 02:46:10,464 INFO [rotate_logs] running in process 1497473
2025-12-18 02:46:10,465 INFO [rotate_logs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-18 02:46:10,466 INFO [rotate_logs] publishing log messages to tcp://*:5564
2025-12-18 02:46:10,467 INFO [rotate_logs] connected to skookum.eos.ubc.ca port 5555
2025-12-18 02:46:10,468 INFO [rotate_logs] rotating log files