2025-06-08 03:57:06,555 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/nowcast.log
2025-06-08 03:57:06,558 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/nowcast.debug.log
2025-06-08 03:57:06,561 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/checklist.log
2025-06-08 03:57:06,562 INFO [rotate_logs] log files rotated
2025-06-08 03:57:06,566 INFO [manager] checklist updated with [log rotation] items from rotate_logs worker
2025-06-08 03:59:07,246 INFO [make_surface_current_tiles] finished PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast2/07jun25
2025-06-08 03:59:07,270 INFO [make_surface_current_tiles] finished rendering of tiles for 2025-06-07 forecast2 into /results/nowcast-sys/figures/surface_currents/forecast2/07jun25
2025-06-08 03:59:07,271 INFO [make_surface_current_tiles] surface current tile figures for 2025-06-07 forecast2 completed
2025-06-08 03:59:07,581 INFO [manager] checklist updated with [surface current tiles] items from make_surface_current_tiles worker
2025-06-08 04:01:54,463 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2025-06-08 04:06:54,464 INFO [watch_ww3] forecast2 WWATCH3 run on arbutus.cloud-nowcast completed
2025-06-08 04:06:54,469 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2025-06-08 04:06:54,832 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2025-06-08'], host='localhost')
2025-06-08 04:06:55,957 INFO [download_wwatch3_results] running in process 746970
2025-06-08 04:06:55,959 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-08 04:06:55,960 INFO [download_wwatch3_results] publishing log messages to tcp://*:5562
2025-06-08 04:06:55,961 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2025-06-08 04:07:37,035 INFO [download_wwatch3_results] forecast2 2025-06-08 results files from arbutus.cloud-nowcast downloaded
2025-06-08 04:07:37,037 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2025-06-08 04:07:37,194 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['wwatch3', 'forecast2', '--run-date', '2025-06-08'], host='localhost')
2025-06-08 04:07:38,154 INFO [update_forecast_datasets] running in process 747002
2025-06-08 04:07:38,156 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-08 04:07:38,156 INFO [update_forecast_datasets] publishing log messages to tcp://*:5562
2025-06-08 04:07:38,157 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2025-06-08 04:07:38,158 INFO [update_forecast_datasets] updating wwatch3 most_recent_forecast directory from forecast2/08jun25 run
2025-06-08 04:07:38,162 INFO [update_forecast_datasets] updated wwatch3 most_recent_forecast directory from forecast2/08jun25 run
2025-06-08 04:07:38,162 INFO [update_forecast_datasets] updating wwatch3 forecast directory for forecast2/08jun25 run
2025-06-08 04:08:01,338 INFO [update_forecast_datasets] updated wwatch3 forecast directory for forecast2/08jun25 run: /results/SalishSea/rolling-forecasts/wwatch3
2025-06-08 04:08:01,339 INFO [update_forecast_datasets] wwatch3 2025-06-08 forecast2 rolling forecast datasets updated
2025-06-08 04:08:01,342 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2025-06-08 04:08:01,496 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['wwatch3-forecast'], host='localhost')
2025-06-08 04:08:02,526 INFO [ping_erddap] running in process 747013
2025-06-08 04:08:02,527 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-08 04:08:02,528 INFO [ping_erddap] publishing log messages to tcp://*:5562
2025-06-08 04:08:02,529 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2025-06-08 04:08:02,531 INFO [ping_erddap] wwatch3-forecast ERDDAP dataset flag file(s) created
2025-06-08 04:08:02,533 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2025-06-08 04:08:02,656 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['wwatch3', 'forecast2', 'publish', '--run-date', '2025-06-08'], host='localhost')
2025-06-08 04:08:05,198 INFO [make_plots] running in process 747018
2025-06-08 04:08:05,199 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-08 04:08:05,199 INFO [make_plots] publishing log messages to tcp://*:5562
2025-06-08 04:08:05,200 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2025-06-08 04:08:05,201 INFO [make_plots] preparing render list for 2025-06-08 WaveWatch3 forecast2 publish figures
2025-06-08 04:08:05,202 INFO [make_plots] added Halibut Bank figure to 2025-06-08 WaveWatch3 forecast2 publish render list
2025-06-08 04:08:05,202 INFO [make_plots] added Sentry Shoal figure to 2025-06-08 WaveWatch3 forecast2 publish render list
2025-06-08 04:08:05,203 INFO [make_plots] starting to render wwatch3 forecast2 publish 08jun25 figures
2025-06-08 04:08:05,556 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46146 Halibut Bank
2025-06-08 04:08:10,416 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46131 Sentry Shoal
2025-06-08 04:08:14,597 INFO [make_plots] finished rendering wwatch3 forecast2 publish 08jun25 figures
2025-06-08 04:08:14,598 INFO [make_plots] wwatch3 publish plots for 2025-06-08 forecast2 completed
2025-06-08 04:08:14,602 INFO [manager] checklist updated with [plots] items from make_plots worker
2025-06-08 08:55:55,410 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/12/ to /results/forcing/atmospheric/continental2.5/GRIB/20250608/12/
2025-06-08 08:55:55,411 INFO [collect_weather] 2025-06-08 2.5km weather forecast 12 collection complete
2025-06-08 08:55:55,414 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2025-06-08 08:55:55,727 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'SkagitMountVernon', '--data-date', '2025-06-07'], host='localhost')
2025-06-08 08:55:55,733 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'SnohomishMonroe', '--data-date', '2025-06-07'], host='localhost')
2025-06-08 08:55:55,737 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'NisquallyMcKenna', '--data-date', '2025-06-07'], host='localhost')
2025-06-08 08:55:55,741 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'GreenwaterGreenwater', '--data-date', '2025-06-07'], host='localhost')
2025-06-08 08:55:55,745 INFO [manager] launching NextWorker(module='nowcast.workers.make_turbidity_file', args=[], host='localhost')
2025-06-08 08:55:55,749 INFO [manager] launching NextWorker(module='nowcast.workers.collect_NeahBay_ssh', args=['06'], host='localhost')
2025-06-08 08:55:55,754 INFO [manager] launching NextWorker(module='nowcast.workers.download_live_ocean', args=[], host='localhost')
2025-06-08 08:55:55,758 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['18', '2.5km'], host='localhost')
2025-06-08 08:55:55,764 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['18'], host='localhost')
2025-06-08 08:55:56,414 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20250608/12/
2025-06-08 08:55:56,415 INFO [crop_gribs] 2025-06-08 12 GRIBs cropping complete
2025-06-08 08:55:56,418 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2025-06-08 08:55:56,926 INFO [manager] launching NextWorker(module='nowcast.workers.grib_to_netcdf', args=['nowcast+'], host='localhost')
2025-06-08 08:55:57,312 INFO [collect_weather] running in process 750356
2025-06-08 08:55:57,314 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-08 08:55:57,315 INFO [collect_weather] publishing log messages to tcp://*:5562
2025-06-08 08:55:57,315 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2025-06-08 08:55:57,333 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/18/
2025-06-08 08:55:57,356 INFO [collect_NeahBay_ssh] running in process 750354
2025-06-08 08:55:57,358 INFO [collect_NeahBay_ssh] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-08 08:55:57,359 INFO [collect_NeahBay_ssh] publishing log messages to tcp://*:5563
2025-06-08 08:55:57,359 INFO [collect_NeahBay_ssh] connected to skookum.eos.ubc.ca port 5555
2025-06-08 08:55:57,360 INFO [collect_NeahBay_ssh] collecting Neah Bay ssh 06Z obs/forecast for 2025-06-08
2025-06-08 08:55:57,592 INFO [download_live_ocean] running in process 750355
2025-06-08 08:55:57,593 INFO [download_live_ocean] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-08 08:55:57,593 INFO [download_live_ocean] publishing log messages to tcp://*:5564
2025-06-08 08:55:57,597 INFO [download_live_ocean] connected to skookum.eos.ubc.ca port 5555
2025-06-08 08:55:57,600 INFO [download_live_ocean] downloading Salish Sea western boundary day-averaged LiveOcean file for 2025-06-08
2025-06-08 08:55:57,879 INFO [collect_river_data] running in process 750350
2025-06-08 08:55:57,880 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-08 08:55:57,880 INFO [collect_river_data] publishing log messages to tcp://*:5565
2025-06-08 08:55:57,881 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-06-08 08:55:57,882 INFO [collect_river_data] Collecting USGS SnohomishMonroe river data for 2025-06-07
2025-06-08 08:55:57,907 INFO [collect_river_data] running in process 750351
2025-06-08 08:55:57,908 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-08 08:55:57,909 INFO [collect_river_data] publishing log messages to tcp://*:5566
2025-06-08 08:55:57,910 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-06-08 08:55:57,911 INFO [collect_river_data] Collecting USGS NisquallyMcKenna river data for 2025-06-07
2025-06-08 08:55:57,964 INFO [make_turbidity_file] running in process 750353
2025-06-08 08:55:57,965 INFO [make_turbidity_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-08 08:55:57,966 INFO [make_turbidity_file] publishing log messages to tcp://*:5567
2025-06-08 08:55:57,966 INFO [make_turbidity_file] connected to skookum.eos.ubc.ca port 5555
2025-06-08 08:55:57,967 INFO [make_turbidity_file] Creating Fraser River turbidity forcing file for 2025-06-08
2025-06-08 08:55:57,968 INFO [collect_river_data] running in process 750352
2025-06-08 08:55:57,969 INFO [collect_river_data] running in process 750348
2025-06-08 08:55:57,970 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-08 08:55:57,971 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-08 08:55:57,972 INFO [collect_river_data] publishing log messages to tcp://*:5568
2025-06-08 08:55:57,973 INFO [collect_river_data] publishing log messages to tcp://*:5569
2025-06-08 08:55:57,973 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-06-08 08:55:57,974 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-06-08 08:55:57,975 INFO [collect_river_data] Collecting USGS GreenwaterGreenwater river data for 2025-06-07
2025-06-08 08:55:57,976 INFO [collect_river_data] Collecting USGS SkagitMountVernon river data for 2025-06-07
2025-06-08 08:55:58,195 INFO [crop_gribs] running in process 750357
2025-06-08 08:55:58,196 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-08 08:55:58,197 INFO [crop_gribs] publishing log messages to tcp://*:5570
2025-06-08 08:55:58,198 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2025-06-08 08:55:58,198 INFO [crop_gribs] cropping 2025-06-08 ECCC HRDPS 2.5km continental 18Z GRIB files to SalishSeaCast subdomain
2025-06-08 08:55:58,211 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20250608/18/
2025-06-08 08:55:58,310 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12097500&startDT=2025-06-07&endDT=2025-06-07 "HTTP/1.1 200 "
2025-06-08 08:55:58,314 INFO [collect_river_data] Appended USGS GreenwaterGreenwater river average discharge for 2025-06-07 to: /results/forcing/rivers/observations/Greenwater_Greenwater_flow
2025-06-08 08:55:58,315 INFO [collect_river_data] USGS GreenwaterGreenwater river data collection for 2025-06-07 completed
2025-06-08 08:55:58,317 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-06-08 08:55:58,436 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12150800&startDT=2025-06-07&endDT=2025-06-07 "HTTP/1.1 200 "
2025-06-08 08:55:58,440 INFO [collect_river_data] Appended USGS SnohomishMonroe river average discharge for 2025-06-07 to: /results/forcing/rivers/observations/Snohomish_Monroe_flow
2025-06-08 08:55:58,440 INFO [collect_river_data] USGS SnohomishMonroe river data collection for 2025-06-07 completed
2025-06-08 08:55:58,441 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12089500&startDT=2025-06-07&endDT=2025-06-07 "HTTP/1.1 200 "
2025-06-08 08:55:58,444 INFO [collect_river_data] Appended USGS NisquallyMcKenna river average discharge for 2025-06-07 to: /results/forcing/rivers/observations/Nisqually_McKenna_flow
2025-06-08 08:55:58,445 INFO [collect_river_data] USGS NisquallyMcKenna river data collection for 2025-06-07 completed
2025-06-08 08:55:58,466 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-06-08 08:55:58,490 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12200500&startDT=2025-06-07&endDT=2025-06-07 "HTTP/1.1 200 "
2025-06-08 08:55:58,494 INFO [collect_river_data] Appended USGS SkagitMountVernon river average discharge for 2025-06-07 to: /results/forcing/rivers/observations/Skagit_MountVernon_flow
2025-06-08 08:55:58,495 INFO [collect_river_data] USGS SkagitMountVernon river data collection for 2025-06-07 completed
2025-06-08 08:55:58,597 INFO [download_live_ocean] downloaded 10229400 bytes from https://liveocean.apl.uw.edu/output/f2025.06.08/ubc.nc to /results/forcing/LiveOcean/downloaded/20250608/low_passed_UBC.nc
2025-06-08 08:55:58,598 INFO [download_live_ocean] Deflating in up to 1 concurrent sub-processes
2025-06-08 08:55:58,618 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-06-08 08:55:58,681 INFO [grib_to_netcdf] running in process 750435
2025-06-08 08:55:58,682 INFO [grib_to_netcdf] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-08 08:55:58,683 INFO [grib_to_netcdf] publishing log messages to tcp://*:5571
2025-06-08 08:55:58,684 INFO [grib_to_netcdf] connected to skookum.eos.ubc.ca port 5555
2025-06-08 08:55:58,685 INFO [grib_to_netcdf] creating NEMO-atmos forcing files for 2025-06-08 nowcast and forecast runs
2025-06-08 08:55:58,766 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-06-08 08:55:59,451 INFO [collect_NeahBay_ssh] 2025-06-08T00:00:00-07:00 Neah Bay ssh 06Z obs/forecast data collection complete
2025-06-08 08:55:59,453 INFO [manager] checklist updated with [Neah Bay ssh data] items from collect_NeahBay_ssh worker
2025-06-08 08:55:59,603 INFO [manager] launching NextWorker(module='nowcast.workers.make_ssh_files', args=['nowcast', '--run-date', '2025-06-08'], host='localhost')
2025-06-08 08:55:59,639 INFO [download_live_ocean] netCDF4 deflated /results/forcing/LiveOcean/downloaded/20250608/low_passed_UBC.nc
2025-06-08 08:55:59,639 INFO [download_live_ocean] 2025-06-08 Live Ocean file for Salish Sea western boundary downloaded
2025-06-08 08:55:59,643 INFO [manager] checklist updated with [Live Ocean products] items from download_live_ocean worker
2025-06-08 08:55:59,697 INFO [make_turbidity_file] 2025-06-08 Fraser River turbidity file creation complete
2025-06-08 08:55:59,986 INFO [manager] launching NextWorker(module='nowcast.workers.make_live_ocean_files', args=['--run-date', '2025-06-08'], host='localhost')
2025-06-08 08:55:59,992 INFO [manager] checklist updated with [Fraser River turbidity] items from make_turbidity_file worker
2025-06-08 08:56:01,875 INFO [make_live_ocean_files] running in process 750501
2025-06-08 08:56:01,876 INFO [make_live_ocean_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-08 08:56:01,877 INFO [make_live_ocean_files] publishing log messages to tcp://*:5563
2025-06-08 08:56:01,878 INFO [make_live_ocean_files] connected to skookum.eos.ubc.ca port 5555
2025-06-08 08:56:01,879 INFO [make_live_ocean_files] Creating T&S western boundary conditions file from 2025-06-08 Live Ocean run
2025-06-08 08:56:01,951 INFO [make_ssh_files] running in process 750488
2025-06-08 08:56:01,952 INFO [make_ssh_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-08 08:56:01,953 INFO [make_ssh_files] publishing log messages to tcp://*:5564
2025-06-08 08:56:01,954 INFO [make_ssh_files] connected to skookum.eos.ubc.ca port 5555
2025-06-08 08:56:01,962 INFO [make_ssh_files] building nowcast sea surface height boundary conditions file(s) from 2025-06-08 NOAA Neah Bay 06Z observation and forecast values
2025-06-08 08:56:03,727 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2025m06d04.nc
2025-06-08 08:56:03,881 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2025m06d05.nc
2025-06-08 08:56:04,053 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2025m06d06.nc
2025-06-08 08:56:04,233 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2025m06d07.nc
2025-06-08 08:56:04,427 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2025m06d08.nc
2025-06-08 08:56:04,602 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2025m06d09.nc
2025-06-08 08:56:04,775 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2025m06d10.nc
2025-06-08 08:56:04,946 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2025m06d11.nc
2025-06-08 08:56:05,047 INFO [make_ssh_files] sea surface height boundary file for 2025-06-08 nowcast run created
2025-06-08 08:56:05,051 INFO [manager] checklist updated with [sea surface height forcing] items from make_ssh_files worker
2025-06-08 08:56:05,210 INFO [manager] launching NextWorker(module='nowcast.workers.make_runoff_file', args=['v202108'], host='localhost')
2025-06-08 08:56:05,216 INFO [manager] launching NextWorker(module='nowcast.workers.make_201702_runoff_file', args=[], host='localhost')
2025-06-08 08:56:06,425 INFO [make_201702_runoff_file] running in process 750543
2025-06-08 08:56:06,428 INFO [make_201702_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-08 08:56:06,429 INFO [make_201702_runoff_file] publishing log messages to tcp://*:5564
2025-06-08 08:56:06,430 INFO [make_201702_runoff_file] connected to skookum.eos.ubc.ca port 5555
2025-06-08 08:56:06,456 INFO [make_201702_runoff_file] calculating runoff forcing for b201702 bathymetry
2025-06-08 08:56:06,535 INFO [make_201702_runoff_file] 2025-06-08 runoff file creation from Fraser at Hope and climatology elsewhere complete
2025-06-08 08:56:06,538 INFO [manager] checklist updated with [rivers forcing] items from make_201702_runoff_file worker
2025-06-08 08:56:07,064 INFO [make_runoff_file] running in process 750541
2025-06-08 08:56:07,065 INFO [make_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-08 08:56:07,066 INFO [make_runoff_file] publishing log messages to tcp://*:5564
2025-06-08 08:56:07,066 INFO [make_runoff_file] connected to skookum.eos.ubc.ca port 5555
2025-06-08 08:56:07,068 INFO [make_runoff_file] calculating NEMO runoff forcing for v202108 bathymetry for 2025-06-07
2025-06-08 08:56:07,357 ERROR [make_runoff_file] no 2025-06-07 discharge obs for SalmonSayward: patching
2025-06-08 08:56:08,136 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/hrdps_y2025m06d08.nc
2025-06-08 08:56:09,798 INFO [make_runoff_file] stored NEMO runoff forcing for v202108 bathymetry for 2025-06-07: /results/forcing/rivers/R202108Dailies_y2025m06d07.nc
2025-06-08 08:56:09,799 INFO [make_runoff_file] 2025-06-07 runoff file creation completed
2025-06-08 08:56:09,801 INFO [manager] checklist updated with [rivers forcing] items from make_runoff_file worker
2025-06-08 08:56:15,473 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2025m06d09.nc
2025-06-08 08:56:19,509 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2025m06d10.nc
2025-06-08 08:56:19,523 INFO [grib_to_netcdf] 2025-06-08 NEMO-atmos forcing file for nowcast+ created
2025-06-08 08:56:19,525 INFO [manager] checklist updated with [weather forcing] items from grib_to_netcdf worker
2025-06-08 08:56:43,664 INFO [make_live_ocean_files] Stored T&S western boundary conditions file: /results/forcing/LiveOcean/boundary_conditions/LiveOcean_v201905_y2025m06d08.nc
2025-06-08 08:56:43,665 INFO [make_live_ocean_files] 2025-06-08 Live Ocean western boundary conditions files created
2025-06-08 08:56:43,667 INFO [manager] checklist updated with [Live Ocean boundary conditions] items from make_live_ocean_files worker
2025-06-08 08:56:43,829 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['weather'], host='localhost')
2025-06-08 08:56:43,833 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'nowcast+'], host='localhost')
2025-06-08 08:56:43,839 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'nowcast+'], host='localhost')
2025-06-08 08:56:43,844 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.graham', 'nowcast+'], host='localhost')
2025-06-08 08:56:43,849 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'nowcast+'], host='localhost')
2025-06-08 08:56:44,832 INFO [ping_erddap] running in process 750596
2025-06-08 08:56:44,833 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-08 08:56:44,834 INFO [ping_erddap] publishing log messages to tcp://*:5563
2025-06-08 08:56:44,835 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2025-06-08 08:56:44,836 INFO [ping_erddap] weather ERDDAP dataset flag file(s) created
2025-06-08 08:56:44,838 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2025-06-08 08:56:44,917 INFO [upload_forcing] running in process 750601
2025-06-08 08:56:44,919 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-08 08:56:44,920 INFO [upload_forcing] publishing log messages to tcp://*:5564
2025-06-08 08:56:44,921 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-06-08 08:56:44,948 INFO [upload_forcing] running in process 750598
2025-06-08 08:56:44,949 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-08 08:56:44,950 INFO [upload_forcing] publishing log messages to tcp://*:5565
2025-06-08 08:56:44,951 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-06-08 08:56:44,956 INFO [upload_forcing] running in process 750600
2025-06-08 08:56:44,957 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-08 08:56:44,958 INFO [upload_forcing] publishing log messages to tcp://*:5566
2025-06-08 08:56:44,959 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-06-08 08:56:45,007 INFO [upload_forcing] running in process 750599
2025-06-08 08:56:45,008 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-08 08:56:45,009 INFO [upload_forcing] publishing log messages to tcp://*:5567
2025-06-08 08:56:45,010 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-06-08 08:56:47,261 INFO [upload_forcing] nowcast+ 2025-06-08 forcing files upload to optimum-hindcast completed
2025-06-08 08:56:47,265 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-06-08 08:56:47,370 INFO [upload_forcing] nowcast+ 2025-06-08 forcing files upload to orcinus-nowcast-agrif completed
2025-06-08 08:56:47,454 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-06-08 08:56:48,732 INFO [upload_forcing] nowcast+ 2025-06-08 forcing files upload to arbutus.cloud-nowcast completed
2025-06-08 08:56:48,736 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-06-08 08:56:48,897 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'nowcast+', '--run-date', '2025-06-08'], host='localhost')
2025-06-08 08:56:49,947 INFO [make_forcing_links] running in process 750635
2025-06-08 08:56:49,948 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-08 08:56:49,949 INFO [make_forcing_links] publishing log messages to tcp://*:5563
2025-06-08 08:56:49,950 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2025-06-08 08:56:51,170 INFO [make_forcing_links] nowcast+ 2025-06-08 forcing file links on arbutus.cloud-nowcast created
2025-06-08 08:56:51,174 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2025-06-08 08:56:51,333 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2025-06-08'], host='arbutus.cloud-nowcast')
2025-06-08 08:56:53,385 INFO [run_NEMO] running in process 8884
2025-06-08 08:56:53,386 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-06-08 08:56:53,387 INFO [run_NEMO] publishing log messages to tcp://*:5556
2025-06-08 08:56:53,405 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2025-06-08 08:56:54,093 WARNING [run_NEMO] There are uncommitted changes in /nemoShare/MEOPAR/nowcast-sys/SS-run-sets
2025-06-08 08:56:54,401 INFO [run_NEMO] nowcast: launching /nemoShare/MEOPAR/nowcast-sys/runs/08jun25nowcast_2025-06-08T085653.435373-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2025-06-08 08:56:54,414 INFO [run_NEMO] nowcast NEMO run for 2025-06-08 on arbutus.cloud-nowcast started
2025-06-08 08:56:54,432 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2025-06-08 08:56:54,598 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'nowcast'], host='arbutus.cloud-nowcast')
2025-06-08 08:56:55,270 INFO [upload_forcing] nowcast+ 2025-06-08 forcing files upload to robot.graham completed
2025-06-08 08:56:55,274 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-06-08 08:56:56,587 INFO [watch_NEMO] running in process 9159
2025-06-08 08:56:56,588 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-06-08 08:56:56,589 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2025-06-08 08:56:56,590 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2025-06-08 08:56:56,747 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2025-06-08 09:01:56,756 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 18494370 = 2025-06-08 04:59:20 UTC, 20.8% complete
2025-06-08 09:06:56,757 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 18494843 = 2025-06-08 10:14:40 UTC, 42.7% complete
2025-06-08 09:11:56,759 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 18495343 = 2025-06-08 15:48:00 UTC, 65.9% complete
2025-06-08 09:16:56,762 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 18495835 = 2025-06-08 21:16:00 UTC, 88.7% complete
2025-06-08 09:21:56,771 INFO [watch_NEMO] nowcast NEMO run on arbutus.cloud-nowcast completed
2025-06-08 09:21:56,774 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2025-06-08 09:21:57,106 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'ssh', '--run-date', '2025-06-08'], host='localhost')
2025-06-08 09:21:57,112 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2025-06-08'], host='localhost')
2025-06-08 09:21:58,255 INFO [make_forcing_links] running in process 750891
2025-06-08 09:21:58,256 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-08 09:21:58,257 INFO [make_forcing_links] publishing log messages to tcp://*:5563
2025-06-08 09:21:58,258 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2025-06-08 09:21:58,260 INFO [download_results] running in process 750892
2025-06-08 09:21:58,262 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-08 09:21:58,262 INFO [download_results] publishing log messages to tcp://*:5564
2025-06-08 09:21:58,263 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2025-06-08 09:21:58,264 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/nowcast/08jun25 to /results/SalishSea/nowcast-blue.202111
2025-06-08 09:21:59,259 INFO [make_forcing_links] ssh 2025-06-08 forcing file links on arbutus.cloud-nowcast created
2025-06-08 09:21:59,262 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2025-06-08 09:21:59,408 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2025-06-08'], host='arbutus.cloud-nowcast')
2025-06-08 09:22:01,570 INFO [run_NEMO] running in process 9731
2025-06-08 09:22:01,571 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-06-08 09:22:01,571 INFO [run_NEMO] publishing log messages to tcp://*:5556
2025-06-08 09:22:01,572 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2025-06-08 09:22:02,362 WARNING [run_NEMO] There are uncommitted changes in /nemoShare/MEOPAR/nowcast-sys/SS-run-sets
2025-06-08 09:22:02,683 INFO [run_NEMO] forecast: launching /nemoShare/MEOPAR/nowcast-sys/runs/08jun25forecast_2025-06-08T092201.688852-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2025-06-08 09:22:02,689 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2025-06-08 09:22:02,707 INFO [run_NEMO] forecast NEMO run for 2025-06-08 on arbutus.cloud-nowcast started
2025-06-08 09:22:02,868 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'forecast'], host='arbutus.cloud-nowcast')
2025-06-08 09:23:04,736 INFO [download_results] nowcast 2025-06-08 results files from arbutus.cloud-nowcast downloaded
2025-06-08 09:23:04,740 INFO [manager] checklist updated with [results files] items from download_results worker
2025-06-08 09:23:04,904 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast', 'research', '--run-date', '2025-06-08'], host='localhost')
2025-06-08 09:23:04,908 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast', 'comparison', '--run-date', '2025-06-07'], host='localhost')
2025-06-08 09:23:04,913 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['nowcast', '--run-date', '2025-06-08'], host='localhost')
2025-06-08 09:23:06,866 INFO [make_CHS_currents_file] running in process 750916
2025-06-08 09:23:06,867 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-08 09:23:06,868 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5563
2025-06-08 09:23:06,869 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555
2025-06-08 09:23:07,712 INFO [make_plots] running in process 750915
2025-06-08 09:23:07,713 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-08 09:23:07,714 INFO [make_plots] publishing log messages to tcp://*:5564
2025-06-08 09:23:07,715 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2025-06-08 09:23:07,724 INFO [make_plots] running in process 750913
2025-06-08 09:23:07,747 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-08 09:23:07,756 INFO [make_plots] publishing log messages to tcp://*:5565
2025-06-08 09:23:07,762 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2025-06-08 09:23:08,087 INFO [make_plots] preparing render list for 2025-06-07 NEMO nowcast-blue comparison figures
2025-06-08 09:23:08,088 INFO [make_plots] preparing render list for 2025-06-08 NEMO nowcast-blue research figures
2025-06-08 09:23:08,257 INFO [make_plots] added SH_wind figure to 2025-06-07 NEMO nowcast-blue comparison render list
2025-06-08 09:23:08,258 INFO [make_plots] added Compare_VENUS_East figure to 2025-06-07 NEMO nowcast-blue comparison render list
2025-06-08 09:23:08,259 INFO [make_plots] added Compare_VENUS_Central figure to 2025-06-07 NEMO nowcast-blue comparison render list
2025-06-08 09:23:08,259 INFO [make_plots] starting to render nemo nowcast comparison 07jun25 figures
2025-06-08 09:23:08,674 ERROR [make_plots] unexpected KeyError in make_figure:
Traceback (most recent call last):
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/pandas/core/indexes/datetimes.py", line 610, in get_loc
    return self._partial_date_slice(reso, parsed)
           ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/pandas/core/indexes/datetimelike.py", line 333, in _partial_date_slice
    raise KeyError
KeyError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/xarray/core/indexes.py", line 789, in sel
    indexer = self.index.get_loc(label_value)
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/pandas/core/indexes/datetimes.py", line 612, in get_loc
    raise KeyError(key) from err
KeyError: '2025-06-07'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 833, in _calc_figure
    fig = fig_func(*args, **kwargs)
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/comparison/sandheads_winds.py", line 76, in make_figure
    plot_data = _prep_plot_data(hrdps_dataset_url, run_type, run_date)
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/comparison/sandheads_winds.py", line 92, in _prep_plot_data
    u_hrdps = hrdps.u_wind.sel(time=run_date.format("YYYY-MM-DD")).isel(
              ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/xarray/core/dataarray.py", line 1681, in sel
    ds = self._to_temp_dataset().sel(
        indexers=indexers,
    ...<3 lines>...
        **indexers_kwargs,
    )
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/xarray/core/dataset.py", line 3242, in sel
    query_results = map_index_queries(
        self, indexers=indexers, method=method, tolerance=tolerance
    )
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/xarray/core/indexing.py", line 195, in map_index_queries
    results.append(index.sel(labels, **options))
                   ~~~~~~~~~^^^^^^^^^^^^^^^^^^^
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/xarray/core/indexes.py", line 791, in sel
    raise KeyError(
    ...<2 lines>...
    ) from e
KeyError: "not all values found in index 'time'. Try setting the `method` keyword argument (example: method='nearest')."
2025-06-08 09:23:09,970 INFO [make_plots] added salinity_thalweg_and_surface figures to 2025-06-08 NEMO nowcast-blue research render list
2025-06-08 09:23:10,809 INFO [make_plots] added temperature_thalweg_and_surface figures to 2025-06-08 NEMO nowcast-blue research render list
2025-06-08 09:23:10,810 INFO [make_plots] added Currents_sections_and_surface figure to 2025-06-08 NEMO nowcast-blue research render list
2025-06-08 09:23:10,810 INFO [make_plots] added Currents_at_VENUS_Central figure to 2025-06-08 NEMO nowcast-blue research render list
2025-06-08 09:23:10,811 INFO [make_plots] added Currents_at_VENUS_East figure to 2025-06-08 NEMO nowcast-blue research render list
2025-06-08 09:23:10,812 INFO [make_plots] starting to render nemo nowcast research 08jun25 figures
2025-06-08 09:23:39,059 INFO [make_CHS_currents_file] Made CHS currents file for 2025-06-08 for nowcast
2025-06-08 09:23:39,063 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker
2025-06-08 09:24:05,785 INFO [make_plots] finished rendering nemo nowcast comparison 07jun25 figures
2025-06-08 09:24:05,785 INFO [make_plots] nemo comparison plots for 2025-06-07 nowcast completed
2025-06-08 09:24:05,790 INFO [manager] checklist updated with [plots] items from make_plots worker
2025-06-08 09:24:17,510 INFO [make_plots] finished rendering nemo nowcast research 08jun25 figures
2025-06-08 09:24:17,512 INFO [make_plots] nemo research plots for 2025-06-08 nowcast completed
2025-06-08 09:24:17,532 INFO [manager] checklist updated with [plots] items from make_plots worker
2025-06-08 09:27:05,020 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 18496538 = 2025-06-09 05:04:40 UTC, 14.1% complete
2025-06-08 09:32:05,023 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 18497053 = 2025-06-09 10:48:00 UTC, 30.0% complete
2025-06-08 09:37:05,074 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 18497556 = 2025-06-09 16:23:20 UTC, 45.5% complete
2025-06-08 09:42:05,027 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 18498050 = 2025-06-09 21:52:40 UTC, 60.8% complete
2025-06-08 09:47:05,023 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 18498557 = 2025-06-10 03:30:40 UTC, 76.4% complete
2025-06-08 09:52:05,019 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 18499053 = 2025-06-10 09:01:20 UTC, 91.8% complete
2025-06-08 09:57:05,017 INFO [watch_NEMO] forecast NEMO run on arbutus.cloud-nowcast completed
2025-06-08 09:57:05,021 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2025-06-08 09:57:05,339 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_wind_file', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2025-06-08'], host='arbutus.cloud-nowcast')
2025-06-08 09:57:05,348 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_current_file', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2025-06-08'], host='arbutus.cloud-nowcast')
2025-06-08 09:57:05,353 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'turbidity', '--run-date', '2025-06-08'], host='localhost')
2025-06-08 09:57:05,359 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'turbidity', '--run-date', '2025-06-08'], host='localhost')
2025-06-08 09:57:05,364 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.graham', 'turbidity', '--run-date', '2025-06-08'], host='localhost')
2025-06-08 09:57:05,370 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'turbidity', '--run-date', '2025-06-08'], host='localhost')
2025-06-08 09:57:05,375 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2025-06-08'], host='localhost')
2025-06-08 09:57:06,541 INFO [download_results] running in process 751296
2025-06-08 09:57:06,542 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-08 09:57:06,543 INFO [download_results] publishing log messages to tcp://*:5563
2025-06-08 09:57:06,544 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2025-06-08 09:57:06,545 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/forecast/08jun25 to /results/SalishSea/forecast.202111
2025-06-08 09:57:06,575 INFO [upload_forcing] running in process 751293
2025-06-08 09:57:06,576 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-08 09:57:06,577 INFO [upload_forcing] publishing log messages to tcp://*:5564
2025-06-08 09:57:06,580 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-06-08 09:57:06,581 INFO [upload_forcing] running in process 751294
2025-06-08 09:57:06,582 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-08 09:57:06,583 INFO [upload_forcing] publishing log messages to tcp://*:5565
2025-06-08 09:57:06,583 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-06-08 09:57:06,586 INFO [upload_forcing] running in process 751295
2025-06-08 09:57:06,587 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-08 09:57:06,588 INFO [upload_forcing] publishing log messages to tcp://*:5566
2025-06-08 09:57:06,589 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-06-08 09:57:06,590 INFO [upload_forcing] running in process 751292
2025-06-08 09:57:06,590 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-08 09:57:06,591 INFO [upload_forcing] publishing log messages to tcp://*:5567
2025-06-08 09:57:06,592 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-06-08 09:57:07,227 INFO [upload_forcing] turbidity 2025-06-08 forcing files upload to orcinus-nowcast-agrif completed
2025-06-08 09:57:07,232 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-06-08 09:57:07,289 INFO [upload_forcing] turbidity 2025-06-08 forcing files upload to optimum-hindcast completed
2025-06-08 09:57:07,397 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['orcinus-nowcast-agrif', 'nowcast-agrif', '--run-date', '2025-06-08'], host='localhost')
2025-06-08 09:57:07,403 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-06-08 09:57:08,213 INFO [upload_forcing] turbidity 2025-06-08 forcing files upload to arbutus.cloud-nowcast completed
2025-06-08 09:57:08,217 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-06-08 09:57:08,384 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2025-06-08'], host='localhost')
2025-06-08 09:57:08,403 INFO [make_forcing_links] running in process 751326
2025-06-08 09:57:08,404 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-08 09:57:08,405 INFO [make_forcing_links] publishing log messages to tcp://*:5564
2025-06-08 09:57:08,406 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2025-06-08 09:57:08,424 INFO [make_ww3_wind_file] running in process 10725
2025-06-08 09:57:08,425 INFO [make_ww3_wind_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-06-08 09:57:08,426 INFO [make_ww3_wind_file] publishing log messages to tcp://*:5570
2025-06-08 09:57:08,432 INFO [make_ww3_wind_file] connected to skookum.eos.ubc.ca port 5555
2025-06-08 09:57:08,433 INFO [make_ww3_wind_file] Creating wwatch3 wind forcing file for 2025-06-08 forecast run
2025-06-08 09:57:08,462 INFO [make_ww3_current_file] running in process 10723
2025-06-08 09:57:08,463 INFO [make_ww3_current_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-06-08 09:57:08,463 INFO [make_ww3_current_file] publishing log messages to tcp://*:5571
2025-06-08 09:57:08,465 INFO [make_ww3_current_file] connected to skookum.eos.ubc.ca port 5555
2025-06-08 09:57:08,466 INFO [make_ww3_current_file] Creating wwatch3 currents forcing file for 2025-06-08 forecast run
2025-06-08 09:57:09,143 INFO [make_forcing_links] nowcast-agrif 2025-06-08 forcing file links on orcinus-nowcast-agrif created
2025-06-08 09:57:09,147 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2025-06-08 09:57:09,318 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_agrif', args=['orcinus-nowcast-agrif', 'nowcast-agrif', '--run-date', '2025-06-08'], host='localhost')
2025-06-08 09:57:09,422 INFO [make_forcing_links] running in process 751332
2025-06-08 09:57:09,424 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-08 09:57:09,425 INFO [make_forcing_links] publishing log messages to tcp://*:5566
2025-06-08 09:57:09,425 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2025-06-08 09:57:10,357 INFO [run_NEMO_agrif] running in process 751341
2025-06-08 09:57:10,358 INFO [run_NEMO_agrif] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-08 09:57:10,359 INFO [run_NEMO_agrif] publishing log messages to tcp://*:5564
2025-06-08 09:57:10,360 INFO [run_NEMO_agrif] connected to skookum.eos.ubc.ca port 5555
2025-06-08 09:57:11,122 INFO [upload_forcing] turbidity 2025-06-08 forcing files upload to robot.graham completed
2025-06-08 09:57:11,127 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-06-08 09:57:11,314 INFO [make_forcing_links] nowcast-green 2025-06-08 forcing file links on arbutus.cloud-nowcast created
2025-06-08 09:57:11,318 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2025-06-08 09:57:11,489 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2025-06-08'], host='arbutus.cloud-nowcast')
2025-06-08 09:57:17,764 WARNING [run_NEMO] There are uncommitted changes in /nemoShare/MEOPAR/nowcast-sys/SS-run-sets
2025-06-08 09:57:18,572 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2025-06-08 09:57:18,593 INFO [run_NEMO] nowcast-green: launching /nemoShare/MEOPAR/nowcast-sys/runs/08jun25nowcast-green_2025-06-08T095715.465022-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2025-06-08 09:57:18,596 INFO [run_NEMO] nowcast-green NEMO run for 2025-06-08 on arbutus.cloud-nowcast started
2025-06-08 09:57:18,735 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'nowcast-green'], host='arbutus.cloud-nowcast')
2025-06-08 09:57:19,898 INFO [make_ww3_wind_file] wwatch3 wind forcing file created on arbutus.cloud-nowcast for 2025-06-08 forecast run
2025-06-08 09:57:19,917 INFO [manager] checklist updated with [WW3 wind forcing] items from make_ww3_wind_file worker
2025-06-08 09:57:20,836 INFO [watch_NEMO] running in process 11634
2025-06-08 09:57:20,836 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-06-08 09:57:20,839 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2025-06-08 09:57:20,840 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2025-06-08 09:57:20,935 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2025-06-08 09:57:25,219 INFO [run_NEMO_agrif] job id for 08jun25nowcast-agrif: 11429482.orca2.ibb
2025-06-08 09:57:25,219 INFO [run_NEMO_agrif] NEMO nowcast-agrif run for 2025-06-08 queued on orcinus-nowcast-agrif
2025-06-08 09:57:25,224 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_agrif worker
2025-06-08 09:57:25,396 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_agrif', args=['orcinus-nowcast-agrif', '11429482.orca2.ibb'], host='localhost')
2025-06-08 09:57:26,517 INFO [watch_NEMO_agrif] running in process 751363
2025-06-08 09:57:26,518 INFO [watch_NEMO_agrif] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-08 09:57:26,521 INFO [watch_NEMO_agrif] publishing log messages to tcp://*:5564
2025-06-08 09:57:26,522 INFO [watch_NEMO_agrif] connected to skookum.eos.ubc.ca port 5555
2025-06-08 09:57:27,384 INFO [watch_NEMO_agrif] watching 08jun25nowcast-agrif job 11429482 on orcinus-nowcast-agrif
2025-06-08 09:57:27,620 INFO [watch_NEMO_agrif] 08jun25nowcast-agrif job 11429482 is queued on orcinus-nowcast-agrif
2025-06-08 09:58:27,829 INFO [watch_NEMO_agrif] 08jun25nowcast-agrif job 11429482 is queued on orcinus-nowcast-agrif
2025-06-08 09:59:28,039 INFO [watch_NEMO_agrif] 08jun25nowcast-agrif job 11429482 is queued on orcinus-nowcast-agrif
2025-06-08 09:59:47,588 INFO [make_ww3_current_file] wwatch3 currents forcing file created on arbutus.cloud-nowcast for 2025-06-08 forecast run
2025-06-08 09:59:47,598 INFO [manager] checklist updated with [WW3 currents forcing] items from make_ww3_current_file worker
2025-06-08 09:59:47,773 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2025-06-08'], host='arbutus.cloud-nowcast')
2025-06-08 09:59:50,209 INFO [run_ww3] running in process 11736
2025-06-08 09:59:50,210 INFO [run_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-06-08 09:59:50,211 INFO [run_ww3] publishing log messages to tcp://*:5572
2025-06-08 09:59:50,212 INFO [run_ww3] connected to skookum.eos.ubc.ca port 5555
2025-06-08 09:59:50,225 INFO [run_ww3] Created run directory /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/nowcast_2025-06-08T095950.203984-0700
2025-06-08 09:59:50,226 INFO [run_ww3] nowcast: launching /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/nowcast_2025-06-08T095950.203984-0700/SoGWW3.sh on arbutus.cloud-nowcast
2025-06-08 09:59:50,232 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker
2025-06-08 09:59:50,233 INFO [run_ww3] nowcast WaveWatch3 run for 2025-06-08 on arbutus.cloud-nowcast started
2025-06-08 09:59:50,406 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'nowcast'], host='arbutus.cloud-nowcast')
2025-06-08 09:59:52,552 INFO [watch_ww3] running in process 11823
2025-06-08 09:59:52,553 INFO [watch_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-06-08 09:59:52,554 INFO [watch_ww3] publishing log messages to tcp://*:5573
2025-06-08 09:59:52,555 INFO [watch_ww3] connected to skookum.eos.ubc.ca port 5555
2025-06-08 09:59:52,588 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2025-06-08 10:00:28,957 INFO [watch_NEMO_agrif] 08jun25nowcast-agrif on orcinus-nowcast-agrif: time.step not found; continuing to watch...
2025-06-08 10:02:20,953 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18494078 = 2025-06-08 01:44:40 UTC, 7.3% complete
2025-06-08 10:04:52,587 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 84, 19.4% complete
2025-06-08 10:05:29,434 INFO [watch_NEMO_agrif] 08jun25nowcast-agrif on orcinus-nowcast-agrif: timestep: 8514898 = 2025-06-08 01:58:00 UTC, 8.2% complete
2025-06-08 10:07:20,933 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18494256 = 2025-06-08 03:43:20 UTC, 15.5% complete
2025-06-08 10:09:52,594 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 174, 40.3% complete
2025-06-08 10:09:58,763 INFO [download_results] forecast 2025-06-08 results files from arbutus.cloud-nowcast downloaded
2025-06-08 10:09:58,768 INFO [manager] checklist updated with [results files] items from download_results worker
2025-06-08 10:09:58,947 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['forecast', '--run-date', '2025-06-08'], host='localhost')
2025-06-08 10:10:00,754 INFO [make_CHS_currents_file] running in process 751518
2025-06-08 10:10:00,755 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-08 10:10:00,756 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5563
2025-06-08 10:10:00,756 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555
2025-06-08 10:10:29,927 INFO [watch_NEMO_agrif] 08jun25nowcast-agrif on orcinus-nowcast-agrif: timestep: 8515101 = 2025-06-08 04:13:20 UTC, 17.6% complete
2025-06-08 10:10:46,720 INFO [make_CHS_currents_file] Made CHS currents file for 2025-06-08 for forecast
2025-06-08 10:10:46,723 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker
2025-06-08 10:10:46,890 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['nemo', 'forecast', '--run-date', '2025-06-08'], host='localhost')
2025-06-08 10:10:48,205 INFO [update_forecast_datasets] running in process 751542
2025-06-08 10:10:48,206 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-08 10:10:48,207 INFO [update_forecast_datasets] publishing log messages to tcp://*:5563
2025-06-08 10:10:48,208 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2025-06-08 10:10:48,209 INFO [update_forecast_datasets] updating nemo forecast directory for forecast/08jun25 run
2025-06-08 10:10:48,504 INFO [update_forecast_datasets] updated nemo forecast directory for forecast/08jun25 run: /results/SalishSea/rolling-forecasts/nemo
2025-06-08 10:10:48,504 INFO [update_forecast_datasets] nemo 2025-06-08 forecast rolling forecast datasets updated
2025-06-08 10:10:48,507 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2025-06-08 10:10:48,674 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nemo-forecast'], host='localhost')
2025-06-08 10:10:48,678 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'forecast', 'publish', '--run-date', '2025-06-08'], host='localhost')
2025-06-08 10:10:48,683 INFO [manager] launching NextWorker(module='nowcast.workers.make_surface_current_tiles', args=['forecast', '--run-date', '2025-06-08'], host='localhost')
2025-06-08 10:10:49,751 INFO [ping_erddap] running in process 751547
2025-06-08 10:10:49,753 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-08 10:10:49,754 INFO [ping_erddap] publishing log messages to tcp://*:5563
2025-06-08 10:10:49,755 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2025-06-08 10:10:49,769 INFO [ping_erddap] nemo-forecast ERDDAP dataset flag file(s) created
2025-06-08 10:10:49,771 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2025-06-08 10:10:51,208 INFO [make_surface_current_tiles] running in process 751550
2025-06-08 10:10:51,209 INFO [make_surface_current_tiles] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-08 10:10:51,210 INFO [make_surface_current_tiles] publishing log messages to tcp://*:5563
2025-06-08 10:10:51,211 INFO [make_surface_current_tiles] connected to skookum.eos.ubc.ca port 5555
2025-06-08 10:10:51,213 INFO [make_surface_current_tiles] starting rendering of tiles for 2025-06-08 forecast in 6 processes into /results/nowcast-sys/figures/surface_currents/forecast/08jun25
2025-06-08 10:10:51,839 INFO [make_plots] running in process 751549
2025-06-08 10:10:51,842 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-08 10:10:51,843 INFO [make_plots] publishing log messages to tcp://*:5565
2025-06-08 10:10:51,843 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2025-06-08 10:10:52,210 INFO [make_plots] preparing render list for 2025-06-08 NEMO forecast publish figures
2025-06-08 10:10:52,313 INFO [make_plots] added Website_thumbnail figure to 2025-06-08 NEMO forecast publish render list
2025-06-08 10:10:52,313 INFO [make_plots] added Threshold_website figure to 2025-06-08 NEMO forecast publish render list
2025-06-08 10:10:52,314 INFO [make_plots] added PA_tidal_predictions figure to 2025-06-08 NEMO forecast publish render list
2025-06-08 10:10:52,315 INFO [make_plots] added Boundary Bay figure to 2025-06-08 NEMO forecast publish render list
2025-06-08 10:10:52,315 INFO [make_plots] added Campbell River figure to 2025-06-08 NEMO forecast publish render list
2025-06-08 10:10:52,316 INFO [make_plots] added Cherry Point figure to 2025-06-08 NEMO forecast publish render list
2025-06-08 10:10:52,316 INFO [make_plots] added Friday Harbor figure to 2025-06-08 NEMO forecast publish render list
2025-06-08 10:10:52,317 INFO [make_plots] added Halfmoon Bay figure to 2025-06-08 NEMO forecast publish render list
2025-06-08 10:10:52,318 INFO [make_plots] added Nanaimo figure to 2025-06-08 NEMO forecast publish render list
2025-06-08 10:10:52,318 INFO [make_plots] added Neah Bay figure to 2025-06-08 NEMO forecast publish render list
2025-06-08 10:10:52,319 INFO [make_plots] added New Westminster figure to 2025-06-08 NEMO forecast publish render list
2025-06-08 10:10:52,320 INFO [make_plots] added Patricia Bay figure to 2025-06-08 NEMO forecast publish render list
2025-06-08 10:10:52,321 INFO [make_plots] added Point Atkinson figure to 2025-06-08 NEMO forecast publish render list
2025-06-08 10:10:52,322 INFO [make_plots] added Port Renfrew figure to 2025-06-08 NEMO forecast publish render list
2025-06-08 10:10:52,323 INFO [make_plots] added Sand Heads figure to 2025-06-08 NEMO forecast publish render list
2025-06-08 10:10:52,324 INFO [make_plots] added Sandy Cove figure to 2025-06-08 NEMO forecast publish render list
2025-06-08 10:10:52,326 INFO [make_plots] added Squamish figure to 2025-06-08 NEMO forecast publish render list
2025-06-08 10:10:52,336 INFO [make_plots] added Victoria figure to 2025-06-08 NEMO forecast publish render list
2025-06-08 10:10:52,345 INFO [make_plots] added Woodwards Landing figure to 2025-06-08 NEMO forecast publish render list
2025-06-08 10:10:52,346 INFO [make_plots] starting to render nemo forecast publish 08jun25 figures
2025-06-08 10:12:20,933 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18494433 = 2025-06-08 05:41:20 UTC, 23.7% complete
2025-06-08 10:14:52,605 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 270, 62.5% complete
2025-06-08 10:15:30,440 INFO [watch_NEMO_agrif] 08jun25nowcast-agrif on orcinus-nowcast-agrif: timestep: 8515295 = 2025-06-08 06:22:40 UTC, 26.6% complete
2025-06-08 10:17:20,937 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18494611 = 2025-06-08 07:40:00 UTC, 32.0% complete
2025-06-08 10:18:20,548 WARNING [make_plots] invalid station number for Boundary Bay station: None
2025-06-08 10:18:20,549 WARNING [make_plots] can't resolve a valid CHS station code for Boundary Bay
2025-06-08 10:18:42,163 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb996/data for station 08074 Campbell River from 2025-06-03 00:00:00Z to 2025-06-10 11:55:00Z
2025-06-08 10:19:42,489 WARNING [make_plots] invalid station number for Halfmoon Bay station: None
2025-06-08 10:19:42,490 WARNING [make_plots] can't resolve a valid CHS station code for Halfmoon Bay
2025-06-08 10:19:52,591 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 363, 84.0% complete
2025-06-08 10:20:04,117 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb96d/data for station 07917 Nanaimo from 2025-06-03 00:00:00Z to 2025-06-10 11:55:00Z
2025-06-08 10:20:30,927 INFO [watch_NEMO_agrif] 08jun25nowcast-agrif on orcinus-nowcast-agrif: timestep: 8515480 = 2025-06-08 08:26:00 UTC, 35.2% complete
2025-06-08 10:20:45,366 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e13d0f4a073c4bbf93/data for station 07654 New Westminster from 2025-06-03 00:00:00Z to 2025-06-10 11:55:00Z
2025-06-08 10:21:06,128 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd2d/data for station 07277 Patricia Bay from 2025-06-03 00:00:00Z to 2025-06-10 11:55:00Z
2025-06-08 10:21:23,061 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94c/data for station 07795 Point Atkinson from 2025-06-03 00:00:00Z to 2025-06-10 11:55:00Z
2025-06-08 10:21:27,818 INFO [make_surface_current_tiles] starting PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast/08jun25
2025-06-08 10:21:39,642 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e23d0f4a073c4bc060/data for station 08525 Port Renfrew from 2025-06-03 00:00:00Z to 2025-06-10 11:55:00Z
2025-06-08 10:21:55,517 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb93c/data for station 07594 Sand Heads from 2025-06-03 00:00:00Z to 2025-06-10 11:55:00Z
2025-06-08 10:22:12,606 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc43b/data for station 07786 Sandy Cove from 2025-06-03 00:00:00Z to 2025-06-10 11:55:00Z
2025-06-08 10:22:20,938 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18494789 = 2025-06-08 09:38:40 UTC, 40.2% complete
2025-06-08 10:22:28,735 WARNING [make_plots] invalid station number for Squamish station: None
2025-06-08 10:22:28,736 WARNING [make_plots] can't resolve a valid CHS station code for Squamish
2025-06-08 10:22:44,367 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd1e/data for station 07120 Victoria from 2025-06-03 00:00:00Z to 2025-06-10 11:55:00Z
2025-06-08 10:23:00,320 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc42d/data for station 07610 Woodwards Landing from 2025-06-03 00:00:00Z to 2025-06-10 11:55:00Z
2025-06-08 10:23:16,447 INFO [make_plots] finished rendering nemo forecast publish 08jun25 figures
2025-06-08 10:23:16,450 INFO [make_plots] nemo publish plots for 2025-06-08 forecast completed
2025-06-08 10:23:16,458 INFO [manager] checklist updated with [plots] items from make_plots worker
2025-06-08 10:23:16,631 INFO [manager] launching NextWorker(module='nowcast.workers.make_feeds', args=['forecast', '--run-date', '2025-06-08'], host='localhost')
2025-06-08 10:23:19,477 INFO [make_feeds] running in process 753475
2025-06-08 10:23:19,479 INFO [make_feeds] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-08 10:23:19,480 INFO [make_feeds] publishing log messages to tcp://*:5565
2025-06-08 10:23:19,481 INFO [make_feeds] connected to skookum.eos.ubc.ca port 5555
2025-06-08 10:23:34,799 INFO [make_feeds] ATOM feeds for 2025-06-08 forecast run completed
2025-06-08 10:23:34,801 INFO [manager] checklist updated with [storm surge alert feeds] items from make_feeds worker
2025-06-08 10:24:52,591 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2025-06-08 10:25:33,962 INFO [watch_NEMO_agrif] 08jun25nowcast-agrif on orcinus-nowcast-agrif: timestep: 8515659 = 2025-06-08 10:25:20 UTC, 43.4% complete
2025-06-08 10:26:06,228 INFO [make_surface_current_tiles] finished PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast/08jun25
2025-06-08 10:26:06,254 INFO [make_surface_current_tiles] finished rendering of tiles for 2025-06-08 forecast into /results/nowcast-sys/figures/surface_currents/forecast/08jun25
2025-06-08 10:26:06,254 INFO [make_surface_current_tiles] surface current tile figures for 2025-06-08 forecast completed
2025-06-08 10:26:06,574 INFO [manager] checklist updated with [surface current tiles] items from make_surface_current_tiles worker
2025-06-08 10:27:20,938 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18494967 = 2025-06-08 11:37:20 UTC, 48.4% complete
2025-06-08 10:29:52,593 INFO [watch_ww3] nowcast WWATCH3 run on arbutus.cloud-nowcast completed
2025-06-08 10:29:52,596 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2025-06-08 10:29:53,033 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2025-06-08'], host='localhost')
2025-06-08 10:29:53,039 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2025-06-08'], host='arbutus.cloud-nowcast')
2025-06-08 10:29:54,030 INFO [download_wwatch3_results] running in process 753550
2025-06-08 10:29:54,031 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-08 10:29:54,032 INFO [download_wwatch3_results] publishing log messages to tcp://*:5563
2025-06-08 10:29:54,033 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2025-06-08 10:29:55,648 INFO [run_ww3] running in process 12172
2025-06-08 10:29:55,649 INFO [run_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-06-08 10:29:55,650 INFO [run_ww3] publishing log messages to tcp://*:5572
2025-06-08 10:29:55,651 INFO [run_ww3] connected to skookum.eos.ubc.ca port 5555
2025-06-08 10:29:55,653 INFO [run_ww3] Created run directory /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast_2025-06-08T102955.644922-0700
2025-06-08 10:29:55,671 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker
2025-06-08 10:29:55,677 INFO [run_ww3] forecast: launching /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast_2025-06-08T102955.644922-0700/SoGWW3.sh on arbutus.cloud-nowcast
2025-06-08 10:29:55,679 INFO [run_ww3] forecast WaveWatch3 run for 2025-06-08 on arbutus.cloud-nowcast started
2025-06-08 10:29:56,014 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'forecast'], host='arbutus.cloud-nowcast')
2025-06-08 10:29:58,094 INFO [watch_ww3] running in process 12308
2025-06-08 10:29:58,096 INFO [watch_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-06-08 10:29:58,096 INFO [watch_ww3] publishing log messages to tcp://*:5573
2025-06-08 10:29:58,097 INFO [watch_ww3] connected to skookum.eos.ubc.ca port 5555
2025-06-08 10:29:58,137 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2025-06-08 10:30:28,843 INFO [download_wwatch3_results] nowcast 2025-06-08 results files from arbutus.cloud-nowcast downloaded
2025-06-08 10:30:28,847 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2025-06-08 10:30:34,445 INFO [watch_NEMO_agrif] 08jun25nowcast-agrif on orcinus-nowcast-agrif: timestep: 8515854 = 2025-06-08 12:35:20 UTC, 52.5% complete
2025-06-08 10:32:20,939 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18495144 = 2025-06-08 13:35:20 UTC, 56.6% complete
2025-06-08 10:34:58,139 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 84, 13.0% complete
2025-06-08 10:35:35,239 INFO [watch_NEMO_agrif] 08jun25nowcast-agrif on orcinus-nowcast-agrif: timestep: 8516063 = 2025-06-08 14:54:40 UTC, 62.2% complete
2025-06-08 10:37:20,943 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18495322 = 2025-06-08 15:34:00 UTC, 64.9% complete
2025-06-08 10:39:58,139 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 174, 26.9% complete
2025-06-08 10:40:35,726 INFO [watch_NEMO_agrif] 08jun25nowcast-agrif on orcinus-nowcast-agrif: timestep: 8516268 = 2025-06-08 17:11:20 UTC, 71.7% complete
2025-06-08 10:42:20,945 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18495499 = 2025-06-08 17:32:00 UTC, 73.1% complete
2025-06-08 10:44:58,142 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 264, 40.7% complete
2025-06-08 10:45:36,215 INFO [watch_NEMO_agrif] 08jun25nowcast-agrif on orcinus-nowcast-agrif: timestep: 8516475 = 2025-06-08 19:29:20 UTC, 81.2% complete
2025-06-08 10:47:20,947 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18495677 = 2025-06-08 19:30:40 UTC, 81.3% complete
2025-06-08 10:49:58,143 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 354, 54.6% complete
2025-06-08 10:50:36,741 INFO [watch_NEMO_agrif] 08jun25nowcast-agrif on orcinus-nowcast-agrif: timestep: 8516683 = 2025-06-08 21:48:00 UTC, 90.9% complete
2025-06-08 10:52:20,948 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18495854 = 2025-06-08 21:28:40 UTC, 89.5% complete
2025-06-08 10:54:58,144 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 444, 68.5% complete
2025-06-08 10:55:37,238 INFO [watch_NEMO_agrif] 08jun25nowcast-agrif on orcinus-nowcast-agrif: timestep: 8516879 = 2025-06-08 23:58:40 UTC, 100.0% complete
2025-06-08 10:57:20,988 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18496031 = 2025-06-08 23:26:40 UTC, 97.7% complete
2025-06-08 10:59:58,149 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 534, 82.4% complete
2025-06-08 11:00:37,465 INFO [watch_NEMO_agrif] NEMO AGRIF run 11429482.orca2.ibb on orcinus-nowcast-agrif completed
2025-06-08 11:00:37,469 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_agrif worker
2025-06-08 11:00:37,924 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['orcinus-nowcast-agrif', 'nowcast-agrif', '--run-date', '2025-06-08'], host='localhost')
2025-06-08 11:00:39,032 INFO [download_results] running in process 753808
2025-06-08 11:00:39,034 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-08 11:00:39,035 INFO [download_results] publishing log messages to tcp://*:5563
2025-06-08 11:00:39,036 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2025-06-08 11:00:39,037 INFO [download_results] downloading results from orcinus-nowcast-agrif:/global/scratch/dlatorne/nowcast-agrif/08jun25 to /results/SalishSea/nowcast-agrif.201702
2025-06-08 11:01:01,049 INFO [download_results] nowcast-agrif 2025-06-08 results files from orcinus-nowcast-agrif downloaded
2025-06-08 11:01:01,058 INFO [manager] checklist updated with [results files] items from download_results worker
2025-06-08 11:01:01,420 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast-agrif', 'research', '--run-date', '2025-06-08'], host='localhost')
2025-06-08 11:01:04,359 INFO [make_plots] running in process 753816
2025-06-08 11:01:04,360 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-08 11:01:04,361 INFO [make_plots] publishing log messages to tcp://*:5563
2025-06-08 11:01:04,362 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2025-06-08 11:01:04,699 INFO [make_plots] preparing render list for 2025-06-08 NEMO nowcast-agrif research figures
2025-06-08 11:01:04,700 INFO [make_plots] added baynes_sound_surface figure to 2025-06-08 NEMO nowcast-agrif research render list
2025-06-08 11:01:04,700 INFO [make_plots] starting to render nemo nowcast-agrif research 08jun25 figures
2025-06-08 11:01:14,390 INFO [make_plots] finished rendering nemo nowcast-agrif research 08jun25 figures
2025-06-08 11:01:14,392 INFO [make_plots] nemo research plots for 2025-06-08 nowcast-agrif completed
2025-06-08 11:01:14,396 INFO [manager] checklist updated with [plots] items from make_plots worker
2025-06-08 11:02:20,952 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18496080 = 2025-06-08 23:59:20 UTC, 100.0% complete
2025-06-08 11:04:58,156 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 624, 96.3% complete
2025-06-08 11:07:20,958 INFO [watch_NEMO] nowcast-green NEMO run on arbutus.cloud-nowcast completed
2025-06-08 11:07:20,962 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2025-06-08 11:07:21,450 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2025-06-08'], host='localhost')
2025-06-08 11:07:22,523 INFO [download_results] running in process 754095
2025-06-08 11:07:22,524 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-08 11:07:22,525 INFO [download_results] publishing log messages to tcp://*:5563
2025-06-08 11:07:22,526 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2025-06-08 11:07:22,526 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/nowcast-green/08jun25 to /results2/SalishSea/nowcast-green.202111
2025-06-08 11:09:58,153 INFO [watch_ww3] forecast WWATCH3 run on arbutus.cloud-nowcast completed
2025-06-08 11:09:58,191 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2025-06-08 11:09:58,670 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2025-06-08'], host='localhost')
2025-06-08 11:09:59,811 INFO [download_wwatch3_results] running in process 754119
2025-06-08 11:09:59,812 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-08 11:09:59,813 INFO [download_wwatch3_results] publishing log messages to tcp://*:5564
2025-06-08 11:09:59,814 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2025-06-08 11:10:45,952 INFO [download_wwatch3_results] forecast 2025-06-08 results files from arbutus.cloud-nowcast downloaded
2025-06-08 11:10:45,954 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2025-06-08 11:10:46,272 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['wwatch3', 'forecast', '--run-date', '2025-06-08'], host='localhost')
2025-06-08 11:10:47,220 INFO [update_forecast_datasets] running in process 754130
2025-06-08 11:10:47,221 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-08 11:10:47,222 INFO [update_forecast_datasets] publishing log messages to tcp://*:5564
2025-06-08 11:10:47,223 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2025-06-08 11:10:47,224 INFO [update_forecast_datasets] updating wwatch3 most_recent_forecast directory from forecast/08jun25 run
2025-06-08 11:10:47,227 INFO [update_forecast_datasets] updated wwatch3 most_recent_forecast directory from forecast/08jun25 run
2025-06-08 11:10:47,228 INFO [update_forecast_datasets] updating wwatch3 forecast directory for forecast/08jun25 run
2025-06-08 11:10:47,261 INFO [update_forecast_datasets] updated wwatch3 forecast directory for forecast/08jun25 run: /results/SalishSea/rolling-forecasts/wwatch3
2025-06-08 11:10:47,262 INFO [update_forecast_datasets] wwatch3 2025-06-08 forecast rolling forecast datasets updated
2025-06-08 11:10:47,264 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2025-06-08 11:10:47,592 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['wwatch3-forecast'], host='localhost')
2025-06-08 11:10:48,614 INFO [ping_erddap] running in process 754135
2025-06-08 11:10:48,615 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-08 11:10:48,616 INFO [ping_erddap] publishing log messages to tcp://*:5564
2025-06-08 11:10:48,617 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2025-06-08 11:10:48,619 INFO [ping_erddap] wwatch3-forecast ERDDAP dataset flag file(s) created
2025-06-08 11:10:48,621 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2025-06-08 11:10:48,896 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['wwatch3', 'forecast', 'publish', '--run-date', '2025-06-08'], host='localhost')
2025-06-08 11:10:51,760 INFO [make_plots] running in process 754140
2025-06-08 11:10:51,761 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-08 11:10:51,761 INFO [make_plots] publishing log messages to tcp://*:5564
2025-06-08 11:10:51,762 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2025-06-08 11:10:51,763 INFO [make_plots] preparing render list for 2025-06-08 WaveWatch3 forecast publish figures
2025-06-08 11:10:51,764 INFO [make_plots] added Halibut Bank figure to 2025-06-08 WaveWatch3 forecast publish render list
2025-06-08 11:10:51,765 INFO [make_plots] added Sentry Shoal figure to 2025-06-08 WaveWatch3 forecast publish render list
2025-06-08 11:10:51,766 INFO [make_plots] starting to render wwatch3 forecast publish 08jun25 figures
2025-06-08 11:10:52,192 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46146 Halibut Bank
2025-06-08 11:10:58,510 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46131 Sentry Shoal
2025-06-08 11:11:03,869 INFO [make_plots] finished rendering wwatch3 forecast publish 08jun25 figures
2025-06-08 11:11:03,870 INFO [make_plots] wwatch3 publish plots for 2025-06-08 forecast completed
2025-06-08 11:11:03,873 INFO [manager] checklist updated with [plots] items from make_plots worker
2025-06-08 11:17:06,622 INFO [download_results] nowcast-green 2025-06-08 results files from arbutus.cloud-nowcast downloaded
2025-06-08 11:17:06,627 INFO [manager] checklist updated with [results files] items from download_results worker
2025-06-08 11:17:06,970 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast-green', 'research', '--run-date', '2025-06-08'], host='localhost')
2025-06-08 11:17:06,976 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nowcast-green'], host='localhost')
2025-06-08 11:17:06,981 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'chemistry', '--run-date', '2025-06-08'], host='localhost')
2025-06-08 11:17:06,987 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'physics', '--run-date', '2025-06-08'], host='localhost')
2025-06-08 11:17:06,993 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'biology', '--run-date', '2025-06-08'], host='localhost')
2025-06-08 11:17:08,430 INFO [ping_erddap] running in process 754216
2025-06-08 11:17:08,431 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-08 11:17:08,431 INFO [ping_erddap] publishing log messages to tcp://*:5563
2025-06-08 11:17:08,432 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2025-06-08 11:17:08,436 INFO [ping_erddap] nowcast-green ERDDAP dataset flag file(s) created
2025-06-08 11:17:08,437 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2025-06-08 11:17:09,362 INFO [make_averaged_dataset] running in process 754218
2025-06-08 11:17:09,363 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-08 11:17:09,364 INFO [make_averaged_dataset] publishing log messages to tcp://*:5563
2025-06-08 11:17:09,364 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555
2025-06-08 11:17:09,365 INFO [make_averaged_dataset] creating day-averaged dataset for 08-Jun-2025 physics
2025-06-08 11:17:09,418 INFO [make_averaged_dataset] running in process 754217
2025-06-08 11:17:09,419 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-08 11:17:09,420 INFO [make_averaged_dataset] publishing log messages to tcp://*:5564
2025-06-08 11:17:09,421 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555
2025-06-08 11:17:09,422 INFO [make_averaged_dataset] creating day-averaged dataset for 08-Jun-2025 chemistry
2025-06-08 11:17:09,431 INFO [make_averaged_dataset] running in process 754219
2025-06-08 11:17:09,432 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-08 11:17:09,432 INFO [make_averaged_dataset] publishing log messages to tcp://*:5565
2025-06-08 11:17:09,433 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555
2025-06-08 11:17:09,434 INFO [make_averaged_dataset] creating day-averaged dataset for 08-Jun-2025 biology
2025-06-08 11:17:09,994 INFO [make_plots] running in process 754214
2025-06-08 11:17:09,995 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-08 11:17:09,996 INFO [make_plots] publishing log messages to tcp://*:5566
2025-06-08 11:17:09,997 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2025-06-08 11:17:10,289 INFO [make_plots] preparing render list for 2025-06-08 NEMO nowcast-green research figures
2025-06-08 11:17:12,469 INFO [make_plots] added temperature_thalweg_and_surface figures to 2025-06-08 NEMO nowcast-green research render list
2025-06-08 11:17:13,235 INFO [make_plots] added nitrate_thalweg_and_surface figures to 2025-06-08 NEMO nowcast-green research render list
2025-06-08 11:17:14,011 INFO [make_plots] added ammonium_thalweg_and_surface figures to 2025-06-08 NEMO nowcast-green research render list
2025-06-08 11:17:14,783 INFO [make_plots] added silicon_thalweg_and_surface figures to 2025-06-08 NEMO nowcast-green research render list
2025-06-08 11:17:15,566 INFO [make_plots] added dissolved_organic_nitrogen_thalweg_and_surface figures to 2025-06-08 NEMO nowcast-green research render list
2025-06-08 11:17:16,344 INFO [make_plots] added particulate_organic_nitrogen_thalweg_and_surface figures to 2025-06-08 NEMO nowcast-green research render list
2025-06-08 11:17:17,145 INFO [make_plots] added biogenic_silicon_thalweg_and_surface figures to 2025-06-08 NEMO nowcast-green research render list
2025-06-08 11:17:18,040 INFO [make_plots] added diatoms_thalweg_and_surface figures to 2025-06-08 NEMO nowcast-green research render list
2025-06-08 11:17:18,905 INFO [make_plots] added flagellates_thalweg_and_surface figures to 2025-06-08 NEMO nowcast-green research render list
2025-06-08 11:17:19,778 INFO [make_plots] added microzooplankton_thalweg_and_surface figures to 2025-06-08 NEMO nowcast-green research render list
2025-06-08 11:17:20,670 INFO [make_plots] added mesozooplankton_thalweg_and_surface figures to 2025-06-08 NEMO nowcast-green research render list
2025-06-08 11:17:21,477 INFO [make_plots] added turbidity_thalweg_and_surface figures to 2025-06-08 NEMO nowcast-green research render list
2025-06-08 11:17:22,354 INFO [make_plots] added temperature_salinity_timeseries figure to 2025-06-08 NEMO nowcast-green research render list
2025-06-08 11:17:22,354 INFO [make_plots] added nitrate_diatoms_timeseries figure to 2025-06-08 NEMO nowcast-green research render list
2025-06-08 11:17:22,355 INFO [make_plots] added diatoms_flagellates_timeseries figure to 2025-06-08 NEMO nowcast-green research render list
2025-06-08 11:17:22,355 INFO [make_plots] added z1_z2_zooplankton_timeseries figure to 2025-06-08 NEMO nowcast-green research render list
2025-06-08 11:17:22,356 INFO [make_plots] starting to render nemo nowcast-green research 08jun25 figures
2025-06-08 11:18:10,695 INFO [make_averaged_dataset] day-averaged dataset for 08-Jun-2025 chemistry created
2025-06-08 11:18:10,697 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker
2025-06-08 11:18:16,592 INFO [make_averaged_dataset] day-averaged dataset for 08-Jun-2025 physics created
2025-06-08 11:18:16,595 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker
2025-06-08 11:18:33,074 INFO [make_averaged_dataset] day-averaged dataset for 08-Jun-2025 biology created
2025-06-08 11:18:33,077 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker
2025-06-08 12:19:04,209 INFO [make_plots] finished rendering nemo nowcast-green research 08jun25 figures
2025-06-08 12:19:04,219 INFO [make_plots] nemo research plots for 2025-06-08 nowcast-green completed
2025-06-08 12:19:04,279 INFO [manager] checklist updated with [plots] items from make_plots worker
2025-06-08 15:04:27,062 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/18/ to /results/forcing/atmospheric/continental2.5/GRIB/20250608/18/
2025-06-08 15:04:27,063 INFO [collect_weather] 2025-06-08 2.5km weather forecast 18 collection complete
2025-06-08 15:04:27,067 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2025-06-08 15:04:27,548 INFO [manager] launching NextWorker(module='nowcast.workers.download_weather', args=['00', '1km'], host='localhost')
2025-06-08 15:04:27,558 INFO [manager] launching NextWorker(module='nowcast.workers.download_weather', args=['12', '1km'], host='localhost')
2025-06-08 15:04:27,562 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['00', '2.5km'], host='localhost')
2025-06-08 15:04:27,567 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['00', '--fcst-date', '2025-06-09'], host='localhost')
2025-06-08 15:04:27,806 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20250608/18/
2025-06-08 15:04:27,807 INFO [crop_gribs] 2025-06-08 18 GRIBs cropping complete
2025-06-08 15:04:27,809 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2025-06-08 15:04:29,189 INFO [collect_weather] running in process 756181
2025-06-08 15:04:29,190 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-08 15:04:29,191 INFO [collect_weather] publishing log messages to tcp://*:5562
2025-06-08 15:04:29,191 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2025-06-08 15:04:29,194 INFO [download_weather] running in process 756180
2025-06-08 15:04:29,196 INFO [download_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-08 15:04:29,196 INFO [download_weather] publishing log messages to tcp://*:5563
2025-06-08 15:04:29,197 INFO [download_weather] connected to skookum.eos.ubc.ca port 5555
2025-06-08 15:04:29,198 INFO [download_weather] downloading 12 1 km forecast GRIB2 files for 20250608
2025-06-08 15:04:29,206 INFO [download_weather] running in process 756178
2025-06-08 15:04:29,207 INFO [download_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-08 15:04:29,208 INFO [download_weather] publishing log messages to tcp://*:5564
2025-06-08 15:04:29,209 INFO [download_weather] connected to skookum.eos.ubc.ca port 5555
2025-06-08 15:04:29,210 INFO [download_weather] downloading 00 1 km forecast GRIB2 files for 20250608
2025-06-08 15:04:29,211 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/00/
2025-06-08 15:04:29,964 INFO [crop_gribs] running in process 756182
2025-06-08 15:04:29,965 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-08 15:04:29,965 INFO [crop_gribs] publishing log messages to tcp://*:5565
2025-06-08 15:04:29,966 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2025-06-08 15:04:29,967 INFO [crop_gribs] cropping 2025-06-09 ECCC HRDPS 2.5km continental 00Z GRIB files to SalishSeaCast subdomain
2025-06-08 15:04:29,982 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20250609/00/
2025-06-08 15:08:04,034 INFO [download_weather] 2025-06-08 1km weather forecast 00 downloads complete
2025-06-08 15:08:04,037 INFO [manager] checklist updated with [weather forecast] items from download_weather worker
2025-06-08 15:08:04,049 INFO [download_weather] 2025-06-08 1km weather forecast 12 downloads complete
2025-06-08 15:08:04,528 INFO [manager] checklist updated with [weather forecast] items from download_weather worker
2025-06-08 15:10:44,644 INFO [ping_erddap] running in process 756661
2025-06-08 15:10:44,645 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-08 15:10:44,646 INFO [ping_erddap] publishing log messages to tcp://*:5563
2025-06-08 15:10:44,647 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2025-06-08 15:10:44,648 INFO [ping_erddap] weather ERDDAP dataset flag file(s) created
2025-06-08 15:10:44,649 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2025-06-08 20:59:39,898 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/00/ to /results/forcing/atmospheric/continental2.5/GRIB/20250609/00/
2025-06-08 20:59:39,899 INFO [collect_weather] 2025-06-09 2.5km weather forecast 00 collection complete
2025-06-08 20:59:39,901 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2025-06-08 20:59:40,358 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20250609/00/
2025-06-08 20:59:40,359 INFO [crop_gribs] 2025-06-09 00 GRIBs cropping complete
2025-06-08 20:59:40,438 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['06', '2.5km'], host='localhost')
2025-06-08 20:59:40,445 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['06', '--fcst-date', '2025-06-09'], host='localhost')
2025-06-08 20:59:40,451 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2025-06-08 20:59:41,645 INFO [collect_weather] running in process 759765
2025-06-08 20:59:41,646 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-08 20:59:41,647 INFO [collect_weather] publishing log messages to tcp://*:5562
2025-06-08 20:59:41,648 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2025-06-08 20:59:41,663 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/06/
2025-06-08 20:59:42,394 INFO [crop_gribs] running in process 759767
2025-06-08 20:59:42,395 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-08 20:59:42,396 INFO [crop_gribs] publishing log messages to tcp://*:5563
2025-06-08 20:59:42,397 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2025-06-08 20:59:42,398 INFO [crop_gribs] cropping 2025-06-09 ECCC HRDPS 2.5km continental 06Z GRIB files to SalishSeaCast subdomain
2025-06-08 20:59:42,413 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20250609/06/
2025-06-09 02:59:19,374 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/06/ to /results/forcing/atmospheric/continental2.5/GRIB/20250609/06/
2025-06-09 02:59:19,375 INFO [collect_weather] 2025-06-09 2.5km weather forecast 06 collection complete
2025-06-09 02:59:19,379 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2025-06-09 02:59:19,937 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Capilano', '--data-date', '2025-06-08'], host='localhost')
2025-06-09 02:59:19,943 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'ChilliwackVedder', '--data-date', '2025-06-08'], host='localhost')
2025-06-09 02:59:19,948 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'ClowhomClowhomLake', '--data-date', '2025-06-08'], host='localhost')
2025-06-09 02:59:19,952 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Englishman', '--data-date', '2025-06-08'], host='localhost')
2025-06-09 02:59:19,957 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Fraser', '--data-date', '2025-06-08'], host='localhost')
2025-06-09 02:59:19,962 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'HomathkoMouth', '--data-date', '2025-06-08'], host='localhost')
2025-06-09 02:59:19,965 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'NicomeklLangley', '--data-date', '2025-06-08'], host='localhost')
2025-06-09 02:59:19,969 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'RobertsCreek', '--data-date', '2025-06-08'], host='localhost')
2025-06-09 02:59:19,979 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SalmonSayward', '--data-date', '2025-06-08'], host='localhost')
2025-06-09 02:59:19,983 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SanJuanPortRenfrew', '--data-date', '2025-06-08'], host='localhost')
2025-06-09 02:59:19,990 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SquamishBrackendale', '--data-date', '2025-06-08'], host='localhost')
2025-06-09 02:59:20,004 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaScotty', '--data-date', '2025-06-08'], host='localhost')
2025-06-09 02:59:20,016 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaBypass', '--data-date', '2025-06-08'], host='localhost')
2025-06-09 02:59:20,047 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaDiversion', '--data-date', '2025-06-08'], host='localhost')
2025-06-09 02:59:20,061 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ctd', args=['SCVIP'], host='localhost')
2025-06-09 02:59:20,077 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ctd', args=['SEVIP'], host='localhost')
2025-06-09 02:59:20,088 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ferry', args=['TWDP'], host='localhost')
2025-06-09 02:59:20,094 INFO [manager] launching NextWorker(module='nowcast.workers.get_vfpa_hadcp', args=['--data-date', '2025-06-08'], host='localhost')
2025-06-09 02:59:20,119 INFO [manager] launching NextWorker(module='nowcast.workers.collect_NeahBay_ssh', args=['00'], host='localhost')
2025-06-09 02:59:20,150 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['12', '2.5km'], host='localhost')
2025-06-09 02:59:20,169 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['12'], host='localhost')
2025-06-09 02:59:22,076 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20250609/06/
2025-06-09 02:59:22,076 INFO [crop_gribs] 2025-06-09 06 GRIBs cropping complete
2025-06-09 02:59:22,078 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2025-06-09 02:59:23,128 INFO [manager] launching NextWorker(module='nowcast.workers.grib_to_netcdf', args=['forecast2'], host='localhost')
2025-06-09 02:59:23,146 INFO [collect_NeahBay_ssh] running in process 762714
2025-06-09 02:59:23,147 INFO [collect_weather] running in process 762715
2025-06-09 02:59:23,149 INFO [collect_NeahBay_ssh] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-09 02:59:23,152 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-09 02:59:23,152 INFO [collect_NeahBay_ssh] publishing log messages to tcp://*:5564
2025-06-09 02:59:23,153 INFO [collect_weather] publishing log messages to tcp://*:5562
2025-06-09 02:59:23,154 INFO [collect_NeahBay_ssh] connected to skookum.eos.ubc.ca port 5555
2025-06-09 02:59:23,154 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2025-06-09 02:59:23,155 INFO [collect_NeahBay_ssh] collecting Neah Bay ssh 00Z obs/forecast for 2025-06-09
2025-06-09 02:59:23,176 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/12/
2025-06-09 02:59:24,001 INFO [collect_river_data] running in process 762702
2025-06-09 02:59:24,003 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-09 02:59:24,006 INFO [collect_river_data] publishing log messages to tcp://*:5563
2025-06-09 02:59:24,008 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-06-09 02:59:24,009 INFO [collect_river_data] Collecting ECCC RobertsCreek river data for 2025-06-08
2025-06-09 02:59:24,014 INFO [collect_river_data] running in process 762700
2025-06-09 02:59:24,016 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-09 02:59:24,021 INFO [collect_river_data] publishing log messages to tcp://*:5565
2025-06-09 02:59:24,022 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-06-09 02:59:24,022 INFO [collect_river_data] Collecting ECCC HomathkoMouth river data for 2025-06-08
2025-06-09 02:59:24,097 INFO [collect_river_data] Appended ECCC HomathkoMouth river average discharge for 2025-06-08 to: /results/forcing/rivers/observations/Homathko_Mouth_flow
2025-06-09 02:59:24,098 INFO [collect_river_data] ECCC HomathkoMouth river data collection for 2025-06-08 completed
2025-06-09 02:59:24,100 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-06-09 02:59:24,103 INFO [collect_river_data] running in process 762703
2025-06-09 02:59:24,104 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-09 02:59:24,105 INFO [collect_river_data] publishing log messages to tcp://*:5566
2025-06-09 02:59:24,106 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-06-09 02:59:24,107 INFO [collect_river_data] Collecting ECCC SalmonSayward river data for 2025-06-08
2025-06-09 02:59:24,107 INFO [collect_river_data] Appended ECCC RobertsCreek river average discharge for 2025-06-08 to: /results/forcing/rivers/observations/RobertsCreek_flow
2025-06-09 02:59:24,108 INFO [collect_river_data] ECCC RobertsCreek river data collection for 2025-06-08 completed
2025-06-09 02:59:24,117 INFO [collect_river_data] running in process 762701
2025-06-09 02:59:24,118 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-09 02:59:24,119 INFO [collect_river_data] publishing log messages to tcp://*:5567
2025-06-09 02:59:24,119 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-06-09 02:59:24,120 INFO [collect_river_data] Collecting ECCC NicomeklLangley river data for 2025-06-08
2025-06-09 02:59:24,142 CRITICAL [collect_river_data] unhandled exception:
Traceback (most recent call last):
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/pandas/core/indexes/datetimes.py", line 610, in get_loc
    return self._partial_date_slice(reso, parsed)
           ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/pandas/core/indexes/datetimelike.py", line 333, in _partial_date_slice
    raise KeyError
KeyError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/SalishSeaCast/NEMO_Nowcast/nemo_nowcast/worker.py", line 387, in _do_work
    checklist = self.worker_func(
        self._parsed_args, self.config, self.tell_manager
    )
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/collect_river_data.py", line 108, in collect_river_data
    day_avg_discharge = day_avg_discharge_funcs[data_src](river_name, data_date, config)
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/collect_river_data.py", line 141, in _calc_eccc_day_avg_discharge
    day_avg_discharge = df.loc[f"{data_date.format('YYYY-MM-DD')}"].mean()[
                        ~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/pandas/core/indexing.py", line 1191, in __getitem__
    return self._getitem_axis(maybe_callable, axis=axis)
           ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/pandas/core/indexing.py", line 1431, in _getitem_axis
    return self._get_label(key, axis=axis)
           ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/pandas/core/indexing.py", line 1381, in _get_label
    return self.obj.xs(label, axis=axis)
           ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/pandas/core/generic.py", line 4301, in xs
    loc = index.get_loc(key)
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/pandas/core/indexes/datetimes.py", line 612, in get_loc
    raise KeyError(key) from err
KeyError: '2025-06-08'
2025-06-09 02:59:24,377 INFO [collect_river_data] running in process 762708
2025-06-09 02:59:24,377 INFO [collect_river_data] running in process 762704
2025-06-09 02:59:24,378 INFO [collect_river_data] running in process 762696
2025-06-09 02:59:24,379 INFO [collect_river_data] running in process 762707
2025-06-09 02:59:24,379 INFO [collect_river_data] running in process 762699
2025-06-09 02:59:24,380 INFO [collect_river_data] running in process 762706
2025-06-09 02:59:24,380 INFO [collect_river_data] running in process 762705
2025-06-09 02:59:24,381 INFO [collect_river_data] running in process 762698
2025-06-09 02:59:24,382 INFO [collect_river_data] running in process 762694
2025-06-09 02:59:24,383 INFO [collect_river_data] running in process 762697
2025-06-09 02:59:24,383 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-09 02:59:24,384 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-09 02:59:24,385 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-09 02:59:24,386 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-09 02:59:24,386 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-09 02:59:24,387 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-09 02:59:24,388 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-09 02:59:24,388 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-09 02:59:24,389 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-09 02:59:24,390 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-09 02:59:24,390 INFO [collect_river_data] Appended ECCC NicomeklLangley river average discharge for 2025-06-08 to: /results/forcing/rivers/observations/Nicomekl_Langley_flow
2025-06-09 02:59:24,391 INFO [collect_river_data] publishing log messages to tcp://*:5568
2025-06-09 02:59:24,392 INFO [collect_river_data] publishing log messages to tcp://*:5569
2025-06-09 02:59:24,392 INFO [collect_river_data] publishing log messages to tcp://*:5570
2025-06-09 02:59:24,393 INFO [collect_river_data] publishing log messages to tcp://*:5571
2025-06-09 02:59:24,394 INFO [collect_river_data] publishing log messages to tcp://*:5572
2025-06-09 02:59:24,394 INFO [collect_river_data] publishing log messages to tcp://*:5573
2025-06-09 02:59:24,395 INFO [collect_river_data] publishing log messages to tcp://*:5574
2025-06-09 02:59:24,396 INFO [collect_river_data] publishing log messages to tcp://*:5576
2025-06-09 02:59:24,397 INFO [collect_river_data] publishing log messages to tcp://*:5575
2025-06-09 02:59:24,397 INFO [collect_river_data] publishing log messages to tcp://*:5577
2025-06-09 02:59:24,398 INFO [collect_river_data] ECCC NicomeklLangley river data collection for 2025-06-08 completed
2025-06-09 02:59:24,398 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-06-09 02:59:24,399 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-06-09 02:59:24,400 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-06-09 02:59:24,400 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-06-09 02:59:24,401 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-06-09 02:59:24,401 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-06-09 02:59:24,402 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-06-09 02:59:24,403 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-06-09 02:59:24,403 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-06-09 02:59:24,404 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-06-09 02:59:24,405 INFO [collect_river_data] Collecting ECCC TheodosiaDiversion river data for 2025-06-08
2025-06-09 02:59:24,406 INFO [collect_river_data] Collecting ECCC SanJuanPortRenfrew river data for 2025-06-08
2025-06-09 02:59:24,406 INFO [collect_river_data] Collecting ECCC ChilliwackVedder river data for 2025-06-08
2025-06-09 02:59:24,407 INFO [collect_river_data] Collecting ECCC TheodosiaBypass river data for 2025-06-08
2025-06-09 02:59:24,408 INFO [collect_river_data] Collecting ECCC Fraser river data for 2025-06-08
2025-06-09 02:59:24,408 INFO [collect_river_data] Collecting ECCC TheodosiaScotty river data for 2025-06-08
2025-06-09 02:59:24,410 INFO [collect_river_data] Collecting ECCC SquamishBrackendale river data for 2025-06-08
2025-06-09 02:59:24,410 INFO [collect_river_data] Collecting ECCC Englishman river data for 2025-06-08
2025-06-09 02:59:24,411 INFO [collect_river_data] Collecting ECCC Capilano river data for 2025-06-08
2025-06-09 02:59:24,412 INFO [collect_river_data] Collecting ECCC ClowhomClowhomLake river data for 2025-06-08
2025-06-09 02:59:24,420 INFO [collect_river_data] Appended ECCC ClowhomClowhomLake river average discharge for 2025-06-08 to: /results/forcing/rivers/observations/Clowhom_ClowhomLake_flow
2025-06-09 02:59:24,420 INFO [collect_river_data] Appended ECCC TheodosiaDiversion river average discharge for 2025-06-08 to: /results/forcing/rivers/observations/Theodosia_Diversion_flow
2025-06-09 02:59:24,421 INFO [collect_river_data] Appended ECCC SanJuanPortRenfrew river average discharge for 2025-06-08 to: /results/forcing/rivers/observations/SanJuan_PortRenfrew_flow
2025-06-09 02:59:24,422 INFO [collect_river_data] Appended ECCC ChilliwackVedder river average discharge for 2025-06-08 to: /results/forcing/rivers/observations/Chilliwack_Vedder_flow
2025-06-09 02:59:24,422 INFO [collect_river_data] Appended ECCC TheodosiaBypass river average discharge for 2025-06-08 to: /results/forcing/rivers/observations/Theodosia_Bypass_flow
2025-06-09 02:59:24,423 INFO [collect_river_data] Appended ECCC Fraser river average discharge for 2025-06-08 to: /data/dlatorne/SOG-projects/SOG-forcing/ECget/Fraser_flow
2025-06-09 02:59:24,424 INFO [collect_river_data] Appended ECCC TheodosiaScotty river average discharge for 2025-06-08 to: /results/forcing/rivers/observations/Theodosia_Scotty_flow
2025-06-09 02:59:24,424 INFO [collect_river_data] Appended ECCC SquamishBrackendale river average discharge for 2025-06-08 to: /results/forcing/rivers/observations/Squamish_Brackendale_flow
2025-06-09 02:59:24,425 INFO [collect_river_data] Appended ECCC Englishman river average discharge for 2025-06-08 to: /data/dlatorne/SOG-projects/SOG-forcing/ECget/Englishman_flow
2025-06-09 02:59:24,426 INFO [collect_river_data] ECCC ClowhomClowhomLake river data collection for 2025-06-08 completed
2025-06-09 02:59:24,426 INFO [collect_river_data] ECCC TheodosiaDiversion river data collection for 2025-06-08 completed
2025-06-09 02:59:24,427 INFO [collect_river_data] ECCC SanJuanPortRenfrew river data collection for 2025-06-08 completed
2025-06-09 02:59:24,428 INFO [collect_river_data] ECCC ChilliwackVedder river data collection for 2025-06-08 completed
2025-06-09 02:59:24,428 INFO [collect_river_data] ECCC TheodosiaBypass river data collection for 2025-06-08 completed
2025-06-09 02:59:24,429 INFO [collect_river_data] ECCC Fraser river data collection for 2025-06-08 completed
2025-06-09 02:59:24,430 INFO [collect_river_data] ECCC TheodosiaScotty river data collection for 2025-06-08 completed
2025-06-09 02:59:24,430 INFO [collect_river_data] ECCC SquamishBrackendale river data collection for 2025-06-08 completed
2025-06-09 02:59:24,431 INFO [collect_river_data] ECCC Englishman river data collection for 2025-06-08 completed
2025-06-09 02:59:24,501 INFO [collect_river_data] Appended ECCC Capilano river average discharge for 2025-06-08 to: /opp/observations/rivers/Capilano/Caplilano_08GA010_day_avg_flow
2025-06-09 02:59:24,501 INFO [collect_river_data] ECCC Capilano river data collection for 2025-06-08 completed
2025-06-09 02:59:24,509 INFO [get_vfpa_hadcp] running in process 762713
2025-06-09 02:59:24,510 INFO [get_vfpa_hadcp] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-09 02:59:24,510 INFO [get_vfpa_hadcp] publishing log messages to tcp://*:5578
2025-06-09 02:59:24,512 INFO [get_vfpa_hadcp] connected to skookum.eos.ubc.ca port 5555
2025-06-09 02:59:24,512 INFO [get_vfpa_hadcp] processing VFPA HADCP data from 2nd Narrows Rail Bridge for 2025-06-08
2025-06-09 02:59:24,527 INFO [collect_NeahBay_ssh] 2025-06-09T00:00:00-07:00 Neah Bay ssh 00Z obs/forecast data collection complete
2025-06-09 02:59:24,672 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-06-09 02:59:24,760 INFO [crop_gribs] running in process 762716
2025-06-09 02:59:24,761 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-09 02:59:24,761 INFO [crop_gribs] publishing log messages to tcp://*:5579
2025-06-09 02:59:24,763 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2025-06-09 02:59:24,763 INFO [crop_gribs] cropping 2025-06-09 ECCC HRDPS 2.5km continental 12Z GRIB files to SalishSeaCast subdomain
2025-06-09 02:59:24,774 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20250609/12/
2025-06-09 02:59:25,090 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-06-09 02:59:25,113 INFO [get_onc_ferry] running in process 762712
2025-06-09 02:59:25,114 INFO [get_onc_ferry] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-09 02:59:25,115 INFO [get_onc_ferry] publishing log messages to tcp://*:5565
2025-06-09 02:59:25,116 INFO [get_onc_ferry] connected to skookum.eos.ubc.ca port 5555
2025-06-09 02:59:25,117 INFO [get_onc_ferry] requesting ONC TWDP.N1 NAV data for 2025-06-08
2025-06-09 02:59:25,125 INFO [get_onc_ctd] running in process 762710
2025-06-09 02:59:25,127 INFO [get_onc_ctd] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-09 02:59:25,128 INFO [get_onc_ctd] publishing log messages to tcp://*:5590
2025-06-09 02:59:25,129 INFO [get_onc_ctd] connected to skookum.eos.ubc.ca port 5555
2025-06-09 02:59:25,129 INFO [get_onc_ctd] requesting ONC SCVIP CTD T&S data for 2025-06-08
2025-06-09 02:59:25,148 INFO [get_onc_ctd] running in process 762711
2025-06-09 02:59:25,150 INFO [get_onc_ctd] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-09 02:59:25,150 INFO [get_onc_ctd] publishing log messages to tcp://*:5591
2025-06-09 02:59:25,151 INFO [get_onc_ctd] connected to skookum.eos.ubc.ca port 5555
2025-06-09 02:59:25,152 INFO [get_onc_ctd] requesting ONC SEVIP CTD T&S data for 2025-06-08
2025-06-09 02:59:25,452 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-06-09 02:59:25,603 INFO [grib_to_netcdf] running in process 762951
2025-06-09 02:59:25,604 INFO [grib_to_netcdf] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-09 02:59:25,604 INFO [grib_to_netcdf] publishing log messages to tcp://*:5563
2025-06-09 02:59:25,605 INFO [grib_to_netcdf] connected to skookum.eos.ubc.ca port 5555
2025-06-09 02:59:25,606 INFO [grib_to_netcdf] creating NEMO-atmos forcing files for 2025-06-09 forecast2 run
2025-06-09 02:59:25,775 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-06-09 02:59:26,106 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-06-09 02:59:26,451 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-06-09 02:59:26,545 INFO [get_vfpa_hadcp] added VFPA HADCP data from 2nd Narrows Rail Bridge for 2025-06-08 to /opp/observations/AISDATA/netcdf/VFPA_2ND_NARROWS_HADCP_2s_202506.nc
2025-06-09 02:59:26,546 INFO [get_vfpa_hadcp] VFPA HADCP observations added to 2025-06 netcdf file
2025-06-09 02:59:26,803 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-06-09 02:59:27,143 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-06-09 02:59:27,450 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-06-09 02:59:27,756 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-06-09 02:59:28,136 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-06-09 02:59:28,471 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-06-09 02:59:28,805 INFO [manager] checklist updated with [Neah Bay ssh data] items from collect_NeahBay_ssh worker
2025-06-09 02:59:29,147 INFO [manager] launching NextWorker(module='nowcast.workers.make_ssh_files', args=['forecast2', '--run-date', '2025-06-09'], host='localhost')
2025-06-09 02:59:29,157 INFO [manager] checklist updated with [VFPA HADCP data] items from get_vfpa_hadcp worker
2025-06-09 02:59:29,573 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['VFPA-HADCP'], host='localhost')
2025-06-09 02:59:30,645 INFO [ping_erddap] running in process 763101
2025-06-09 02:59:30,646 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-09 02:59:30,647 INFO [ping_erddap] publishing log messages to tcp://*:5564
2025-06-09 02:59:30,648 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2025-06-09 02:59:30,649 INFO [ping_erddap] VFPA-HADCP ERDDAP dataset flag file(s) created
2025-06-09 02:59:30,651 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2025-06-09 02:59:31,838 INFO [make_ssh_files] running in process 763088
2025-06-09 02:59:31,841 INFO [make_ssh_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-09 02:59:31,841 INFO [make_ssh_files] publishing log messages to tcp://*:5564
2025-06-09 02:59:31,842 INFO [make_ssh_files] connected to skookum.eos.ubc.ca port 5555
2025-06-09 02:59:31,843 INFO [make_ssh_files] building forecast2 sea surface height boundary conditions file(s) from 2025-06-09 NOAA Neah Bay 00Z observation and forecast values
2025-06-09 02:59:34,115 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2025m06d04.nc
2025-06-09 02:59:34,287 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2025m06d05.nc
2025-06-09 02:59:34,502 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2025m06d06.nc
2025-06-09 02:59:34,681 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2025m06d07.nc
2025-06-09 02:59:34,868 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2025m06d10.nc
2025-06-09 02:59:34,900 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2025m06d08.nc
2025-06-09 02:59:35,123 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2025m06d09.nc
2025-06-09 02:59:35,340 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2025m06d10.nc
2025-06-09 02:59:35,575 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2025m06d11.nc
2025-06-09 02:59:35,831 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2025m06d12.nc
2025-06-09 02:59:35,999 INFO [make_ssh_files] sea surface height boundary file for 2025-06-09 forecast2 run created
2025-06-09 02:59:36,004 INFO [manager] checklist updated with [sea surface height forcing] items from make_ssh_files worker
2025-06-09 02:59:36,442 INFO [manager] launching NextWorker(module='nowcast.workers.make_runoff_file', args=['v202108'], host='localhost')
2025-06-09 02:59:36,446 INFO [manager] launching NextWorker(module='nowcast.workers.make_201702_runoff_file', args=[], host='localhost')
2025-06-09 02:59:37,691 INFO [make_201702_runoff_file] running in process 763137
2025-06-09 02:59:37,692 INFO [make_201702_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-09 02:59:37,693 INFO [make_201702_runoff_file] publishing log messages to tcp://*:5564
2025-06-09 02:59:37,693 INFO [make_201702_runoff_file] connected to skookum.eos.ubc.ca port 5555
2025-06-09 02:59:37,742 INFO [make_201702_runoff_file] calculating runoff forcing for b201702 bathymetry
2025-06-09 02:59:37,839 INFO [make_201702_runoff_file] 2025-06-09 runoff file creation from Fraser at Hope and climatology elsewhere complete
2025-06-09 02:59:37,841 INFO [manager] checklist updated with [rivers forcing] items from make_201702_runoff_file worker
2025-06-09 02:59:37,907 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2025m06d11.nc
2025-06-09 02:59:37,914 INFO [grib_to_netcdf] 2025-06-09 NEMO-atmos forcing file for forecast2 created
2025-06-09 02:59:38,242 INFO [manager] checklist updated with [weather forcing] items from grib_to_netcdf worker
2025-06-09 02:59:38,310 INFO [make_runoff_file] running in process 763135
2025-06-09 02:59:38,311 INFO [make_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-09 02:59:38,312 INFO [make_runoff_file] publishing log messages to tcp://*:5566
2025-06-09 02:59:38,312 INFO [make_runoff_file] connected to skookum.eos.ubc.ca port 5555
2025-06-09 02:59:38,316 INFO [make_runoff_file] calculating NEMO runoff forcing for v202108 bathymetry for 2025-06-08
2025-06-09 02:59:38,668 ERROR [make_runoff_file] no 2025-06-08 discharge obs for SalmonSayward: patching
2025-06-09 02:59:39,847 ERROR [make_runoff_file] no 2025-06-08 discharge obs for SkagitMountVernon: patching
2025-06-09 02:59:39,959 ERROR [make_runoff_file] no 2025-06-08 discharge obs for SnohomishMonroe: patching
2025-06-09 02:59:40,167 ERROR [make_runoff_file] no 2025-06-08 discharge obs for NisquallyMcKenna: patching
2025-06-09 02:59:40,261 ERROR [make_runoff_file] no 2025-06-08 discharge obs for GreenwaterGreenwater: patching
2025-06-09 02:59:41,515 INFO [make_runoff_file] stored NEMO runoff forcing for v202108 bathymetry for 2025-06-08: /results/forcing/rivers/R202108Dailies_y2025m06d08.nc
2025-06-09 02:59:41,516 INFO [make_runoff_file] 2025-06-08 runoff file creation completed
2025-06-09 02:59:41,517 INFO [manager] checklist updated with [rivers forcing] items from make_runoff_file worker
2025-06-09 02:59:41,858 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'forecast2'], host='localhost')
2025-06-09 02:59:41,863 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'forecast2'], host='localhost')
2025-06-09 02:59:41,866 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.graham', 'forecast2'], host='localhost')
2025-06-09 02:59:41,871 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'forecast2'], host='localhost')
2025-06-09 02:59:43,157 INFO [upload_forcing] running in process 763171
2025-06-09 02:59:43,158 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-09 02:59:43,159 INFO [upload_forcing] publishing log messages to tcp://*:5563
2025-06-09 02:59:43,160 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-06-09 02:59:43,178 INFO [upload_forcing] running in process 763173
2025-06-09 02:59:43,179 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-09 02:59:43,180 INFO [upload_forcing] publishing log messages to tcp://*:5564
2025-06-09 02:59:43,180 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-06-09 02:59:43,191 INFO [upload_forcing] running in process 763172
2025-06-09 02:59:43,192 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-09 02:59:43,192 INFO [upload_forcing] publishing log messages to tcp://*:5566
2025-06-09 02:59:43,193 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-06-09 02:59:43,246 INFO [upload_forcing] running in process 763170
2025-06-09 02:59:43,247 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-09 02:59:43,247 INFO [upload_forcing] publishing log messages to tcp://*:5567
2025-06-09 02:59:43,248 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-06-09 02:59:43,582 INFO [get_onc_ctd] 2025-06-08 ONC SEVIP CTD T&S file created
2025-06-09 02:59:43,585 INFO [manager] checklist updated with [ONC CTD data] items from get_onc_ctd worker
2025-06-09 02:59:43,914 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['SEVIP-CTD'], host='localhost')
2025-06-09 02:59:43,986 INFO [get_onc_ctd] 2025-06-08 ONC SCVIP CTD T&S file created
2025-06-09 02:59:43,988 INFO [manager] checklist updated with [ONC CTD data] items from get_onc_ctd worker
2025-06-09 02:59:44,505 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['SCVIP-CTD'], host='localhost')
2025-06-09 02:59:44,637 INFO [upload_forcing] LiveOcean boundary conditions file not found; created symlink to /results/forcing/LiveOcean/boundary_conditions/LiveOcean_v201905_y2025m06d08.nc
2025-06-09 02:59:44,705 INFO [upload_forcing] forecast2 2025-06-09 forcing files upload to optimum-hindcast completed
2025-06-09 02:59:44,707 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-06-09 02:59:44,763 INFO [upload_forcing] forecast2 2025-06-09 forcing files upload to orcinus-nowcast-agrif completed
2025-06-09 02:59:45,065 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-06-09 02:59:45,081 INFO [ping_erddap] running in process 763198
2025-06-09 02:59:45,082 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-09 02:59:45,083 INFO [ping_erddap] publishing log messages to tcp://*:5568
2025-06-09 02:59:45,084 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2025-06-09 02:59:45,085 INFO [ping_erddap] SEVIP-CTD ERDDAP dataset flag file(s) created
2025-06-09 02:59:45,391 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2025-06-09 02:59:45,585 INFO [ping_erddap] running in process 763200
2025-06-09 02:59:45,586 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-09 02:59:45,587 INFO [ping_erddap] publishing log messages to tcp://*:5564
2025-06-09 02:59:45,588 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2025-06-09 02:59:45,589 INFO [ping_erddap] SCVIP-CTD ERDDAP dataset flag file(s) created
2025-06-09 02:59:45,696 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2025-06-09 02:59:46,190 INFO [get_onc_ferry] requesting ONC TWDP TSG data for 2025-06-08
2025-06-09 02:59:51,158 INFO [upload_forcing] forecast2 2025-06-09 forcing files upload to robot.graham completed
2025-06-09 02:59:51,163 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-06-09 03:00:09,207 INFO [get_onc_ferry] requesting ONC TWDP OXYSENSOR data for 2025-06-08
2025-06-09 03:00:10,375 INFO [upload_forcing] forecast2 2025-06-09 forcing files upload to arbutus.cloud-nowcast completed
2025-06-09 03:00:10,379 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-06-09 03:00:10,723 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2025-06-09'], host='localhost')
2025-06-09 03:00:11,719 INFO [make_forcing_links] running in process 763217
2025-06-09 03:00:11,720 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-09 03:00:11,721 INFO [make_forcing_links] publishing log messages to tcp://*:5563
2025-06-09 03:00:11,722 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2025-06-09 03:00:13,133 INFO [make_forcing_links] forecast2 2025-06-09 forcing file links on arbutus.cloud-nowcast created
2025-06-09 03:00:13,137 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2025-06-09 03:00:13,525 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2025-06-09'], host='arbutus.cloud-nowcast')
2025-06-09 03:00:15,847 INFO [run_NEMO] running in process 19537
2025-06-09 03:00:15,850 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-06-09 03:00:15,851 INFO [run_NEMO] publishing log messages to tcp://*:5556
2025-06-09 03:00:15,893 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2025-06-09 03:00:16,725 WARNING [run_NEMO] There are uncommitted changes in /nemoShare/MEOPAR/nowcast-sys/SS-run-sets
2025-06-09 03:00:17,114 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2025-06-09 03:00:17,153 INFO [run_NEMO] forecast2: launching /nemoShare/MEOPAR/nowcast-sys/runs/08jun25forecast2_2025-06-09T030015.967064-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2025-06-09 03:00:17,169 INFO [run_NEMO] forecast2 NEMO run for 2025-06-09 on arbutus.cloud-nowcast started
2025-06-09 03:00:17,484 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'forecast2'], host='arbutus.cloud-nowcast')
2025-06-09 03:00:19,472 INFO [watch_NEMO] running in process 19810
2025-06-09 03:00:19,476 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-06-09 03:00:19,481 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2025-06-09 03:00:19,482 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2025-06-09 03:00:19,865 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2025-06-09 03:00:31,530 INFO [get_onc_ferry] requesting ONC TWDP TURBCHLFL data for 2025-06-08
2025-06-09 03:00:55,385 INFO [get_onc_ferry] requesting ONC TWDP CO2SENSOR data for 2025-06-08
2025-06-09 03:00:55,426 WARNING [get_onc_ferry] No ONC TWDP CO2SENSOR partial_pressure,co2 data for 2025-06-08; substituting empty dataset
2025-06-09 03:00:55,430 WARNING [get_onc_ferry] No ONC TWDP CO2SENSOR partial_pressure data for 2025-06-08; substituting empty dataset
2025-06-09 03:00:55,431 WARNING [get_onc_ferry] No ONC TWDP CO2SENSOR co2 data for 2025-06-08; substituting empty dataset
2025-06-09 03:00:55,432 INFO [get_onc_ferry] requesting ONC TWDP TEMPHUMID data for 2025-06-08
2025-06-09 03:00:55,475 WARNING [get_onc_ferry] No ONC TWDP TEMPHUMID air_temperature,rel_humidity data for 2025-06-08; substituting empty dataset
2025-06-09 03:00:55,477 WARNING [get_onc_ferry] No ONC TWDP TEMPHUMID air_temperature data for 2025-06-08; substituting empty dataset
2025-06-09 03:00:55,479 WARNING [get_onc_ferry] No ONC TWDP TEMPHUMID rel_humidity data for 2025-06-08; substituting empty dataset
2025-06-09 03:00:55,480 INFO [get_onc_ferry] requesting ONC TWDP BARPRESS data for 2025-06-08
2025-06-09 03:00:55,515 WARNING [get_onc_ferry] No ONC TWDP BARPRESS barometric_pressure data for 2025-06-08; substituting empty dataset
2025-06-09 03:00:55,517 WARNING [get_onc_ferry] No ONC TWDP BARPRESS barometric_pressure data for 2025-06-08; substituting empty dataset
2025-06-09 03:00:55,518 INFO [get_onc_ferry] requesting ONC TWDP PYRANOMETER data for 2025-06-08
2025-06-09 03:00:55,563 WARNING [get_onc_ferry] No ONC TWDP PYRANOMETER solar_radiation data for 2025-06-08; substituting empty dataset
2025-06-09 03:00:55,565 WARNING [get_onc_ferry] No ONC TWDP PYRANOMETER solar_radiation data for 2025-06-08; substituting empty dataset
2025-06-09 03:00:55,566 INFO [get_onc_ferry] requesting ONC TWDP PYRGEOMETER data for 2025-06-08
2025-06-09 03:00:55,638 WARNING [get_onc_ferry] No ONC TWDP PYRGEOMETER downward_radiation data for 2025-06-08; substituting empty dataset
2025-06-09 03:00:55,641 WARNING [get_onc_ferry] No ONC TWDP PYRGEOMETER downward_radiation data for 2025-06-08; substituting empty dataset
2025-06-09 03:00:55,831 WARNING [get_onc_ferry] ONC TWDP CO2SENSOR partial_pressure data for 2025-06-08 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2025-06-09 03:00:55,846 WARNING [get_onc_ferry] ONC TWDP CO2SENSOR co2 data for 2025-06-08 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2025-06-09 03:00:55,862 WARNING [get_onc_ferry] ONC TWDP TEMPHUMID air_temperature data for 2025-06-08 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2025-06-09 03:00:55,877 WARNING [get_onc_ferry] ONC TWDP TEMPHUMID rel_humidity data for 2025-06-08 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2025-06-09 03:00:55,893 WARNING [get_onc_ferry] ONC TWDP BARPRESS barometric_pressure data for 2025-06-08 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2025-06-09 03:00:55,909 WARNING [get_onc_ferry] ONC TWDP PYRANOMETER solar_radiation data for 2025-06-08 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2025-06-09 03:00:55,924 WARNING [get_onc_ferry] ONC TWDP PYRGEOMETER downward_radiation data for 2025-06-08 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2025-06-09 03:02:03,505 INFO [get_onc_ferry] 2025-06-08 ONC TWDP ferry data file created
2025-06-09 03:02:03,508 INFO [manager] checklist updated with [ONC ferry data] items from get_onc_ferry worker
2025-06-09 03:02:03,861 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['TWDP-ferry'], host='localhost')
2025-06-09 03:02:04,902 INFO [ping_erddap] running in process 763238
2025-06-09 03:02:04,903 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-09 03:02:04,904 INFO [ping_erddap] publishing log messages to tcp://*:5563
2025-06-09 03:02:04,905 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2025-06-09 03:02:04,907 INFO [ping_erddap] TWDP-ferry ERDDAP dataset flag file(s) created
2025-06-09 03:02:04,908 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2025-06-09 03:05:19,871 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 18498704 = 2025-06-10 05:08:40 UTC, 17.2% complete
2025-06-09 03:10:19,868 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 18499217 = 2025-06-10 10:50:40 UTC, 36.2% complete
2025-06-09 03:15:19,871 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 18499738 = 2025-06-10 16:38:00 UTC, 55.5% complete
2025-06-09 03:20:19,868 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 18500257 = 2025-06-10 22:24:00 UTC, 74.7% complete
2025-06-09 03:25:19,888 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 18500788 = 2025-06-11 04:18:00 UTC, 94.4% complete
2025-06-09 03:30:19,880 INFO [watch_NEMO] forecast2 NEMO run on arbutus.cloud-nowcast completed
2025-06-09 03:30:19,886 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2025-06-09 03:30:20,391 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_wind_file', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2025-06-09'], host='arbutus.cloud-nowcast')
2025-06-09 03:30:20,399 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_current_file', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2025-06-09'], host='arbutus.cloud-nowcast')
2025-06-09 03:30:20,404 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2025-06-08'], host='localhost')
2025-06-09 03:30:21,475 INFO [download_results] running in process 763487
2025-06-09 03:30:21,476 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-09 03:30:21,477 INFO [download_results] publishing log messages to tcp://*:5563
2025-06-09 03:30:21,478 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2025-06-09 03:30:21,479 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/forecast2/08jun25 to /results/SalishSea/forecast2.202111
2025-06-09 03:30:23,255 INFO [make_ww3_wind_file] running in process 20218
2025-06-09 03:30:23,256 INFO [make_ww3_wind_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-06-09 03:30:23,257 INFO [make_ww3_wind_file] publishing log messages to tcp://*:5570
2025-06-09 03:30:23,258 INFO [make_ww3_wind_file] connected to skookum.eos.ubc.ca port 5555
2025-06-09 03:30:23,259 INFO [make_ww3_wind_file] Creating wwatch3 wind forcing file for 2025-06-09 forecast2 run
2025-06-09 03:30:31,714 INFO [manager] checklist updated with [WW3 wind forcing] items from make_ww3_wind_file worker
2025-06-09 03:30:31,715 INFO [make_ww3_wind_file] wwatch3 wind forcing file created on arbutus.cloud-nowcast for 2025-06-09 forecast2 run
2025-06-09 03:32:10,212 INFO [download_results] forecast2 2025-06-08 results files from arbutus.cloud-nowcast downloaded
2025-06-09 03:32:10,216 INFO [manager] checklist updated with [results files] items from download_results worker
2025-06-09 03:32:10,590 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['forecast2', '--run-date', '2025-06-08'], host='localhost')
2025-06-09 03:32:12,348 INFO [make_CHS_currents_file] running in process 763516
2025-06-09 03:32:12,349 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-09 03:32:12,350 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5563
2025-06-09 03:32:12,351 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555
2025-06-09 03:32:48,600 INFO [make_CHS_currents_file] Made CHS currents file for 2025-06-08 for forecast2
2025-06-09 03:32:48,603 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker
2025-06-09 03:32:48,975 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['nemo', 'forecast2', '--run-date', '2025-06-08'], host='localhost')
2025-06-09 03:32:49,943 INFO [update_forecast_datasets] running in process 763539
2025-06-09 03:32:49,944 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-09 03:32:49,945 INFO [update_forecast_datasets] publishing log messages to tcp://*:5563
2025-06-09 03:32:49,946 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2025-06-09 03:32:49,946 INFO [update_forecast_datasets] updating nemo forecast directory for forecast2/08jun25 run
2025-06-09 03:33:58,921 INFO [make_ww3_current_file] wwatch3 currents forcing file created on arbutus.cloud-nowcast for 2025-06-09 forecast2 run
2025-06-09 03:33:58,926 INFO [manager] checklist updated with [WW3 currents forcing] items from make_ww3_current_file worker
2025-06-09 03:33:59,294 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2025-06-09'], host='arbutus.cloud-nowcast')
2025-06-09 03:34:01,230 INFO [run_ww3] running in process 20840
2025-06-09 03:34:01,231 INFO [run_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-06-09 03:34:01,232 INFO [run_ww3] publishing log messages to tcp://*:5572
2025-06-09 03:34:01,233 INFO [run_ww3] connected to skookum.eos.ubc.ca port 5555
2025-06-09 03:34:01,235 INFO [run_ww3] Created run directory /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast2_2025-06-09T033401.229079-0700
2025-06-09 03:34:01,236 INFO [run_ww3] forecast2: launching /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast2_2025-06-09T033401.229079-0700/SoGWW3.sh on arbutus.cloud-nowcast
2025-06-09 03:34:01,247 INFO [run_ww3] forecast2 WaveWatch3 run for 2025-06-09 on arbutus.cloud-nowcast started
2025-06-09 03:34:01,334 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker
2025-06-09 03:34:01,713 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'forecast2'], host='arbutus.cloud-nowcast')
2025-06-09 03:34:03,601 INFO [watch_ww3] running in process 20927
2025-06-09 03:34:03,604 INFO [watch_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-06-09 03:34:03,605 INFO [watch_ww3] publishing log messages to tcp://*:5573
2025-06-09 03:34:03,606 INFO [watch_ww3] connected to skookum.eos.ubc.ca port 5555
2025-06-09 03:34:03,644 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2025-06-09 03:37:02,670 INFO [update_forecast_datasets] updated nemo forecast directory for forecast2/08jun25 run: /results/SalishSea/rolling-forecasts/nemo
2025-06-09 03:37:02,671 INFO [update_forecast_datasets] nemo 2025-06-08 forecast2 rolling forecast datasets updated
2025-06-09 03:37:02,675 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2025-06-09 03:37:02,969 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nemo-forecast'], host='localhost')
2025-06-09 03:37:02,974 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'forecast2', 'publish', '--run-date', '2025-06-08'], host='localhost')
2025-06-09 03:37:02,980 INFO [manager] launching NextWorker(module='nowcast.workers.make_surface_current_tiles', args=['forecast2', '--run-date', '2025-06-08'], host='localhost')
2025-06-09 03:37:04,171 INFO [ping_erddap] running in process 763650
2025-06-09 03:37:04,172 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-09 03:37:04,173 INFO [ping_erddap] publishing log messages to tcp://*:5563
2025-06-09 03:37:04,174 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2025-06-09 03:37:04,184 INFO [ping_erddap] nemo-forecast ERDDAP dataset flag file(s) created
2025-06-09 03:37:04,185 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2025-06-09 03:37:05,376 INFO [make_surface_current_tiles] running in process 763653
2025-06-09 03:37:05,377 INFO [make_surface_current_tiles] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-09 03:37:05,378 INFO [make_surface_current_tiles] publishing log messages to tcp://*:5563
2025-06-09 03:37:05,379 INFO [make_surface_current_tiles] connected to skookum.eos.ubc.ca port 5555
2025-06-09 03:37:05,380 INFO [make_surface_current_tiles] starting rendering of tiles for 2025-06-08 forecast2 in 6 processes into /results/nowcast-sys/figures/surface_currents/forecast2/08jun25
2025-06-09 03:37:05,812 INFO [make_plots] running in process 763652
2025-06-09 03:37:05,815 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-09 03:37:05,817 INFO [make_plots] publishing log messages to tcp://*:5564
2025-06-09 03:37:05,823 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2025-06-09 03:37:06,255 INFO [make_plots] preparing render list for 2025-06-08 NEMO forecast2 publish figures
2025-06-09 03:37:06,382 INFO [make_plots] added Website_thumbnail figure to 2025-06-08 NEMO forecast2 publish render list
2025-06-09 03:37:06,384 INFO [make_plots] added Threshold_website figure to 2025-06-08 NEMO forecast2 publish render list
2025-06-09 03:37:06,385 INFO [make_plots] added PA_tidal_predictions figure to 2025-06-08 NEMO forecast2 publish render list
2025-06-09 03:37:06,387 INFO [make_plots] added Boundary Bay figure to 2025-06-08 NEMO forecast2 publish render list
2025-06-09 03:37:06,389 INFO [make_plots] added Campbell River figure to 2025-06-08 NEMO forecast2 publish render list
2025-06-09 03:37:06,391 INFO [make_plots] added Cherry Point figure to 2025-06-08 NEMO forecast2 publish render list
2025-06-09 03:37:06,393 INFO [make_plots] added Friday Harbor figure to 2025-06-08 NEMO forecast2 publish render list
2025-06-09 03:37:06,397 INFO [make_plots] added Halfmoon Bay figure to 2025-06-08 NEMO forecast2 publish render list
2025-06-09 03:37:06,402 INFO [make_plots] added Nanaimo figure to 2025-06-08 NEMO forecast2 publish render list
2025-06-09 03:37:06,405 INFO [make_plots] added Neah Bay figure to 2025-06-08 NEMO forecast2 publish render list
2025-06-09 03:37:06,409 INFO [make_plots] added New Westminster figure to 2025-06-08 NEMO forecast2 publish render list
2025-06-09 03:37:06,414 INFO [make_plots] added Patricia Bay figure to 2025-06-08 NEMO forecast2 publish render list
2025-06-09 03:37:06,415 INFO [make_plots] added Point Atkinson figure to 2025-06-08 NEMO forecast2 publish render list
2025-06-09 03:37:06,416 INFO [make_plots] added Port Renfrew figure to 2025-06-08 NEMO forecast2 publish render list
2025-06-09 03:37:06,417 INFO [make_plots] added Sand Heads figure to 2025-06-08 NEMO forecast2 publish render list
2025-06-09 03:37:06,418 INFO [make_plots] added Sandy Cove figure to 2025-06-08 NEMO forecast2 publish render list
2025-06-09 03:37:06,419 INFO [make_plots] added Squamish figure to 2025-06-08 NEMO forecast2 publish render list
2025-06-09 03:37:06,420 INFO [make_plots] added Victoria figure to 2025-06-08 NEMO forecast2 publish render list
2025-06-09 03:37:06,421 INFO [make_plots] added Woodwards Landing figure to 2025-06-08 NEMO forecast2 publish render list
2025-06-09 03:37:06,422 INFO [make_plots] starting to render nemo forecast2 publish 08jun25 figures
2025-06-09 03:39:03,644 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 84, 15.6% complete
2025-06-09 03:44:03,645 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 183, 33.9% complete
2025-06-09 03:44:43,551 WARNING [make_plots] invalid station number for Boundary Bay station: None
2025-06-09 03:44:43,552 WARNING [make_plots] can't resolve a valid CHS station code for Boundary Bay
2025-06-09 03:45:08,739 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb996/data for station 08074 Campbell River from 2025-06-04 00:00:00Z to 2025-06-11 05:55:00Z
2025-06-09 03:46:19,626 WARNING [make_plots] invalid station number for Halfmoon Bay station: None
2025-06-09 03:46:19,627 WARNING [make_plots] can't resolve a valid CHS station code for Halfmoon Bay
2025-06-09 03:46:41,623 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb96d/data for station 07917 Nanaimo from 2025-06-04 00:00:00Z to 2025-06-11 05:55:00Z
2025-06-09 03:47:26,767 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e13d0f4a073c4bbf93/data for station 07654 New Westminster from 2025-06-04 00:00:00Z to 2025-06-11 05:55:00Z
2025-06-09 03:47:48,978 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd2d/data for station 07277 Patricia Bay from 2025-06-04 00:00:00Z to 2025-06-11 05:55:00Z
2025-06-09 03:48:12,809 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94c/data for station 07795 Point Atkinson from 2025-06-04 00:00:00Z to 2025-06-11 05:55:00Z
2025-06-09 03:48:23,822 INFO [make_surface_current_tiles] starting PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast2/08jun25
2025-06-09 03:48:30,163 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e23d0f4a073c4bc060/data for station 08525 Port Renfrew from 2025-06-04 00:00:00Z to 2025-06-11 05:55:00Z
2025-06-09 03:48:46,446 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb93c/data for station 07594 Sand Heads from 2025-06-04 00:00:00Z to 2025-06-11 05:55:00Z
2025-06-09 03:49:03,390 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc43b/data for station 07786 Sandy Cove from 2025-06-04 00:00:00Z to 2025-06-11 05:55:00Z
2025-06-09 03:49:03,646 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 279, 51.7% complete
2025-06-09 03:49:20,339 WARNING [make_plots] invalid station number for Squamish station: None
2025-06-09 03:49:20,339 WARNING [make_plots] can't resolve a valid CHS station code for Squamish
2025-06-09 03:49:36,881 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd1e/data for station 07120 Victoria from 2025-06-04 00:00:00Z to 2025-06-11 05:55:00Z
2025-06-09 03:49:53,383 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc42d/data for station 07610 Woodwards Landing from 2025-06-04 00:00:00Z to 2025-06-11 05:55:00Z
2025-06-09 03:50:10,245 INFO [make_plots] finished rendering nemo forecast2 publish 08jun25 figures
2025-06-09 03:50:10,247 INFO [make_plots] nemo publish plots for 2025-06-08 forecast2 completed
2025-06-09 03:50:10,255 INFO [manager] checklist updated with [plots] items from make_plots worker
2025-06-09 03:50:10,628 INFO [manager] launching NextWorker(module='nowcast.workers.make_feeds', args=['forecast2', '--run-date', '2025-06-08'], host='localhost')
2025-06-09 03:50:13,367 INFO [make_feeds] running in process 765662
2025-06-09 03:50:13,368 INFO [make_feeds] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-09 03:50:13,369 INFO [make_feeds] publishing log messages to tcp://*:5564
2025-06-09 03:50:13,370 INFO [make_feeds] connected to skookum.eos.ubc.ca port 5555
2025-06-09 03:50:28,039 INFO [make_feeds] ATOM feeds for 2025-06-08 forecast2 run completed
2025-06-09 03:50:28,042 INFO [manager] checklist updated with [storm surge alert feeds] items from make_feeds worker
2025-06-09 03:50:28,386 INFO [manager] launching NextWorker(module='nemo_nowcast.workers.clear_checklist', args=[], host='localhost')
2025-06-09 03:50:29,338 INFO [clear_checklist] running in process 765685
2025-06-09 03:50:29,339 INFO [clear_checklist] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-09 03:50:29,340 INFO [clear_checklist] publishing log messages to tcp://*:5564
2025-06-09 03:50:29,340 INFO [clear_checklist] connected to skookum.eos.ubc.ca port 5555
2025-06-09 03:50:29,341 INFO [clear_checklist] requesting that manager clear system state checklist
2025-06-09 03:50:29,343 INFO [manager] writing checklist to log file
2025-06-09 03:50:29,386 INFO [manager] checklist cleared
2025-06-09 03:50:29,388 INFO [clear_checklist] nowcast system checklist cleared
2025-06-09 03:50:29,390 INFO [manager] launching NextWorker(module='nemo_nowcast.workers.rotate_logs', args=[], host='localhost')
2025-06-09 03:50:30,310 INFO [rotate_logs] running in process 765690
2025-06-09 03:50:30,311 INFO [rotate_logs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-09 03:50:30,312 INFO [rotate_logs] publishing log messages to tcp://*:5564
2025-06-09 03:50:30,313 INFO [rotate_logs] connected to skookum.eos.ubc.ca port 5555
2025-06-09 03:50:30,313 INFO [rotate_logs] rotating log files