2025-06-13 04:15:14,804 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/nowcast.log
2025-06-13 04:15:14,811 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/nowcast.debug.log
2025-06-13 04:15:14,816 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/checklist.log
2025-06-13 04:15:14,817 INFO [rotate_logs] log files rotated
2025-06-13 04:15:14,821 INFO [manager] checklist updated with [log rotation] items from rotate_logs worker
2025-06-13 04:17:10,207 INFO [make_surface_current_tiles] finished PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast2/12jun25
2025-06-13 04:17:10,231 INFO [make_surface_current_tiles] finished rendering of tiles for 2025-06-12 forecast2 into /results/nowcast-sys/figures/surface_currents/forecast2/12jun25
2025-06-13 04:17:10,232 INFO [make_surface_current_tiles] surface current tile figures for 2025-06-12 forecast2 completed
2025-06-13 04:17:10,541 INFO [manager] checklist updated with [surface current tiles] items from make_surface_current_tiles worker
2025-06-13 09:00:07,662 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/12/ to /results/forcing/atmospheric/continental2.5/GRIB/20250613/12/
2025-06-13 09:00:07,663 INFO [collect_weather] 2025-06-13 2.5km weather forecast 12 collection complete
2025-06-13 09:00:07,666 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2025-06-13 09:00:08,001 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'SkagitMountVernon', '--data-date', '2025-06-12'], host='localhost')
2025-06-13 09:00:08,009 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'SnohomishMonroe', '--data-date', '2025-06-12'], host='localhost')
2025-06-13 09:00:08,014 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'NisquallyMcKenna', '--data-date', '2025-06-12'], host='localhost')
2025-06-13 09:00:08,020 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'GreenwaterGreenwater', '--data-date', '2025-06-12'], host='localhost')
2025-06-13 09:00:08,024 INFO [manager] launching NextWorker(module='nowcast.workers.make_turbidity_file', args=[], host='localhost')
2025-06-13 09:00:08,028 INFO [manager] launching NextWorker(module='nowcast.workers.collect_NeahBay_ssh', args=['06'], host='localhost')
2025-06-13 09:00:08,033 INFO [manager] launching NextWorker(module='nowcast.workers.download_live_ocean', args=[], host='localhost')
2025-06-13 09:00:08,039 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['18', '2.5km'], host='localhost')
2025-06-13 09:00:08,043 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['18'], host='localhost')
2025-06-13 09:00:08,558 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20250613/12/
2025-06-13 09:00:08,559 INFO [crop_gribs] 2025-06-13 12 GRIBs cropping complete
2025-06-13 09:00:08,561 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2025-06-13 09:00:09,214 INFO [manager] launching NextWorker(module='nowcast.workers.grib_to_netcdf', args=['nowcast+'], host='localhost')
2025-06-13 09:00:09,679 INFO [collect_NeahBay_ssh] running in process 855611
2025-06-13 09:00:09,679 INFO [collect_NeahBay_ssh] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-13 09:00:09,680 INFO [collect_NeahBay_ssh] publishing log messages to tcp://*:5562
2025-06-13 09:00:09,681 INFO [collect_NeahBay_ssh] connected to skookum.eos.ubc.ca port 5555
2025-06-13 09:00:09,682 INFO [collect_NeahBay_ssh] collecting Neah Bay ssh 06Z obs/forecast for 2025-06-13
2025-06-13 09:00:09,682 INFO [collect_weather] running in process 855613
2025-06-13 09:00:09,683 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-13 09:00:09,684 INFO [collect_weather] publishing log messages to tcp://*:5563
2025-06-13 09:00:09,685 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2025-06-13 09:00:09,700 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/18/
2025-06-13 09:00:09,806 INFO [download_live_ocean] running in process 855612
2025-06-13 09:00:09,806 INFO [download_live_ocean] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-13 09:00:09,807 INFO [download_live_ocean] publishing log messages to tcp://*:5564
2025-06-13 09:00:09,808 INFO [download_live_ocean] connected to skookum.eos.ubc.ca port 5555
2025-06-13 09:00:09,809 INFO [download_live_ocean] downloading Salish Sea western boundary day-averaged LiveOcean file for 2025-06-13
2025-06-13 09:00:10,156 INFO [collect_river_data] running in process 855605
2025-06-13 09:00:10,157 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-13 09:00:10,157 INFO [collect_river_data] publishing log messages to tcp://*:5565
2025-06-13 09:00:10,158 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-06-13 09:00:10,159 INFO [collect_river_data] Collecting USGS SkagitMountVernon river data for 2025-06-12
2025-06-13 09:00:10,168 INFO [collect_river_data] running in process 855609
2025-06-13 09:00:10,169 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-13 09:00:10,169 INFO [collect_river_data] publishing log messages to tcp://*:5566
2025-06-13 09:00:10,170 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-06-13 09:00:10,171 INFO [collect_river_data] Collecting USGS GreenwaterGreenwater river data for 2025-06-12
2025-06-13 09:00:10,192 INFO [collect_river_data] running in process 855608
2025-06-13 09:00:10,193 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-13 09:00:10,194 INFO [collect_river_data] publishing log messages to tcp://*:5567
2025-06-13 09:00:10,194 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-06-13 09:00:10,195 INFO [collect_river_data] Collecting USGS NisquallyMcKenna river data for 2025-06-12
2025-06-13 09:00:10,301 INFO [collect_river_data] running in process 855607
2025-06-13 09:00:10,302 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-13 09:00:10,303 INFO [collect_river_data] publishing log messages to tcp://*:5568
2025-06-13 09:00:10,304 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-06-13 09:00:10,304 INFO [collect_river_data] Collecting USGS SnohomishMonroe river data for 2025-06-12
2025-06-13 09:00:10,406 INFO [make_turbidity_file] running in process 855610
2025-06-13 09:00:10,407 INFO [make_turbidity_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-13 09:00:10,408 INFO [make_turbidity_file] publishing log messages to tcp://*:5569
2025-06-13 09:00:10,409 INFO [make_turbidity_file] connected to skookum.eos.ubc.ca port 5555
2025-06-13 09:00:10,410 INFO [make_turbidity_file] Creating Fraser River turbidity forcing file for 2025-06-13
2025-06-13 09:00:10,624 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12150800&startDT=2025-06-12&endDT=2025-06-12 "HTTP/1.1 200 "
2025-06-13 09:00:10,644 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12089500&startDT=2025-06-12&endDT=2025-06-12 "HTTP/1.1 200 "
2025-06-13 09:00:10,649 INFO [collect_river_data] Appended USGS SnohomishMonroe river average discharge for 2025-06-12 to: /results/forcing/rivers/observations/Snohomish_Monroe_flow
2025-06-13 09:00:10,650 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12097500&startDT=2025-06-12&endDT=2025-06-12 "HTTP/1.1 200 "
2025-06-13 09:00:10,651 INFO [collect_river_data] USGS SnohomishMonroe river data collection for 2025-06-12 completed
2025-06-13 09:00:10,655 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-06-13 09:00:10,657 INFO [crop_gribs] running in process 855614
2025-06-13 09:00:10,658 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-13 09:00:10,659 INFO [crop_gribs] publishing log messages to tcp://*:5570
2025-06-13 09:00:10,660 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2025-06-13 09:00:10,660 INFO [crop_gribs] cropping 2025-06-13 ECCC HRDPS 2.5km continental 18Z GRIB files to SalishSeaCast subdomain
2025-06-13 09:00:10,668 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12200500&startDT=2025-06-12&endDT=2025-06-12 "HTTP/1.1 200 "
2025-06-13 09:00:10,672 INFO [collect_river_data] Appended USGS SkagitMountVernon river average discharge for 2025-06-12 to: /results/forcing/rivers/observations/Skagit_MountVernon_flow
2025-06-13 09:00:10,673 INFO [collect_river_data] USGS SkagitMountVernon river data collection for 2025-06-12 completed
2025-06-13 09:00:10,674 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20250613/18/
2025-06-13 09:00:10,675 INFO [collect_river_data] Appended USGS GreenwaterGreenwater river average discharge for 2025-06-12 to: /results/forcing/rivers/observations/Greenwater_Greenwater_flow
2025-06-13 09:00:10,676 INFO [collect_river_data] USGS GreenwaterGreenwater river data collection for 2025-06-12 completed
2025-06-13 09:00:10,690 INFO [download_live_ocean] downloaded 10208018 bytes from https://liveocean.apl.uw.edu/output/f2025.06.13/ubc.nc to /results/forcing/LiveOcean/downloaded/20250613/low_passed_UBC.nc
2025-06-13 09:00:10,691 INFO [download_live_ocean] Deflating in up to 1 concurrent sub-processes
2025-06-13 09:00:10,775 INFO [collect_river_data] Appended USGS NisquallyMcKenna river average discharge for 2025-06-12 to: /results/forcing/rivers/observations/Nisqually_McKenna_flow
2025-06-13 09:00:10,775 INFO [collect_river_data] USGS NisquallyMcKenna river data collection for 2025-06-12 completed
2025-06-13 09:00:10,802 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-06-13 09:00:10,962 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-06-13 09:00:11,132 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-06-13 09:00:11,202 INFO [collect_NeahBay_ssh] 2025-06-13T00:00:00-07:00 Neah Bay ssh 06Z obs/forecast data collection complete
2025-06-13 09:00:11,228 INFO [grib_to_netcdf] running in process 855692
2025-06-13 09:00:11,229 INFO [grib_to_netcdf] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-13 09:00:11,230 INFO [grib_to_netcdf] publishing log messages to tcp://*:5565
2025-06-13 09:00:11,231 INFO [grib_to_netcdf] connected to skookum.eos.ubc.ca port 5555
2025-06-13 09:00:11,232 INFO [grib_to_netcdf] creating NEMO-atmos forcing files for 2025-06-13 nowcast and forecast runs
2025-06-13 09:00:11,313 INFO [manager] checklist updated with [Neah Bay ssh data] items from collect_NeahBay_ssh worker
2025-06-13 09:00:11,471 INFO [manager] launching NextWorker(module='nowcast.workers.make_ssh_files', args=['nowcast', '--run-date', '2025-06-13'], host='localhost')
2025-06-13 09:00:11,734 INFO [download_live_ocean] netCDF4 deflated /results/forcing/LiveOcean/downloaded/20250613/low_passed_UBC.nc
2025-06-13 09:00:11,735 INFO [download_live_ocean] 2025-06-13 Live Ocean file for Salish Sea western boundary downloaded
2025-06-13 09:00:11,736 INFO [manager] checklist updated with [Live Ocean products] items from download_live_ocean worker
2025-06-13 09:00:12,089 INFO [manager] launching NextWorker(module='nowcast.workers.make_live_ocean_files', args=['--run-date', '2025-06-13'], host='localhost')
2025-06-13 09:00:12,895 INFO [make_turbidity_file] 2025-06-13 Fraser River turbidity file creation complete
2025-06-13 09:00:12,897 INFO [manager] checklist updated with [Fraser River turbidity] items from make_turbidity_file worker
2025-06-13 09:00:14,069 INFO [make_ssh_files] running in process 855744
2025-06-13 09:00:14,070 INFO [make_ssh_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-13 09:00:14,071 INFO [make_ssh_files] publishing log messages to tcp://*:5562
2025-06-13 09:00:14,072 INFO [make_ssh_files] connected to skookum.eos.ubc.ca port 5555
2025-06-13 09:00:14,073 INFO [make_ssh_files] building nowcast sea surface height boundary conditions file(s) from 2025-06-13 NOAA Neah Bay 06Z observation and forecast values
2025-06-13 09:00:14,095 INFO [make_live_ocean_files] running in process 855757
2025-06-13 09:00:14,097 INFO [make_live_ocean_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-13 09:00:14,097 INFO [make_live_ocean_files] publishing log messages to tcp://*:5564
2025-06-13 09:00:14,099 INFO [make_live_ocean_files] connected to skookum.eos.ubc.ca port 5555
2025-06-13 09:00:14,101 INFO [make_live_ocean_files] Creating T&S western boundary conditions file from 2025-06-13 Live Ocean run
2025-06-13 09:00:15,851 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2025m06d09.nc
2025-06-13 09:00:16,006 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2025m06d10.nc
2025-06-13 09:00:16,172 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2025m06d11.nc
2025-06-13 09:00:16,339 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2025m06d12.nc
2025-06-13 09:00:16,527 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2025m06d13.nc
2025-06-13 09:00:16,708 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2025m06d14.nc
2025-06-13 09:00:16,921 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2025m06d15.nc
2025-06-13 09:00:17,117 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2025m06d16.nc
2025-06-13 09:00:17,243 INFO [make_ssh_files] sea surface height boundary file for 2025-06-13 nowcast run created
2025-06-13 09:00:17,247 INFO [manager] checklist updated with [sea surface height forcing] items from make_ssh_files worker
2025-06-13 09:00:17,409 INFO [manager] launching NextWorker(module='nowcast.workers.make_runoff_file', args=['v202108'], host='localhost')
2025-06-13 09:00:17,414 INFO [manager] launching NextWorker(module='nowcast.workers.make_201702_runoff_file', args=[], host='localhost')
2025-06-13 09:00:18,751 INFO [make_201702_runoff_file] running in process 855790
2025-06-13 09:00:18,752 INFO [make_201702_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-13 09:00:18,753 INFO [make_201702_runoff_file] publishing log messages to tcp://*:5562
2025-06-13 09:00:18,754 INFO [make_201702_runoff_file] connected to skookum.eos.ubc.ca port 5555
2025-06-13 09:00:18,788 INFO [make_201702_runoff_file] calculating runoff forcing for b201702 bathymetry
2025-06-13 09:00:18,886 INFO [make_201702_runoff_file] 2025-06-13 runoff file creation from Fraser at Hope and climatology elsewhere complete
2025-06-13 09:00:18,889 INFO [manager] checklist updated with [rivers forcing] items from make_201702_runoff_file worker
2025-06-13 09:00:19,318 INFO [make_runoff_file] running in process 855788
2025-06-13 09:00:19,320 INFO [make_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-13 09:00:19,320 INFO [make_runoff_file] publishing log messages to tcp://*:5562
2025-06-13 09:00:19,321 INFO [make_runoff_file] connected to skookum.eos.ubc.ca port 5555
2025-06-13 09:00:19,325 INFO [make_runoff_file] calculating NEMO runoff forcing for v202108 bathymetry for 2025-06-12
2025-06-13 09:00:22,388 INFO [make_runoff_file] stored NEMO runoff forcing for v202108 bathymetry for 2025-06-12: /results/forcing/rivers/R202108Dailies_y2025m06d12.nc
2025-06-13 09:00:22,391 INFO [make_runoff_file] 2025-06-12 runoff file creation completed
2025-06-13 09:00:22,392 INFO [manager] checklist updated with [rivers forcing] items from make_runoff_file worker
2025-06-13 09:00:22,428 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/hrdps_y2025m06d13.nc
2025-06-13 09:00:29,940 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2025m06d14.nc
2025-06-13 09:00:34,139 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2025m06d15.nc
2025-06-13 09:00:34,154 INFO [grib_to_netcdf] 2025-06-13 NEMO-atmos forcing file for nowcast+ created
2025-06-13 09:00:34,157 INFO [manager] checklist updated with [weather forcing] items from grib_to_netcdf worker
2025-06-13 09:00:55,802 INFO [make_live_ocean_files] Stored T&S western boundary conditions file: /results/forcing/LiveOcean/boundary_conditions/LiveOcean_v201905_y2025m06d13.nc
2025-06-13 09:00:55,803 INFO [make_live_ocean_files] 2025-06-13 Live Ocean western boundary conditions files created
2025-06-13 09:00:55,805 INFO [manager] checklist updated with [Live Ocean boundary conditions] items from make_live_ocean_files worker
2025-06-13 09:00:55,967 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['weather'], host='localhost')
2025-06-13 09:00:55,970 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'nowcast+'], host='localhost')
2025-06-13 09:00:55,977 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'nowcast+'], host='localhost')
2025-06-13 09:00:55,982 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.graham', 'nowcast+'], host='localhost')
2025-06-13 09:00:55,987 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'nowcast+'], host='localhost')
2025-06-13 09:00:57,065 INFO [ping_erddap] running in process 855847
2025-06-13 09:00:57,066 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-13 09:00:57,067 INFO [ping_erddap] publishing log messages to tcp://*:5562
2025-06-13 09:00:57,068 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2025-06-13 09:00:57,070 INFO [ping_erddap] weather ERDDAP dataset flag file(s) created
2025-06-13 09:00:57,073 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2025-06-13 09:00:57,163 INFO [upload_forcing] running in process 855851
2025-06-13 09:00:57,164 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-13 09:00:57,165 INFO [upload_forcing] publishing log messages to tcp://*:5564
2025-06-13 09:00:57,166 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-06-13 09:00:57,167 INFO [upload_forcing] running in process 855849
2025-06-13 09:00:57,168 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-13 09:00:57,169 INFO [upload_forcing] publishing log messages to tcp://*:5565
2025-06-13 09:00:57,170 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-06-13 09:00:57,179 INFO [upload_forcing] running in process 855852
2025-06-13 09:00:57,181 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-13 09:00:57,182 INFO [upload_forcing] publishing log messages to tcp://*:5566
2025-06-13 09:00:57,183 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-06-13 09:00:57,235 INFO [upload_forcing] running in process 855850
2025-06-13 09:00:57,235 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-13 09:00:57,236 INFO [upload_forcing] publishing log messages to tcp://*:5567
2025-06-13 09:00:57,237 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-06-13 09:00:59,306 INFO [upload_forcing] nowcast+ 2025-06-13 forcing files upload to optimum-hindcast completed
2025-06-13 09:00:59,310 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-06-13 09:00:59,438 INFO [upload_forcing] nowcast+ 2025-06-13 forcing files upload to orcinus-nowcast-agrif completed
2025-06-13 09:00:59,475 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-06-13 09:01:07,696 INFO [upload_forcing] nowcast+ 2025-06-13 forcing files upload to robot.graham completed
2025-06-13 09:01:07,701 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-06-13 09:01:11,121 INFO [upload_forcing] nowcast+ 2025-06-13 forcing files upload to arbutus.cloud-nowcast completed
2025-06-13 09:01:11,126 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-06-13 09:01:11,284 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'nowcast+', '--run-date', '2025-06-13'], host='localhost')
2025-06-13 09:01:12,426 INFO [make_forcing_links] running in process 855889
2025-06-13 09:01:12,429 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-13 09:01:12,431 INFO [make_forcing_links] publishing log messages to tcp://*:5562
2025-06-13 09:01:12,432 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2025-06-13 09:01:15,061 INFO [make_forcing_links] nowcast+ 2025-06-13 forcing file links on arbutus.cloud-nowcast created
2025-06-13 09:01:15,065 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2025-06-13 09:01:15,231 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2025-06-13'], host='arbutus.cloud-nowcast')
2025-06-13 09:01:17,596 INFO [run_NEMO] running in process 23482
2025-06-13 09:01:17,596 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-06-13 09:01:17,599 INFO [run_NEMO] publishing log messages to tcp://*:5556
2025-06-13 09:01:17,819 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2025-06-13 09:01:18,305 WARNING [run_NEMO] There are uncommitted changes in /nemoShare/MEOPAR/nowcast-sys/SS-run-sets
2025-06-13 09:01:18,604 INFO [run_NEMO] nowcast: launching /nemoShare/MEOPAR/nowcast-sys/runs/13jun25nowcast_2025-06-13T090117.636284-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2025-06-13 09:01:18,619 INFO [run_NEMO] nowcast NEMO run for 2025-06-13 on arbutus.cloud-nowcast started
2025-06-13 09:01:18,624 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2025-06-13 09:01:18,790 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'nowcast'], host='arbutus.cloud-nowcast')
2025-06-13 09:01:20,815 INFO [watch_NEMO] running in process 23755
2025-06-13 09:01:20,816 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-06-13 09:01:20,817 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2025-06-13 09:01:20,818 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2025-06-13 09:01:20,987 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2025-06-13 09:06:20,986 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 18505181 = 2025-06-13 05:06:40 UTC, 21.3% complete
2025-06-13 09:11:20,985 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 18505655 = 2025-06-13 10:22:40 UTC, 43.3% complete
2025-06-13 09:16:20,986 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 18506160 = 2025-06-13 15:59:20 UTC, 66.7% complete
2025-06-13 09:21:20,986 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 18506647 = 2025-06-13 21:24:00 UTC, 89.2% complete
2025-06-13 09:26:20,991 INFO [watch_NEMO] nowcast NEMO run on arbutus.cloud-nowcast completed
2025-06-13 09:26:21,000 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2025-06-13 09:26:21,319 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'ssh', '--run-date', '2025-06-13'], host='localhost')
2025-06-13 09:26:21,326 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2025-06-13'], host='localhost')
2025-06-13 09:26:22,554 INFO [download_results] running in process 856203
2025-06-13 09:26:22,555 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-13 09:26:22,557 INFO [download_results] publishing log messages to tcp://*:5562
2025-06-13 09:26:22,558 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2025-06-13 09:26:22,559 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/nowcast/13jun25 to /results/SalishSea/nowcast-blue.202111
2025-06-13 09:26:22,576 INFO [make_forcing_links] running in process 856202
2025-06-13 09:26:22,578 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-13 09:26:22,579 INFO [make_forcing_links] publishing log messages to tcp://*:5564
2025-06-13 09:26:22,580 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2025-06-13 09:26:24,008 INFO [make_forcing_links] ssh 2025-06-13 forcing file links on arbutus.cloud-nowcast created
2025-06-13 09:26:24,012 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2025-06-13 09:26:24,169 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2025-06-13'], host='arbutus.cloud-nowcast')
2025-06-13 09:26:26,350 INFO [run_NEMO] running in process 24152
2025-06-13 09:26:26,351 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-06-13 09:26:26,353 INFO [run_NEMO] publishing log messages to tcp://*:5556
2025-06-13 09:26:26,355 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2025-06-13 09:26:27,404 WARNING [run_NEMO] There are uncommitted changes in /nemoShare/MEOPAR/nowcast-sys/SS-run-sets
2025-06-13 09:26:27,719 INFO [run_NEMO] forecast: launching /nemoShare/MEOPAR/nowcast-sys/runs/13jun25forecast_2025-06-13T092626.604589-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2025-06-13 09:26:27,743 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2025-06-13 09:26:27,748 INFO [run_NEMO] forecast NEMO run for 2025-06-13 on arbutus.cloud-nowcast started
2025-06-13 09:26:27,910 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'forecast'], host='arbutus.cloud-nowcast')
2025-06-13 09:26:29,954 INFO [watch_NEMO] running in process 24427
2025-06-13 09:26:29,956 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-06-13 09:26:29,958 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2025-06-13 09:26:29,959 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2025-06-13 09:26:30,055 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2025-06-13 09:31:30,051 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 18507336 = 2025-06-14 05:03:20 UTC, 14.0% complete
2025-06-13 09:32:37,357 INFO [download_results] nowcast 2025-06-13 results files from arbutus.cloud-nowcast downloaded
2025-06-13 09:32:37,362 INFO [manager] checklist updated with [results files] items from download_results worker
2025-06-13 09:32:37,521 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast', 'research', '--run-date', '2025-06-13'], host='localhost')
2025-06-13 09:32:37,527 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast', 'comparison', '--run-date', '2025-06-12'], host='localhost')
2025-06-13 09:32:37,532 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['nowcast', '--run-date', '2025-06-13'], host='localhost')
2025-06-13 09:32:39,562 INFO [make_CHS_currents_file] running in process 856299
2025-06-13 09:32:39,563 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-13 09:32:39,564 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5562
2025-06-13 09:32:39,565 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555
2025-06-13 09:32:40,693 INFO [make_plots] running in process 856298
2025-06-13 09:32:40,694 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-13 09:32:40,695 INFO [make_plots] publishing log messages to tcp://*:5564
2025-06-13 09:32:40,696 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2025-06-13 09:32:40,710 INFO [make_plots] running in process 856296
2025-06-13 09:32:40,740 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-13 09:32:40,742 INFO [make_plots] publishing log messages to tcp://*:5565
2025-06-13 09:32:40,743 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2025-06-13 09:32:41,097 INFO [make_plots] preparing render list for 2025-06-13 NEMO nowcast-blue research figures
2025-06-13 09:32:41,122 INFO [make_plots] preparing render list for 2025-06-12 NEMO nowcast-blue comparison figures
2025-06-13 09:32:41,162 INFO [make_plots] added SH_wind figure to 2025-06-12 NEMO nowcast-blue comparison render list
2025-06-13 09:32:41,163 INFO [make_plots] added Compare_VENUS_East figure to 2025-06-12 NEMO nowcast-blue comparison render list
2025-06-13 09:32:41,163 INFO [make_plots] added Compare_VENUS_Central figure to 2025-06-12 NEMO nowcast-blue comparison render list
2025-06-13 09:32:41,164 INFO [make_plots] starting to render nemo nowcast comparison 12jun25 figures
2025-06-13 09:32:42,605 INFO [make_plots] added salinity_thalweg_and_surface figures to 2025-06-13 NEMO nowcast-blue research render list
2025-06-13 09:32:43,442 INFO [make_plots] added temperature_thalweg_and_surface figures to 2025-06-13 NEMO nowcast-blue research render list
2025-06-13 09:32:43,443 INFO [make_plots] added Currents_sections_and_surface figure to 2025-06-13 NEMO nowcast-blue research render list
2025-06-13 09:32:43,444 INFO [make_plots] added Currents_at_VENUS_Central figure to 2025-06-13 NEMO nowcast-blue research render list
2025-06-13 09:32:43,444 INFO [make_plots] added Currents_at_VENUS_East figure to 2025-06-13 NEMO nowcast-blue research render list
2025-06-13 09:32:43,445 INFO [make_plots] starting to render nemo nowcast research 13jun25 figures
2025-06-13 09:33:10,882 INFO [make_CHS_currents_file] Made CHS currents file for 2025-06-13 for nowcast
2025-06-13 09:33:10,885 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker
2025-06-13 09:33:49,680 INFO [make_plots] finished rendering nemo nowcast research 13jun25 figures
2025-06-13 09:33:49,683 INFO [make_plots] nemo research plots for 2025-06-13 nowcast completed
2025-06-13 09:33:49,699 INFO [manager] checklist updated with [plots] items from make_plots worker
2025-06-13 09:34:10,311 INFO [make_plots] finished rendering nemo nowcast comparison 12jun25 figures
2025-06-13 09:34:10,314 INFO [make_plots] nemo comparison plots for 2025-06-12 nowcast completed
2025-06-13 09:34:10,318 INFO [manager] checklist updated with [plots] items from make_plots worker
2025-06-13 09:36:30,053 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 18507802 = 2025-06-14 10:14:00 UTC, 28.4% complete
2025-06-13 09:41:30,055 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 18508305 = 2025-06-14 15:49:20 UTC, 44.0% complete
2025-06-13 09:46:30,059 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 18508779 = 2025-06-14 21:05:20 UTC, 58.6% complete
2025-06-13 09:51:30,058 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 18509243 = 2025-06-15 02:14:40 UTC, 72.9% complete
2025-06-13 09:56:30,061 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 18509717 = 2025-06-15 07:30:40 UTC, 87.6% complete
2025-06-13 10:01:30,064 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 18510120 = 2025-06-15 11:59:20 UTC, 100.0% complete
2025-06-13 10:06:30,068 INFO [watch_NEMO] forecast NEMO run on arbutus.cloud-nowcast completed
2025-06-13 10:06:30,072 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2025-06-13 10:06:30,412 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_wind_file', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2025-06-13'], host='arbutus.cloud-nowcast')
2025-06-13 10:06:30,421 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_current_file', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2025-06-13'], host='arbutus.cloud-nowcast')
2025-06-13 10:06:30,426 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'turbidity', '--run-date', '2025-06-13'], host='localhost')
2025-06-13 10:06:30,432 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'turbidity', '--run-date', '2025-06-13'], host='localhost')
2025-06-13 10:06:30,437 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.graham', 'turbidity', '--run-date', '2025-06-13'], host='localhost')
2025-06-13 10:06:30,441 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'turbidity', '--run-date', '2025-06-13'], host='localhost')
2025-06-13 10:06:30,445 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2025-06-13'], host='localhost')
2025-06-13 10:06:31,650 INFO [download_results] running in process 856807
2025-06-13 10:06:31,652 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-13 10:06:31,653 INFO [download_results] publishing log messages to tcp://*:5562
2025-06-13 10:06:31,654 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2025-06-13 10:06:31,655 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/forecast/13jun25 to /results/SalishSea/forecast.202111
2025-06-13 10:06:31,656 INFO [upload_forcing] running in process 856805
2025-06-13 10:06:31,657 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-13 10:06:31,658 INFO [upload_forcing] publishing log messages to tcp://*:5564
2025-06-13 10:06:31,659 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-06-13 10:06:31,676 INFO [upload_forcing] running in process 856803
2025-06-13 10:06:31,677 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-13 10:06:31,678 INFO [upload_forcing] publishing log messages to tcp://*:5565
2025-06-13 10:06:31,679 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-06-13 10:06:31,686 INFO [upload_forcing] running in process 856806
2025-06-13 10:06:31,687 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-13 10:06:31,688 INFO [upload_forcing] publishing log messages to tcp://*:5566
2025-06-13 10:06:31,689 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-06-13 10:06:31,708 INFO [upload_forcing] running in process 856804
2025-06-13 10:06:31,709 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-13 10:06:31,710 INFO [upload_forcing] publishing log messages to tcp://*:5567
2025-06-13 10:06:31,711 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-06-13 10:06:32,377 INFO [upload_forcing] turbidity 2025-06-13 forcing files upload to optimum-hindcast completed
2025-06-13 10:06:32,381 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-06-13 10:06:32,436 INFO [upload_forcing] turbidity 2025-06-13 forcing files upload to orcinus-nowcast-agrif completed
2025-06-13 10:06:32,550 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-06-13 10:06:32,710 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['orcinus-nowcast-agrif', 'nowcast-agrif', '--run-date', '2025-06-13'], host='localhost')
2025-06-13 10:06:33,509 INFO [make_ww3_wind_file] running in process 25019
2025-06-13 10:06:33,510 INFO [make_ww3_wind_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-06-13 10:06:33,511 INFO [make_ww3_wind_file] publishing log messages to tcp://*:5570
2025-06-13 10:06:33,511 INFO [make_ww3_wind_file] connected to skookum.eos.ubc.ca port 5555
2025-06-13 10:06:33,512 INFO [make_ww3_wind_file] Creating wwatch3 wind forcing file for 2025-06-13 forecast run
2025-06-13 10:06:33,850 INFO [make_forcing_links] running in process 856839
2025-06-13 10:06:33,851 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-13 10:06:33,853 INFO [make_forcing_links] publishing log messages to tcp://*:5566
2025-06-13 10:06:33,854 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2025-06-13 10:06:34,001 INFO [upload_forcing] turbidity 2025-06-13 forcing files upload to arbutus.cloud-nowcast completed
2025-06-13 10:06:34,005 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-06-13 10:06:34,176 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2025-06-13'], host='localhost')
2025-06-13 10:06:34,663 INFO [make_forcing_links] nowcast-agrif 2025-06-13 forcing file links on orcinus-nowcast-agrif created
2025-06-13 10:06:34,667 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2025-06-13 10:06:34,840 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_agrif', args=['orcinus-nowcast-agrif', 'nowcast-agrif', '--run-date', '2025-06-13'], host='localhost')
2025-06-13 10:06:35,236 INFO [make_forcing_links] running in process 856848
2025-06-13 10:06:35,239 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-13 10:06:35,240 INFO [make_forcing_links] publishing log messages to tcp://*:5565
2025-06-13 10:06:35,240 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2025-06-13 10:06:35,966 INFO [run_NEMO_agrif] running in process 856851
2025-06-13 10:06:35,967 INFO [run_NEMO_agrif] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-13 10:06:35,969 INFO [run_NEMO_agrif] publishing log messages to tcp://*:5566
2025-06-13 10:06:35,970 INFO [run_NEMO_agrif] connected to skookum.eos.ubc.ca port 5555
2025-06-13 10:06:36,377 INFO [upload_forcing] turbidity 2025-06-13 forcing files upload to robot.graham completed
2025-06-13 10:06:36,380 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-06-13 10:06:37,453 INFO [make_forcing_links] nowcast-green 2025-06-13 forcing file links on arbutus.cloud-nowcast created
2025-06-13 10:06:37,458 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2025-06-13 10:06:37,636 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2025-06-13'], host='arbutus.cloud-nowcast')
2025-06-13 10:06:41,984 INFO [run_NEMO] running in process 25629
2025-06-13 10:06:41,985 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-06-13 10:06:41,986 INFO [run_NEMO] publishing log messages to tcp://*:5556
2025-06-13 10:06:41,987 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2025-06-13 10:06:43,491 WARNING [run_NEMO] There are uncommitted changes in /nemoShare/MEOPAR/nowcast-sys/SS-run-sets
2025-06-13 10:06:43,823 INFO [run_NEMO] nowcast-green: launching /nemoShare/MEOPAR/nowcast-sys/runs/13jun25nowcast-green_2025-06-13T100642.082459-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2025-06-13 10:06:43,836 INFO [run_NEMO] nowcast-green NEMO run for 2025-06-13 on arbutus.cloud-nowcast started
2025-06-13 10:06:43,846 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2025-06-13 10:06:44,028 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'nowcast-green'], host='arbutus.cloud-nowcast')
2025-06-13 10:06:44,609 INFO [make_ww3_wind_file] wwatch3 wind forcing file created on arbutus.cloud-nowcast for 2025-06-13 forecast run
2025-06-13 10:06:44,628 INFO [manager] checklist updated with [WW3 wind forcing] items from make_ww3_wind_file worker
2025-06-13 10:06:46,324 INFO [watch_NEMO] running in process 25915
2025-06-13 10:06:46,325 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-06-13 10:06:46,326 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2025-06-13 10:06:46,327 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2025-06-13 10:06:46,419 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2025-06-13 10:06:51,000 INFO [run_NEMO_agrif] job id for 13jun25nowcast-agrif: 11429894.orca2.ibb
2025-06-13 10:06:51,001 INFO [run_NEMO_agrif] NEMO nowcast-agrif run for 2025-06-13 queued on orcinus-nowcast-agrif
2025-06-13 10:06:51,006 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_agrif worker
2025-06-13 10:06:51,174 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_agrif', args=['orcinus-nowcast-agrif', '11429894.orca2.ibb'], host='localhost')
2025-06-13 10:06:52,607 INFO [watch_NEMO_agrif] running in process 856877
2025-06-13 10:06:52,608 INFO [watch_NEMO_agrif] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-13 10:06:52,609 INFO [watch_NEMO_agrif] publishing log messages to tcp://*:5564
2025-06-13 10:06:52,610 INFO [watch_NEMO_agrif] connected to skookum.eos.ubc.ca port 5555
2025-06-13 10:06:53,429 INFO [watch_NEMO_agrif] watching 13jun25nowcast-agrif job 11429894 on orcinus-nowcast-agrif
2025-06-13 10:06:54,553 INFO [watch_NEMO_agrif] 13jun25nowcast-agrif on orcinus-nowcast-agrif: time.step not found; continuing to watch...
2025-06-13 10:09:07,722 INFO [manager] checklist updated with [WW3 currents forcing] items from make_ww3_current_file worker
2025-06-13 10:09:07,723 INFO [make_ww3_current_file] wwatch3 currents forcing file created on arbutus.cloud-nowcast for 2025-06-13 forecast run
2025-06-13 10:09:07,798 INFO [download_results] forecast 2025-06-13 results files from arbutus.cloud-nowcast downloaded
2025-06-13 10:09:07,893 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2025-06-13'], host='arbutus.cloud-nowcast')
2025-06-13 10:09:07,902 INFO [manager] checklist updated with [results files] items from download_results worker
2025-06-13 10:09:08,066 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['forecast', '--run-date', '2025-06-13'], host='localhost')
2025-06-13 10:09:10,277 INFO [make_CHS_currents_file] running in process 856930
2025-06-13 10:09:10,278 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-13 10:09:10,279 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5562
2025-06-13 10:09:10,280 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555
2025-06-13 10:09:10,412 INFO [run_ww3] running in process 26032
2025-06-13 10:09:10,413 INFO [run_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-06-13 10:09:10,414 INFO [run_ww3] publishing log messages to tcp://*:5572
2025-06-13 10:09:10,415 INFO [run_ww3] connected to skookum.eos.ubc.ca port 5555
2025-06-13 10:09:10,420 INFO [run_ww3] Created run directory /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/nowcast_2025-06-13T100910.408056-0700
2025-06-13 10:09:10,422 INFO [run_ww3] nowcast: launching /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/nowcast_2025-06-13T100910.408056-0700/SoGWW3.sh on arbutus.cloud-nowcast
2025-06-13 10:09:10,433 INFO [run_ww3] nowcast WaveWatch3 run for 2025-06-13 on arbutus.cloud-nowcast started
2025-06-13 10:09:10,541 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker
2025-06-13 10:09:10,707 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'nowcast'], host='arbutus.cloud-nowcast')
2025-06-13 10:09:12,761 INFO [watch_ww3] running in process 26119
2025-06-13 10:09:12,764 INFO [watch_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-06-13 10:09:12,765 INFO [watch_ww3] publishing log messages to tcp://*:5573
2025-06-13 10:09:12,766 INFO [watch_ww3] connected to skookum.eos.ubc.ca port 5555
2025-06-13 10:09:12,869 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2025-06-13 10:09:56,039 INFO [make_CHS_currents_file] Made CHS currents file for 2025-06-13 for forecast
2025-06-13 10:09:56,042 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker
2025-06-13 10:09:56,214 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['nemo', 'forecast', '--run-date', '2025-06-13'], host='localhost')
2025-06-13 10:09:57,230 INFO [update_forecast_datasets] running in process 856955
2025-06-13 10:09:57,231 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-13 10:09:57,232 INFO [update_forecast_datasets] publishing log messages to tcp://*:5562
2025-06-13 10:09:57,233 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2025-06-13 10:09:57,234 INFO [update_forecast_datasets] updating nemo forecast directory for forecast/13jun25 run
2025-06-13 10:09:57,516 INFO [update_forecast_datasets] updated nemo forecast directory for forecast/13jun25 run: /results/SalishSea/rolling-forecasts/nemo
2025-06-13 10:09:57,517 INFO [update_forecast_datasets] nemo 2025-06-13 forecast rolling forecast datasets updated
2025-06-13 10:09:57,520 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2025-06-13 10:09:57,685 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nemo-forecast'], host='localhost')
2025-06-13 10:09:57,691 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'forecast', 'publish', '--run-date', '2025-06-13'], host='localhost')
2025-06-13 10:09:57,698 INFO [manager] launching NextWorker(module='nowcast.workers.make_surface_current_tiles', args=['forecast', '--run-date', '2025-06-13'], host='localhost')
2025-06-13 10:09:58,885 INFO [ping_erddap] running in process 856960
2025-06-13 10:09:58,885 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-13 10:09:58,886 INFO [ping_erddap] publishing log messages to tcp://*:5562
2025-06-13 10:09:58,887 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2025-06-13 10:09:58,896 INFO [ping_erddap] nemo-forecast ERDDAP dataset flag file(s) created
2025-06-13 10:09:58,897 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2025-06-13 10:09:59,952 INFO [make_surface_current_tiles] running in process 856963
2025-06-13 10:09:59,953 INFO [make_surface_current_tiles] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-13 10:09:59,954 INFO [make_surface_current_tiles] publishing log messages to tcp://*:5562
2025-06-13 10:09:59,955 INFO [make_surface_current_tiles] connected to skookum.eos.ubc.ca port 5555
2025-06-13 10:09:59,957 INFO [make_surface_current_tiles] starting rendering of tiles for 2025-06-13 forecast in 6 processes into /results/nowcast-sys/figures/surface_currents/forecast/13jun25
2025-06-13 10:10:00,908 INFO [make_plots] running in process 856962
2025-06-13 10:10:00,910 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-13 10:10:00,911 INFO [make_plots] publishing log messages to tcp://*:5565
2025-06-13 10:10:00,917 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2025-06-13 10:10:01,381 INFO [make_plots] preparing render list for 2025-06-13 NEMO forecast publish figures
2025-06-13 10:10:01,453 INFO [make_plots] added Website_thumbnail figure to 2025-06-13 NEMO forecast publish render list
2025-06-13 10:10:01,454 INFO [make_plots] added Threshold_website figure to 2025-06-13 NEMO forecast publish render list
2025-06-13 10:10:01,455 INFO [make_plots] added PA_tidal_predictions figure to 2025-06-13 NEMO forecast publish render list
2025-06-13 10:10:01,455 INFO [make_plots] added Boundary Bay figure to 2025-06-13 NEMO forecast publish render list
2025-06-13 10:10:01,456 INFO [make_plots] added Campbell River figure to 2025-06-13 NEMO forecast publish render list
2025-06-13 10:10:01,457 INFO [make_plots] added Cherry Point figure to 2025-06-13 NEMO forecast publish render list
2025-06-13 10:10:01,457 INFO [make_plots] added Friday Harbor figure to 2025-06-13 NEMO forecast publish render list
2025-06-13 10:10:01,458 INFO [make_plots] added Halfmoon Bay figure to 2025-06-13 NEMO forecast publish render list
2025-06-13 10:10:01,459 INFO [make_plots] added Nanaimo figure to 2025-06-13 NEMO forecast publish render list
2025-06-13 10:10:01,459 INFO [make_plots] added Neah Bay figure to 2025-06-13 NEMO forecast publish render list
2025-06-13 10:10:01,460 INFO [make_plots] added New Westminster figure to 2025-06-13 NEMO forecast publish render list
2025-06-13 10:10:01,461 INFO [make_plots] added Patricia Bay figure to 2025-06-13 NEMO forecast publish render list
2025-06-13 10:10:01,461 INFO [make_plots] added Point Atkinson figure to 2025-06-13 NEMO forecast publish render list
2025-06-13 10:10:01,462 INFO [make_plots] added Port Renfrew figure to 2025-06-13 NEMO forecast publish render list
2025-06-13 10:10:01,462 INFO [make_plots] added Sand Heads figure to 2025-06-13 NEMO forecast publish render list
2025-06-13 10:10:01,463 INFO [make_plots] added Sandy Cove figure to 2025-06-13 NEMO forecast publish render list
2025-06-13 10:10:01,464 INFO [make_plots] added Squamish figure to 2025-06-13 NEMO forecast publish render list
2025-06-13 10:10:01,464 INFO [make_plots] added Victoria figure to 2025-06-13 NEMO forecast publish render list
2025-06-13 10:10:01,465 INFO [make_plots] added Woodwards Landing figure to 2025-06-13 NEMO forecast publish render list
2025-06-13 10:10:01,466 INFO [make_plots] starting to render nemo forecast publish 13jun25 figures
2025-06-13 10:11:46,424 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18504883 = 2025-06-13 01:48:00 UTC, 7.5% complete
2025-06-13 10:11:55,029 INFO [watch_NEMO_agrif] 13jun25nowcast-agrif on orcinus-nowcast-agrif: timestep: 8525682 = 2025-06-13 01:47:20 UTC, 7.5% complete
2025-06-13 10:14:12,884 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 93, 21.5% complete
2025-06-13 10:16:46,423 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18505060 = 2025-06-13 03:46:00 UTC, 15.7% complete
2025-06-13 10:16:55,475 INFO [watch_NEMO_agrif] 13jun25nowcast-agrif on orcinus-nowcast-agrif: timestep: 8525886 = 2025-06-13 04:03:20 UTC, 16.9% complete
2025-06-13 10:18:45,182 WARNING [make_plots] invalid station number for Boundary Bay station: None
2025-06-13 10:18:45,183 WARNING [make_plots] can't resolve a valid CHS station code for Boundary Bay
2025-06-13 10:19:12,637 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb996/data for station 08074 Campbell River from 2025-06-08 00:00:00Z to 2025-06-15 11:55:00Z
2025-06-13 10:19:12,871 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 201, 46.5% complete
2025-06-13 10:20:29,739 WARNING [make_plots] invalid station number for Halfmoon Bay station: None
2025-06-13 10:20:29,739 WARNING [make_plots] can't resolve a valid CHS station code for Halfmoon Bay
2025-06-13 10:20:50,957 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb96d/data for station 07917 Nanaimo from 2025-06-08 00:00:00Z to 2025-06-15 11:55:00Z
2025-06-13 10:21:09,446 INFO [make_surface_current_tiles] starting PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast/13jun25
2025-06-13 10:21:28,609 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e13d0f4a073c4bbf93/data for station 07654 New Westminster from 2025-06-08 00:00:00Z to 2025-06-15 11:55:00Z
2025-06-13 10:21:46,424 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18505240 = 2025-06-13 05:46:00 UTC, 24.0% complete
2025-06-13 10:21:46,886 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd2d/data for station 07277 Patricia Bay from 2025-06-08 00:00:00Z to 2025-06-15 11:55:00Z
2025-06-13 10:21:55,918 INFO [watch_NEMO_agrif] 13jun25nowcast-agrif on orcinus-nowcast-agrif: timestep: 8526093 = 2025-06-13 06:21:20 UTC, 26.5% complete
2025-06-13 10:22:05,505 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94c/data for station 07795 Point Atkinson from 2025-06-08 00:00:00Z to 2025-06-15 11:55:00Z
2025-06-13 10:22:24,176 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e23d0f4a073c4bc060/data for station 08525 Port Renfrew from 2025-06-08 00:00:00Z to 2025-06-15 11:55:00Z
2025-06-13 10:22:41,405 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb93c/data for station 07594 Sand Heads from 2025-06-08 00:00:00Z to 2025-06-15 11:55:00Z
2025-06-13 10:23:00,519 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc43b/data for station 07786 Sandy Cove from 2025-06-08 00:00:00Z to 2025-06-15 11:55:00Z
2025-06-13 10:23:19,025 WARNING [make_plots] invalid station number for Squamish station: None
2025-06-13 10:23:19,025 WARNING [make_plots] can't resolve a valid CHS station code for Squamish
2025-06-13 10:23:36,803 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd1e/data for station 07120 Victoria from 2025-06-08 00:00:00Z to 2025-06-15 11:55:00Z
2025-06-13 10:23:53,903 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc42d/data for station 07610 Woodwards Landing from 2025-06-08 00:00:00Z to 2025-06-15 11:55:00Z
2025-06-13 10:24:12,759 INFO [make_plots] finished rendering nemo forecast publish 13jun25 figures
2025-06-13 10:24:12,762 INFO [make_plots] nemo publish plots for 2025-06-13 forecast completed
2025-06-13 10:24:12,768 INFO [manager] checklist updated with [plots] items from make_plots worker
2025-06-13 10:24:12,875 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 306, 70.8% complete
2025-06-13 10:24:12,964 INFO [manager] launching NextWorker(module='nowcast.workers.make_feeds', args=['forecast', '--run-date', '2025-06-13'], host='localhost')
2025-06-13 10:24:18,029 INFO [make_feeds] running in process 858904
2025-06-13 10:24:18,030 INFO [make_feeds] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-13 10:24:18,032 INFO [make_feeds] publishing log messages to tcp://*:5565
2025-06-13 10:24:18,033 INFO [make_feeds] connected to skookum.eos.ubc.ca port 5555
2025-06-13 10:24:35,615 INFO [make_feeds] ATOM feeds for 2025-06-13 forecast run completed
2025-06-13 10:24:35,618 INFO [manager] checklist updated with [storm surge alert feeds] items from make_feeds worker
2025-06-13 10:26:09,318 INFO [make_surface_current_tiles] finished PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast/13jun25
2025-06-13 10:26:09,343 INFO [make_surface_current_tiles] finished rendering of tiles for 2025-06-13 forecast into /results/nowcast-sys/figures/surface_currents/forecast/13jun25
2025-06-13 10:26:09,344 INFO [make_surface_current_tiles] surface current tile figures for 2025-06-13 forecast completed
2025-06-13 10:26:09,628 INFO [manager] checklist updated with [surface current tiles] items from make_surface_current_tiles worker
2025-06-13 10:26:46,427 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18505419 = 2025-06-13 07:45:20 UTC, 32.3% complete
2025-06-13 10:26:56,360 INFO [watch_NEMO_agrif] 13jun25nowcast-agrif on orcinus-nowcast-agrif: timestep: 8526302 = 2025-06-13 08:40:40 UTC, 36.2% complete
2025-06-13 10:29:12,877 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 414, 95.8% complete
2025-06-13 10:31:46,435 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18505598 = 2025-06-13 09:44:40 UTC, 40.6% complete
2025-06-13 10:31:56,802 INFO [watch_NEMO_agrif] 13jun25nowcast-agrif on orcinus-nowcast-agrif: timestep: 8526509 = 2025-06-13 10:58:40 UTC, 45.8% complete
2025-06-13 10:34:12,879 INFO [watch_ww3] nowcast WWATCH3 run on arbutus.cloud-nowcast completed
2025-06-13 10:34:12,894 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2025-06-13 10:34:13,390 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2025-06-13'], host='localhost')
2025-06-13 10:34:13,398 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2025-06-13'], host='arbutus.cloud-nowcast')
2025-06-13 10:34:15,166 INFO [download_wwatch3_results] running in process 859061
2025-06-13 10:34:15,189 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-13 10:34:15,194 INFO [download_wwatch3_results] publishing log messages to tcp://*:5562
2025-06-13 10:34:15,195 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2025-06-13 10:34:15,621 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker
2025-06-13 10:34:15,902 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'forecast'], host='arbutus.cloud-nowcast')
2025-06-13 10:34:19,562 INFO [watch_ww3] running in process 26542
2025-06-13 10:34:19,565 INFO [watch_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-06-13 10:34:19,567 INFO [watch_ww3] publishing log messages to tcp://*:5573
2025-06-13 10:34:19,568 INFO [watch_ww3] connected to skookum.eos.ubc.ca port 5555
2025-06-13 10:34:19,651 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2025-06-13 10:34:46,762 INFO [download_wwatch3_results] nowcast 2025-06-13 results files from arbutus.cloud-nowcast downloaded
2025-06-13 10:34:46,765 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2025-06-13 10:36:46,433 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18505774 = 2025-06-13 11:42:00 UTC, 48.8% complete
2025-06-13 10:36:57,245 INFO [watch_NEMO_agrif] 13jun25nowcast-agrif on orcinus-nowcast-agrif: timestep: 8526714 = 2025-06-13 13:15:20 UTC, 55.3% complete
2025-06-13 10:39:19,639 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 93, 14.4% complete
2025-06-13 10:41:46,433 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18505951 = 2025-06-13 13:40:00 UTC, 57.0% complete
2025-06-13 10:41:57,688 INFO [watch_NEMO_agrif] 13jun25nowcast-agrif on orcinus-nowcast-agrif: timestep: 8526922 = 2025-06-13 15:34:00 UTC, 64.9% complete
2025-06-13 10:44:19,643 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 201, 31.0% complete
2025-06-13 10:46:46,437 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18506137 = 2025-06-13 15:44:00 UTC, 65.6% complete
2025-06-13 10:46:58,164 INFO [watch_NEMO_agrif] 13jun25nowcast-agrif on orcinus-nowcast-agrif: timestep: 8527129 = 2025-06-13 17:52:00 UTC, 74.5% complete
2025-06-13 10:49:19,654 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 309, 47.7% complete
2025-06-13 10:51:46,434 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18506316 = 2025-06-13 17:43:20 UTC, 73.9% complete
2025-06-13 10:51:58,603 INFO [watch_NEMO_agrif] 13jun25nowcast-agrif on orcinus-nowcast-agrif: timestep: 8527334 = 2025-06-13 20:08:40 UTC, 84.0% complete
2025-06-13 10:54:19,644 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 417, 64.4% complete
2025-06-13 10:56:46,440 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18506492 = 2025-06-13 19:40:40 UTC, 82.0% complete
2025-06-13 10:56:59,036 INFO [watch_NEMO_agrif] 13jun25nowcast-agrif on orcinus-nowcast-agrif: timestep: 8527541 = 2025-06-13 22:26:40 UTC, 93.6% complete
2025-06-13 10:59:19,641 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 522, 80.6% complete
2025-06-13 11:01:46,452 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18506670 = 2025-06-13 21:39:20 UTC, 90.3% complete
2025-06-13 11:01:59,463 INFO [watch_NEMO_agrif] 13jun25nowcast-agrif on orcinus-nowcast-agrif: timestep: 8527680 = 2025-06-13 23:59:20 UTC, 100.0% complete
2025-06-13 11:04:19,643 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 630, 97.2% complete
2025-06-13 11:06:46,435 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18506846 = 2025-06-13 23:36:40 UTC, 98.4% complete
2025-06-13 11:06:59,685 INFO [watch_NEMO_agrif] NEMO AGRIF run 11429894.orca2.ibb on orcinus-nowcast-agrif completed
2025-06-13 11:06:59,689 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_agrif worker
2025-06-13 11:07:00,182 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['orcinus-nowcast-agrif', 'nowcast-agrif', '--run-date', '2025-06-13'], host='localhost')
2025-06-13 11:07:02,281 INFO [download_results] running in process 859499
2025-06-13 11:07:02,298 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-13 11:07:02,333 INFO [download_results] publishing log messages to tcp://*:5562
2025-06-13 11:07:02,343 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2025-06-13 11:07:02,344 INFO [download_results] downloading results from orcinus-nowcast-agrif:/global/scratch/dlatorne/nowcast-agrif/13jun25 to /results/SalishSea/nowcast-agrif.201702
2025-06-13 11:07:24,896 INFO [download_results] nowcast-agrif 2025-06-13 results files from orcinus-nowcast-agrif downloaded
2025-06-13 11:07:24,902 INFO [manager] checklist updated with [results files] items from download_results worker
2025-06-13 11:07:25,254 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast-agrif', 'research', '--run-date', '2025-06-13'], host='localhost')
2025-06-13 11:07:30,113 INFO [make_plots] running in process 859533
2025-06-13 11:07:30,116 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-13 11:07:30,117 INFO [make_plots] publishing log messages to tcp://*:5562
2025-06-13 11:07:30,118 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2025-06-13 11:07:30,497 INFO [make_plots] preparing render list for 2025-06-13 NEMO nowcast-agrif research figures
2025-06-13 11:07:30,499 INFO [make_plots] added baynes_sound_surface figure to 2025-06-13 NEMO nowcast-agrif research render list
2025-06-13 11:07:30,500 INFO [make_plots] starting to render nemo nowcast-agrif research 13jun25 figures
2025-06-13 11:07:40,085 INFO [make_plots] finished rendering nemo nowcast-agrif research 13jun25 figures
2025-06-13 11:07:40,088 INFO [make_plots] nemo research plots for 2025-06-13 nowcast-agrif completed
2025-06-13 11:07:40,092 INFO [manager] checklist updated with [plots] items from make_plots worker
2025-06-13 11:09:19,656 INFO [watch_ww3] forecast WWATCH3 run on arbutus.cloud-nowcast completed
2025-06-13 11:09:19,658 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2025-06-13 11:09:20,146 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2025-06-13'], host='localhost')
2025-06-13 11:09:21,694 INFO [download_wwatch3_results] running in process 859567
2025-06-13 11:09:21,715 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-13 11:09:21,732 INFO [download_wwatch3_results] publishing log messages to tcp://*:5562
2025-06-13 11:09:21,746 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2025-06-13 11:10:29,803 INFO [download_wwatch3_results] forecast 2025-06-13 results files from arbutus.cloud-nowcast downloaded
2025-06-13 11:10:29,807 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2025-06-13 11:10:30,144 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['wwatch3', 'forecast', '--run-date', '2025-06-13'], host='localhost')
2025-06-13 11:10:31,773 INFO [update_forecast_datasets] running in process 859583
2025-06-13 11:10:31,774 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-13 11:10:31,776 INFO [update_forecast_datasets] publishing log messages to tcp://*:5562
2025-06-13 11:10:31,777 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2025-06-13 11:10:31,777 INFO [update_forecast_datasets] updating wwatch3 most_recent_forecast directory from forecast/13jun25 run
2025-06-13 11:10:31,908 INFO [update_forecast_datasets] updated wwatch3 most_recent_forecast directory from forecast/13jun25 run
2025-06-13 11:10:31,909 INFO [update_forecast_datasets] updating wwatch3 forecast directory for forecast/13jun25 run
2025-06-13 11:10:32,032 INFO [update_forecast_datasets] updated wwatch3 forecast directory for forecast/13jun25 run: /results/SalishSea/rolling-forecasts/wwatch3
2025-06-13 11:10:32,033 INFO [update_forecast_datasets] wwatch3 2025-06-13 forecast rolling forecast datasets updated
2025-06-13 11:10:32,036 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2025-06-13 11:10:32,325 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['wwatch3-forecast'], host='localhost')
2025-06-13 11:10:34,045 INFO [ping_erddap] running in process 859589
2025-06-13 11:10:34,046 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-13 11:10:34,048 INFO [ping_erddap] publishing log messages to tcp://*:5562
2025-06-13 11:10:34,049 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2025-06-13 11:10:34,050 INFO [ping_erddap] wwatch3-forecast ERDDAP dataset flag file(s) created
2025-06-13 11:10:34,051 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2025-06-13 11:10:34,391 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['wwatch3', 'forecast', 'publish', '--run-date', '2025-06-13'], host='localhost')
2025-06-13 11:10:38,689 INFO [make_plots] running in process 859594
2025-06-13 11:10:38,691 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-13 11:10:38,692 INFO [make_plots] publishing log messages to tcp://*:5562
2025-06-13 11:10:38,693 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2025-06-13 11:10:38,694 INFO [make_plots] preparing render list for 2025-06-13 WaveWatch3 forecast publish figures
2025-06-13 11:10:38,695 INFO [make_plots] added Halibut Bank figure to 2025-06-13 WaveWatch3 forecast publish render list
2025-06-13 11:10:38,696 INFO [make_plots] added Sentry Shoal figure to 2025-06-13 WaveWatch3 forecast publish render list
2025-06-13 11:10:38,697 INFO [make_plots] starting to render wwatch3 forecast publish 13jun25 figures
2025-06-13 11:10:39,224 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46146 Halibut Bank
2025-06-13 11:10:47,834 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46131 Sentry Shoal
2025-06-13 11:10:53,053 INFO [make_plots] finished rendering wwatch3 forecast publish 13jun25 figures
2025-06-13 11:10:53,054 INFO [make_plots] wwatch3 publish plots for 2025-06-13 forecast completed
2025-06-13 11:10:53,058 INFO [manager] checklist updated with [plots] items from make_plots worker
2025-06-13 11:11:46,452 INFO [watch_NEMO] nowcast-green NEMO run on arbutus.cloud-nowcast completed
2025-06-13 11:11:46,458 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2025-06-13 11:11:47,021 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2025-06-13'], host='localhost')
2025-06-13 11:11:48,403 INFO [download_results] running in process 859637
2025-06-13 11:11:48,443 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-13 11:11:48,478 INFO [download_results] publishing log messages to tcp://*:5562
2025-06-13 11:11:48,509 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2025-06-13 11:11:48,514 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/nowcast-green/13jun25 to /results2/SalishSea/nowcast-green.202111
2025-06-13 11:16:50,665 INFO [download_results] nowcast-green 2025-06-13 results files from arbutus.cloud-nowcast downloaded
2025-06-13 11:16:50,672 INFO [manager] checklist updated with [results files] items from download_results worker
2025-06-13 11:16:51,007 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast-green', 'research', '--run-date', '2025-06-13'], host='localhost')
2025-06-13 11:16:51,012 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nowcast-green'], host='localhost')
2025-06-13 11:16:51,017 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'physics', '--run-date', '2025-06-13'], host='localhost')
2025-06-13 11:16:51,022 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'biology', '--run-date', '2025-06-13'], host='localhost')
2025-06-13 11:16:51,028 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'chemistry', '--run-date', '2025-06-13'], host='localhost')
2025-06-13 11:16:52,505 INFO [ping_erddap] running in process 859687
2025-06-13 11:16:52,506 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-13 11:16:52,507 INFO [ping_erddap] publishing log messages to tcp://*:5562
2025-06-13 11:16:52,508 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2025-06-13 11:16:52,513 INFO [ping_erddap] nowcast-green ERDDAP dataset flag file(s) created
2025-06-13 11:16:52,518 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2025-06-13 11:16:53,706 INFO [make_averaged_dataset] running in process 859689
2025-06-13 11:16:53,707 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-13 11:16:53,708 INFO [make_averaged_dataset] publishing log messages to tcp://*:5562
2025-06-13 11:16:53,709 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555
2025-06-13 11:16:53,710 INFO [make_averaged_dataset] creating day-averaged dataset for 13-Jun-2025 biology
2025-06-13 11:16:53,711 INFO [make_averaged_dataset] running in process 859690
2025-06-13 11:16:53,712 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-13 11:16:53,712 INFO [make_averaged_dataset] publishing log messages to tcp://*:5564
2025-06-13 11:16:53,713 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555
2025-06-13 11:16:53,714 INFO [make_averaged_dataset] creating day-averaged dataset for 13-Jun-2025 chemistry
2025-06-13 11:16:53,726 INFO [make_averaged_dataset] running in process 859688
2025-06-13 11:16:53,727 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-13 11:16:53,728 INFO [make_averaged_dataset] publishing log messages to tcp://*:5565
2025-06-13 11:16:53,729 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555
2025-06-13 11:16:53,730 INFO [make_averaged_dataset] creating day-averaged dataset for 13-Jun-2025 physics
2025-06-13 11:16:54,483 INFO [make_plots] running in process 859685
2025-06-13 11:16:54,484 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-13 11:16:54,485 INFO [make_plots] publishing log messages to tcp://*:5566
2025-06-13 11:16:54,486 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2025-06-13 11:16:54,811 INFO [make_plots] preparing render list for 2025-06-13 NEMO nowcast-green research figures
2025-06-13 11:16:56,716 INFO [make_plots] added temperature_thalweg_and_surface figures to 2025-06-13 NEMO nowcast-green research render list
2025-06-13 11:16:57,443 INFO [make_plots] added nitrate_thalweg_and_surface figures to 2025-06-13 NEMO nowcast-green research render list
2025-06-13 11:16:58,209 INFO [make_plots] added ammonium_thalweg_and_surface figures to 2025-06-13 NEMO nowcast-green research render list
2025-06-13 11:16:58,934 INFO [make_plots] added silicon_thalweg_and_surface figures to 2025-06-13 NEMO nowcast-green research render list
2025-06-13 11:16:59,666 INFO [make_plots] added dissolved_organic_nitrogen_thalweg_and_surface figures to 2025-06-13 NEMO nowcast-green research render list
2025-06-13 11:17:00,394 INFO [make_plots] added particulate_organic_nitrogen_thalweg_and_surface figures to 2025-06-13 NEMO nowcast-green research render list
2025-06-13 11:17:01,125 INFO [make_plots] added biogenic_silicon_thalweg_and_surface figures to 2025-06-13 NEMO nowcast-green research render list
2025-06-13 11:17:01,953 INFO [make_plots] added diatoms_thalweg_and_surface figures to 2025-06-13 NEMO nowcast-green research render list
2025-06-13 11:17:02,787 INFO [make_plots] added flagellates_thalweg_and_surface figures to 2025-06-13 NEMO nowcast-green research render list
2025-06-13 11:17:03,626 INFO [make_plots] added microzooplankton_thalweg_and_surface figures to 2025-06-13 NEMO nowcast-green research render list
2025-06-13 11:17:04,538 INFO [make_plots] added mesozooplankton_thalweg_and_surface figures to 2025-06-13 NEMO nowcast-green research render list
2025-06-13 11:17:05,374 INFO [make_plots] added turbidity_thalweg_and_surface figures to 2025-06-13 NEMO nowcast-green research render list
2025-06-13 11:17:06,248 INFO [make_plots] added temperature_salinity_timeseries figure to 2025-06-13 NEMO nowcast-green research render list
2025-06-13 11:17:06,249 INFO [make_plots] added nitrate_diatoms_timeseries figure to 2025-06-13 NEMO nowcast-green research render list
2025-06-13 11:17:06,249 INFO [make_plots] added diatoms_flagellates_timeseries figure to 2025-06-13 NEMO nowcast-green research render list
2025-06-13 11:17:06,250 INFO [make_plots] added z1_z2_zooplankton_timeseries figure to 2025-06-13 NEMO nowcast-green research render list
2025-06-13 11:17:06,251 INFO [make_plots] starting to render nemo nowcast-green research 13jun25 figures
2025-06-13 11:17:27,389 INFO [make_averaged_dataset] day-averaged dataset for 13-Jun-2025 physics created
2025-06-13 11:17:27,392 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker
2025-06-13 11:18:10,091 INFO [make_averaged_dataset] day-averaged dataset for 13-Jun-2025 chemistry created
2025-06-13 11:18:10,094 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker
2025-06-13 11:18:20,948 INFO [make_averaged_dataset] day-averaged dataset for 13-Jun-2025 biology created
2025-06-13 11:18:20,951 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker
2025-06-13 12:28:26,229 INFO [make_plots] finished rendering nemo nowcast-green research 13jun25 figures
2025-06-13 12:28:26,230 INFO [make_plots] nemo research plots for 2025-06-13 nowcast-green completed
2025-06-13 12:28:26,307 INFO [manager] checklist updated with [plots] items from make_plots worker
2025-06-13 14:49:08,445 INFO [ping_erddap] running in process 865101
2025-06-13 14:49:08,447 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-13 14:49:08,449 INFO [ping_erddap] publishing log messages to tcp://*:5562
2025-06-13 14:49:08,451 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2025-06-13 14:49:08,452 INFO [ping_erddap] weather ERDDAP dataset flag file(s) created
2025-06-13 14:49:08,455 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2025-06-13 14:56:17,073 INFO [ping_erddap] running in process 866150
2025-06-13 14:56:17,075 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-13 14:56:17,075 INFO [ping_erddap] publishing log messages to tcp://*:5562
2025-06-13 14:56:17,085 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2025-06-13 14:56:17,112 INFO [ping_erddap] weather ERDDAP dataset flag file(s) created
2025-06-13 14:56:17,143 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2025-06-13 15:02:54,382 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/18/ to /results/forcing/atmospheric/continental2.5/GRIB/20250613/18/
2025-06-13 15:02:54,383 INFO [collect_weather] 2025-06-13 2.5km weather forecast 18 collection complete
2025-06-13 15:02:54,386 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2025-06-13 15:02:54,883 INFO [manager] launching NextWorker(module='nowcast.workers.download_weather', args=['00', '1km'], host='localhost')
2025-06-13 15:02:54,890 INFO [manager] launching NextWorker(module='nowcast.workers.download_weather', args=['12', '1km'], host='localhost')
2025-06-13 15:02:54,896 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['00', '2.5km'], host='localhost')
2025-06-13 15:02:54,900 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['00', '--fcst-date', '2025-06-14'], host='localhost')
2025-06-13 15:02:56,548 INFO [download_weather] running in process 866859
2025-06-13 15:02:56,549 INFO [download_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-13 15:02:56,550 INFO [download_weather] publishing log messages to tcp://*:5562
2025-06-13 15:02:56,551 INFO [download_weather] connected to skookum.eos.ubc.ca port 5555
2025-06-13 15:02:56,552 INFO [download_weather] downloading 00 1 km forecast GRIB2 files for 20250613
2025-06-13 15:02:56,557 INFO [download_weather] running in process 866861
2025-06-13 15:02:56,559 INFO [download_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-13 15:02:56,560 INFO [download_weather] publishing log messages to tcp://*:5563
2025-06-13 15:02:56,561 INFO [download_weather] connected to skookum.eos.ubc.ca port 5555
2025-06-13 15:02:56,562 INFO [download_weather] downloading 12 1 km forecast GRIB2 files for 20250613
2025-06-13 15:02:56,562 INFO [collect_weather] running in process 866862
2025-06-13 15:02:56,563 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-13 15:02:56,564 INFO [collect_weather] publishing log messages to tcp://*:5564
2025-06-13 15:02:56,565 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2025-06-13 15:02:56,624 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/00/
2025-06-13 15:02:56,838 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20250613/18/
2025-06-13 15:02:56,853 INFO [crop_gribs] 2025-06-13 18 GRIBs cropping complete
2025-06-13 15:02:56,869 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2025-06-13 15:02:58,697 INFO [crop_gribs] running in process 866863
2025-06-13 15:02:58,699 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-13 15:02:58,700 INFO [crop_gribs] publishing log messages to tcp://*:5565
2025-06-13 15:02:58,701 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2025-06-13 15:02:58,702 INFO [crop_gribs] cropping 2025-06-14 ECCC HRDPS 2.5km continental 00Z GRIB files to SalishSeaCast subdomain
2025-06-13 15:02:58,715 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20250614/00/
2025-06-13 15:06:33,945 INFO [download_weather] 2025-06-13 1km weather forecast 00 downloads complete
2025-06-13 15:06:33,947 INFO [manager] checklist updated with [weather forecast] items from download_weather worker
2025-06-13 15:06:37,078 INFO [download_weather] 2025-06-13 1km weather forecast 12 downloads complete
2025-06-13 15:06:37,081 INFO [manager] checklist updated with [weather forecast] items from download_weather worker
2025-06-13 21:05:58,394 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/00/ to /results/forcing/atmospheric/continental2.5/GRIB/20250614/00/
2025-06-13 21:05:58,396 INFO [collect_weather] 2025-06-14 2.5km weather forecast 00 collection complete
2025-06-13 21:05:58,401 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2025-06-13 21:05:58,762 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20250614/00/
2025-06-13 21:05:58,763 INFO [crop_gribs] 2025-06-14 00 GRIBs cropping complete
2025-06-13 21:05:58,929 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['06', '2.5km'], host='localhost')
2025-06-13 21:05:58,950 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['06', '--fcst-date', '2025-06-14'], host='localhost')
2025-06-13 21:05:58,966 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2025-06-13 21:06:00,815 INFO [collect_weather] running in process 869980
2025-06-13 21:06:00,816 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-13 21:06:00,817 INFO [collect_weather] publishing log messages to tcp://*:5562
2025-06-13 21:06:00,818 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2025-06-13 21:06:00,835 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/06/
2025-06-13 21:06:02,091 INFO [crop_gribs] running in process 869982
2025-06-13 21:06:02,092 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-13 21:06:02,095 INFO [crop_gribs] publishing log messages to tcp://*:5563
2025-06-13 21:06:02,096 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2025-06-13 21:06:02,097 INFO [crop_gribs] cropping 2025-06-14 ECCC HRDPS 2.5km continental 06Z GRIB files to SalishSeaCast subdomain
2025-06-13 21:06:02,109 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20250614/06/
2025-06-14 03:00:56,538 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/06/ to /results/forcing/atmospheric/continental2.5/GRIB/20250614/06/
2025-06-14 03:00:56,540 INFO [collect_weather] 2025-06-14 2.5km weather forecast 06 collection complete
2025-06-14 03:00:56,544 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2025-06-14 03:00:57,098 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Capilano', '--data-date', '2025-06-13'], host='localhost')
2025-06-14 03:00:57,137 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'ChilliwackVedder', '--data-date', '2025-06-13'], host='localhost')
2025-06-14 03:00:57,143 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'ClowhomClowhomLake', '--data-date', '2025-06-13'], host='localhost')
2025-06-14 03:00:57,148 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Englishman', '--data-date', '2025-06-13'], host='localhost')
2025-06-14 03:00:57,153 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Fraser', '--data-date', '2025-06-13'], host='localhost')
2025-06-14 03:00:57,158 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'HomathkoMouth', '--data-date', '2025-06-13'], host='localhost')
2025-06-14 03:00:57,164 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'NicomeklLangley', '--data-date', '2025-06-13'], host='localhost')
2025-06-14 03:00:57,168 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'RobertsCreek', '--data-date', '2025-06-13'], host='localhost')
2025-06-14 03:00:57,173 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SalmonSayward', '--data-date', '2025-06-13'], host='localhost')
2025-06-14 03:00:57,177 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SanJuanPortRenfrew', '--data-date', '2025-06-13'], host='localhost')
2025-06-14 03:00:57,184 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SquamishBrackendale', '--data-date', '2025-06-13'], host='localhost')
2025-06-14 03:00:57,190 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaScotty', '--data-date', '2025-06-13'], host='localhost')
2025-06-14 03:00:57,207 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaBypass', '--data-date', '2025-06-13'], host='localhost')
2025-06-14 03:00:57,212 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaDiversion', '--data-date', '2025-06-13'], host='localhost')
2025-06-14 03:00:57,222 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ctd', args=['SCVIP'], host='localhost')
2025-06-14 03:00:57,234 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ctd', args=['SEVIP'], host='localhost')
2025-06-14 03:00:57,240 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ferry', args=['TWDP'], host='localhost')
2025-06-14 03:00:57,265 INFO [manager] launching NextWorker(module='nowcast.workers.get_vfpa_hadcp', args=['--data-date', '2025-06-13'], host='localhost')
2025-06-14 03:00:57,277 INFO [manager] launching NextWorker(module='nowcast.workers.collect_NeahBay_ssh', args=['00'], host='localhost')
2025-06-14 03:00:57,292 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['12', '2.5km'], host='localhost')
2025-06-14 03:00:57,305 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['12'], host='localhost')
2025-06-14 03:00:58,170 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20250614/06/
2025-06-14 03:00:58,171 INFO [crop_gribs] 2025-06-14 06 GRIBs cropping complete
2025-06-14 03:00:58,178 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2025-06-14 03:00:59,222 INFO [manager] launching NextWorker(module='nowcast.workers.grib_to_netcdf', args=['forecast2'], host='localhost')
2025-06-14 03:01:00,396 INFO [collect_NeahBay_ssh] running in process 872529
2025-06-14 03:01:00,397 INFO [collect_NeahBay_ssh] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-14 03:01:00,398 INFO [collect_NeahBay_ssh] publishing log messages to tcp://*:5562
2025-06-14 03:01:00,399 INFO [collect_NeahBay_ssh] connected to skookum.eos.ubc.ca port 5555
2025-06-14 03:01:00,400 INFO [collect_NeahBay_ssh] collecting Neah Bay ssh 00Z obs/forecast for 2025-06-14
2025-06-14 03:01:00,402 INFO [collect_weather] running in process 872530
2025-06-14 03:01:00,405 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-14 03:01:00,409 INFO [collect_weather] publishing log messages to tcp://*:5563
2025-06-14 03:01:00,409 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2025-06-14 03:01:00,434 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/12/
2025-06-14 03:01:01,166 INFO [collect_river_data] running in process 872515
2025-06-14 03:01:01,167 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-14 03:01:01,168 INFO [collect_river_data] publishing log messages to tcp://*:5564
2025-06-14 03:01:01,169 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-06-14 03:01:01,170 INFO [collect_river_data] Collecting ECCC HomathkoMouth river data for 2025-06-13
2025-06-14 03:01:01,253 INFO [collect_river_data] running in process 872517
2025-06-14 03:01:01,254 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-14 03:01:01,255 INFO [collect_river_data] publishing log messages to tcp://*:5565
2025-06-14 03:01:01,256 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-06-14 03:01:01,257 INFO [collect_river_data] Collecting ECCC RobertsCreek river data for 2025-06-13
2025-06-14 03:01:01,314 INFO [collect_river_data] running in process 872512
2025-06-14 03:01:01,316 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-14 03:01:01,317 INFO [collect_river_data] publishing log messages to tcp://*:5566
2025-06-14 03:01:01,318 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-06-14 03:01:01,320 INFO [collect_river_data] Collecting ECCC ClowhomClowhomLake river data for 2025-06-13
2025-06-14 03:01:01,324 INFO [collect_river_data] running in process 872511
2025-06-14 03:01:01,326 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-14 03:01:01,329 INFO [collect_river_data] publishing log messages to tcp://*:5567
2025-06-14 03:01:01,331 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-06-14 03:01:01,332 INFO [collect_river_data] Collecting ECCC ChilliwackVedder river data for 2025-06-13
2025-06-14 03:01:01,337 INFO [collect_river_data] Appended ECCC HomathkoMouth river average discharge for 2025-06-13 to: /results/forcing/rivers/observations/Homathko_Mouth_flow
2025-06-14 03:01:01,338 INFO [collect_river_data] ECCC HomathkoMouth river data collection for 2025-06-13 completed
2025-06-14 03:01:01,341 INFO [collect_river_data] running in process 872509
2025-06-14 03:01:01,341 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-14 03:01:01,342 INFO [collect_river_data] publishing log messages to tcp://*:5568
2025-06-14 03:01:01,343 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-06-14 03:01:01,344 INFO [collect_river_data] Collecting ECCC Capilano river data for 2025-06-13
2025-06-14 03:01:01,345 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-06-14 03:01:01,346 INFO [collect_river_data] Appended ECCC RobertsCreek river average discharge for 2025-06-13 to: /results/forcing/rivers/observations/RobertsCreek_flow
2025-06-14 03:01:01,347 INFO [collect_river_data] ECCC RobertsCreek river data collection for 2025-06-13 completed
2025-06-14 03:01:01,352 INFO [collect_river_data] running in process 872516
2025-06-14 03:01:01,353 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-14 03:01:01,354 INFO [collect_river_data] publishing log messages to tcp://*:5569
2025-06-14 03:01:01,355 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-06-14 03:01:01,356 INFO [collect_river_data] Collecting ECCC NicomeklLangley river data for 2025-06-13
2025-06-14 03:01:01,360 INFO [collect_river_data] running in process 872519
2025-06-14 03:01:01,361 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-14 03:01:01,364 INFO [collect_river_data] Appended ECCC ChilliwackVedder river average discharge for 2025-06-13 to: /results/forcing/rivers/observations/Chilliwack_Vedder_flow
2025-06-14 03:01:01,365 INFO [collect_river_data] publishing log messages to tcp://*:5570
2025-06-14 03:01:01,366 INFO [collect_river_data] ECCC ChilliwackVedder river data collection for 2025-06-13 completed
2025-06-14 03:01:01,367 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-06-14 03:01:01,370 INFO [collect_river_data] Collecting ECCC SanJuanPortRenfrew river data for 2025-06-13
2025-06-14 03:01:01,373 INFO [collect_river_data] Appended ECCC ClowhomClowhomLake river average discharge for 2025-06-13 to: /results/forcing/rivers/observations/Clowhom_ClowhomLake_flow
2025-06-14 03:01:01,374 INFO [collect_river_data] ECCC ClowhomClowhomLake river data collection for 2025-06-13 completed
2025-06-14 03:01:01,378 INFO [collect_river_data] running in process 872513
2025-06-14 03:01:01,378 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-14 03:01:01,381 INFO [collect_river_data] publishing log messages to tcp://*:5571
2025-06-14 03:01:01,381 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-06-14 03:01:01,383 INFO [collect_river_data] Collecting ECCC Englishman river data for 2025-06-13
2025-06-14 03:01:01,384 INFO [collect_river_data] running in process 872514
2025-06-14 03:01:01,385 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-14 03:01:01,388 INFO [collect_river_data] publishing log messages to tcp://*:5572
2025-06-14 03:01:01,396 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-06-14 03:01:01,399 INFO [collect_river_data] Collecting ECCC Fraser river data for 2025-06-13
2025-06-14 03:01:01,401 INFO [collect_river_data] Appended ECCC NicomeklLangley river average discharge for 2025-06-13 to: /results/forcing/rivers/observations/Nicomekl_Langley_flow
2025-06-14 03:01:01,403 INFO [collect_river_data] ECCC NicomeklLangley river data collection for 2025-06-13 completed
2025-06-14 03:01:01,408 INFO [collect_river_data] Appended ECCC SanJuanPortRenfrew river average discharge for 2025-06-13 to: /results/forcing/rivers/observations/SanJuan_PortRenfrew_flow
2025-06-14 03:01:01,409 INFO [collect_river_data] ECCC SanJuanPortRenfrew river data collection for 2025-06-13 completed
2025-06-14 03:01:01,436 INFO [collect_river_data] running in process 872518
2025-06-14 03:01:01,437 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-14 03:01:01,438 INFO [collect_river_data] publishing log messages to tcp://*:5573
2025-06-14 03:01:01,438 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-06-14 03:01:01,439 INFO [collect_river_data] Collecting ECCC SalmonSayward river data for 2025-06-13
2025-06-14 03:01:01,446 INFO [collect_river_data] Appended ECCC Fraser river average discharge for 2025-06-13 to: /data/dlatorne/SOG-projects/SOG-forcing/ECget/Fraser_flow
2025-06-14 03:01:01,447 INFO [collect_river_data] ECCC Fraser river data collection for 2025-06-13 completed
2025-06-14 03:01:01,451 INFO [collect_river_data] Appended ECCC Englishman river average discharge for 2025-06-13 to: /data/dlatorne/SOG-projects/SOG-forcing/ECget/Englishman_flow
2025-06-14 03:01:01,452 INFO [collect_river_data] ECCC Englishman river data collection for 2025-06-13 completed
2025-06-14 03:01:01,454 INFO [collect_river_data] Appended ECCC Capilano river average discharge for 2025-06-13 to: /opp/observations/rivers/Capilano/Caplilano_08GA010_day_avg_flow
2025-06-14 03:01:01,455 INFO [collect_river_data] ECCC Capilano river data collection for 2025-06-13 completed
2025-06-14 03:01:01,471 INFO [collect_river_data] Appended ECCC SalmonSayward river average discharge for 2025-06-13 to: /results/forcing/rivers/observations/Salmon_Sayward_flow
2025-06-14 03:01:01,472 INFO [collect_river_data] ECCC SalmonSayward river data collection for 2025-06-13 completed
2025-06-14 03:01:01,475 INFO [collect_river_data] running in process 872522
2025-06-14 03:01:01,476 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-14 03:01:01,477 INFO [collect_river_data] publishing log messages to tcp://*:5574
2025-06-14 03:01:01,478 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-06-14 03:01:01,479 INFO [collect_river_data] Collecting ECCC TheodosiaBypass river data for 2025-06-13
2025-06-14 03:01:01,493 INFO [collect_river_data] running in process 872520
2025-06-14 03:01:01,495 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-14 03:01:01,496 INFO [collect_river_data] publishing log messages to tcp://*:5575
2025-06-14 03:01:01,497 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-06-14 03:01:01,497 INFO [collect_river_data] Collecting ECCC SquamishBrackendale river data for 2025-06-13
2025-06-14 03:01:01,499 INFO [collect_river_data] running in process 872523
2025-06-14 03:01:01,500 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-14 03:01:01,501 INFO [collect_river_data] publishing log messages to tcp://*:5576
2025-06-14 03:01:01,502 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-06-14 03:01:01,507 INFO [collect_river_data] Collecting ECCC TheodosiaDiversion river data for 2025-06-13
2025-06-14 03:01:01,510 INFO [collect_river_data] running in process 872521
2025-06-14 03:01:01,511 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-14 03:01:01,513 INFO [collect_river_data] publishing log messages to tcp://*:5577
2025-06-14 03:01:01,514 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-06-14 03:01:01,515 INFO [collect_river_data] Collecting ECCC TheodosiaScotty river data for 2025-06-13
2025-06-14 03:01:01,517 CRITICAL [collect_river_data] unhandled exception:
Traceback (most recent call last):
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/pandas/core/indexes/datetimes.py", line 610, in get_loc
    return self._partial_date_slice(reso, parsed)
           ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/pandas/core/indexes/datetimelike.py", line 333, in _partial_date_slice
    raise KeyError
KeyError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/SalishSeaCast/NEMO_Nowcast/nemo_nowcast/worker.py", line 387, in _do_work
    checklist = self.worker_func(
        self._parsed_args, self.config, self.tell_manager
    )
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/collect_river_data.py", line 108, in collect_river_data
    day_avg_discharge = day_avg_discharge_funcs[data_src](river_name, data_date, config)
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/collect_river_data.py", line 141, in _calc_eccc_day_avg_discharge
    day_avg_discharge = df.loc[f"{data_date.format('YYYY-MM-DD')}"].mean()[
                        ~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/pandas/core/indexing.py", line 1191, in __getitem__
    return self._getitem_axis(maybe_callable, axis=axis)
           ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/pandas/core/indexing.py", line 1431, in _getitem_axis
    return self._get_label(key, axis=axis)
           ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/pandas/core/indexing.py", line 1381, in _get_label
    return self.obj.xs(label, axis=axis)
           ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/pandas/core/generic.py", line 4301, in xs
    loc = index.get_loc(key)
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/pandas/core/indexes/datetimes.py", line 612, in get_loc
    raise KeyError(key) from err
KeyError: '2025-06-13'
2025-06-14 03:01:01,829 INFO [get_vfpa_hadcp] running in process 872528
2025-06-14 03:01:01,843 INFO [get_vfpa_hadcp] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-14 03:01:01,847 INFO [collect_NeahBay_ssh] 2025-06-14T00:00:00-07:00 Neah Bay ssh 00Z obs/forecast data collection complete
2025-06-14 03:01:01,851 INFO [collect_river_data] Appended ECCC SquamishBrackendale river average discharge for 2025-06-13 to: /results/forcing/rivers/observations/Squamish_Brackendale_flow
2025-06-14 03:01:01,854 INFO [collect_river_data] Appended ECCC TheodosiaDiversion river average discharge for 2025-06-13 to: /results/forcing/rivers/observations/Theodosia_Diversion_flow
2025-06-14 03:01:01,859 INFO [collect_river_data] Appended ECCC TheodosiaScotty river average discharge for 2025-06-13 to: /results/forcing/rivers/observations/Theodosia_Scotty_flow
2025-06-14 03:01:01,865 INFO [get_vfpa_hadcp] publishing log messages to tcp://*:5578
2025-06-14 03:01:01,871 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-06-14 03:01:01,871 INFO [collect_river_data] ECCC SquamishBrackendale river data collection for 2025-06-13 completed
2025-06-14 03:01:01,875 INFO [collect_river_data] ECCC TheodosiaDiversion river data collection for 2025-06-13 completed
2025-06-14 03:01:01,876 INFO [collect_river_data] ECCC TheodosiaScotty river data collection for 2025-06-13 completed
2025-06-14 03:01:01,878 INFO [get_vfpa_hadcp] connected to skookum.eos.ubc.ca port 5555
2025-06-14 03:01:01,881 INFO [get_vfpa_hadcp] processing VFPA HADCP data from 2nd Narrows Rail Bridge for 2025-06-13
2025-06-14 03:01:01,915 INFO [crop_gribs] running in process 872531
2025-06-14 03:01:01,916 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-14 03:01:01,917 INFO [crop_gribs] publishing log messages to tcp://*:5579
2025-06-14 03:01:01,918 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2025-06-14 03:01:01,919 INFO [crop_gribs] cropping 2025-06-14 ECCC HRDPS 2.5km continental 12Z GRIB files to SalishSeaCast subdomain
2025-06-14 03:01:01,931 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20250614/12/
2025-06-14 03:01:02,175 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-06-14 03:01:02,433 INFO [grib_to_netcdf] running in process 872760
2025-06-14 03:01:02,434 INFO [grib_to_netcdf] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-14 03:01:02,435 INFO [grib_to_netcdf] publishing log messages to tcp://*:5564
2025-06-14 03:01:02,436 INFO [grib_to_netcdf] connected to skookum.eos.ubc.ca port 5555
2025-06-14 03:01:02,437 INFO [grib_to_netcdf] creating NEMO-atmos forcing files for 2025-06-14 forecast2 run
2025-06-14 03:01:02,603 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-06-14 03:01:02,873 INFO [get_onc_ctd] running in process 872525
2025-06-14 03:01:02,874 INFO [get_onc_ctd] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-14 03:01:02,875 INFO [get_onc_ctd] publishing log messages to tcp://*:5565
2025-06-14 03:01:02,879 INFO [get_onc_ctd] connected to skookum.eos.ubc.ca port 5555
2025-06-14 03:01:02,880 INFO [get_onc_ctd] requesting ONC SCVIP CTD T&S data for 2025-06-13
2025-06-14 03:01:02,916 INFO [get_onc_ctd] running in process 872526
2025-06-14 03:01:02,917 INFO [get_onc_ctd] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-14 03:01:02,918 INFO [get_onc_ctd] publishing log messages to tcp://*:5567
2025-06-14 03:01:02,919 INFO [get_onc_ctd] connected to skookum.eos.ubc.ca port 5555
2025-06-14 03:01:02,920 INFO [get_onc_ctd] requesting ONC SEVIP CTD T&S data for 2025-06-13
2025-06-14 03:01:02,946 INFO [get_onc_ferry] running in process 872527
2025-06-14 03:01:02,948 INFO [get_onc_ferry] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-14 03:01:02,949 INFO [get_onc_ferry] publishing log messages to tcp://*:5590
2025-06-14 03:01:02,950 INFO [get_onc_ferry] connected to skookum.eos.ubc.ca port 5555
2025-06-14 03:01:02,951 INFO [get_onc_ferry] requesting ONC TWDP.N1 NAV data for 2025-06-13
2025-06-14 03:01:02,972 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-06-14 03:01:03,310 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-06-14 03:01:03,676 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-06-14 03:01:04,022 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-06-14 03:01:04,321 INFO [get_vfpa_hadcp] added VFPA HADCP data from 2nd Narrows Rail Bridge for 2025-06-13 to /opp/observations/AISDATA/netcdf/VFPA_2ND_NARROWS_HADCP_2s_202506.nc
2025-06-14 03:01:04,322 INFO [get_vfpa_hadcp] VFPA HADCP observations added to 2025-06 netcdf file
2025-06-14 03:01:04,357 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-06-14 03:01:04,665 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-06-14 03:01:04,970 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-06-14 03:01:05,281 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-06-14 03:01:05,587 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-06-14 03:01:05,949 INFO [manager] checklist updated with [Neah Bay ssh data] items from collect_NeahBay_ssh worker
2025-06-14 03:01:06,272 INFO [manager] launching NextWorker(module='nowcast.workers.make_ssh_files', args=['forecast2', '--run-date', '2025-06-14'], host='localhost')
2025-06-14 03:01:06,281 INFO [manager] checklist updated with [VFPA HADCP data] items from get_vfpa_hadcp worker
2025-06-14 03:01:06,755 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['VFPA-HADCP'], host='localhost')
2025-06-14 03:01:08,000 INFO [ping_erddap] running in process 872882
2025-06-14 03:01:08,002 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-14 03:01:08,003 INFO [ping_erddap] publishing log messages to tcp://*:5562
2025-06-14 03:01:08,004 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2025-06-14 03:01:08,006 INFO [ping_erddap] VFPA-HADCP ERDDAP dataset flag file(s) created
2025-06-14 03:01:08,008 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2025-06-14 03:01:10,170 INFO [make_ssh_files] running in process 872869
2025-06-14 03:01:10,171 INFO [make_ssh_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-14 03:01:10,172 INFO [make_ssh_files] publishing log messages to tcp://*:5562
2025-06-14 03:01:10,173 INFO [make_ssh_files] connected to skookum.eos.ubc.ca port 5555
2025-06-14 03:01:10,174 INFO [make_ssh_files] building forecast2 sea surface height boundary conditions file(s) from 2025-06-14 NOAA Neah Bay 00Z observation and forecast values
2025-06-14 03:01:12,009 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2025m06d15.nc
2025-06-14 03:01:12,496 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2025m06d09.nc
2025-06-14 03:01:12,710 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2025m06d10.nc
2025-06-14 03:01:12,911 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2025m06d11.nc
2025-06-14 03:01:13,168 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2025m06d12.nc
2025-06-14 03:01:13,369 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2025m06d13.nc
2025-06-14 03:01:13,619 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2025m06d14.nc
2025-06-14 03:01:13,849 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2025m06d15.nc
2025-06-14 03:01:14,065 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2025m06d16.nc
2025-06-14 03:01:14,316 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2025m06d17.nc
2025-06-14 03:01:14,552 INFO [make_ssh_files] sea surface height boundary file for 2025-06-14 forecast2 run created
2025-06-14 03:01:14,555 INFO [manager] checklist updated with [sea surface height forcing] items from make_ssh_files worker
2025-06-14 03:01:14,683 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2025m06d16.nc
2025-06-14 03:01:14,689 INFO [grib_to_netcdf] 2025-06-14 NEMO-atmos forcing file for forecast2 created
2025-06-14 03:01:14,942 INFO [manager] launching NextWorker(module='nowcast.workers.make_runoff_file', args=['v202108'], host='localhost')
2025-06-14 03:01:14,948 INFO [manager] launching NextWorker(module='nowcast.workers.make_201702_runoff_file', args=[], host='localhost')
2025-06-14 03:01:14,953 INFO [manager] checklist updated with [weather forcing] items from grib_to_netcdf worker
2025-06-14 03:01:16,299 INFO [make_201702_runoff_file] running in process 872918
2025-06-14 03:01:16,301 INFO [make_201702_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-14 03:01:16,302 INFO [make_201702_runoff_file] publishing log messages to tcp://*:5562
2025-06-14 03:01:16,302 INFO [make_201702_runoff_file] connected to skookum.eos.ubc.ca port 5555
2025-06-14 03:01:16,358 INFO [make_201702_runoff_file] calculating runoff forcing for b201702 bathymetry
2025-06-14 03:01:16,489 INFO [make_201702_runoff_file] 2025-06-14 runoff file creation from Fraser at Hope and climatology elsewhere complete
2025-06-14 03:01:16,491 INFO [manager] checklist updated with [rivers forcing] items from make_201702_runoff_file worker
2025-06-14 03:01:16,879 INFO [make_runoff_file] running in process 872916
2025-06-14 03:01:16,880 INFO [make_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-14 03:01:16,881 INFO [make_runoff_file] publishing log messages to tcp://*:5564
2025-06-14 03:01:16,882 INFO [make_runoff_file] connected to skookum.eos.ubc.ca port 5555
2025-06-14 03:01:16,885 INFO [make_runoff_file] calculating NEMO runoff forcing for v202108 bathymetry for 2025-06-13
2025-06-14 03:01:18,103 ERROR [make_runoff_file] no 2025-06-13 discharge obs for SkagitMountVernon: patching
2025-06-14 03:01:18,572 ERROR [make_runoff_file] no 2025-06-13 discharge obs for SnohomishMonroe: patching
2025-06-14 03:01:18,837 ERROR [make_runoff_file] no 2025-06-13 discharge obs for NisquallyMcKenna: patching
2025-06-14 03:01:18,977 ERROR [make_runoff_file] no 2025-06-13 discharge obs for GreenwaterGreenwater: patching
2025-06-14 03:01:20,450 INFO [make_runoff_file] stored NEMO runoff forcing for v202108 bathymetry for 2025-06-13: /results/forcing/rivers/R202108Dailies_y2025m06d13.nc
2025-06-14 03:01:20,451 INFO [make_runoff_file] 2025-06-13 runoff file creation completed
2025-06-14 03:01:20,452 INFO [manager] checklist updated with [rivers forcing] items from make_runoff_file worker
2025-06-14 03:01:20,812 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'forecast2'], host='localhost')
2025-06-14 03:01:20,816 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'forecast2'], host='localhost')
2025-06-14 03:01:20,821 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.graham', 'forecast2'], host='localhost')
2025-06-14 03:01:20,826 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'forecast2'], host='localhost')
2025-06-14 03:01:21,699 INFO [get_onc_ctd] 2025-06-13 ONC SCVIP CTD T&S file created
2025-06-14 03:01:21,701 INFO [manager] checklist updated with [ONC CTD data] items from get_onc_ctd worker
2025-06-14 03:01:22,171 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['SCVIP-CTD'], host='localhost')
2025-06-14 03:01:22,225 INFO [get_onc_ctd] 2025-06-13 ONC SEVIP CTD T&S file created
2025-06-14 03:01:22,228 INFO [manager] checklist updated with [ONC CTD data] items from get_onc_ctd worker
2025-06-14 03:01:22,259 INFO [upload_forcing] running in process 872954
2025-06-14 03:01:22,260 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-14 03:01:22,261 INFO [upload_forcing] publishing log messages to tcp://*:5562
2025-06-14 03:01:22,262 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-06-14 03:01:22,284 INFO [upload_forcing] running in process 872952
2025-06-14 03:01:22,285 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-14 03:01:22,286 INFO [upload_forcing] publishing log messages to tcp://*:5564
2025-06-14 03:01:22,287 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-06-14 03:01:22,299 INFO [upload_forcing] running in process 872953
2025-06-14 03:01:22,300 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-14 03:01:22,301 INFO [upload_forcing] publishing log messages to tcp://*:5566
2025-06-14 03:01:22,302 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-06-14 03:01:22,325 INFO [upload_forcing] running in process 872951
2025-06-14 03:01:22,326 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-14 03:01:22,327 INFO [upload_forcing] publishing log messages to tcp://*:5568
2025-06-14 03:01:22,327 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-06-14 03:01:22,618 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['SEVIP-CTD'], host='localhost')
2025-06-14 03:01:23,313 INFO [ping_erddap] running in process 872967
2025-06-14 03:01:23,317 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-14 03:01:23,321 INFO [ping_erddap] publishing log messages to tcp://*:5565
2025-06-14 03:01:23,330 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2025-06-14 03:01:23,339 INFO [ping_erddap] SCVIP-CTD ERDDAP dataset flag file(s) created
2025-06-14 03:01:23,350 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2025-06-14 03:01:23,784 INFO [upload_forcing] LiveOcean boundary conditions file not found; created symlink to /results/forcing/LiveOcean/boundary_conditions/LiveOcean_v201905_y2025m06d13.nc
2025-06-14 03:01:23,839 INFO [upload_forcing] forecast2 2025-06-14 forcing files upload to orcinus-nowcast-agrif completed
2025-06-14 03:01:23,842 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-06-14 03:01:23,872 INFO [upload_forcing] forecast2 2025-06-14 forcing files upload to optimum-hindcast completed
2025-06-14 03:01:24,145 INFO [ping_erddap] running in process 872978
2025-06-14 03:01:24,146 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-14 03:01:24,147 INFO [ping_erddap] publishing log messages to tcp://*:5565
2025-06-14 03:01:24,148 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2025-06-14 03:01:24,150 INFO [ping_erddap] SEVIP-CTD ERDDAP dataset flag file(s) created
2025-06-14 03:01:24,204 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-06-14 03:01:24,538 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2025-06-14 03:01:26,580 INFO [get_onc_ferry] requesting ONC TWDP TSG data for 2025-06-13
2025-06-14 03:01:30,171 INFO [upload_forcing] forecast2 2025-06-14 forcing files upload to robot.graham completed
2025-06-14 03:01:30,176 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-06-14 03:01:50,832 INFO [get_onc_ferry] requesting ONC TWDP OXYSENSOR data for 2025-06-13
2025-06-14 03:02:14,048 INFO [get_onc_ferry] requesting ONC TWDP TURBCHLFL data for 2025-06-13
2025-06-14 03:02:52,327 INFO [get_onc_ferry] requesting ONC TWDP CO2SENSOR data for 2025-06-13
2025-06-14 03:02:52,396 WARNING [get_onc_ferry] No ONC TWDP CO2SENSOR partial_pressure,co2 data for 2025-06-13; substituting empty dataset
2025-06-14 03:02:52,401 WARNING [get_onc_ferry] No ONC TWDP CO2SENSOR partial_pressure data for 2025-06-13; substituting empty dataset
2025-06-14 03:02:52,402 WARNING [get_onc_ferry] No ONC TWDP CO2SENSOR co2 data for 2025-06-13; substituting empty dataset
2025-06-14 03:02:52,403 INFO [get_onc_ferry] requesting ONC TWDP TEMPHUMID data for 2025-06-13
2025-06-14 03:02:52,451 WARNING [get_onc_ferry] No ONC TWDP TEMPHUMID air_temperature,rel_humidity data for 2025-06-13; substituting empty dataset
2025-06-14 03:02:52,455 WARNING [get_onc_ferry] No ONC TWDP TEMPHUMID air_temperature data for 2025-06-13; substituting empty dataset
2025-06-14 03:02:52,457 WARNING [get_onc_ferry] No ONC TWDP TEMPHUMID rel_humidity data for 2025-06-13; substituting empty dataset
2025-06-14 03:02:52,457 INFO [get_onc_ferry] requesting ONC TWDP BARPRESS data for 2025-06-13
2025-06-14 03:02:52,503 WARNING [get_onc_ferry] No ONC TWDP BARPRESS barometric_pressure data for 2025-06-13; substituting empty dataset
2025-06-14 03:02:52,505 WARNING [get_onc_ferry] No ONC TWDP BARPRESS barometric_pressure data for 2025-06-13; substituting empty dataset
2025-06-14 03:02:52,506 INFO [get_onc_ferry] requesting ONC TWDP PYRANOMETER data for 2025-06-13
2025-06-14 03:02:52,559 WARNING [get_onc_ferry] No ONC TWDP PYRANOMETER solar_radiation data for 2025-06-13; substituting empty dataset
2025-06-14 03:02:52,562 WARNING [get_onc_ferry] No ONC TWDP PYRANOMETER solar_radiation data for 2025-06-13; substituting empty dataset
2025-06-14 03:02:52,563 INFO [get_onc_ferry] requesting ONC TWDP PYRGEOMETER data for 2025-06-13
2025-06-14 03:02:52,602 WARNING [get_onc_ferry] No ONC TWDP PYRGEOMETER downward_radiation data for 2025-06-13; substituting empty dataset
2025-06-14 03:02:52,604 WARNING [get_onc_ferry] No ONC TWDP PYRGEOMETER downward_radiation data for 2025-06-13; substituting empty dataset
2025-06-14 03:02:52,865 WARNING [get_onc_ferry] ONC TWDP CO2SENSOR partial_pressure data for 2025-06-13 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2025-06-14 03:02:52,882 WARNING [get_onc_ferry] ONC TWDP CO2SENSOR co2 data for 2025-06-13 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2025-06-14 03:02:52,898 WARNING [get_onc_ferry] ONC TWDP TEMPHUMID air_temperature data for 2025-06-13 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2025-06-14 03:02:52,915 WARNING [get_onc_ferry] ONC TWDP TEMPHUMID rel_humidity data for 2025-06-13 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2025-06-14 03:02:52,930 WARNING [get_onc_ferry] ONC TWDP BARPRESS barometric_pressure data for 2025-06-13 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2025-06-14 03:02:52,948 WARNING [get_onc_ferry] ONC TWDP PYRANOMETER solar_radiation data for 2025-06-13 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2025-06-14 03:02:52,964 WARNING [get_onc_ferry] ONC TWDP PYRGEOMETER downward_radiation data for 2025-06-13 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2025-06-14 03:05:01,053 INFO [get_onc_ferry] 2025-06-13 ONC TWDP ferry data file created
2025-06-14 03:05:01,056 INFO [manager] checklist updated with [ONC ferry data] items from get_onc_ferry worker
2025-06-14 03:05:01,400 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['TWDP-ferry'], host='localhost')
2025-06-14 03:05:02,646 INFO [ping_erddap] running in process 873022
2025-06-14 03:05:02,647 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-14 03:05:02,648 INFO [ping_erddap] publishing log messages to tcp://*:5562
2025-06-14 03:05:02,649 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2025-06-14 03:05:02,651 INFO [ping_erddap] TWDP-ferry ERDDAP dataset flag file(s) created
2025-06-14 03:05:02,653 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2025-06-14 03:06:28,037 INFO [upload_forcing] forecast2 2025-06-14 forcing files upload to arbutus.cloud-nowcast completed
2025-06-14 03:06:28,043 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-06-14 03:06:28,406 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2025-06-14'], host='localhost')
2025-06-14 03:06:29,844 INFO [make_forcing_links] running in process 873041
2025-06-14 03:06:29,846 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-14 03:06:29,847 INFO [make_forcing_links] publishing log messages to tcp://*:5562
2025-06-14 03:06:29,848 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2025-06-14 03:06:34,572 INFO [make_forcing_links] forecast2 2025-06-14 forcing file links on arbutus.cloud-nowcast created
2025-06-14 03:06:34,576 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2025-06-14 03:06:34,932 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2025-06-14'], host='arbutus.cloud-nowcast')
2025-06-14 03:06:40,083 WARNING [run_NEMO] There are uncommitted changes in /nemoShare/MEOPAR/nowcast-sys/SS-run-sets
2025-06-14 03:06:40,285 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2025-06-14 03:06:40,389 INFO [run_NEMO] forecast2: launching /nemoShare/MEOPAR/nowcast-sys/runs/13jun25forecast2_2025-06-14T030639.289654-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2025-06-14 03:06:40,405 INFO [run_NEMO] forecast2 NEMO run for 2025-06-14 on arbutus.cloud-nowcast started
2025-06-14 03:06:40,616 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'forecast2'], host='arbutus.cloud-nowcast')
2025-06-14 03:06:47,008 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2025-06-14 03:11:46,994 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 18509504 = 2025-06-15 05:08:40 UTC, 17.2% complete
2025-06-14 03:16:47,090 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 18509989 = 2025-06-15 10:32:00 UTC, 35.1% complete
2025-06-14 03:21:47,111 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 18510495 = 2025-06-15 16:09:20 UTC, 53.9% complete
2025-06-14 03:26:47,131 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 18510980 = 2025-06-15 21:32:40 UTC, 71.8% complete
2025-06-14 03:31:47,132 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 18511484 = 2025-06-16 03:08:40 UTC, 90.5% complete
2025-06-14 03:36:47,125 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2025-06-14 03:36:47,143 INFO [watch_NEMO] forecast2 NEMO run on arbutus.cloud-nowcast completed
2025-06-14 03:36:47,616 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_wind_file', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2025-06-14'], host='arbutus.cloud-nowcast')
2025-06-14 03:36:47,624 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_current_file', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2025-06-14'], host='arbutus.cloud-nowcast')
2025-06-14 03:36:47,629 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2025-06-13'], host='localhost')
2025-06-14 03:36:48,884 INFO [download_results] running in process 873268
2025-06-14 03:36:48,885 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-14 03:36:48,886 INFO [download_results] publishing log messages to tcp://*:5562
2025-06-14 03:36:48,887 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2025-06-14 03:36:48,888 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/forecast2/13jun25 to /results/SalishSea/forecast2.202111
2025-06-14 03:37:01,872 INFO [make_ww3_wind_file] wwatch3 wind forcing file created on arbutus.cloud-nowcast for 2025-06-14 forecast2 run
2025-06-14 03:37:02,226 INFO [manager] checklist updated with [WW3 wind forcing] items from make_ww3_wind_file worker
2025-06-14 03:40:33,169 INFO [make_ww3_current_file] wwatch3 currents forcing file created on arbutus.cloud-nowcast for 2025-06-14 forecast2 run
2025-06-14 03:40:33,247 INFO [manager] checklist updated with [WW3 currents forcing] items from make_ww3_current_file worker
2025-06-14 03:40:33,622 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2025-06-14'], host='arbutus.cloud-nowcast')
2025-06-14 03:40:38,928 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker
2025-06-14 03:40:39,296 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'forecast2'], host='arbutus.cloud-nowcast')
2025-06-14 03:45:44,516 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 84, 15.6% complete
2025-06-14 03:50:44,491 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 189, 35.0% complete
2025-06-14 03:55:44,524 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 291, 53.9% complete
2025-06-14 04:00:44,519 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 396, 73.3% complete
2025-06-14 04:05:44,499 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 495, 91.7% complete
2025-06-14 04:10:44,408 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2025-06-14 04:10:44,528 INFO [watch_ww3] forecast2 WWATCH3 run on arbutus.cloud-nowcast completed
2025-06-14 04:10:44,975 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2025-06-14'], host='localhost')
2025-06-14 04:10:46,253 INFO [download_wwatch3_results] running in process 873524
2025-06-14 04:10:46,255 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-14 04:10:46,256 INFO [download_wwatch3_results] publishing log messages to tcp://*:5564
2025-06-14 04:10:46,258 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2025-06-14 04:16:42,366 INFO [download_wwatch3_results] forecast2 2025-06-14 results files from arbutus.cloud-nowcast downloaded
2025-06-14 04:16:42,369 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2025-06-14 04:16:42,730 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['wwatch3', 'forecast2', '--run-date', '2025-06-14'], host='localhost')
2025-06-14 04:16:44,232 INFO [update_forecast_datasets] running in process 873563
2025-06-14 04:16:44,233 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-14 04:16:44,235 INFO [update_forecast_datasets] publishing log messages to tcp://*:5564
2025-06-14 04:16:44,236 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2025-06-14 04:16:44,237 INFO [update_forecast_datasets] updating wwatch3 most_recent_forecast directory from forecast2/14jun25 run
2025-06-14 04:16:44,248 INFO [update_forecast_datasets] updated wwatch3 most_recent_forecast directory from forecast2/14jun25 run
2025-06-14 04:16:44,249 INFO [update_forecast_datasets] updating wwatch3 forecast directory for forecast2/14jun25 run
2025-06-14 04:16:44,281 INFO [download_results] forecast2 2025-06-13 results files from arbutus.cloud-nowcast downloaded
2025-06-14 04:16:44,285 INFO [manager] checklist updated with [results files] items from download_results worker
2025-06-14 04:16:44,651 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['forecast2', '--run-date', '2025-06-13'], host='localhost')
2025-06-14 04:16:46,776 INFO [make_CHS_currents_file] running in process 873569
2025-06-14 04:16:46,778 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-14 04:16:46,779 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5562
2025-06-14 04:16:46,780 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555
2025-06-14 04:17:14,818 INFO [update_forecast_datasets] updated wwatch3 forecast directory for forecast2/14jun25 run: /results/SalishSea/rolling-forecasts/wwatch3
2025-06-14 04:17:14,819 INFO [update_forecast_datasets] wwatch3 2025-06-14 forecast2 rolling forecast datasets updated
2025-06-14 04:17:14,823 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2025-06-14 04:17:15,184 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['wwatch3-forecast'], host='localhost')
2025-06-14 04:17:16,413 INFO [ping_erddap] running in process 873594
2025-06-14 04:17:16,415 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-14 04:17:16,416 INFO [ping_erddap] publishing log messages to tcp://*:5564
2025-06-14 04:17:16,474 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2025-06-14 04:17:16,476 INFO [ping_erddap] wwatch3-forecast ERDDAP dataset flag file(s) created
2025-06-14 04:17:16,478 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2025-06-14 04:17:16,834 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['wwatch3', 'forecast', 'publish', '--run-date', '2025-06-13'], host='localhost')
2025-06-14 04:17:20,431 INFO [make_plots] running in process 873599
2025-06-14 04:17:20,434 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-14 04:17:20,435 INFO [make_plots] publishing log messages to tcp://*:5564
2025-06-14 04:17:20,436 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2025-06-14 04:17:20,437 INFO [make_plots] preparing render list for 2025-06-13 WaveWatch3 forecast publish figures
2025-06-14 04:17:20,438 INFO [make_plots] added Halibut Bank figure to 2025-06-13 WaveWatch3 forecast publish render list
2025-06-14 04:17:20,439 INFO [make_plots] added Sentry Shoal figure to 2025-06-13 WaveWatch3 forecast publish render list
2025-06-14 04:17:20,439 INFO [make_plots] starting to render wwatch3 forecast publish 13jun25 figures
2025-06-14 04:17:20,826 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46146 Halibut Bank
2025-06-14 04:17:27,573 INFO [make_CHS_currents_file] Made CHS currents file for 2025-06-13 for forecast2
2025-06-14 04:17:27,576 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker
2025-06-14 04:17:27,949 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['nemo', 'forecast2', '--run-date', '2025-06-13'], host='localhost')
2025-06-14 04:17:28,148 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46131 Sentry Shoal
2025-06-14 04:17:29,041 INFO [update_forecast_datasets] running in process 873629
2025-06-14 04:17:29,043 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-14 04:17:29,044 INFO [update_forecast_datasets] publishing log messages to tcp://*:5562
2025-06-14 04:17:29,045 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2025-06-14 04:17:29,046 INFO [update_forecast_datasets] updating nemo forecast directory for forecast2/13jun25 run
2025-06-14 04:17:32,768 INFO [make_plots] finished rendering wwatch3 forecast publish 13jun25 figures
2025-06-14 04:17:32,769 INFO [make_plots] wwatch3 publish plots for 2025-06-13 forecast completed
2025-06-14 04:17:32,772 INFO [manager] checklist updated with [plots] items from make_plots worker
2025-06-14 04:21:46,610 INFO [update_forecast_datasets] updated nemo forecast directory for forecast2/13jun25 run: /results/SalishSea/rolling-forecasts/nemo
2025-06-14 04:21:46,611 INFO [update_forecast_datasets] nemo 2025-06-13 forecast2 rolling forecast datasets updated
2025-06-14 04:21:46,614 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2025-06-14 04:21:46,952 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nemo-forecast'], host='localhost')
2025-06-14 04:21:46,956 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'forecast2', 'publish', '--run-date', '2025-06-13'], host='localhost')
2025-06-14 04:21:46,961 INFO [manager] launching NextWorker(module='nowcast.workers.make_surface_current_tiles', args=['forecast2', '--run-date', '2025-06-13'], host='localhost')
2025-06-14 04:21:48,287 INFO [ping_erddap] running in process 873723
2025-06-14 04:21:48,288 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-14 04:21:48,289 INFO [ping_erddap] publishing log messages to tcp://*:5562
2025-06-14 04:21:48,290 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2025-06-14 04:21:48,299 INFO [ping_erddap] nemo-forecast ERDDAP dataset flag file(s) created
2025-06-14 04:21:48,300 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2025-06-14 04:21:49,628 INFO [make_surface_current_tiles] running in process 873726
2025-06-14 04:21:49,629 INFO [make_surface_current_tiles] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-14 04:21:49,630 INFO [make_surface_current_tiles] publishing log messages to tcp://*:5562
2025-06-14 04:21:49,630 INFO [make_surface_current_tiles] connected to skookum.eos.ubc.ca port 5555
2025-06-14 04:21:49,689 INFO [make_surface_current_tiles] starting rendering of tiles for 2025-06-13 forecast2 in 6 processes into /results/nowcast-sys/figures/surface_currents/forecast2/13jun25
2025-06-14 04:21:50,386 INFO [make_plots] running in process 873725
2025-06-14 04:21:50,388 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-14 04:21:50,389 INFO [make_plots] publishing log messages to tcp://*:5564
2025-06-14 04:21:50,390 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2025-06-14 04:21:51,046 INFO [make_plots] preparing render list for 2025-06-13 NEMO forecast2 publish figures
2025-06-14 04:21:51,193 INFO [make_plots] added Website_thumbnail figure to 2025-06-13 NEMO forecast2 publish render list
2025-06-14 04:21:51,196 INFO [make_plots] added Threshold_website figure to 2025-06-13 NEMO forecast2 publish render list
2025-06-14 04:21:51,206 INFO [make_plots] added PA_tidal_predictions figure to 2025-06-13 NEMO forecast2 publish render list
2025-06-14 04:21:51,211 INFO [make_plots] added Boundary Bay figure to 2025-06-13 NEMO forecast2 publish render list
2025-06-14 04:21:51,212 INFO [make_plots] added Campbell River figure to 2025-06-13 NEMO forecast2 publish render list
2025-06-14 04:21:51,213 INFO [make_plots] added Cherry Point figure to 2025-06-13 NEMO forecast2 publish render list
2025-06-14 04:21:51,213 INFO [make_plots] added Friday Harbor figure to 2025-06-13 NEMO forecast2 publish render list
2025-06-14 04:21:51,214 INFO [make_plots] added Halfmoon Bay figure to 2025-06-13 NEMO forecast2 publish render list
2025-06-14 04:21:51,215 INFO [make_plots] added Nanaimo figure to 2025-06-13 NEMO forecast2 publish render list
2025-06-14 04:21:51,216 INFO [make_plots] added Neah Bay figure to 2025-06-13 NEMO forecast2 publish render list
2025-06-14 04:21:51,216 INFO [make_plots] added New Westminster figure to 2025-06-13 NEMO forecast2 publish render list
2025-06-14 04:21:51,217 INFO [make_plots] added Patricia Bay figure to 2025-06-13 NEMO forecast2 publish render list
2025-06-14 04:21:51,218 INFO [make_plots] added Point Atkinson figure to 2025-06-13 NEMO forecast2 publish render list
2025-06-14 04:21:51,220 INFO [make_plots] added Port Renfrew figure to 2025-06-13 NEMO forecast2 publish render list
2025-06-14 04:21:51,221 INFO [make_plots] added Sand Heads figure to 2025-06-13 NEMO forecast2 publish render list
2025-06-14 04:21:51,222 INFO [make_plots] added Sandy Cove figure to 2025-06-13 NEMO forecast2 publish render list
2025-06-14 04:21:51,224 INFO [make_plots] added Squamish figure to 2025-06-13 NEMO forecast2 publish render list
2025-06-14 04:21:51,224 INFO [make_plots] added Victoria figure to 2025-06-13 NEMO forecast2 publish render list
2025-06-14 04:21:51,225 INFO [make_plots] added Woodwards Landing figure to 2025-06-13 NEMO forecast2 publish render list
2025-06-14 04:21:51,226 INFO [make_plots] starting to render nemo forecast2 publish 13jun25 figures
2025-06-14 04:29:57,847 WARNING [make_plots] invalid station number for Boundary Bay station: None
2025-06-14 04:29:57,848 WARNING [make_plots] can't resolve a valid CHS station code for Boundary Bay
2025-06-14 04:30:22,935 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb996/data for station 08074 Campbell River from 2025-06-09 00:00:00Z to 2025-06-16 05:55:00Z
2025-06-14 04:31:28,322 WARNING [make_plots] invalid station number for Halfmoon Bay station: None
2025-06-14 04:31:28,323 WARNING [make_plots] can't resolve a valid CHS station code for Halfmoon Bay
2025-06-14 04:31:51,071 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb96d/data for station 07917 Nanaimo from 2025-06-09 00:00:00Z to 2025-06-16 05:55:00Z
2025-06-14 04:32:33,775 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e13d0f4a073c4bbf93/data for station 07654 New Westminster from 2025-06-09 00:00:00Z to 2025-06-16 05:55:00Z
2025-06-14 04:32:54,206 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd2d/data for station 07277 Patricia Bay from 2025-06-09 00:00:00Z to 2025-06-16 05:55:00Z
2025-06-14 04:33:09,072 INFO [make_surface_current_tiles] starting PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast2/13jun25
2025-06-14 04:33:11,603 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94c/data for station 07795 Point Atkinson from 2025-06-09 00:00:00Z to 2025-06-16 05:55:00Z
2025-06-14 04:33:29,242 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e23d0f4a073c4bc060/data for station 08525 Port Renfrew from 2025-06-09 00:00:00Z to 2025-06-16 05:55:00Z
2025-06-14 04:33:44,609 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb93c/data for station 07594 Sand Heads from 2025-06-09 00:00:00Z to 2025-06-16 05:55:00Z
2025-06-14 04:34:02,132 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc43b/data for station 07786 Sandy Cove from 2025-06-09 00:00:00Z to 2025-06-16 05:55:00Z
2025-06-14 04:34:19,451 WARNING [make_plots] invalid station number for Squamish station: None
2025-06-14 04:34:19,452 WARNING [make_plots] can't resolve a valid CHS station code for Squamish
2025-06-14 04:34:36,957 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd1e/data for station 07120 Victoria from 2025-06-09 00:00:00Z to 2025-06-16 05:55:00Z
2025-06-14 04:34:52,505 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc42d/data for station 07610 Woodwards Landing from 2025-06-09 00:00:00Z to 2025-06-16 05:55:00Z
2025-06-14 04:35:10,448 INFO [make_plots] finished rendering nemo forecast2 publish 13jun25 figures
2025-06-14 04:35:10,450 INFO [make_plots] nemo publish plots for 2025-06-13 forecast2 completed
2025-06-14 04:35:10,459 INFO [manager] checklist updated with [plots] items from make_plots worker
2025-06-14 04:35:10,813 INFO [manager] launching NextWorker(module='nowcast.workers.make_feeds', args=['forecast2', '--run-date', '2025-06-13'], host='localhost')
2025-06-14 04:35:14,397 INFO [make_feeds] running in process 875712
2025-06-14 04:35:14,398 INFO [make_feeds] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-14 04:35:14,400 INFO [make_feeds] publishing log messages to tcp://*:5564
2025-06-14 04:35:14,401 INFO [make_feeds] connected to skookum.eos.ubc.ca port 5555
2025-06-14 04:35:28,666 INFO [make_feeds] ATOM feeds for 2025-06-13 forecast2 run completed
2025-06-14 04:35:28,670 INFO [manager] checklist updated with [storm surge alert feeds] items from make_feeds worker
2025-06-14 04:35:29,029 INFO [manager] launching NextWorker(module='nemo_nowcast.workers.clear_checklist', args=[], host='localhost')
2025-06-14 04:35:30,180 INFO [clear_checklist] running in process 875733
2025-06-14 04:35:30,181 INFO [clear_checklist] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-14 04:35:30,182 INFO [clear_checklist] publishing log messages to tcp://*:5564
2025-06-14 04:35:30,183 INFO [clear_checklist] connected to skookum.eos.ubc.ca port 5555
2025-06-14 04:35:30,184 INFO [clear_checklist] requesting that manager clear system state checklist
2025-06-14 04:35:30,186 INFO [manager] writing checklist to log file
2025-06-14 04:35:30,236 INFO [manager] checklist cleared
2025-06-14 04:35:30,239 INFO [clear_checklist] nowcast system checklist cleared
2025-06-14 04:35:30,242 INFO [manager] launching NextWorker(module='nemo_nowcast.workers.rotate_logs', args=[], host='localhost')
2025-06-14 04:35:31,347 INFO [rotate_logs] running in process 875739
2025-06-14 04:35:31,348 INFO [rotate_logs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-14 04:35:31,349 INFO [rotate_logs] publishing log messages to tcp://*:5564
2025-06-14 04:35:31,350 INFO [rotate_logs] connected to skookum.eos.ubc.ca port 5555
2025-06-14 04:35:31,351 INFO [rotate_logs] rotating log files