2025-06-23 03:56:43,201 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/nowcast.log
2025-06-23 03:56:43,206 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/nowcast.debug.log
2025-06-23 03:56:43,210 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/checklist.log
2025-06-23 03:56:43,210 INFO [rotate_logs] log files rotated
2025-06-23 03:56:43,214 INFO [manager] checklist updated with [log rotation] items from rotate_logs worker
2025-06-23 03:58:52,022 INFO [make_surface_current_tiles] finished PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast2/22jun25
2025-06-23 03:58:52,045 INFO [make_surface_current_tiles] finished rendering of tiles for 2025-06-22 forecast2 into /results/nowcast-sys/figures/surface_currents/forecast2/22jun25
2025-06-23 03:58:52,046 INFO [make_surface_current_tiles] surface current tile figures for 2025-06-22 forecast2 completed
2025-06-23 03:58:52,325 INFO [manager] checklist updated with [surface current tiles] items from make_surface_current_tiles worker
2025-06-23 03:59:38,253 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 372, 68.9% complete
2025-06-23 04:04:38,250 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 450, 83.3% complete
2025-06-23 04:09:38,251 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 525, 97.2% complete
2025-06-23 04:14:38,253 INFO [watch_ww3] forecast2 WWATCH3 run on arbutus.cloud-nowcast completed
2025-06-23 04:14:38,255 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2025-06-23 04:14:38,634 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2025-06-23'], host='localhost')
2025-06-23 04:14:39,692 INFO [download_wwatch3_results] running in process 1066257
2025-06-23 04:14:39,693 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-23 04:14:39,693 INFO [download_wwatch3_results] publishing log messages to tcp://*:5562
2025-06-23 04:14:39,694 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2025-06-23 04:15:13,921 INFO [download_wwatch3_results] forecast2 2025-06-23 results files from arbutus.cloud-nowcast downloaded
2025-06-23 04:15:13,924 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2025-06-23 04:15:14,085 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['wwatch3', 'forecast2', '--run-date', '2025-06-23'], host='localhost')
2025-06-23 04:15:15,057 INFO [update_forecast_datasets] running in process 1066269
2025-06-23 04:15:15,058 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-23 04:15:15,059 INFO [update_forecast_datasets] publishing log messages to tcp://*:5562
2025-06-23 04:15:15,060 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2025-06-23 04:15:15,061 INFO [update_forecast_datasets] updating wwatch3 most_recent_forecast directory from forecast2/23jun25 run
2025-06-23 04:15:15,097 INFO [update_forecast_datasets] updated wwatch3 most_recent_forecast directory from forecast2/23jun25 run
2025-06-23 04:15:15,098 INFO [update_forecast_datasets] updating wwatch3 forecast directory for forecast2/23jun25 run
2025-06-23 04:15:36,727 INFO [update_forecast_datasets] updated wwatch3 forecast directory for forecast2/23jun25 run: /results/SalishSea/rolling-forecasts/wwatch3
2025-06-23 04:15:36,728 INFO [update_forecast_datasets] wwatch3 2025-06-23 forecast2 rolling forecast datasets updated
2025-06-23 04:15:36,731 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2025-06-23 04:15:36,892 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['wwatch3-forecast'], host='localhost')
2025-06-23 04:15:37,853 INFO [ping_erddap] running in process 1066284
2025-06-23 04:15:37,854 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-23 04:15:37,855 INFO [ping_erddap] publishing log messages to tcp://*:5562
2025-06-23 04:15:37,856 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2025-06-23 04:15:37,857 INFO [ping_erddap] wwatch3-forecast ERDDAP dataset flag file(s) created
2025-06-23 04:15:37,859 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2025-06-23 04:15:37,987 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['wwatch3', 'forecast2', 'publish', '--run-date', '2025-06-23'], host='localhost')
2025-06-23 04:15:40,641 INFO [make_plots] running in process 1066289
2025-06-23 04:15:40,642 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-23 04:15:40,644 INFO [make_plots] publishing log messages to tcp://*:5562
2025-06-23 04:15:40,645 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2025-06-23 04:15:40,645 INFO [make_plots] preparing render list for 2025-06-23 WaveWatch3 forecast2 publish figures
2025-06-23 04:15:40,646 INFO [make_plots] added Halibut Bank figure to 2025-06-23 WaveWatch3 forecast2 publish render list
2025-06-23 04:15:40,647 INFO [make_plots] added Sentry Shoal figure to 2025-06-23 WaveWatch3 forecast2 publish render list
2025-06-23 04:15:40,648 INFO [make_plots] starting to render wwatch3 forecast2 publish 23jun25 figures
2025-06-23 04:15:41,060 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46146 Halibut Bank
2025-06-23 04:15:47,691 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46131 Sentry Shoal
2025-06-23 04:15:52,099 INFO [make_plots] finished rendering wwatch3 forecast2 publish 23jun25 figures
2025-06-23 04:15:52,100 INFO [make_plots] wwatch3 publish plots for 2025-06-23 forecast2 completed
2025-06-23 04:15:52,103 INFO [manager] checklist updated with [plots] items from make_plots worker
2025-06-23 09:07:38,098 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/12/ to /results/forcing/atmospheric/continental2.5/GRIB/20250623/12/
2025-06-23 09:07:38,099 INFO [collect_weather] 2025-06-23 2.5km weather forecast 12 collection complete
2025-06-23 09:07:38,102 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2025-06-23 09:07:38,424 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'SkagitMountVernon', '--data-date', '2025-06-22'], host='localhost')
2025-06-23 09:07:38,431 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'SnohomishMonroe', '--data-date', '2025-06-22'], host='localhost')
2025-06-23 09:07:38,436 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'NisquallyMcKenna', '--data-date', '2025-06-22'], host='localhost')
2025-06-23 09:07:38,441 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'GreenwaterGreenwater', '--data-date', '2025-06-22'], host='localhost')
2025-06-23 09:07:38,446 INFO [manager] launching NextWorker(module='nowcast.workers.make_turbidity_file', args=[], host='localhost')
2025-06-23 09:07:38,451 INFO [manager] launching NextWorker(module='nowcast.workers.collect_NeahBay_ssh', args=['06'], host='localhost')
2025-06-23 09:07:38,455 INFO [manager] launching NextWorker(module='nowcast.workers.download_live_ocean', args=[], host='localhost')
2025-06-23 09:07:38,461 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['18', '2.5km'], host='localhost')
2025-06-23 09:07:38,471 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['18'], host='localhost')
2025-06-23 09:07:38,743 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20250623/12/
2025-06-23 09:07:38,744 INFO [crop_gribs] 2025-06-23 12 GRIBs cropping complete
2025-06-23 09:07:38,746 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2025-06-23 09:07:39,308 INFO [manager] launching NextWorker(module='nowcast.workers.grib_to_netcdf', args=['nowcast+'], host='localhost')
2025-06-23 09:07:40,174 INFO [collect_NeahBay_ssh] running in process 1068779
2025-06-23 09:07:40,175 INFO [collect_NeahBay_ssh] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-23 09:07:40,175 INFO [collect_NeahBay_ssh] publishing log messages to tcp://*:5562
2025-06-23 09:07:40,176 INFO [collect_NeahBay_ssh] connected to skookum.eos.ubc.ca port 5555
2025-06-23 09:07:40,177 INFO [collect_NeahBay_ssh] collecting Neah Bay ssh 06Z obs/forecast for 2025-06-23
2025-06-23 09:07:40,178 INFO [collect_weather] running in process 1068781
2025-06-23 09:07:40,179 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-23 09:07:40,180 INFO [collect_weather] publishing log messages to tcp://*:5563
2025-06-23 09:07:40,181 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2025-06-23 09:07:40,219 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/18/
2025-06-23 09:07:40,286 INFO [download_live_ocean] running in process 1068780
2025-06-23 09:07:40,287 INFO [download_live_ocean] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-23 09:07:40,288 INFO [download_live_ocean] publishing log messages to tcp://*:5564
2025-06-23 09:07:40,289 INFO [download_live_ocean] connected to skookum.eos.ubc.ca port 5555
2025-06-23 09:07:40,289 INFO [download_live_ocean] downloading Salish Sea western boundary day-averaged LiveOcean file for 2025-06-23
2025-06-23 09:07:40,600 INFO [collect_river_data] running in process 1068777
2025-06-23 09:07:40,601 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-23 09:07:40,602 INFO [collect_river_data] publishing log messages to tcp://*:5565
2025-06-23 09:07:40,603 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-06-23 09:07:40,603 INFO [collect_river_data] Collecting USGS GreenwaterGreenwater river data for 2025-06-22
2025-06-23 09:07:40,643 INFO [collect_river_data] running in process 1068773
2025-06-23 09:07:40,644 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-23 09:07:40,645 INFO [collect_river_data] publishing log messages to tcp://*:5566
2025-06-23 09:07:40,645 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-06-23 09:07:40,646 INFO [collect_river_data] Collecting USGS SkagitMountVernon river data for 2025-06-22
2025-06-23 09:07:40,773 INFO [collect_river_data] running in process 1068775
2025-06-23 09:07:40,774 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-23 09:07:40,775 INFO [collect_river_data] publishing log messages to tcp://*:5567
2025-06-23 09:07:40,776 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-06-23 09:07:40,776 INFO [collect_river_data] Collecting USGS SnohomishMonroe river data for 2025-06-22
2025-06-23 09:07:40,810 INFO [collect_river_data] running in process 1068776
2025-06-23 09:07:40,810 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-23 09:07:40,811 INFO [collect_river_data] publishing log messages to tcp://*:5568
2025-06-23 09:07:40,812 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-06-23 09:07:40,813 INFO [collect_river_data] Collecting USGS NisquallyMcKenna river data for 2025-06-22
2025-06-23 09:07:40,829 INFO [make_turbidity_file] running in process 1068778
2025-06-23 09:07:40,830 INFO [make_turbidity_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-23 09:07:40,831 INFO [make_turbidity_file] publishing log messages to tcp://*:5569
2025-06-23 09:07:40,832 INFO [make_turbidity_file] connected to skookum.eos.ubc.ca port 5555
2025-06-23 09:07:40,832 INFO [make_turbidity_file] Creating Fraser River turbidity forcing file for 2025-06-23
2025-06-23 09:07:40,998 INFO [crop_gribs] running in process 1068782
2025-06-23 09:07:40,999 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-23 09:07:41,000 INFO [crop_gribs] publishing log messages to tcp://*:5570
2025-06-23 09:07:41,001 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2025-06-23 09:07:41,001 INFO [crop_gribs] cropping 2025-06-23 ECCC HRDPS 2.5km continental 18Z GRIB files to SalishSeaCast subdomain
2025-06-23 09:07:41,013 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20250623/18/
2025-06-23 09:07:41,163 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12089500&startDT=2025-06-22&endDT=2025-06-22 "HTTP/1.1 200 "
2025-06-23 09:07:41,183 INFO [collect_river_data] Appended USGS NisquallyMcKenna river average discharge for 2025-06-22 to: /results/forcing/rivers/observations/Nisqually_McKenna_flow
2025-06-23 09:07:41,183 INFO [collect_river_data] USGS NisquallyMcKenna river data collection for 2025-06-22 completed
2025-06-23 09:07:41,186 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-06-23 09:07:41,191 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12200500&startDT=2025-06-22&endDT=2025-06-22 "HTTP/1.1 200 "
2025-06-23 09:07:41,195 INFO [collect_river_data] Appended USGS SkagitMountVernon river average discharge for 2025-06-22 to: /results/forcing/rivers/observations/Skagit_MountVernon_flow
2025-06-23 09:07:41,195 INFO [collect_river_data] USGS SkagitMountVernon river data collection for 2025-06-22 completed
2025-06-23 09:07:41,196 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12097500&startDT=2025-06-22&endDT=2025-06-22 "HTTP/1.1 200 "
2025-06-23 09:07:41,211 INFO [collect_NeahBay_ssh] 2025-06-23T00:00:00-07:00 Neah Bay ssh 06Z obs/forecast data collection complete
2025-06-23 09:07:41,216 INFO [grib_to_netcdf] running in process 1068855
2025-06-23 09:07:41,217 INFO [grib_to_netcdf] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-23 09:07:41,218 INFO [grib_to_netcdf] publishing log messages to tcp://*:5571
2025-06-23 09:07:41,219 INFO [grib_to_netcdf] connected to skookum.eos.ubc.ca port 5555
2025-06-23 09:07:41,220 INFO [grib_to_netcdf] creating NEMO-atmos forcing files for 2025-06-23 nowcast and forecast runs
2025-06-23 09:07:41,224 INFO [collect_river_data] Appended USGS GreenwaterGreenwater river average discharge for 2025-06-22 to: /results/forcing/rivers/observations/Greenwater_Greenwater_flow
2025-06-23 09:07:41,225 INFO [collect_river_data] USGS GreenwaterGreenwater river data collection for 2025-06-22 completed
2025-06-23 09:07:41,299 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12150800&startDT=2025-06-22&endDT=2025-06-22 "HTTP/1.1 200 "
2025-06-23 09:07:41,303 INFO [collect_river_data] Appended USGS SnohomishMonroe river average discharge for 2025-06-22 to: /results/forcing/rivers/observations/Snohomish_Monroe_flow
2025-06-23 09:07:41,304 INFO [collect_river_data] USGS SnohomishMonroe river data collection for 2025-06-22 completed
2025-06-23 09:07:41,350 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-06-23 09:07:41,509 INFO [manager] checklist updated with [Neah Bay ssh data] items from collect_NeahBay_ssh worker
2025-06-23 09:07:41,667 INFO [manager] launching NextWorker(module='nowcast.workers.make_ssh_files', args=['nowcast', '--run-date', '2025-06-23'], host='localhost')
2025-06-23 09:07:41,672 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-06-23 09:07:41,853 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-06-23 09:07:41,908 INFO [download_live_ocean] downloaded 10210603 bytes from https://liveocean.apl.uw.edu/output/f2025.06.23/ubc.nc to /results/forcing/LiveOcean/downloaded/20250623/low_passed_UBC.nc
2025-06-23 09:07:41,908 INFO [download_live_ocean] Deflating in up to 1 concurrent sub-processes
2025-06-23 09:07:42,937 INFO [download_live_ocean] netCDF4 deflated /results/forcing/LiveOcean/downloaded/20250623/low_passed_UBC.nc
2025-06-23 09:07:42,937 INFO [download_live_ocean] 2025-06-23 Live Ocean file for Salish Sea western boundary downloaded
2025-06-23 09:07:42,940 INFO [manager] checklist updated with [Live Ocean products] items from download_live_ocean worker
2025-06-23 09:07:43,113 INFO [make_turbidity_file] 2025-06-23 Fraser River turbidity file creation complete
2025-06-23 09:07:43,241 INFO [manager] launching NextWorker(module='nowcast.workers.make_live_ocean_files', args=['--run-date', '2025-06-23'], host='localhost')
2025-06-23 09:07:43,246 INFO [manager] checklist updated with [Fraser River turbidity] items from make_turbidity_file worker
2025-06-23 09:07:44,349 INFO [make_ssh_files] running in process 1068911
2025-06-23 09:07:44,351 INFO [make_ssh_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-23 09:07:44,351 INFO [make_ssh_files] publishing log messages to tcp://*:5562
2025-06-23 09:07:44,352 INFO [make_ssh_files] connected to skookum.eos.ubc.ca port 5555
2025-06-23 09:07:44,352 INFO [make_ssh_files] building nowcast sea surface height boundary conditions file(s) from 2025-06-23 NOAA Neah Bay 06Z observation and forecast values
2025-06-23 09:07:45,245 INFO [make_live_ocean_files] running in process 1068929
2025-06-23 09:07:45,246 INFO [make_live_ocean_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-23 09:07:45,247 INFO [make_live_ocean_files] publishing log messages to tcp://*:5564
2025-06-23 09:07:45,248 INFO [make_live_ocean_files] connected to skookum.eos.ubc.ca port 5555
2025-06-23 09:07:45,249 INFO [make_live_ocean_files] Creating T&S western boundary conditions file from 2025-06-23 Live Ocean run
2025-06-23 09:07:46,150 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2025m06d19.nc
2025-06-23 09:07:46,324 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2025m06d20.nc
2025-06-23 09:07:46,501 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2025m06d21.nc
2025-06-23 09:07:46,674 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2025m06d22.nc
2025-06-23 09:07:46,846 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2025m06d23.nc
2025-06-23 09:07:47,037 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2025m06d24.nc
2025-06-23 09:07:47,221 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2025m06d25.nc
2025-06-23 09:07:47,409 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2025m06d26.nc
2025-06-23 09:07:47,552 INFO [make_ssh_files] sea surface height boundary file for 2025-06-23 nowcast run created
2025-06-23 09:07:47,555 INFO [manager] checklist updated with [sea surface height forcing] items from make_ssh_files worker
2025-06-23 09:07:47,718 INFO [manager] launching NextWorker(module='nowcast.workers.make_runoff_file', args=['v202108'], host='localhost')
2025-06-23 09:07:47,722 INFO [manager] launching NextWorker(module='nowcast.workers.make_201702_runoff_file', args=[], host='localhost')
2025-06-23 09:07:48,943 INFO [make_201702_runoff_file] running in process 1068960
2025-06-23 09:07:48,944 INFO [make_201702_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-23 09:07:48,945 INFO [make_201702_runoff_file] publishing log messages to tcp://*:5562
2025-06-23 09:07:48,946 INFO [make_201702_runoff_file] connected to skookum.eos.ubc.ca port 5555
2025-06-23 09:07:48,967 INFO [make_201702_runoff_file] calculating runoff forcing for b201702 bathymetry
2025-06-23 09:07:49,071 INFO [make_201702_runoff_file] 2025-06-23 runoff file creation from Fraser at Hope and climatology elsewhere complete
2025-06-23 09:07:49,074 INFO [manager] checklist updated with [rivers forcing] items from make_201702_runoff_file worker
2025-06-23 09:07:49,421 INFO [make_runoff_file] running in process 1068958
2025-06-23 09:07:49,423 INFO [make_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-23 09:07:49,423 INFO [make_runoff_file] publishing log messages to tcp://*:5565
2025-06-23 09:07:49,424 INFO [make_runoff_file] connected to skookum.eos.ubc.ca port 5555
2025-06-23 09:07:49,429 INFO [make_runoff_file] calculating NEMO runoff forcing for v202108 bathymetry for 2025-06-22
2025-06-23 09:07:51,768 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/hrdps_y2025m06d23.nc
2025-06-23 09:07:52,181 INFO [make_runoff_file] stored NEMO runoff forcing for v202108 bathymetry for 2025-06-22: /results/forcing/rivers/R202108Dailies_y2025m06d22.nc
2025-06-23 09:07:52,183 INFO [make_runoff_file] 2025-06-22 runoff file creation completed
2025-06-23 09:07:52,184 INFO [manager] checklist updated with [rivers forcing] items from make_runoff_file worker
2025-06-23 09:07:59,096 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2025m06d24.nc
2025-06-23 09:08:03,124 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2025m06d25.nc
2025-06-23 09:08:03,139 INFO [grib_to_netcdf] 2025-06-23 NEMO-atmos forcing file for nowcast+ created
2025-06-23 09:08:03,142 INFO [manager] checklist updated with [weather forcing] items from grib_to_netcdf worker
2025-06-23 09:08:24,897 INFO [make_live_ocean_files] Stored T&S western boundary conditions file: /results/forcing/LiveOcean/boundary_conditions/LiveOcean_v201905_y2025m06d23.nc
2025-06-23 09:08:24,897 INFO [make_live_ocean_files] 2025-06-23 Live Ocean western boundary conditions files created
2025-06-23 09:08:24,899 INFO [manager] checklist updated with [Live Ocean boundary conditions] items from make_live_ocean_files worker
2025-06-23 09:08:25,063 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['weather'], host='localhost')
2025-06-23 09:08:25,067 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'nowcast+'], host='localhost')
2025-06-23 09:08:25,072 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'nowcast+'], host='localhost')
2025-06-23 09:08:25,077 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.graham', 'nowcast+'], host='localhost')
2025-06-23 09:08:25,082 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'nowcast+'], host='localhost')
2025-06-23 09:08:26,074 INFO [ping_erddap] running in process 1069005
2025-06-23 09:08:26,075 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-23 09:08:26,076 INFO [ping_erddap] publishing log messages to tcp://*:5562
2025-06-23 09:08:26,077 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2025-06-23 09:08:26,078 INFO [ping_erddap] weather ERDDAP dataset flag file(s) created
2025-06-23 09:08:26,080 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2025-06-23 09:08:26,173 INFO [upload_forcing] running in process 1069008
2025-06-23 09:08:26,174 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-23 09:08:26,175 INFO [upload_forcing] publishing log messages to tcp://*:5564
2025-06-23 09:08:26,176 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-06-23 09:08:26,265 INFO [upload_forcing] running in process 1069007
2025-06-23 09:08:26,266 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-23 09:08:26,267 INFO [upload_forcing] publishing log messages to tcp://*:5565
2025-06-23 09:08:26,268 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-06-23 09:08:26,287 INFO [upload_forcing] running in process 1069010
2025-06-23 09:08:26,289 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-23 09:08:26,289 INFO [upload_forcing] publishing log messages to tcp://*:5566
2025-06-23 09:08:26,290 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-06-23 09:08:26,291 INFO [upload_forcing] running in process 1069009
2025-06-23 09:08:26,292 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-23 09:08:26,292 INFO [upload_forcing] publishing log messages to tcp://*:5567
2025-06-23 09:08:26,293 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-06-23 09:08:28,445 INFO [upload_forcing] nowcast+ 2025-06-23 forcing files upload to optimum-hindcast completed
2025-06-23 09:08:28,449 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-06-23 09:08:28,498 INFO [upload_forcing] nowcast+ 2025-06-23 forcing files upload to orcinus-nowcast-agrif completed
2025-06-23 09:08:28,616 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-06-23 09:08:35,898 INFO [upload_forcing] nowcast+ 2025-06-23 forcing files upload to robot.graham completed
2025-06-23 09:08:35,905 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-06-23 09:09:29,519 INFO [upload_forcing] nowcast+ 2025-06-23 forcing files upload to arbutus.cloud-nowcast completed
2025-06-23 09:09:29,524 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-06-23 09:09:29,688 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'nowcast+', '--run-date', '2025-06-23'], host='localhost')
2025-06-23 09:09:30,828 INFO [make_forcing_links] running in process 1069089
2025-06-23 09:09:30,829 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-23 09:09:30,830 INFO [make_forcing_links] publishing log messages to tcp://*:5562
2025-06-23 09:09:30,831 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2025-06-23 09:09:32,445 INFO [make_forcing_links] nowcast+ 2025-06-23 forcing file links on arbutus.cloud-nowcast created
2025-06-23 09:09:32,449 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2025-06-23 09:09:32,605 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2025-06-23'], host='arbutus.cloud-nowcast')
2025-06-23 09:09:35,171 INFO [run_NEMO] running in process 1699
2025-06-23 09:09:35,172 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-06-23 09:09:35,173 INFO [run_NEMO] publishing log messages to tcp://*:5556
2025-06-23 09:09:35,189 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2025-06-23 09:09:35,859 WARNING [run_NEMO] There are uncommitted changes in /nemoShare/MEOPAR/nowcast-sys/SS-run-sets
2025-06-23 09:09:36,501 INFO [run_NEMO] nowcast: launching /nemoShare/MEOPAR/nowcast-sys/runs/23jun25nowcast_2025-06-23T090935.212133-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2025-06-23 09:09:36,513 INFO [run_NEMO] nowcast NEMO run for 2025-06-23 on arbutus.cloud-nowcast started
2025-06-23 09:09:36,521 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2025-06-23 09:09:36,688 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'nowcast'], host='arbutus.cloud-nowcast')
2025-06-23 09:09:38,997 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2025-06-23 09:14:38,996 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 18526784 = 2025-06-23 05:08:40 UTC, 21.4% complete
2025-06-23 09:19:38,997 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 18527324 = 2025-06-23 11:08:40 UTC, 46.5% complete
2025-06-23 09:24:38,998 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 18527846 = 2025-06-23 16:56:40 UTC, 70.6% complete
2025-06-23 09:29:38,999 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 18528359 = 2025-06-23 22:38:40 UTC, 94.4% complete
2025-06-23 09:34:39,004 INFO [watch_NEMO] nowcast NEMO run on arbutus.cloud-nowcast completed
2025-06-23 09:34:39,011 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2025-06-23 09:34:39,343 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'ssh', '--run-date', '2025-06-23'], host='localhost')
2025-06-23 09:34:39,350 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2025-06-23'], host='localhost')
2025-06-23 09:34:40,478 INFO [download_results] running in process 1069294
2025-06-23 09:34:40,480 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-23 09:34:40,481 INFO [download_results] publishing log messages to tcp://*:5562
2025-06-23 09:34:40,482 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2025-06-23 09:34:40,482 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/nowcast/23jun25 to /results/SalishSea/nowcast-blue.202111
2025-06-23 09:34:40,484 INFO [make_forcing_links] running in process 1069293
2025-06-23 09:34:40,485 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-23 09:34:40,485 INFO [make_forcing_links] publishing log messages to tcp://*:5564
2025-06-23 09:34:40,486 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2025-06-23 09:34:41,475 INFO [make_forcing_links] ssh 2025-06-23 forcing file links on arbutus.cloud-nowcast created
2025-06-23 09:34:41,479 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2025-06-23 09:34:41,646 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2025-06-23'], host='arbutus.cloud-nowcast')
2025-06-23 09:34:45,150 WARNING [run_NEMO] There are uncommitted changes in /nemoShare/MEOPAR/nowcast-sys/SS-run-sets
2025-06-23 09:34:45,459 INFO [run_NEMO] forecast: launching /nemoShare/MEOPAR/nowcast-sys/runs/23jun25forecast_2025-06-23T093444.435413-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2025-06-23 09:34:45,463 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2025-06-23 09:34:45,471 INFO [run_NEMO] forecast NEMO run for 2025-06-23 on arbutus.cloud-nowcast started
2025-06-23 09:34:45,668 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'forecast'], host='arbutus.cloud-nowcast')
2025-06-23 09:34:48,025 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2025-06-23 09:35:50,237 INFO [download_results] nowcast 2025-06-23 results files from arbutus.cloud-nowcast downloaded
2025-06-23 09:35:50,250 INFO [manager] checklist updated with [results files] items from download_results worker
2025-06-23 09:35:50,414 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast', 'research', '--run-date', '2025-06-23'], host='localhost')
2025-06-23 09:35:50,419 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast', 'comparison', '--run-date', '2025-06-22'], host='localhost')
2025-06-23 09:35:50,423 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['nowcast', '--run-date', '2025-06-23'], host='localhost')
2025-06-23 09:35:52,223 INFO [make_CHS_currents_file] running in process 1069324
2025-06-23 09:35:52,224 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-23 09:35:52,225 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5562
2025-06-23 09:35:52,225 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555
2025-06-23 09:35:53,205 INFO [make_plots] running in process 1069321
2025-06-23 09:35:53,206 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-23 09:35:53,207 INFO [make_plots] publishing log messages to tcp://*:5564
2025-06-23 09:35:53,208 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2025-06-23 09:35:53,232 INFO [make_plots] running in process 1069323
2025-06-23 09:35:53,250 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-23 09:35:53,252 INFO [make_plots] publishing log messages to tcp://*:5565
2025-06-23 09:35:53,253 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2025-06-23 09:35:53,574 INFO [make_plots] preparing render list for 2025-06-22 NEMO nowcast-blue comparison figures
2025-06-23 09:35:53,595 INFO [make_plots] preparing render list for 2025-06-23 NEMO nowcast-blue research figures
2025-06-23 09:35:53,613 INFO [make_plots] added SH_wind figure to 2025-06-22 NEMO nowcast-blue comparison render list
2025-06-23 09:35:53,614 INFO [make_plots] added Compare_VENUS_East figure to 2025-06-22 NEMO nowcast-blue comparison render list
2025-06-23 09:35:53,615 INFO [make_plots] added Compare_VENUS_Central figure to 2025-06-22 NEMO nowcast-blue comparison render list
2025-06-23 09:35:53,615 INFO [make_plots] starting to render nemo nowcast comparison 22jun25 figures
2025-06-23 09:35:55,089 INFO [make_plots] added salinity_thalweg_and_surface figures to 2025-06-23 NEMO nowcast-blue research render list
2025-06-23 09:35:56,194 INFO [make_plots] added temperature_thalweg_and_surface figures to 2025-06-23 NEMO nowcast-blue research render list
2025-06-23 09:35:56,194 INFO [make_plots] added Currents_sections_and_surface figure to 2025-06-23 NEMO nowcast-blue research render list
2025-06-23 09:35:56,195 INFO [make_plots] added Currents_at_VENUS_Central figure to 2025-06-23 NEMO nowcast-blue research render list
2025-06-23 09:35:56,196 INFO [make_plots] added Currents_at_VENUS_East figure to 2025-06-23 NEMO nowcast-blue research render list
2025-06-23 09:35:56,197 INFO [make_plots] starting to render nemo nowcast research 23jun25 figures
2025-06-23 09:36:26,011 INFO [make_CHS_currents_file] Made CHS currents file for 2025-06-23 for nowcast
2025-06-23 09:36:26,014 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker
2025-06-23 09:37:04,942 INFO [make_plots] finished rendering nemo nowcast research 23jun25 figures
2025-06-23 09:37:04,944 INFO [make_plots] nemo research plots for 2025-06-23 nowcast completed
2025-06-23 09:37:04,965 INFO [manager] checklist updated with [plots] items from make_plots worker
2025-06-23 09:37:23,563 INFO [make_plots] finished rendering nemo nowcast comparison 22jun25 figures
2025-06-23 09:37:23,565 INFO [make_plots] nemo comparison plots for 2025-06-22 nowcast completed
2025-06-23 09:37:23,570 INFO [manager] checklist updated with [plots] items from make_plots worker
2025-06-23 09:39:48,048 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 18528945 = 2025-06-24 05:09:20 UTC, 14.3% complete
2025-06-23 09:44:48,061 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 18529429 = 2025-06-24 10:32:00 UTC, 29.3% complete
2025-06-23 09:49:48,002 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 18529923 = 2025-06-24 16:01:20 UTC, 44.5% complete
2025-06-23 09:54:48,124 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 18530405 = 2025-06-24 21:22:40 UTC, 59.4% complete
2025-06-23 09:59:48,010 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 18530896 = 2025-06-25 02:50:00 UTC, 74.6% complete
2025-06-23 10:04:48,072 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 18531383 = 2025-06-25 08:14:40 UTC, 89.6% complete
2025-06-23 10:09:48,017 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2025-06-23 10:09:48,060 INFO [watch_NEMO] forecast NEMO run on arbutus.cloud-nowcast completed
2025-06-23 10:09:48,355 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_wind_file', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2025-06-23'], host='arbutus.cloud-nowcast')
2025-06-23 10:09:48,364 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_current_file', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2025-06-23'], host='arbutus.cloud-nowcast')
2025-06-23 10:09:48,369 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'turbidity', '--run-date', '2025-06-23'], host='localhost')
2025-06-23 10:09:48,375 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'turbidity', '--run-date', '2025-06-23'], host='localhost')
2025-06-23 10:09:48,378 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.graham', 'turbidity', '--run-date', '2025-06-23'], host='localhost')
2025-06-23 10:09:48,383 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'turbidity', '--run-date', '2025-06-23'], host='localhost')
2025-06-23 10:09:48,388 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2025-06-23'], host='localhost')
2025-06-23 10:09:49,501 INFO [download_results] running in process 1069663
2025-06-23 10:09:49,502 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-23 10:09:49,503 INFO [download_results] publishing log messages to tcp://*:5562
2025-06-23 10:09:49,504 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2025-06-23 10:09:49,505 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/forecast/23jun25 to /results/SalishSea/forecast.202111
2025-06-23 10:09:49,539 INFO [upload_forcing] running in process 1069659
2025-06-23 10:09:49,540 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-23 10:09:49,541 INFO [upload_forcing] publishing log messages to tcp://*:5564
2025-06-23 10:09:49,542 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-06-23 10:09:49,544 INFO [upload_forcing] running in process 1069660
2025-06-23 10:09:49,545 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-23 10:09:49,546 INFO [upload_forcing] publishing log messages to tcp://*:5565
2025-06-23 10:09:49,547 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-06-23 10:09:49,566 INFO [upload_forcing] running in process 1069661
2025-06-23 10:09:49,568 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-23 10:09:49,569 INFO [upload_forcing] publishing log messages to tcp://*:5566
2025-06-23 10:09:49,570 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-06-23 10:09:49,581 INFO [upload_forcing] running in process 1069662
2025-06-23 10:09:49,582 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-23 10:09:49,583 INFO [upload_forcing] publishing log messages to tcp://*:5567
2025-06-23 10:09:49,584 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-06-23 10:09:50,188 INFO [upload_forcing] turbidity 2025-06-23 forcing files upload to orcinus-nowcast-agrif completed
2025-06-23 10:09:50,192 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-06-23 10:09:50,217 INFO [upload_forcing] turbidity 2025-06-23 forcing files upload to optimum-hindcast completed
2025-06-23 10:09:50,338 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['orcinus-nowcast-agrif', 'nowcast-agrif', '--run-date', '2025-06-23'], host='localhost')
2025-06-23 10:09:50,344 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-06-23 10:09:51,124 INFO [upload_forcing] turbidity 2025-06-23 forcing files upload to arbutus.cloud-nowcast completed
2025-06-23 10:09:51,128 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-06-23 10:09:51,278 INFO [make_ww3_current_file] running in process 3056
2025-06-23 10:09:51,280 INFO [make_ww3_current_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-06-23 10:09:51,296 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2025-06-23'], host='localhost')
2025-06-23 10:09:51,306 INFO [make_ww3_current_file] publishing log messages to tcp://*:5571
2025-06-23 10:09:51,306 INFO [make_ww3_current_file] connected to skookum.eos.ubc.ca port 5555
2025-06-23 10:09:51,307 INFO [make_ww3_current_file] Creating wwatch3 currents forcing file for 2025-06-23 forecast run
2025-06-23 10:09:51,416 INFO [make_forcing_links] running in process 1069694
2025-06-23 10:09:51,417 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-23 10:09:51,417 INFO [make_forcing_links] publishing log messages to tcp://*:5565
2025-06-23 10:09:51,418 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2025-06-23 10:09:52,206 INFO [make_forcing_links] nowcast-agrif 2025-06-23 forcing file links on orcinus-nowcast-agrif created
2025-06-23 10:09:52,210 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2025-06-23 10:09:52,356 INFO [make_forcing_links] running in process 1069700
2025-06-23 10:09:52,357 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-23 10:09:52,358 INFO [make_forcing_links] publishing log messages to tcp://*:5564
2025-06-23 10:09:52,359 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2025-06-23 10:09:52,382 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_agrif', args=['orcinus-nowcast-agrif', 'nowcast-agrif', '--run-date', '2025-06-23'], host='localhost')
2025-06-23 10:09:53,454 INFO [run_NEMO_agrif] running in process 1069709
2025-06-23 10:09:53,455 INFO [run_NEMO_agrif] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-23 10:09:53,456 INFO [run_NEMO_agrif] publishing log messages to tcp://*:5565
2025-06-23 10:09:53,457 INFO [run_NEMO_agrif] connected to skookum.eos.ubc.ca port 5555
2025-06-23 10:09:53,661 INFO [upload_forcing] turbidity 2025-06-23 forcing files upload to robot.graham completed
2025-06-23 10:09:53,665 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-06-23 10:09:54,956 INFO [make_forcing_links] nowcast-green 2025-06-23 forcing file links on arbutus.cloud-nowcast created
2025-06-23 10:09:54,961 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2025-06-23 10:09:55,133 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2025-06-23'], host='arbutus.cloud-nowcast')
2025-06-23 10:10:00,077 INFO [run_NEMO] running in process 3737
2025-06-23 10:10:00,078 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-06-23 10:10:00,079 INFO [run_NEMO] publishing log messages to tcp://*:5556
2025-06-23 10:10:00,081 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2025-06-23 10:10:01,545 WARNING [run_NEMO] There are uncommitted changes in /nemoShare/MEOPAR/nowcast-sys/SS-run-sets
2025-06-23 10:10:01,887 INFO [run_NEMO] nowcast-green: launching /nemoShare/MEOPAR/nowcast-sys/runs/23jun25nowcast-green_2025-06-23T101000.164862-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2025-06-23 10:10:01,897 INFO [run_NEMO] nowcast-green NEMO run for 2025-06-23 on arbutus.cloud-nowcast started
2025-06-23 10:10:01,901 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2025-06-23 10:10:02,072 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'nowcast-green'], host='arbutus.cloud-nowcast')
2025-06-23 10:10:03,375 INFO [make_ww3_wind_file] wwatch3 wind forcing file created on arbutus.cloud-nowcast for 2025-06-23 forecast run
2025-06-23 10:10:03,388 INFO [manager] checklist updated with [WW3 wind forcing] items from make_ww3_wind_file worker
2025-06-23 10:10:04,302 INFO [watch_NEMO] running in process 4021
2025-06-23 10:10:04,303 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-06-23 10:10:04,304 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2025-06-23 10:10:04,305 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2025-06-23 10:10:04,650 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2025-06-23 10:10:05,483 INFO [run_NEMO_agrif] job id for 23jun25nowcast-agrif: 11431978.orca2.ibb
2025-06-23 10:10:05,484 INFO [run_NEMO_agrif] NEMO nowcast-agrif run for 2025-06-23 queued on orcinus-nowcast-agrif
2025-06-23 10:10:05,489 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_agrif worker
2025-06-23 10:10:05,661 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_agrif', args=['orcinus-nowcast-agrif', '11431978.orca2.ibb'], host='localhost')
2025-06-23 10:10:06,714 INFO [watch_NEMO_agrif] running in process 1069730
2025-06-23 10:10:06,715 INFO [watch_NEMO_agrif] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-23 10:10:06,716 INFO [watch_NEMO_agrif] publishing log messages to tcp://*:5564
2025-06-23 10:10:06,717 INFO [watch_NEMO_agrif] connected to skookum.eos.ubc.ca port 5555
2025-06-23 10:10:07,510 INFO [watch_NEMO_agrif] watching 23jun25nowcast-agrif job 11431978 on orcinus-nowcast-agrif
2025-06-23 10:10:07,743 INFO [watch_NEMO_agrif] 23jun25nowcast-agrif job 11431978 is queued on orcinus-nowcast-agrif
2025-06-23 10:11:07,954 INFO [watch_NEMO_agrif] 23jun25nowcast-agrif job 11431978 is queued on orcinus-nowcast-agrif
2025-06-23 10:12:08,862 INFO [watch_NEMO_agrif] 23jun25nowcast-agrif on orcinus-nowcast-agrif: time.step not found; continuing to watch...
2025-06-23 10:12:25,068 INFO [manager] checklist updated with [WW3 currents forcing] items from make_ww3_current_file worker
2025-06-23 10:12:25,084 INFO [make_ww3_current_file] wwatch3 currents forcing file created on arbutus.cloud-nowcast for 2025-06-23 forecast run
2025-06-23 10:12:25,240 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2025-06-23'], host='arbutus.cloud-nowcast')
2025-06-23 10:12:27,684 INFO [run_ww3] running in process 4129
2025-06-23 10:12:27,685 INFO [run_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-06-23 10:12:27,686 INFO [run_ww3] publishing log messages to tcp://*:5572
2025-06-23 10:12:27,690 INFO [run_ww3] connected to skookum.eos.ubc.ca port 5555
2025-06-23 10:12:27,696 INFO [run_ww3] Created run directory /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/nowcast_2025-06-23T101227.671495-0700
2025-06-23 10:12:27,699 INFO [run_ww3] nowcast: launching /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/nowcast_2025-06-23T101227.671495-0700/SoGWW3.sh on arbutus.cloud-nowcast
2025-06-23 10:12:27,712 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker
2025-06-23 10:12:27,713 INFO [run_ww3] nowcast WaveWatch3 run for 2025-06-23 on arbutus.cloud-nowcast started
2025-06-23 10:12:27,880 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'nowcast'], host='arbutus.cloud-nowcast')
2025-06-23 10:15:04,665 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18526487 = 2025-06-23 01:50:40 UTC, 7.7% complete
2025-06-23 10:17:09,313 INFO [watch_NEMO_agrif] 23jun25nowcast-agrif on orcinus-nowcast-agrif: timestep: 8547325 = 2025-06-23 02:16:00 UTC, 9.4% complete
2025-06-23 10:17:30,040 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 66, 15.3% complete
2025-06-23 10:20:04,651 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18526672 = 2025-06-23 03:54:00 UTC, 16.3% complete
2025-06-23 10:21:12,840 INFO [download_results] forecast 2025-06-23 results files from arbutus.cloud-nowcast downloaded
2025-06-23 10:21:12,846 INFO [manager] checklist updated with [results files] items from download_results worker
2025-06-23 10:21:13,017 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['forecast', '--run-date', '2025-06-23'], host='localhost')
2025-06-23 10:21:14,816 INFO [make_CHS_currents_file] running in process 1069807
2025-06-23 10:21:14,818 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-23 10:21:14,818 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5562
2025-06-23 10:21:14,819 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555
2025-06-23 10:22:01,966 INFO [make_CHS_currents_file] Made CHS currents file for 2025-06-23 for forecast
2025-06-23 10:22:01,969 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker
2025-06-23 10:22:02,150 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['nemo', 'forecast', '--run-date', '2025-06-23'], host='localhost')
2025-06-23 10:22:03,141 INFO [update_forecast_datasets] running in process 1069830
2025-06-23 10:22:03,142 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-23 10:22:03,143 INFO [update_forecast_datasets] publishing log messages to tcp://*:5562
2025-06-23 10:22:03,144 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2025-06-23 10:22:03,145 INFO [update_forecast_datasets] updating nemo forecast directory for forecast/23jun25 run
2025-06-23 10:22:03,408 INFO [update_forecast_datasets] updated nemo forecast directory for forecast/23jun25 run: /results/SalishSea/rolling-forecasts/nemo
2025-06-23 10:22:03,409 INFO [update_forecast_datasets] nemo 2025-06-23 forecast rolling forecast datasets updated
2025-06-23 10:22:03,411 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2025-06-23 10:22:03,576 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nemo-forecast'], host='localhost')
2025-06-23 10:22:03,582 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'forecast', 'publish', '--run-date', '2025-06-23'], host='localhost')
2025-06-23 10:22:03,586 INFO [manager] launching NextWorker(module='nowcast.workers.make_surface_current_tiles', args=['forecast', '--run-date', '2025-06-23'], host='localhost')
2025-06-23 10:22:04,670 INFO [ping_erddap] running in process 1069835
2025-06-23 10:22:04,671 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-23 10:22:04,672 INFO [ping_erddap] publishing log messages to tcp://*:5562
2025-06-23 10:22:04,673 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2025-06-23 10:22:04,681 INFO [ping_erddap] nemo-forecast ERDDAP dataset flag file(s) created
2025-06-23 10:22:04,683 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2025-06-23 10:22:05,688 INFO [make_surface_current_tiles] running in process 1069838
2025-06-23 10:22:05,689 INFO [make_surface_current_tiles] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-23 10:22:05,690 INFO [make_surface_current_tiles] publishing log messages to tcp://*:5562
2025-06-23 10:22:05,691 INFO [make_surface_current_tiles] connected to skookum.eos.ubc.ca port 5555
2025-06-23 10:22:05,693 INFO [make_surface_current_tiles] starting rendering of tiles for 2025-06-23 forecast in 6 processes into /results/nowcast-sys/figures/surface_currents/forecast/23jun25
2025-06-23 10:22:06,647 INFO [make_plots] running in process 1069837
2025-06-23 10:22:06,648 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-23 10:22:06,649 INFO [make_plots] publishing log messages to tcp://*:5565
2025-06-23 10:22:06,650 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2025-06-23 10:22:07,120 INFO [make_plots] preparing render list for 2025-06-23 NEMO forecast publish figures
2025-06-23 10:22:07,236 INFO [make_plots] added Website_thumbnail figure to 2025-06-23 NEMO forecast publish render list
2025-06-23 10:22:07,237 INFO [make_plots] added Threshold_website figure to 2025-06-23 NEMO forecast publish render list
2025-06-23 10:22:07,237 INFO [make_plots] added PA_tidal_predictions figure to 2025-06-23 NEMO forecast publish render list
2025-06-23 10:22:07,238 INFO [make_plots] added Boundary Bay figure to 2025-06-23 NEMO forecast publish render list
2025-06-23 10:22:07,239 INFO [make_plots] added Campbell River figure to 2025-06-23 NEMO forecast publish render list
2025-06-23 10:22:07,239 INFO [make_plots] added Cherry Point figure to 2025-06-23 NEMO forecast publish render list
2025-06-23 10:22:07,240 INFO [make_plots] added Friday Harbor figure to 2025-06-23 NEMO forecast publish render list
2025-06-23 10:22:07,241 INFO [make_plots] added Halfmoon Bay figure to 2025-06-23 NEMO forecast publish render list
2025-06-23 10:22:07,241 INFO [make_plots] added Nanaimo figure to 2025-06-23 NEMO forecast publish render list
2025-06-23 10:22:07,242 INFO [make_plots] added Neah Bay figure to 2025-06-23 NEMO forecast publish render list
2025-06-23 10:22:07,243 INFO [make_plots] added New Westminster figure to 2025-06-23 NEMO forecast publish render list
2025-06-23 10:22:07,243 INFO [make_plots] added Patricia Bay figure to 2025-06-23 NEMO forecast publish render list
2025-06-23 10:22:07,244 INFO [make_plots] added Point Atkinson figure to 2025-06-23 NEMO forecast publish render list
2025-06-23 10:22:07,245 INFO [make_plots] added Port Renfrew figure to 2025-06-23 NEMO forecast publish render list
2025-06-23 10:22:07,246 INFO [make_plots] added Sand Heads figure to 2025-06-23 NEMO forecast publish render list
2025-06-23 10:22:07,246 INFO [make_plots] added Sandy Cove figure to 2025-06-23 NEMO forecast publish render list
2025-06-23 10:22:07,247 INFO [make_plots] added Squamish figure to 2025-06-23 NEMO forecast publish render list
2025-06-23 10:22:07,248 INFO [make_plots] added Victoria figure to 2025-06-23 NEMO forecast publish render list
2025-06-23 10:22:07,249 INFO [make_plots] added Woodwards Landing figure to 2025-06-23 NEMO forecast publish render list
2025-06-23 10:22:07,250 INFO [make_plots] starting to render nemo forecast publish 23jun25 figures
2025-06-23 10:22:09,780 INFO [watch_NEMO_agrif] 23jun25nowcast-agrif on orcinus-nowcast-agrif: timestep: 8547536 = 2025-06-23 04:36:40 UTC, 19.2% complete
2025-06-23 10:22:30,041 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 147, 34.0% complete
2025-06-23 10:25:04,663 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18526855 = 2025-06-23 05:56:00 UTC, 24.7% complete
2025-06-23 10:27:10,220 INFO [watch_NEMO_agrif] 23jun25nowcast-agrif on orcinus-nowcast-agrif: timestep: 8547747 = 2025-06-23 06:57:20 UTC, 29.0% complete
2025-06-23 10:27:30,042 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 234, 54.2% complete
2025-06-23 10:29:29,553 WARNING [make_plots] invalid station number for Boundary Bay station: None
2025-06-23 10:29:29,554 WARNING [make_plots] can't resolve a valid CHS station code for Boundary Bay
2025-06-23 10:29:54,473 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb996/data for station 08074 Campbell River from 2025-06-18 00:00:00Z to 2025-06-25 11:55:00Z
2025-06-23 10:30:04,652 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18527038 = 2025-06-23 07:58:00 UTC, 33.2% complete
2025-06-23 10:31:02,366 WARNING [make_plots] invalid station number for Halfmoon Bay station: None
2025-06-23 10:31:02,366 WARNING [make_plots] can't resolve a valid CHS station code for Halfmoon Bay
2025-06-23 10:31:26,505 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb96d/data for station 07917 Nanaimo from 2025-06-18 00:00:00Z to 2025-06-25 11:55:00Z
2025-06-23 10:32:08,641 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e13d0f4a073c4bbf93/data for station 07654 New Westminster from 2025-06-18 00:00:00Z to 2025-06-25 11:55:00Z
2025-06-23 10:32:10,696 INFO [watch_NEMO_agrif] 23jun25nowcast-agrif on orcinus-nowcast-agrif: timestep: 8547957 = 2025-06-23 09:17:20 UTC, 38.7% complete
2025-06-23 10:32:30,043 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 324, 75.0% complete
2025-06-23 10:32:30,169 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd2d/data for station 07277 Patricia Bay from 2025-06-18 00:00:00Z to 2025-06-25 11:55:00Z
2025-06-23 10:32:47,991 INFO [make_surface_current_tiles] starting PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast/23jun25
2025-06-23 10:32:48,259 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94c/data for station 07795 Point Atkinson from 2025-06-18 00:00:00Z to 2025-06-25 11:55:00Z
2025-06-23 10:33:06,359 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e23d0f4a073c4bc060/data for station 08525 Port Renfrew from 2025-06-18 00:00:00Z to 2025-06-25 11:55:00Z
2025-06-23 10:33:22,585 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb93c/data for station 07594 Sand Heads from 2025-06-18 00:00:00Z to 2025-06-25 11:55:00Z
2025-06-23 10:33:40,310 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc43b/data for station 07786 Sandy Cove from 2025-06-18 00:00:00Z to 2025-06-25 11:55:00Z
2025-06-23 10:33:58,437 WARNING [make_plots] invalid station number for Squamish station: None
2025-06-23 10:33:58,438 WARNING [make_plots] can't resolve a valid CHS station code for Squamish
2025-06-23 10:34:15,601 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd1e/data for station 07120 Victoria from 2025-06-18 00:00:00Z to 2025-06-25 11:55:00Z
2025-06-23 10:34:31,958 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc42d/data for station 07610 Woodwards Landing from 2025-06-18 00:00:00Z to 2025-06-25 11:55:00Z
2025-06-23 10:34:50,072 INFO [make_plots] finished rendering nemo forecast publish 23jun25 figures
2025-06-23 10:34:50,075 INFO [make_plots] nemo publish plots for 2025-06-23 forecast completed
2025-06-23 10:34:50,082 INFO [manager] checklist updated with [plots] items from make_plots worker
2025-06-23 10:34:50,252 INFO [manager] launching NextWorker(module='nowcast.workers.make_feeds', args=['forecast', '--run-date', '2025-06-23'], host='localhost')
2025-06-23 10:34:53,444 INFO [make_feeds] running in process 1071721
2025-06-23 10:34:53,445 INFO [make_feeds] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-23 10:34:53,446 INFO [make_feeds] publishing log messages to tcp://*:5565
2025-06-23 10:34:53,447 INFO [make_feeds] connected to skookum.eos.ubc.ca port 5555
2025-06-23 10:35:04,670 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18527220 = 2025-06-23 09:59:20 UTC, 41.6% complete
2025-06-23 10:35:08,969 INFO [make_feeds] ATOM feeds for 2025-06-23 forecast run completed
2025-06-23 10:35:08,972 INFO [manager] checklist updated with [storm surge alert feeds] items from make_feeds worker
2025-06-23 10:37:11,175 INFO [watch_NEMO_agrif] 23jun25nowcast-agrif on orcinus-nowcast-agrif: timestep: 8548170 = 2025-06-23 11:39:20 UTC, 48.6% complete
2025-06-23 10:37:27,375 INFO [make_surface_current_tiles] finished PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast/23jun25
2025-06-23 10:37:27,396 INFO [make_surface_current_tiles] finished rendering of tiles for 2025-06-23 forecast into /results/nowcast-sys/figures/surface_currents/forecast/23jun25
2025-06-23 10:37:27,397 INFO [make_surface_current_tiles] surface current tile figures for 2025-06-23 forecast completed
2025-06-23 10:37:27,682 INFO [manager] checklist updated with [surface current tiles] items from make_surface_current_tiles worker
2025-06-23 10:37:30,045 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 408, 94.4% complete
2025-06-23 10:40:04,661 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18527410 = 2025-06-23 12:06:00 UTC, 50.4% complete
2025-06-23 10:42:11,615 INFO [watch_NEMO_agrif] 23jun25nowcast-agrif on orcinus-nowcast-agrif: timestep: 8548380 = 2025-06-23 13:59:20 UTC, 58.3% complete
2025-06-23 10:42:30,046 INFO [watch_ww3] nowcast WWATCH3 run on arbutus.cloud-nowcast completed
2025-06-23 10:42:30,050 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2025-06-23 10:42:30,592 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2025-06-23'], host='localhost')
2025-06-23 10:42:30,598 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2025-06-23'], host='arbutus.cloud-nowcast')
2025-06-23 10:42:31,634 INFO [download_wwatch3_results] running in process 1071803
2025-06-23 10:42:31,635 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-23 10:42:31,636 INFO [download_wwatch3_results] publishing log messages to tcp://*:5562
2025-06-23 10:42:31,637 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2025-06-23 10:42:32,725 INFO [run_ww3] running in process 4520
2025-06-23 10:42:32,726 INFO [run_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-06-23 10:42:32,727 INFO [run_ww3] publishing log messages to tcp://*:5572
2025-06-23 10:42:32,728 INFO [run_ww3] connected to skookum.eos.ubc.ca port 5555
2025-06-23 10:42:32,733 INFO [run_ww3] Created run directory /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast_2025-06-23T104232.705459-0700
2025-06-23 10:42:32,740 INFO [run_ww3] forecast: launching /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast_2025-06-23T104232.705459-0700/SoGWW3.sh on arbutus.cloud-nowcast
2025-06-23 10:42:32,746 INFO [run_ww3] forecast WaveWatch3 run for 2025-06-23 on arbutus.cloud-nowcast started
2025-06-23 10:42:33,112 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker
2025-06-23 10:42:33,425 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'forecast'], host='arbutus.cloud-nowcast')
2025-06-23 10:42:35,685 INFO [watch_ww3] running in process 4682
2025-06-23 10:42:35,685 INFO [watch_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-06-23 10:42:35,686 INFO [watch_ww3] publishing log messages to tcp://*:5573
2025-06-23 10:42:35,687 INFO [watch_ww3] connected to skookum.eos.ubc.ca port 5555
2025-06-23 10:42:35,734 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2025-06-23 10:44:42,822 INFO [download_wwatch3_results] nowcast 2025-06-23 results files from arbutus.cloud-nowcast downloaded
2025-06-23 10:44:42,826 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2025-06-23 10:45:04,653 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18527597 = 2025-06-23 14:10:40 UTC, 59.1% complete
2025-06-23 10:47:12,063 INFO [watch_NEMO_agrif] 23jun25nowcast-agrif on orcinus-nowcast-agrif: timestep: 8548592 = 2025-06-23 16:20:40 UTC, 68.1% complete
2025-06-23 10:47:35,741 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 81, 12.5% complete
2025-06-23 10:50:04,676 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18527781 = 2025-06-23 16:13:20 UTC, 67.6% complete
2025-06-23 10:52:12,534 INFO [watch_NEMO_agrif] 23jun25nowcast-agrif on orcinus-nowcast-agrif: timestep: 8548804 = 2025-06-23 18:42:00 UTC, 78.0% complete
2025-06-23 10:52:35,728 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 174, 26.9% complete
2025-06-23 10:55:04,676 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18527965 = 2025-06-23 18:16:00 UTC, 76.1% complete
2025-06-23 10:57:12,977 INFO [watch_NEMO_agrif] 23jun25nowcast-agrif on orcinus-nowcast-agrif: timestep: 8549014 = 2025-06-23 21:02:00 UTC, 87.7% complete
2025-06-23 10:57:35,730 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 264, 40.7% complete
2025-06-23 11:00:04,662 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18528155 = 2025-06-23 20:22:40 UTC, 84.9% complete
2025-06-23 11:02:13,448 INFO [watch_NEMO_agrif] 23jun25nowcast-agrif on orcinus-nowcast-agrif: timestep: 8549226 = 2025-06-23 23:23:20 UTC, 97.5% complete
2025-06-23 11:02:35,723 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 360, 55.6% complete
2025-06-23 11:05:04,667 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18528340 = 2025-06-23 22:26:00 UTC, 93.5% complete
2025-06-23 11:07:13,878 INFO [watch_NEMO_agrif] 23jun25nowcast-agrif on orcinus-nowcast-agrif: timestep: 8549280 = 2025-06-23 23:59:20 UTC, 100.0% complete
2025-06-23 11:07:35,737 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 459, 70.8% complete
2025-06-23 11:10:04,655 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18528480 = 2025-06-23 23:59:20 UTC, 100.0% complete
2025-06-23 11:12:14,099 INFO [watch_NEMO_agrif] NEMO AGRIF run 11431978.orca2.ibb on orcinus-nowcast-agrif completed
2025-06-23 11:12:14,104 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_agrif worker
2025-06-23 11:12:14,591 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['orcinus-nowcast-agrif', 'nowcast-agrif', '--run-date', '2025-06-23'], host='localhost')
2025-06-23 11:12:15,616 INFO [download_results] running in process 1072052
2025-06-23 11:12:15,617 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-23 11:12:15,618 INFO [download_results] publishing log messages to tcp://*:5562
2025-06-23 11:12:15,619 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2025-06-23 11:12:15,620 INFO [download_results] downloading results from orcinus-nowcast-agrif:/global/scratch/dlatorne/nowcast-agrif/23jun25 to /results/SalishSea/nowcast-agrif.201702
2025-06-23 11:12:35,735 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 543, 83.8% complete
2025-06-23 11:12:37,266 INFO [download_results] nowcast-agrif 2025-06-23 results files from orcinus-nowcast-agrif downloaded
2025-06-23 11:12:37,275 INFO [manager] checklist updated with [results files] items from download_results worker
2025-06-23 11:12:37,609 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast-agrif', 'research', '--run-date', '2025-06-23'], host='localhost')
2025-06-23 11:12:40,070 INFO [make_plots] running in process 1072064
2025-06-23 11:12:40,071 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-23 11:12:40,072 INFO [make_plots] publishing log messages to tcp://*:5562
2025-06-23 11:12:40,073 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2025-06-23 11:12:40,370 INFO [make_plots] preparing render list for 2025-06-23 NEMO nowcast-agrif research figures
2025-06-23 11:12:40,371 INFO [make_plots] added baynes_sound_surface figure to 2025-06-23 NEMO nowcast-agrif research render list
2025-06-23 11:12:40,371 INFO [make_plots] starting to render nemo nowcast-agrif research 23jun25 figures
2025-06-23 11:12:50,273 INFO [make_plots] finished rendering nemo nowcast-agrif research 23jun25 figures
2025-06-23 11:12:50,275 INFO [make_plots] nemo research plots for 2025-06-23 nowcast-agrif completed
2025-06-23 11:12:50,278 INFO [manager] checklist updated with [plots] items from make_plots worker
2025-06-23 11:15:04,664 INFO [watch_NEMO] nowcast-green NEMO run on arbutus.cloud-nowcast completed
2025-06-23 11:15:04,695 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2025-06-23 11:15:05,169 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2025-06-23'], host='localhost')
2025-06-23 11:15:06,257 INFO [download_results] running in process 1072100
2025-06-23 11:15:06,258 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-23 11:15:06,259 INFO [download_results] publishing log messages to tcp://*:5562
2025-06-23 11:15:06,260 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2025-06-23 11:15:06,261 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/nowcast-green/23jun25 to /results2/SalishSea/nowcast-green.202111
2025-06-23 11:17:35,739 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 624, 96.3% complete
2025-06-23 11:22:19,332 INFO [download_results] nowcast-green 2025-06-23 results files from arbutus.cloud-nowcast downloaded
2025-06-23 11:22:19,339 INFO [manager] checklist updated with [results files] items from download_results worker
2025-06-23 11:22:19,667 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast-green', 'research', '--run-date', '2025-06-23'], host='localhost')
2025-06-23 11:22:19,673 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nowcast-green'], host='localhost')
2025-06-23 11:22:19,677 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'chemistry', '--run-date', '2025-06-23'], host='localhost')
2025-06-23 11:22:19,682 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'biology', '--run-date', '2025-06-23'], host='localhost')
2025-06-23 11:22:19,688 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'physics', '--run-date', '2025-06-23'], host='localhost')
2025-06-23 11:22:20,894 INFO [ping_erddap] running in process 1072158
2025-06-23 11:22:20,895 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-23 11:22:20,895 INFO [ping_erddap] publishing log messages to tcp://*:5562
2025-06-23 11:22:20,896 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2025-06-23 11:22:20,901 INFO [ping_erddap] nowcast-green ERDDAP dataset flag file(s) created
2025-06-23 11:22:20,903 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2025-06-23 11:22:22,026 INFO [make_averaged_dataset] running in process 1072159
2025-06-23 11:22:22,028 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-23 11:22:22,028 INFO [make_averaged_dataset] publishing log messages to tcp://*:5562
2025-06-23 11:22:22,029 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555
2025-06-23 11:22:22,030 INFO [make_averaged_dataset] creating day-averaged dataset for 23-Jun-2025 chemistry
2025-06-23 11:22:22,045 INFO [make_averaged_dataset] running in process 1072161
2025-06-23 11:22:22,046 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-23 11:22:22,047 INFO [make_averaged_dataset] publishing log messages to tcp://*:5564
2025-06-23 11:22:22,048 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555
2025-06-23 11:22:22,049 INFO [make_averaged_dataset] creating day-averaged dataset for 23-Jun-2025 physics
2025-06-23 11:22:22,050 INFO [make_averaged_dataset] running in process 1072160
2025-06-23 11:22:22,051 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-23 11:22:22,052 INFO [make_averaged_dataset] publishing log messages to tcp://*:5565
2025-06-23 11:22:22,052 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555
2025-06-23 11:22:22,053 INFO [make_averaged_dataset] creating day-averaged dataset for 23-Jun-2025 biology
2025-06-23 11:22:22,865 INFO [make_plots] running in process 1072156
2025-06-23 11:22:22,866 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-23 11:22:22,867 INFO [make_plots] publishing log messages to tcp://*:5566
2025-06-23 11:22:22,868 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2025-06-23 11:22:23,139 INFO [make_plots] preparing render list for 2025-06-23 NEMO nowcast-green research figures
2025-06-23 11:22:24,840 INFO [make_plots] added temperature_thalweg_and_surface figures to 2025-06-23 NEMO nowcast-green research render list
2025-06-23 11:22:25,564 INFO [make_plots] added nitrate_thalweg_and_surface figures to 2025-06-23 NEMO nowcast-green research render list
2025-06-23 11:22:26,285 INFO [make_plots] added ammonium_thalweg_and_surface figures to 2025-06-23 NEMO nowcast-green research render list
2025-06-23 11:22:27,011 INFO [make_plots] added silicon_thalweg_and_surface figures to 2025-06-23 NEMO nowcast-green research render list
2025-06-23 11:22:27,758 INFO [make_plots] added dissolved_organic_nitrogen_thalweg_and_surface figures to 2025-06-23 NEMO nowcast-green research render list
2025-06-23 11:22:28,505 INFO [make_plots] added particulate_organic_nitrogen_thalweg_and_surface figures to 2025-06-23 NEMO nowcast-green research render list
2025-06-23 11:22:29,235 INFO [make_plots] added biogenic_silicon_thalweg_and_surface figures to 2025-06-23 NEMO nowcast-green research render list
2025-06-23 11:22:30,073 INFO [make_plots] added diatoms_thalweg_and_surface figures to 2025-06-23 NEMO nowcast-green research render list
2025-06-23 11:22:30,899 INFO [make_plots] added flagellates_thalweg_and_surface figures to 2025-06-23 NEMO nowcast-green research render list
2025-06-23 11:22:31,730 INFO [make_plots] added microzooplankton_thalweg_and_surface figures to 2025-06-23 NEMO nowcast-green research render list
2025-06-23 11:22:32,558 INFO [make_plots] added mesozooplankton_thalweg_and_surface figures to 2025-06-23 NEMO nowcast-green research render list
2025-06-23 11:22:33,291 INFO [make_plots] added turbidity_thalweg_and_surface figures to 2025-06-23 NEMO nowcast-green research render list
2025-06-23 11:22:34,322 INFO [make_plots] added temperature_salinity_timeseries figure to 2025-06-23 NEMO nowcast-green research render list
2025-06-23 11:22:34,323 INFO [make_plots] added nitrate_diatoms_timeseries figure to 2025-06-23 NEMO nowcast-green research render list
2025-06-23 11:22:34,324 INFO [make_plots] added diatoms_flagellates_timeseries figure to 2025-06-23 NEMO nowcast-green research render list
2025-06-23 11:22:34,325 INFO [make_plots] added z1_z2_zooplankton_timeseries figure to 2025-06-23 NEMO nowcast-green research render list
2025-06-23 11:22:34,326 INFO [make_plots] starting to render nemo nowcast-green research 23jun25 figures
2025-06-23 11:22:35,724 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2025-06-23 11:22:35,739 INFO [watch_ww3] forecast WWATCH3 run on arbutus.cloud-nowcast completed
2025-06-23 11:22:36,216 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2025-06-23'], host='localhost')
2025-06-23 11:22:37,298 INFO [download_wwatch3_results] running in process 1072241
2025-06-23 11:22:37,299 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-23 11:22:37,300 INFO [download_wwatch3_results] publishing log messages to tcp://*:5567
2025-06-23 11:22:37,301 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2025-06-23 11:22:53,529 INFO [make_averaged_dataset] day-averaged dataset for 23-Jun-2025 physics created
2025-06-23 11:22:53,532 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker
2025-06-23 11:23:34,527 INFO [make_averaged_dataset] day-averaged dataset for 23-Jun-2025 chemistry created
2025-06-23 11:23:34,531 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker
2025-06-23 11:23:51,438 INFO [make_averaged_dataset] day-averaged dataset for 23-Jun-2025 biology created
2025-06-23 11:23:51,441 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker
2025-06-23 11:24:30,670 INFO [download_wwatch3_results] forecast 2025-06-23 results files from arbutus.cloud-nowcast downloaded
2025-06-23 11:24:30,673 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2025-06-23 11:24:31,026 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['wwatch3', 'forecast', '--run-date', '2025-06-23'], host='localhost')
2025-06-23 11:24:32,181 INFO [update_forecast_datasets] running in process 1072262
2025-06-23 11:24:32,182 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-23 11:24:32,183 INFO [update_forecast_datasets] publishing log messages to tcp://*:5562
2025-06-23 11:24:32,184 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2025-06-23 11:24:32,185 INFO [update_forecast_datasets] updating wwatch3 most_recent_forecast directory from forecast/23jun25 run
2025-06-23 11:24:32,188 INFO [update_forecast_datasets] updated wwatch3 most_recent_forecast directory from forecast/23jun25 run
2025-06-23 11:24:32,188 INFO [update_forecast_datasets] updating wwatch3 forecast directory for forecast/23jun25 run
2025-06-23 11:24:32,229 INFO [update_forecast_datasets] updated wwatch3 forecast directory for forecast/23jun25 run: /results/SalishSea/rolling-forecasts/wwatch3
2025-06-23 11:24:32,230 INFO [update_forecast_datasets] wwatch3 2025-06-23 forecast rolling forecast datasets updated
2025-06-23 11:24:32,233 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2025-06-23 11:24:32,576 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['wwatch3-forecast'], host='localhost')
2025-06-23 11:24:33,662 INFO [ping_erddap] running in process 1072267
2025-06-23 11:24:33,664 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-23 11:24:33,665 INFO [ping_erddap] publishing log messages to tcp://*:5562
2025-06-23 11:24:33,666 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2025-06-23 11:24:33,667 INFO [ping_erddap] wwatch3-forecast ERDDAP dataset flag file(s) created
2025-06-23 11:24:33,669 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2025-06-23 11:24:34,001 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['wwatch3', 'forecast', 'publish', '--run-date', '2025-06-23'], host='localhost')
2025-06-23 11:24:36,957 INFO [make_plots] running in process 1072272
2025-06-23 11:24:36,959 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-23 11:24:36,960 INFO [make_plots] publishing log messages to tcp://*:5562
2025-06-23 11:24:36,961 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2025-06-23 11:24:36,962 INFO [make_plots] preparing render list for 2025-06-23 WaveWatch3 forecast publish figures
2025-06-23 11:24:36,963 INFO [make_plots] added Halibut Bank figure to 2025-06-23 WaveWatch3 forecast publish render list
2025-06-23 11:24:36,964 INFO [make_plots] added Sentry Shoal figure to 2025-06-23 WaveWatch3 forecast publish render list
2025-06-23 11:24:36,965 INFO [make_plots] starting to render wwatch3 forecast publish 23jun25 figures
2025-06-23 11:24:37,330 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46146 Halibut Bank
2025-06-23 11:37:30,266 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46131 Sentry Shoal
2025-06-23 11:37:36,514 INFO [make_plots] finished rendering wwatch3 forecast publish 23jun25 figures
2025-06-23 11:37:36,517 INFO [make_plots] wwatch3 publish plots for 2025-06-23 forecast completed
2025-06-23 11:37:36,519 INFO [manager] checklist updated with [plots] items from make_plots worker
2025-06-23 12:22:47,403 INFO [make_plots] finished rendering nemo nowcast-green research 23jun25 figures
2025-06-23 12:22:47,576 INFO [make_plots] nemo research plots for 2025-06-23 nowcast-green completed
2025-06-23 12:22:47,646 INFO [manager] checklist updated with [plots] items from make_plots worker
2025-06-23 15:29:51,090 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/18/ to /results/forcing/atmospheric/continental2.5/GRIB/20250623/18/
2025-06-23 15:29:51,091 INFO [collect_weather] 2025-06-23 2.5km weather forecast 18 collection complete
2025-06-23 15:29:51,096 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2025-06-23 15:29:51,605 INFO [manager] launching NextWorker(module='nowcast.workers.download_weather', args=['00', '1km'], host='localhost')
2025-06-23 15:29:51,617 INFO [manager] launching NextWorker(module='nowcast.workers.download_weather', args=['12', '1km'], host='localhost')
2025-06-23 15:29:51,621 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['00', '2.5km'], host='localhost')
2025-06-23 15:29:51,626 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['00', '--fcst-date', '2025-06-24'], host='localhost')
2025-06-23 15:29:51,938 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20250623/18/
2025-06-23 15:29:51,939 INFO [crop_gribs] 2025-06-23 18 GRIBs cropping complete
2025-06-23 15:29:51,941 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2025-06-23 15:29:53,239 INFO [download_weather] running in process 1074526
2025-06-23 15:29:53,240 INFO [download_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-23 15:29:53,241 INFO [download_weather] publishing log messages to tcp://*:5562
2025-06-23 15:29:53,242 INFO [download_weather] connected to skookum.eos.ubc.ca port 5555
2025-06-23 15:29:53,243 INFO [download_weather] downloading 12 1 km forecast GRIB2 files for 20250623
2025-06-23 15:29:53,244 INFO [download_weather] running in process 1074524
2025-06-23 15:29:53,246 INFO [download_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-23 15:29:53,247 INFO [download_weather] publishing log messages to tcp://*:5563
2025-06-23 15:29:53,248 INFO [download_weather] connected to skookum.eos.ubc.ca port 5555
2025-06-23 15:29:53,249 INFO [download_weather] downloading 00 1 km forecast GRIB2 files for 20250623
2025-06-23 15:29:53,251 INFO [collect_weather] running in process 1074527
2025-06-23 15:29:53,252 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-23 15:29:53,254 INFO [collect_weather] publishing log messages to tcp://*:5564
2025-06-23 15:29:53,255 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2025-06-23 15:29:53,264 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/00/
2025-06-23 15:29:54,281 INFO [crop_gribs] running in process 1074528
2025-06-23 15:29:54,282 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-23 15:29:54,283 INFO [crop_gribs] publishing log messages to tcp://*:5565
2025-06-23 15:29:54,285 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2025-06-23 15:29:54,286 INFO [crop_gribs] cropping 2025-06-24 ECCC HRDPS 2.5km continental 00Z GRIB files to SalishSeaCast subdomain
2025-06-23 15:29:54,299 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20250624/00/
2025-06-23 15:33:36,269 INFO [download_weather] 2025-06-23 1km weather forecast 12 downloads complete
2025-06-23 15:33:36,273 INFO [manager] checklist updated with [weather forecast] items from download_weather worker
2025-06-23 15:33:42,590 INFO [download_weather] 2025-06-23 1km weather forecast 00 downloads complete
2025-06-23 15:33:42,593 INFO [manager] checklist updated with [weather forecast] items from download_weather worker
2025-06-23 21:04:33,814 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/00/ to /results/forcing/atmospheric/continental2.5/GRIB/20250624/00/
2025-06-23 21:04:33,815 INFO [collect_weather] 2025-06-24 2.5km weather forecast 00 collection complete
2025-06-23 21:04:33,820 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2025-06-23 21:04:34,326 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20250624/00/
2025-06-23 21:04:34,327 INFO [crop_gribs] 2025-06-24 00 GRIBs cropping complete
2025-06-23 21:04:34,349 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['06', '2.5km'], host='localhost')
2025-06-23 21:04:34,357 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['06', '--fcst-date', '2025-06-24'], host='localhost')
2025-06-23 21:04:34,363 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2025-06-23 21:04:35,513 INFO [collect_weather] running in process 1077325
2025-06-23 21:04:35,515 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-23 21:04:35,516 INFO [collect_weather] publishing log messages to tcp://*:5562
2025-06-23 21:04:35,516 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2025-06-23 21:04:35,533 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/06/
2025-06-23 21:04:36,271 INFO [crop_gribs] running in process 1077327
2025-06-23 21:04:36,272 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-23 21:04:36,273 INFO [crop_gribs] publishing log messages to tcp://*:5563
2025-06-23 21:04:36,274 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2025-06-23 21:04:36,275 INFO [crop_gribs] cropping 2025-06-24 ECCC HRDPS 2.5km continental 06Z GRIB files to SalishSeaCast subdomain
2025-06-23 21:04:36,287 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20250624/06/
2025-06-24 03:05:04,394 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/06/ to /results/forcing/atmospheric/continental2.5/GRIB/20250624/06/
2025-06-24 03:05:04,395 INFO [collect_weather] 2025-06-24 2.5km weather forecast 06 collection complete
2025-06-24 03:05:04,399 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2025-06-24 03:05:05,049 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Capilano', '--data-date', '2025-06-23'], host='localhost')
2025-06-24 03:05:05,060 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'ChilliwackVedder', '--data-date', '2025-06-23'], host='localhost')
2025-06-24 03:05:05,066 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'ClowhomClowhomLake', '--data-date', '2025-06-23'], host='localhost')
2025-06-24 03:05:05,072 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Englishman', '--data-date', '2025-06-23'], host='localhost')
2025-06-24 03:05:05,079 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Fraser', '--data-date', '2025-06-23'], host='localhost')
2025-06-24 03:05:05,094 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'HomathkoMouth', '--data-date', '2025-06-23'], host='localhost')
2025-06-24 03:05:05,141 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'NicomeklLangley', '--data-date', '2025-06-23'], host='localhost')
2025-06-24 03:05:05,164 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'RobertsCreek', '--data-date', '2025-06-23'], host='localhost')
2025-06-24 03:05:05,171 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SalmonSayward', '--data-date', '2025-06-23'], host='localhost')
2025-06-24 03:05:05,174 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20250624/06/
2025-06-24 03:05:05,175 INFO [crop_gribs] 2025-06-24 06 GRIBs cropping complete
2025-06-24 03:05:05,177 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SanJuanPortRenfrew', '--data-date', '2025-06-23'], host='localhost')
2025-06-24 03:05:05,181 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SquamishBrackendale', '--data-date', '2025-06-23'], host='localhost')
2025-06-24 03:05:05,188 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaScotty', '--data-date', '2025-06-23'], host='localhost')
2025-06-24 03:05:05,195 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaBypass', '--data-date', '2025-06-23'], host='localhost')
2025-06-24 03:05:05,202 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaDiversion', '--data-date', '2025-06-23'], host='localhost')
2025-06-24 03:05:05,209 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ctd', args=['SCVIP'], host='localhost')
2025-06-24 03:05:05,212 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ctd', args=['SEVIP'], host='localhost')
2025-06-24 03:05:05,218 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ferry', args=['TWDP'], host='localhost')
2025-06-24 03:05:05,229 INFO [manager] launching NextWorker(module='nowcast.workers.get_vfpa_hadcp', args=['--data-date', '2025-06-23'], host='localhost')
2025-06-24 03:05:05,242 INFO [manager] launching NextWorker(module='nowcast.workers.collect_NeahBay_ssh', args=['00'], host='localhost')
2025-06-24 03:05:05,249 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['12', '2.5km'], host='localhost')
2025-06-24 03:05:05,261 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['12'], host='localhost')
2025-06-24 03:05:05,280 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2025-06-24 03:05:06,084 INFO [manager] launching NextWorker(module='nowcast.workers.grib_to_netcdf', args=['forecast2'], host='localhost')
2025-06-24 03:05:08,544 INFO [collect_weather] running in process 1081105
2025-06-24 03:05:08,545 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-24 03:05:08,545 INFO [collect_weather] publishing log messages to tcp://*:5562
2025-06-24 03:05:08,546 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2025-06-24 03:05:08,547 INFO [collect_NeahBay_ssh] running in process 1081104
2025-06-24 03:05:08,548 INFO [collect_NeahBay_ssh] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-24 03:05:08,549 INFO [collect_NeahBay_ssh] publishing log messages to tcp://*:5563
2025-06-24 03:05:08,551 INFO [collect_NeahBay_ssh] connected to skookum.eos.ubc.ca port 5555
2025-06-24 03:05:08,552 INFO [collect_NeahBay_ssh] collecting Neah Bay ssh 00Z obs/forecast for 2025-06-24
2025-06-24 03:05:08,570 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/12/
2025-06-24 03:05:09,602 INFO [collect_river_data] running in process 1081088
2025-06-24 03:05:09,604 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-24 03:05:09,606 INFO [collect_river_data] publishing log messages to tcp://*:5564
2025-06-24 03:05:09,606 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-06-24 03:05:09,608 INFO [collect_river_data] Collecting ECCC Englishman river data for 2025-06-23
2025-06-24 03:05:09,609 INFO [collect_river_data] running in process 1081097
2025-06-24 03:05:09,611 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-24 03:05:09,612 INFO [collect_river_data] publishing log messages to tcp://*:5565
2025-06-24 03:05:09,613 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-06-24 03:05:09,615 INFO [collect_river_data] Collecting ECCC TheodosiaScotty river data for 2025-06-23
2025-06-24 03:05:09,642 INFO [collect_river_data] running in process 1081098
2025-06-24 03:05:09,645 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-24 03:05:09,649 INFO [collect_river_data] publishing log messages to tcp://*:5566
2025-06-24 03:05:09,651 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-06-24 03:05:09,654 INFO [collect_river_data] Collecting ECCC TheodosiaBypass river data for 2025-06-23
2025-06-24 03:05:09,702 INFO [collect_river_data] running in process 1081093
2025-06-24 03:05:09,703 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-24 03:05:09,704 INFO [collect_river_data] publishing log messages to tcp://*:5567
2025-06-24 03:05:09,705 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-06-24 03:05:09,706 INFO [collect_river_data] Collecting ECCC RobertsCreek river data for 2025-06-23
2025-06-24 03:05:09,710 INFO [collect_river_data] Appended ECCC TheodosiaBypass river average discharge for 2025-06-23 to: /results/forcing/rivers/observations/Theodosia_Bypass_flow
2025-06-24 03:05:09,711 INFO [collect_river_data] ECCC TheodosiaBypass river data collection for 2025-06-23 completed
2025-06-24 03:05:09,713 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-06-24 03:05:09,718 INFO [collect_river_data] Appended ECCC Englishman river average discharge for 2025-06-23 to: /data/dlatorne/SOG-projects/SOG-forcing/ECget/Englishman_flow
2025-06-24 03:05:09,719 INFO [collect_river_data] ECCC Englishman river data collection for 2025-06-23 completed
2025-06-24 03:05:09,723 INFO [collect_river_data] running in process 1081095
2025-06-24 03:05:09,724 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-24 03:05:09,725 INFO [collect_river_data] publishing log messages to tcp://*:5568
2025-06-24 03:05:09,726 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-06-24 03:05:09,727 INFO [collect_river_data] Collecting ECCC SanJuanPortRenfrew river data for 2025-06-23
2025-06-24 03:05:09,728 INFO [collect_river_data] Appended ECCC TheodosiaScotty river average discharge for 2025-06-23 to: /results/forcing/rivers/observations/Theodosia_Scotty_flow
2025-06-24 03:05:09,729 INFO [collect_river_data] ECCC TheodosiaScotty river data collection for 2025-06-23 completed
2025-06-24 03:05:09,749 INFO [collect_river_data] Appended ECCC SanJuanPortRenfrew river average discharge for 2025-06-23 to: /results/forcing/rivers/observations/SanJuan_PortRenfrew_flow
2025-06-24 03:05:09,749 INFO [collect_river_data] ECCC SanJuanPortRenfrew river data collection for 2025-06-23 completed
2025-06-24 03:05:09,754 INFO [collect_river_data] Appended ECCC RobertsCreek river average discharge for 2025-06-23 to: /results/forcing/rivers/observations/RobertsCreek_flow
2025-06-24 03:05:09,755 INFO [collect_river_data] ECCC RobertsCreek river data collection for 2025-06-23 completed
2025-06-24 03:05:09,779 INFO [collect_river_data] running in process 1081099
2025-06-24 03:05:09,780 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-24 03:05:09,781 INFO [collect_river_data] publishing log messages to tcp://*:5569
2025-06-24 03:05:09,782 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-06-24 03:05:09,782 INFO [collect_river_data] Collecting ECCC TheodosiaDiversion river data for 2025-06-23
2025-06-24 03:05:09,784 INFO [collect_river_data] running in process 1081086
2025-06-24 03:05:09,784 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-24 03:05:09,785 INFO [collect_river_data] publishing log messages to tcp://*:5570
2025-06-24 03:05:09,786 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-06-24 03:05:09,787 INFO [collect_river_data] Collecting ECCC ChilliwackVedder river data for 2025-06-23
2025-06-24 03:05:09,787 INFO [collect_river_data] running in process 1081087
2025-06-24 03:05:09,788 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-24 03:05:09,789 INFO [collect_river_data] publishing log messages to tcp://*:5571
2025-06-24 03:05:09,789 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-06-24 03:05:09,790 INFO [collect_river_data] Collecting ECCC ClowhomClowhomLake river data for 2025-06-23
2025-06-24 03:05:09,798 INFO [collect_river_data] running in process 1081094
2025-06-24 03:05:09,799 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-24 03:05:09,800 INFO [collect_river_data] publishing log messages to tcp://*:5572
2025-06-24 03:05:09,801 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-06-24 03:05:09,802 INFO [collect_river_data] Collecting ECCC SalmonSayward river data for 2025-06-23
2025-06-24 03:05:09,816 INFO [collect_river_data] Appended ECCC TheodosiaDiversion river average discharge for 2025-06-23 to: /results/forcing/rivers/observations/Theodosia_Diversion_flow
2025-06-24 03:05:09,817 INFO [collect_river_data] ECCC TheodosiaDiversion river data collection for 2025-06-23 completed
2025-06-24 03:05:09,819 INFO [collect_river_data] running in process 1081090
2025-06-24 03:05:09,820 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-24 03:05:09,822 INFO [collect_river_data] publishing log messages to tcp://*:5573
2025-06-24 03:05:09,823 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-06-24 03:05:09,823 INFO [collect_river_data] Collecting ECCC Fraser river data for 2025-06-23
2025-06-24 03:05:09,825 INFO [collect_river_data] running in process 1081091
2025-06-24 03:05:09,826 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-24 03:05:09,826 INFO [collect_river_data] publishing log messages to tcp://*:5574
2025-06-24 03:05:09,827 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-06-24 03:05:09,828 INFO [collect_river_data] Collecting ECCC HomathkoMouth river data for 2025-06-23
2025-06-24 03:05:09,831 INFO [collect_river_data] Appended ECCC ChilliwackVedder river average discharge for 2025-06-23 to: /results/forcing/rivers/observations/Chilliwack_Vedder_flow
2025-06-24 03:05:09,832 INFO [collect_river_data] ECCC ChilliwackVedder river data collection for 2025-06-23 completed
2025-06-24 03:05:09,836 INFO [collect_river_data] Appended ECCC ClowhomClowhomLake river average discharge for 2025-06-23 to: /results/forcing/rivers/observations/Clowhom_ClowhomLake_flow
2025-06-24 03:05:09,837 INFO [collect_river_data] ECCC ClowhomClowhomLake river data collection for 2025-06-23 completed
2025-06-24 03:05:09,838 INFO [collect_river_data] running in process 1081096
2025-06-24 03:05:09,839 INFO [collect_river_data] running in process 1081092
2025-06-24 03:05:09,840 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-24 03:05:09,842 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-24 03:05:09,843 INFO [collect_river_data] publishing log messages to tcp://*:5575
2025-06-24 03:05:09,844 INFO [collect_river_data] publishing log messages to tcp://*:5576
2025-06-24 03:05:09,845 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-06-24 03:05:09,846 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-06-24 03:05:09,846 INFO [collect_river_data] Collecting ECCC SquamishBrackendale river data for 2025-06-23
2025-06-24 03:05:09,847 INFO [collect_river_data] Collecting ECCC NicomeklLangley river data for 2025-06-23
2025-06-24 03:05:09,848 INFO [collect_river_data] running in process 1081084
2025-06-24 03:05:09,849 INFO [collect_river_data] Appended ECCC SalmonSayward river average discharge for 2025-06-23 to: /results/forcing/rivers/observations/Salmon_Sayward_flow
2025-06-24 03:05:09,850 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-24 03:05:09,850 INFO [collect_river_data] ECCC SalmonSayward river data collection for 2025-06-23 completed
2025-06-24 03:05:09,852 INFO [collect_river_data] publishing log messages to tcp://*:5577
2025-06-24 03:05:09,853 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-06-24 03:05:09,854 INFO [collect_river_data] Collecting ECCC Capilano river data for 2025-06-23
2025-06-24 03:05:09,866 INFO [collect_river_data] Appended ECCC HomathkoMouth river average discharge for 2025-06-23 to: /results/forcing/rivers/observations/Homathko_Mouth_flow
2025-06-24 03:05:09,866 INFO [collect_river_data] ECCC HomathkoMouth river data collection for 2025-06-23 completed
2025-06-24 03:05:09,881 INFO [collect_river_data] Appended ECCC NicomeklLangley river average discharge for 2025-06-23 to: /results/forcing/rivers/observations/Nicomekl_Langley_flow
2025-06-24 03:05:09,882 INFO [collect_river_data] ECCC NicomeklLangley river data collection for 2025-06-23 completed
2025-06-24 03:05:09,884 INFO [collect_river_data] Appended ECCC SquamishBrackendale river average discharge for 2025-06-23 to: /results/forcing/rivers/observations/Squamish_Brackendale_flow
2025-06-24 03:05:09,885 INFO [collect_river_data] ECCC SquamishBrackendale river data collection for 2025-06-23 completed
2025-06-24 03:05:09,900 INFO [collect_river_data] Appended ECCC Fraser river average discharge for 2025-06-23 to: /data/dlatorne/SOG-projects/SOG-forcing/ECget/Fraser_flow
2025-06-24 03:05:09,901 INFO [collect_river_data] ECCC Fraser river data collection for 2025-06-23 completed
2025-06-24 03:05:09,906 INFO [collect_NeahBay_ssh] 2025-06-24T00:00:00-07:00 Neah Bay ssh 00Z obs/forecast data collection complete
2025-06-24 03:05:09,985 INFO [collect_river_data] Appended ECCC Capilano river average discharge for 2025-06-23 to: /opp/observations/rivers/Capilano/Caplilano_08GA010_day_avg_flow
2025-06-24 03:05:09,986 INFO [collect_river_data] ECCC Capilano river data collection for 2025-06-23 completed
2025-06-24 03:05:10,123 INFO [grib_to_netcdf] running in process 1081164
2025-06-24 03:05:10,124 INFO [grib_to_netcdf] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-24 03:05:10,125 INFO [grib_to_netcdf] publishing log messages to tcp://*:5578
2025-06-24 03:05:10,125 INFO [grib_to_netcdf] connected to skookum.eos.ubc.ca port 5555
2025-06-24 03:05:10,126 INFO [grib_to_netcdf] creating NEMO-atmos forcing files for 2025-06-24 forecast2 run
2025-06-24 03:05:10,138 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-06-24 03:05:10,226 INFO [get_vfpa_hadcp] running in process 1081103
2025-06-24 03:05:10,227 INFO [get_vfpa_hadcp] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-24 03:05:10,228 INFO [get_vfpa_hadcp] publishing log messages to tcp://*:5579
2025-06-24 03:05:10,229 INFO [get_vfpa_hadcp] connected to skookum.eos.ubc.ca port 5555
2025-06-24 03:05:10,229 INFO [get_vfpa_hadcp] processing VFPA HADCP data from 2nd Narrows Rail Bridge for 2025-06-23
2025-06-24 03:05:10,403 INFO [crop_gribs] running in process 1081106
2025-06-24 03:05:10,405 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-24 03:05:10,405 INFO [crop_gribs] publishing log messages to tcp://*:5590
2025-06-24 03:05:10,406 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2025-06-24 03:05:10,407 INFO [crop_gribs] cropping 2025-06-24 ECCC HRDPS 2.5km continental 12Z GRIB files to SalishSeaCast subdomain
2025-06-24 03:05:10,422 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20250624/12/
2025-06-24 03:05:10,514 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-06-24 03:05:10,889 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-06-24 03:05:11,176 INFO [get_onc_ctd] running in process 1081100
2025-06-24 03:05:11,177 INFO [get_onc_ctd] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-24 03:05:11,178 INFO [get_onc_ctd] publishing log messages to tcp://*:5564
2025-06-24 03:05:11,179 INFO [get_onc_ctd] connected to skookum.eos.ubc.ca port 5555
2025-06-24 03:05:11,180 INFO [get_onc_ctd] requesting ONC SCVIP CTD T&S data for 2025-06-23
2025-06-24 03:05:11,196 INFO [get_onc_ctd] running in process 1081101
2025-06-24 03:05:11,197 INFO [get_onc_ctd] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-24 03:05:11,198 INFO [get_onc_ctd] publishing log messages to tcp://*:5565
2025-06-24 03:05:11,199 INFO [get_onc_ctd] connected to skookum.eos.ubc.ca port 5555
2025-06-24 03:05:11,199 INFO [get_onc_ctd] requesting ONC SEVIP CTD T&S data for 2025-06-23
2025-06-24 03:05:11,219 INFO [get_onc_ferry] running in process 1081102
2025-06-24 03:05:11,220 INFO [get_onc_ferry] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-24 03:05:11,221 INFO [get_onc_ferry] publishing log messages to tcp://*:5566
2025-06-24 03:05:11,221 INFO [get_onc_ferry] connected to skookum.eos.ubc.ca port 5555
2025-06-24 03:05:11,222 INFO [get_onc_ferry] requesting ONC TWDP.N1 NAV data for 2025-06-23
2025-06-24 03:05:11,232 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-06-24 03:05:11,651 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-06-24 03:05:12,005 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-06-24 03:05:12,335 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-06-24 03:05:12,670 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-06-24 03:05:12,684 INFO [get_vfpa_hadcp] added VFPA HADCP data from 2nd Narrows Rail Bridge for 2025-06-23 to /opp/observations/AISDATA/netcdf/VFPA_2ND_NARROWS_HADCP_2s_202506.nc
2025-06-24 03:05:12,685 INFO [get_vfpa_hadcp] VFPA HADCP observations added to 2025-06 netcdf file
2025-06-24 03:05:12,984 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-06-24 03:05:13,358 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-06-24 03:05:13,691 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-06-24 03:05:14,012 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-06-24 03:05:14,360 INFO [manager] checklist updated with [Neah Bay ssh data] items from collect_NeahBay_ssh worker
2025-06-24 03:05:14,719 INFO [manager] launching NextWorker(module='nowcast.workers.make_ssh_files', args=['forecast2', '--run-date', '2025-06-24'], host='localhost')
2025-06-24 03:05:14,727 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-06-24 03:05:15,179 INFO [manager] checklist updated with [VFPA HADCP data] items from get_vfpa_hadcp worker
2025-06-24 03:05:15,646 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['VFPA-HADCP'], host='localhost')
2025-06-24 03:05:16,949 INFO [ping_erddap] running in process 1081454
2025-06-24 03:05:16,950 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-24 03:05:16,951 INFO [ping_erddap] publishing log messages to tcp://*:5563
2025-06-24 03:05:16,951 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2025-06-24 03:05:16,952 INFO [ping_erddap] VFPA-HADCP ERDDAP dataset flag file(s) created
2025-06-24 03:05:16,953 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2025-06-24 03:05:18,901 INFO [make_ssh_files] running in process 1081440
2025-06-24 03:05:18,904 INFO [make_ssh_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-24 03:05:18,905 INFO [make_ssh_files] publishing log messages to tcp://*:5563
2025-06-24 03:05:18,906 INFO [make_ssh_files] connected to skookum.eos.ubc.ca port 5555
2025-06-24 03:05:18,907 INFO [make_ssh_files] building forecast2 sea surface height boundary conditions file(s) from 2025-06-24 NOAA Neah Bay 00Z observation and forecast values
2025-06-24 03:05:20,434 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2025m06d25.nc
2025-06-24 03:05:21,431 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2025m06d19.nc
2025-06-24 03:05:21,664 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2025m06d20.nc
2025-06-24 03:05:21,913 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2025m06d21.nc
2025-06-24 03:05:22,101 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2025m06d22.nc
2025-06-24 03:05:22,324 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2025m06d23.nc
2025-06-24 03:05:22,612 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2025m06d24.nc
2025-06-24 03:05:22,856 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2025m06d25.nc
2025-06-24 03:05:23,024 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2025m06d26.nc
2025-06-24 03:05:23,144 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2025m06d26.nc
2025-06-24 03:05:23,150 INFO [grib_to_netcdf] 2025-06-24 NEMO-atmos forcing file for forecast2 created
2025-06-24 03:05:23,161 INFO [manager] checklist updated with [weather forcing] items from grib_to_netcdf worker
2025-06-24 03:05:23,237 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2025m06d27.nc
2025-06-24 03:05:23,460 INFO [make_ssh_files] sea surface height boundary file for 2025-06-24 forecast2 run created
2025-06-24 03:05:23,556 INFO [manager] checklist updated with [sea surface height forcing] items from make_ssh_files worker
2025-06-24 03:05:23,904 INFO [manager] launching NextWorker(module='nowcast.workers.make_runoff_file', args=['v202108'], host='localhost')
2025-06-24 03:05:23,908 INFO [manager] launching NextWorker(module='nowcast.workers.make_201702_runoff_file', args=[], host='localhost')
2025-06-24 03:05:25,200 INFO [make_201702_runoff_file] running in process 1081489
2025-06-24 03:05:25,201 INFO [make_201702_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-24 03:05:25,202 INFO [make_201702_runoff_file] publishing log messages to tcp://*:5563
2025-06-24 03:05:25,203 INFO [make_201702_runoff_file] connected to skookum.eos.ubc.ca port 5555
2025-06-24 03:05:25,529 INFO [make_201702_runoff_file] calculating runoff forcing for b201702 bathymetry
2025-06-24 03:05:25,649 INFO [make_201702_runoff_file] 2025-06-24 runoff file creation from Fraser at Hope and climatology elsewhere complete
2025-06-24 03:05:25,652 INFO [manager] checklist updated with [rivers forcing] items from make_201702_runoff_file worker
2025-06-24 03:05:25,886 INFO [make_runoff_file] running in process 1081487
2025-06-24 03:05:25,887 INFO [make_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-24 03:05:25,888 INFO [make_runoff_file] publishing log messages to tcp://*:5567
2025-06-24 03:05:25,889 INFO [make_runoff_file] connected to skookum.eos.ubc.ca port 5555
2025-06-24 03:05:25,891 INFO [make_runoff_file] calculating NEMO runoff forcing for v202108 bathymetry for 2025-06-23
2025-06-24 03:05:27,116 ERROR [make_runoff_file] no 2025-06-23 discharge obs for SkagitMountVernon: patching
2025-06-24 03:05:27,800 ERROR [make_runoff_file] no 2025-06-23 discharge obs for SnohomishMonroe: patching
2025-06-24 03:05:28,031 ERROR [make_runoff_file] no 2025-06-23 discharge obs for NisquallyMcKenna: patching
2025-06-24 03:05:28,126 ERROR [make_runoff_file] no 2025-06-23 discharge obs for GreenwaterGreenwater: patching
2025-06-24 03:05:29,528 INFO [make_runoff_file] stored NEMO runoff forcing for v202108 bathymetry for 2025-06-23: /results/forcing/rivers/R202108Dailies_y2025m06d23.nc
2025-06-24 03:05:29,529 INFO [make_runoff_file] 2025-06-23 runoff file creation completed
2025-06-24 03:05:29,531 INFO [manager] checklist updated with [rivers forcing] items from make_runoff_file worker
2025-06-24 03:05:29,927 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'forecast2'], host='localhost')
2025-06-24 03:05:29,932 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'forecast2'], host='localhost')
2025-06-24 03:05:29,937 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.graham', 'forecast2'], host='localhost')
2025-06-24 03:05:29,942 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'forecast2'], host='localhost')
2025-06-24 03:05:30,055 INFO [get_onc_ctd] 2025-06-23 ONC SEVIP CTD T&S file created
2025-06-24 03:05:30,057 INFO [manager] checklist updated with [ONC CTD data] items from get_onc_ctd worker
2025-06-24 03:05:30,057 INFO [get_onc_ctd] 2025-06-23 ONC SCVIP CTD T&S file created
2025-06-24 03:05:30,476 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['SEVIP-CTD'], host='localhost')
2025-06-24 03:05:30,482 INFO [manager] checklist updated with [ONC CTD data] items from get_onc_ctd worker
2025-06-24 03:05:30,867 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['SCVIP-CTD'], host='localhost')
2025-06-24 03:05:31,352 INFO [upload_forcing] running in process 1081525
2025-06-24 03:05:31,353 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-24 03:05:31,354 INFO [upload_forcing] publishing log messages to tcp://*:5563
2025-06-24 03:05:31,354 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-06-24 03:05:31,383 INFO [upload_forcing] running in process 1081527
2025-06-24 03:05:31,383 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-24 03:05:31,384 INFO [upload_forcing] publishing log messages to tcp://*:5564
2025-06-24 03:05:31,385 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-06-24 03:05:31,427 INFO [upload_forcing] running in process 1081528
2025-06-24 03:05:31,428 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-24 03:05:31,429 INFO [upload_forcing] publishing log messages to tcp://*:5565
2025-06-24 03:05:31,430 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-06-24 03:05:31,481 INFO [upload_forcing] running in process 1081526
2025-06-24 03:05:31,482 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-24 03:05:31,482 INFO [upload_forcing] publishing log messages to tcp://*:5567
2025-06-24 03:05:31,483 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-06-24 03:05:31,728 INFO [ping_erddap] running in process 1081529
2025-06-24 03:05:31,729 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-24 03:05:31,730 INFO [ping_erddap] publishing log messages to tcp://*:5568
2025-06-24 03:05:31,731 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2025-06-24 03:05:31,732 INFO [ping_erddap] SEVIP-CTD ERDDAP dataset flag file(s) created
2025-06-24 03:05:31,734 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2025-06-24 03:05:31,945 INFO [ping_erddap] running in process 1081533
2025-06-24 03:05:31,947 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-24 03:05:31,948 INFO [ping_erddap] publishing log messages to tcp://*:5569
2025-06-24 03:05:31,949 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2025-06-24 03:05:31,950 INFO [ping_erddap] SCVIP-CTD ERDDAP dataset flag file(s) created
2025-06-24 03:05:32,102 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2025-06-24 03:05:32,743 INFO [get_onc_ferry] requesting ONC TWDP TSG data for 2025-06-23
2025-06-24 03:05:33,014 INFO [upload_forcing] LiveOcean boundary conditions file not found; created symlink to /results/forcing/LiveOcean/boundary_conditions/LiveOcean_v201905_y2025m06d23.nc
2025-06-24 03:05:33,016 INFO [upload_forcing] LiveOcean boundary conditions file not found; created symlink to /results/forcing/LiveOcean/boundary_conditions/LiveOcean_v201905_y2025m06d23.nc
2025-06-24 03:05:33,068 INFO [upload_forcing] forecast2 2025-06-24 forcing files upload to optimum-hindcast completed
2025-06-24 03:05:33,070 INFO [upload_forcing] forecast2 2025-06-24 forcing files upload to orcinus-nowcast-agrif completed
2025-06-24 03:05:33,072 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-06-24 03:05:33,395 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-06-24 03:05:39,683 INFO [upload_forcing] forecast2 2025-06-24 forcing files upload to robot.graham completed
2025-06-24 03:05:39,688 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-06-24 03:05:48,854 INFO [upload_forcing] forecast2 2025-06-24 forcing files upload to arbutus.cloud-nowcast completed
2025-06-24 03:05:48,859 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-06-24 03:05:49,220 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2025-06-24'], host='localhost')
2025-06-24 03:05:50,383 INFO [make_forcing_links] running in process 1081570
2025-06-24 03:05:50,384 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-24 03:05:50,385 INFO [make_forcing_links] publishing log messages to tcp://*:5563
2025-06-24 03:05:50,386 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2025-06-24 03:05:51,658 INFO [make_forcing_links] forecast2 2025-06-24 forcing file links on arbutus.cloud-nowcast created
2025-06-24 03:05:51,662 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2025-06-24 03:05:52,026 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2025-06-24'], host='arbutus.cloud-nowcast')
2025-06-24 03:05:54,418 INFO [run_NEMO] running in process 12324
2025-06-24 03:05:54,420 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-06-24 03:05:54,421 INFO [run_NEMO] publishing log messages to tcp://*:5556
2025-06-24 03:05:56,441 INFO [get_onc_ferry] requesting ONC TWDP OXYSENSOR data for 2025-06-23
2025-06-24 03:05:57,086 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2025-06-24 03:05:57,840 WARNING [run_NEMO] There are uncommitted changes in /nemoShare/MEOPAR/nowcast-sys/SS-run-sets
2025-06-24 03:05:58,157 INFO [run_NEMO] forecast2: launching /nemoShare/MEOPAR/nowcast-sys/runs/23jun25forecast2_2025-06-24T030557.181262-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2025-06-24 03:05:58,163 INFO [run_NEMO] forecast2 NEMO run for 2025-06-24 on arbutus.cloud-nowcast started
2025-06-24 03:05:58,166 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2025-06-24 03:05:58,518 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'forecast2'], host='arbutus.cloud-nowcast')
2025-06-24 03:06:01,378 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2025-06-24 03:06:19,624 INFO [get_onc_ferry] requesting ONC TWDP TURBCHLFL data for 2025-06-23
2025-06-24 03:06:46,817 INFO [get_onc_ferry] requesting ONC TWDP CO2SENSOR data for 2025-06-23
2025-06-24 03:06:46,880 WARNING [get_onc_ferry] No ONC TWDP CO2SENSOR partial_pressure,co2 data for 2025-06-23; substituting empty dataset
2025-06-24 03:06:46,885 WARNING [get_onc_ferry] No ONC TWDP CO2SENSOR partial_pressure data for 2025-06-23; substituting empty dataset
2025-06-24 03:06:46,886 WARNING [get_onc_ferry] No ONC TWDP CO2SENSOR co2 data for 2025-06-23; substituting empty dataset
2025-06-24 03:06:46,887 INFO [get_onc_ferry] requesting ONC TWDP TEMPHUMID data for 2025-06-23
2025-06-24 03:06:46,948 WARNING [get_onc_ferry] No ONC TWDP TEMPHUMID air_temperature,rel_humidity data for 2025-06-23; substituting empty dataset
2025-06-24 03:06:46,953 WARNING [get_onc_ferry] No ONC TWDP TEMPHUMID air_temperature data for 2025-06-23; substituting empty dataset
2025-06-24 03:06:46,955 WARNING [get_onc_ferry] No ONC TWDP TEMPHUMID rel_humidity data for 2025-06-23; substituting empty dataset
2025-06-24 03:06:46,956 INFO [get_onc_ferry] requesting ONC TWDP BARPRESS data for 2025-06-23
2025-06-24 03:06:46,993 WARNING [get_onc_ferry] No ONC TWDP BARPRESS barometric_pressure data for 2025-06-23; substituting empty dataset
2025-06-24 03:06:46,995 WARNING [get_onc_ferry] No ONC TWDP BARPRESS barometric_pressure data for 2025-06-23; substituting empty dataset
2025-06-24 03:06:46,996 INFO [get_onc_ferry] requesting ONC TWDP PYRANOMETER data for 2025-06-23
2025-06-24 03:06:47,055 WARNING [get_onc_ferry] No ONC TWDP PYRANOMETER solar_radiation data for 2025-06-23; substituting empty dataset
2025-06-24 03:06:47,058 WARNING [get_onc_ferry] No ONC TWDP PYRANOMETER solar_radiation data for 2025-06-23; substituting empty dataset
2025-06-24 03:06:47,059 INFO [get_onc_ferry] requesting ONC TWDP PYRGEOMETER data for 2025-06-23
2025-06-24 03:06:47,099 WARNING [get_onc_ferry] No ONC TWDP PYRGEOMETER downward_radiation data for 2025-06-23; substituting empty dataset
2025-06-24 03:06:47,102 WARNING [get_onc_ferry] No ONC TWDP PYRGEOMETER downward_radiation data for 2025-06-23; substituting empty dataset
2025-06-24 03:06:47,294 WARNING [get_onc_ferry] ONC TWDP CO2SENSOR partial_pressure data for 2025-06-23 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2025-06-24 03:06:47,310 WARNING [get_onc_ferry] ONC TWDP CO2SENSOR co2 data for 2025-06-23 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2025-06-24 03:06:47,328 WARNING [get_onc_ferry] ONC TWDP TEMPHUMID air_temperature data for 2025-06-23 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2025-06-24 03:06:47,345 WARNING [get_onc_ferry] ONC TWDP TEMPHUMID rel_humidity data for 2025-06-23 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2025-06-24 03:06:47,362 WARNING [get_onc_ferry] ONC TWDP BARPRESS barometric_pressure data for 2025-06-23 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2025-06-24 03:06:47,380 WARNING [get_onc_ferry] ONC TWDP PYRANOMETER solar_radiation data for 2025-06-23 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2025-06-24 03:06:47,397 WARNING [get_onc_ferry] ONC TWDP PYRGEOMETER downward_radiation data for 2025-06-23 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2025-06-24 03:07:55,989 INFO [get_onc_ferry] 2025-06-23 ONC TWDP ferry data file created
2025-06-24 03:07:55,992 INFO [manager] checklist updated with [ONC ferry data] items from get_onc_ferry worker
2025-06-24 03:07:56,335 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['TWDP-ferry'], host='localhost')
2025-06-24 03:07:57,416 INFO [ping_erddap] running in process 1081612
2025-06-24 03:07:57,418 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-24 03:07:57,419 INFO [ping_erddap] publishing log messages to tcp://*:5563
2025-06-24 03:07:57,420 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2025-06-24 03:07:57,421 INFO [ping_erddap] TWDP-ferry ERDDAP dataset flag file(s) created
2025-06-24 03:07:57,423 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2025-06-24 03:11:01,402 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 18531092 = 2025-06-25 05:00:40 UTC, 16.7% complete
2025-06-24 03:16:01,351 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 18531568 = 2025-06-25 10:18:00 UTC, 34.3% complete
2025-06-24 03:21:01,412 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 18532070 = 2025-06-25 15:52:40 UTC, 52.9% complete
2025-06-24 03:26:01,364 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 18532545 = 2025-06-25 21:09:20 UTC, 70.5% complete
2025-06-24 03:31:01,387 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 18533020 = 2025-06-26 02:26:00 UTC, 88.1% complete
2025-06-24 03:36:01,475 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2025-06-24 03:36:01,493 INFO [watch_NEMO] forecast2 NEMO run on arbutus.cloud-nowcast completed
2025-06-24 03:36:02,152 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_wind_file', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2025-06-24'], host='arbutus.cloud-nowcast')
2025-06-24 03:36:02,160 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_current_file', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2025-06-24'], host='arbutus.cloud-nowcast')
2025-06-24 03:36:02,165 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2025-06-23'], host='localhost')
2025-06-24 03:36:03,928 INFO [download_results] running in process 1081828
2025-06-24 03:36:03,929 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-24 03:36:03,930 INFO [download_results] publishing log messages to tcp://*:5563
2025-06-24 03:36:03,932 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2025-06-24 03:36:03,933 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/forecast2/23jun25 to /results/SalishSea/forecast2.202111
2025-06-24 03:36:05,132 INFO [make_ww3_current_file] running in process 13149
2025-06-24 03:36:05,133 INFO [make_ww3_current_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-06-24 03:36:05,134 INFO [make_ww3_current_file] publishing log messages to tcp://*:5571
2025-06-24 03:36:05,135 INFO [make_ww3_current_file] connected to skookum.eos.ubc.ca port 5555
2025-06-24 03:36:05,136 INFO [make_ww3_current_file] Creating wwatch3 currents forcing file for 2025-06-24 forecast2 run
2025-06-24 03:36:05,197 INFO [make_ww3_wind_file] running in process 13151
2025-06-24 03:36:05,199 INFO [make_ww3_wind_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-06-24 03:36:05,200 INFO [make_ww3_wind_file] publishing log messages to tcp://*:5570
2025-06-24 03:36:05,201 INFO [make_ww3_wind_file] connected to skookum.eos.ubc.ca port 5555
2025-06-24 03:36:05,202 INFO [make_ww3_wind_file] Creating wwatch3 wind forcing file for 2025-06-24 forecast2 run
2025-06-24 03:36:13,537 INFO [manager] checklist updated with [WW3 wind forcing] items from make_ww3_wind_file worker
2025-06-24 03:36:13,574 INFO [make_ww3_wind_file] wwatch3 wind forcing file created on arbutus.cloud-nowcast for 2025-06-24 forecast2 run
2025-06-24 03:39:35,341 INFO [make_ww3_current_file] wwatch3 currents forcing file created on arbutus.cloud-nowcast for 2025-06-24 forecast2 run
2025-06-24 03:39:35,460 INFO [manager] checklist updated with [WW3 currents forcing] items from make_ww3_current_file worker
2025-06-24 03:39:35,826 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2025-06-24'], host='arbutus.cloud-nowcast')
2025-06-24 03:39:40,285 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker
2025-06-24 03:39:40,659 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'forecast2'], host='arbutus.cloud-nowcast')
2025-06-24 03:39:42,650 INFO [watch_ww3] running in process 13784
2025-06-24 03:39:42,651 INFO [watch_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-06-24 03:39:42,652 INFO [watch_ww3] publishing log messages to tcp://*:5573
2025-06-24 03:39:42,653 INFO [watch_ww3] connected to skookum.eos.ubc.ca port 5555
2025-06-24 03:39:42,686 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2025-06-24 03:44:42,694 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 84, 15.6% complete
2025-06-24 03:49:42,712 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 162, 30.0% complete
2025-06-24 03:54:42,704 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 237, 43.9% complete
2025-06-24 03:59:42,709 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 315, 58.3% complete
2025-06-24 04:04:42,711 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 396, 73.3% complete
2025-06-24 04:09:42,709 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 471, 87.2% complete
2025-06-24 04:14:42,704 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2025-06-24 04:19:42,701 INFO [watch_ww3] forecast2 WWATCH3 run on arbutus.cloud-nowcast completed
2025-06-24 04:19:42,705 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2025-06-24 04:19:43,294 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2025-06-24'], host='localhost')
2025-06-24 04:19:44,587 INFO [download_wwatch3_results] running in process 1082169
2025-06-24 04:19:44,588 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-24 04:19:44,589 INFO [download_wwatch3_results] publishing log messages to tcp://*:5564
2025-06-24 04:19:44,590 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2025-06-24 04:20:23,360 INFO [download_wwatch3_results] forecast2 2025-06-24 results files from arbutus.cloud-nowcast downloaded
2025-06-24 04:20:23,363 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2025-06-24 04:20:23,741 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['wwatch3', 'forecast2', '--run-date', '2025-06-24'], host='localhost')
2025-06-24 04:20:24,843 INFO [update_forecast_datasets] running in process 1082176
2025-06-24 04:20:24,844 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-24 04:20:24,845 INFO [update_forecast_datasets] publishing log messages to tcp://*:5564
2025-06-24 04:20:24,846 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2025-06-24 04:20:24,847 INFO [update_forecast_datasets] updating wwatch3 most_recent_forecast directory from forecast2/24jun25 run
2025-06-24 04:20:24,881 INFO [update_forecast_datasets] updated wwatch3 most_recent_forecast directory from forecast2/24jun25 run
2025-06-24 04:20:24,882 INFO [update_forecast_datasets] updating wwatch3 forecast directory for forecast2/24jun25 run
2025-06-24 04:20:50,010 INFO [update_forecast_datasets] updated wwatch3 forecast directory for forecast2/24jun25 run: /results/SalishSea/rolling-forecasts/wwatch3
2025-06-24 04:20:50,021 INFO [update_forecast_datasets] wwatch3 2025-06-24 forecast2 rolling forecast datasets updated
2025-06-24 04:20:50,057 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2025-06-24 04:20:50,403 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['wwatch3-forecast'], host='localhost')
2025-06-24 04:20:51,769 INFO [ping_erddap] running in process 1082191
2025-06-24 04:20:51,771 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-24 04:20:51,772 INFO [ping_erddap] publishing log messages to tcp://*:5564
2025-06-24 04:20:51,773 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2025-06-24 04:20:51,774 INFO [ping_erddap] wwatch3-forecast ERDDAP dataset flag file(s) created
2025-06-24 04:20:51,776 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2025-06-24 04:20:52,120 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['wwatch3', 'forecast', 'publish', '--run-date', '2025-06-23'], host='localhost')
2025-06-24 04:20:57,235 INFO [make_plots] running in process 1082196
2025-06-24 04:20:57,237 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-24 04:20:57,238 INFO [make_plots] publishing log messages to tcp://*:5564
2025-06-24 04:20:57,239 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2025-06-24 04:20:57,240 INFO [make_plots] preparing render list for 2025-06-23 WaveWatch3 forecast publish figures
2025-06-24 04:20:57,240 INFO [make_plots] added Halibut Bank figure to 2025-06-23 WaveWatch3 forecast publish render list
2025-06-24 04:20:57,241 INFO [make_plots] added Sentry Shoal figure to 2025-06-23 WaveWatch3 forecast publish render list
2025-06-24 04:20:57,241 INFO [make_plots] starting to render wwatch3 forecast publish 23jun25 figures
2025-06-24 04:20:57,783 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46146 Halibut Bank
2025-06-24 04:21:03,907 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46131 Sentry Shoal
2025-06-24 04:21:08,541 INFO [make_plots] finished rendering wwatch3 forecast publish 23jun25 figures
2025-06-24 04:21:08,542 INFO [make_plots] wwatch3 publish plots for 2025-06-23 forecast completed
2025-06-24 04:21:08,545 INFO [manager] checklist updated with [plots] items from make_plots worker
2025-06-24 04:27:01,646 INFO [download_results] forecast2 2025-06-23 results files from arbutus.cloud-nowcast downloaded
2025-06-24 04:27:01,650 INFO [manager] checklist updated with [results files] items from download_results worker
2025-06-24 04:27:01,986 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['forecast2', '--run-date', '2025-06-23'], host='localhost')
2025-06-24 04:27:03,810 INFO [make_CHS_currents_file] running in process 1082256
2025-06-24 04:27:03,811 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-24 04:27:03,812 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5563
2025-06-24 04:27:03,813 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555
2025-06-24 04:27:44,792 INFO [make_CHS_currents_file] Made CHS currents file for 2025-06-23 for forecast2
2025-06-24 04:27:44,794 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker
2025-06-24 04:27:45,099 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['nemo', 'forecast2', '--run-date', '2025-06-23'], host='localhost')
2025-06-24 04:27:46,420 INFO [update_forecast_datasets] running in process 1082279
2025-06-24 04:27:46,421 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-24 04:27:46,423 INFO [update_forecast_datasets] publishing log messages to tcp://*:5563
2025-06-24 04:27:46,423 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2025-06-24 04:27:46,424 INFO [update_forecast_datasets] updating nemo forecast directory for forecast2/23jun25 run
2025-06-24 04:32:06,178 INFO [update_forecast_datasets] updated nemo forecast directory for forecast2/23jun25 run: /results/SalishSea/rolling-forecasts/nemo
2025-06-24 04:32:06,179 INFO [update_forecast_datasets] nemo 2025-06-23 forecast2 rolling forecast datasets updated
2025-06-24 04:32:06,180 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2025-06-24 04:32:06,532 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nemo-forecast'], host='localhost')
2025-06-24 04:32:06,538 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'forecast2', 'publish', '--run-date', '2025-06-23'], host='localhost')
2025-06-24 04:32:06,543 INFO [manager] launching NextWorker(module='nowcast.workers.make_surface_current_tiles', args=['forecast2', '--run-date', '2025-06-23'], host='localhost')
2025-06-24 04:32:07,819 INFO [ping_erddap] running in process 1082367
2025-06-24 04:32:07,820 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-24 04:32:07,821 INFO [ping_erddap] publishing log messages to tcp://*:5563
2025-06-24 04:32:07,822 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2025-06-24 04:32:07,829 INFO [ping_erddap] nemo-forecast ERDDAP dataset flag file(s) created
2025-06-24 04:32:07,830 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2025-06-24 04:32:09,240 INFO [make_surface_current_tiles] running in process 1082370
2025-06-24 04:32:09,259 INFO [make_surface_current_tiles] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-24 04:32:09,260 INFO [make_surface_current_tiles] publishing log messages to tcp://*:5563
2025-06-24 04:32:09,261 INFO [make_surface_current_tiles] connected to skookum.eos.ubc.ca port 5555
2025-06-24 04:32:09,301 INFO [make_surface_current_tiles] starting rendering of tiles for 2025-06-23 forecast2 in 6 processes into /results/nowcast-sys/figures/surface_currents/forecast2/23jun25
2025-06-24 04:32:09,672 INFO [make_plots] running in process 1082369
2025-06-24 04:32:09,674 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-24 04:32:09,679 INFO [make_plots] publishing log messages to tcp://*:5564
2025-06-24 04:32:09,680 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2025-06-24 04:32:10,272 INFO [make_plots] preparing render list for 2025-06-23 NEMO forecast2 publish figures
2025-06-24 04:32:10,568 INFO [make_plots] added Website_thumbnail figure to 2025-06-23 NEMO forecast2 publish render list
2025-06-24 04:32:10,592 INFO [make_plots] added Threshold_website figure to 2025-06-23 NEMO forecast2 publish render list
2025-06-24 04:32:10,609 INFO [make_plots] added PA_tidal_predictions figure to 2025-06-23 NEMO forecast2 publish render list
2025-06-24 04:32:10,616 INFO [make_plots] added Boundary Bay figure to 2025-06-23 NEMO forecast2 publish render list
2025-06-24 04:32:10,617 INFO [make_plots] added Campbell River figure to 2025-06-23 NEMO forecast2 publish render list
2025-06-24 04:32:10,618 INFO [make_plots] added Cherry Point figure to 2025-06-23 NEMO forecast2 publish render list
2025-06-24 04:32:10,619 INFO [make_plots] added Friday Harbor figure to 2025-06-23 NEMO forecast2 publish render list
2025-06-24 04:32:10,621 INFO [make_plots] added Halfmoon Bay figure to 2025-06-23 NEMO forecast2 publish render list
2025-06-24 04:32:10,622 INFO [make_plots] added Nanaimo figure to 2025-06-23 NEMO forecast2 publish render list
2025-06-24 04:32:10,624 INFO [make_plots] added Neah Bay figure to 2025-06-23 NEMO forecast2 publish render list
2025-06-24 04:32:10,625 INFO [make_plots] added New Westminster figure to 2025-06-23 NEMO forecast2 publish render list
2025-06-24 04:32:10,626 INFO [make_plots] added Patricia Bay figure to 2025-06-23 NEMO forecast2 publish render list
2025-06-24 04:32:10,627 INFO [make_plots] added Point Atkinson figure to 2025-06-23 NEMO forecast2 publish render list
2025-06-24 04:32:10,627 INFO [make_plots] added Port Renfrew figure to 2025-06-23 NEMO forecast2 publish render list
2025-06-24 04:32:10,628 INFO [make_plots] added Sand Heads figure to 2025-06-23 NEMO forecast2 publish render list
2025-06-24 04:32:10,629 INFO [make_plots] added Sandy Cove figure to 2025-06-23 NEMO forecast2 publish render list
2025-06-24 04:32:10,629 INFO [make_plots] added Squamish figure to 2025-06-23 NEMO forecast2 publish render list
2025-06-24 04:32:10,630 INFO [make_plots] added Victoria figure to 2025-06-23 NEMO forecast2 publish render list
2025-06-24 04:32:10,630 INFO [make_plots] added Woodwards Landing figure to 2025-06-23 NEMO forecast2 publish render list
2025-06-24 04:32:10,631 INFO [make_plots] starting to render nemo forecast2 publish 23jun25 figures
2025-06-24 04:41:04,657 WARNING [make_plots] invalid station number for Boundary Bay station: None
2025-06-24 04:41:04,658 WARNING [make_plots] can't resolve a valid CHS station code for Boundary Bay
2025-06-24 04:41:30,668 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb996/data for station 08074 Campbell River from 2025-06-19 00:00:00Z to 2025-06-26 05:55:00Z
2025-06-24 04:42:45,486 WARNING [make_plots] invalid station number for Halfmoon Bay station: None
2025-06-24 04:42:45,487 WARNING [make_plots] can't resolve a valid CHS station code for Halfmoon Bay
2025-06-24 04:43:10,529 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb96d/data for station 07917 Nanaimo from 2025-06-19 00:00:00Z to 2025-06-26 05:55:00Z
2025-06-24 04:43:48,414 INFO [make_surface_current_tiles] starting PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast2/23jun25
2025-06-24 04:43:51,438 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e13d0f4a073c4bbf93/data for station 07654 New Westminster from 2025-06-19 00:00:00Z to 2025-06-26 05:55:00Z
2025-06-24 04:44:10,642 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd2d/data for station 07277 Patricia Bay from 2025-06-19 00:00:00Z to 2025-06-26 05:55:00Z
2025-06-24 04:44:29,692 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94c/data for station 07795 Point Atkinson from 2025-06-19 00:00:00Z to 2025-06-26 05:55:00Z
2025-06-24 04:44:48,505 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e23d0f4a073c4bc060/data for station 08525 Port Renfrew from 2025-06-19 00:00:00Z to 2025-06-26 05:55:00Z
2025-06-24 04:45:05,298 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb93c/data for station 07594 Sand Heads from 2025-06-19 00:00:00Z to 2025-06-26 05:55:00Z
2025-06-24 04:45:24,059 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc43b/data for station 07786 Sandy Cove from 2025-06-19 00:00:00Z to 2025-06-26 05:55:00Z
2025-06-24 04:45:42,948 WARNING [make_plots] invalid station number for Squamish station: None
2025-06-24 04:45:42,949 WARNING [make_plots] can't resolve a valid CHS station code for Squamish
2025-06-24 04:46:00,978 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd1e/data for station 07120 Victoria from 2025-06-19 00:00:00Z to 2025-06-26 05:55:00Z
2025-06-24 04:46:17,733 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc42d/data for station 07610 Woodwards Landing from 2025-06-19 00:00:00Z to 2025-06-26 05:55:00Z
2025-06-24 04:46:37,849 INFO [make_plots] finished rendering nemo forecast2 publish 23jun25 figures
2025-06-24 04:46:37,851 INFO [make_plots] nemo publish plots for 2025-06-23 forecast2 completed
2025-06-24 04:46:37,859 INFO [manager] checklist updated with [plots] items from make_plots worker
2025-06-24 04:46:38,233 INFO [manager] launching NextWorker(module='nowcast.workers.make_feeds', args=['forecast2', '--run-date', '2025-06-23'], host='localhost')
2025-06-24 04:46:42,078 INFO [make_feeds] running in process 1085238
2025-06-24 04:46:42,079 INFO [make_feeds] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-24 04:46:42,080 INFO [make_feeds] publishing log messages to tcp://*:5564
2025-06-24 04:46:42,081 INFO [make_feeds] connected to skookum.eos.ubc.ca port 5555
2025-06-24 04:46:58,812 INFO [make_feeds] ATOM feeds for 2025-06-23 forecast2 run completed
2025-06-24 04:46:58,815 INFO [manager] checklist updated with [storm surge alert feeds] items from make_feeds worker
2025-06-24 04:46:59,161 INFO [manager] launching NextWorker(module='nemo_nowcast.workers.clear_checklist', args=[], host='localhost')
2025-06-24 04:47:00,514 INFO [clear_checklist] running in process 1085256
2025-06-24 04:47:00,515 INFO [clear_checklist] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-24 04:47:00,516 INFO [clear_checklist] publishing log messages to tcp://*:5564
2025-06-24 04:47:00,517 INFO [clear_checklist] connected to skookum.eos.ubc.ca port 5555
2025-06-24 04:47:00,518 INFO [clear_checklist] requesting that manager clear system state checklist
2025-06-24 04:47:00,520 INFO [manager] writing checklist to log file
2025-06-24 04:47:00,579 INFO [manager] checklist cleared
2025-06-24 04:47:00,581 INFO [clear_checklist] nowcast system checklist cleared
2025-06-24 04:47:00,584 INFO [manager] launching NextWorker(module='nemo_nowcast.workers.rotate_logs', args=[], host='localhost')
2025-06-24 04:47:01,630 INFO [rotate_logs] running in process 1085261
2025-06-24 04:47:01,631 INFO [rotate_logs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-24 04:47:01,632 INFO [rotate_logs] publishing log messages to tcp://*:5564
2025-06-24 04:47:01,633 INFO [rotate_logs] connected to skookum.eos.ubc.ca port 5555
2025-06-24 04:47:01,634 INFO [rotate_logs] rotating log files