2025-07-18 03:56:44,074 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/nowcast.log
2025-07-18 03:56:44,080 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/nowcast.debug.log
2025-07-18 03:56:44,089 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/checklist.log
2025-07-18 03:56:44,090 INFO [rotate_logs] log files rotated
2025-07-18 03:56:44,101 INFO [manager] checklist updated with [log rotation] items from rotate_logs worker
2025-07-18 03:58:32,295 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 417, 77.2% complete
2025-07-18 03:58:45,324 INFO [make_surface_current_tiles] finished PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast2/17jul25
2025-07-18 03:58:45,347 INFO [make_surface_current_tiles] finished rendering of tiles for 2025-07-17 forecast2 into /results/nowcast-sys/figures/surface_currents/forecast2/17jul25
2025-07-18 03:58:45,348 INFO [make_surface_current_tiles] surface current tile figures for 2025-07-17 forecast2 completed
2025-07-18 03:58:45,663 INFO [manager] checklist updated with [surface current tiles] items from make_surface_current_tiles worker
2025-07-18 04:03:32,297 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 525, 97.2% complete
2025-07-18 04:08:32,299 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2025-07-18 04:08:32,300 INFO [watch_ww3] forecast2 WWATCH3 run on arbutus.cloud-nowcast completed
2025-07-18 04:08:32,685 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2025-07-18'], host='localhost')
2025-07-18 04:08:33,808 INFO [download_wwatch3_results] running in process 1647239
2025-07-18 04:08:33,810 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-18 04:08:33,811 INFO [download_wwatch3_results] publishing log messages to tcp://*:5562
2025-07-18 04:08:33,812 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2025-07-18 04:09:14,148 INFO [download_wwatch3_results] forecast2 2025-07-18 results files from arbutus.cloud-nowcast downloaded
2025-07-18 04:09:14,151 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2025-07-18 04:09:14,309 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['wwatch3', 'forecast2', '--run-date', '2025-07-18'], host='localhost')
2025-07-18 04:09:15,436 INFO [update_forecast_datasets] running in process 1647249
2025-07-18 04:09:15,437 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-18 04:09:15,438 INFO [update_forecast_datasets] publishing log messages to tcp://*:5562
2025-07-18 04:09:15,439 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2025-07-18 04:09:15,439 INFO [update_forecast_datasets] updating wwatch3 most_recent_forecast directory from forecast2/18jul25 run
2025-07-18 04:09:15,564 INFO [update_forecast_datasets] updated wwatch3 most_recent_forecast directory from forecast2/18jul25 run
2025-07-18 04:09:15,565 INFO [update_forecast_datasets] updating wwatch3 forecast directory for forecast2/18jul25 run
2025-07-18 04:09:43,672 INFO [update_forecast_datasets] updated wwatch3 forecast directory for forecast2/18jul25 run: /results/SalishSea/rolling-forecasts/wwatch3
2025-07-18 04:09:43,672 INFO [update_forecast_datasets] wwatch3 2025-07-18 forecast2 rolling forecast datasets updated
2025-07-18 04:09:43,676 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2025-07-18 04:09:43,830 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['wwatch3-forecast'], host='localhost')
2025-07-18 04:09:44,895 INFO [ping_erddap] running in process 1647264
2025-07-18 04:09:44,896 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-18 04:09:44,897 INFO [ping_erddap] publishing log messages to tcp://*:5562
2025-07-18 04:09:44,898 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2025-07-18 04:09:44,899 INFO [ping_erddap] wwatch3-forecast ERDDAP dataset flag file(s) created
2025-07-18 04:09:44,901 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2025-07-18 04:09:45,055 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['wwatch3', 'forecast2', 'publish', '--run-date', '2025-07-18'], host='localhost')
2025-07-18 04:09:47,910 INFO [make_plots] running in process 1647269
2025-07-18 04:09:47,911 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-18 04:09:47,912 INFO [make_plots] publishing log messages to tcp://*:5562
2025-07-18 04:09:47,913 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2025-07-18 04:09:47,913 INFO [make_plots] preparing render list for 2025-07-18 WaveWatch3 forecast2 publish figures
2025-07-18 04:09:47,914 INFO [make_plots] added Halibut Bank figure to 2025-07-18 WaveWatch3 forecast2 publish render list
2025-07-18 04:09:47,915 INFO [make_plots] added Sentry Shoal figure to 2025-07-18 WaveWatch3 forecast2 publish render list
2025-07-18 04:09:47,916 INFO [make_plots] starting to render wwatch3 forecast2 publish 18jul25 figures
2025-07-18 04:09:48,300 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46146 Halibut Bank
2025-07-18 04:09:54,568 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46131 Sentry Shoal
2025-07-18 04:09:59,132 INFO [make_plots] finished rendering wwatch3 forecast2 publish 18jul25 figures
2025-07-18 04:09:59,133 INFO [make_plots] wwatch3 publish plots for 2025-07-18 forecast2 completed
2025-07-18 04:09:59,136 INFO [manager] checklist updated with [plots] items from make_plots worker
2025-07-18 09:08:18,994 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/12/ to /results/forcing/atmospheric/continental2.5/GRIB/20250718/12/
2025-07-18 09:08:18,995 INFO [collect_weather] 2025-07-18 2.5km weather forecast 12 collection complete
2025-07-18 09:08:18,999 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2025-07-18 09:08:19,178 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20250718/12/
2025-07-18 09:08:19,179 INFO [crop_gribs] 2025-07-18 12 GRIBs cropping complete
2025-07-18 09:08:19,311 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'SkagitMountVernon', '--data-date', '2025-07-17'], host='localhost')
2025-07-18 09:08:19,320 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'SnohomishMonroe', '--data-date', '2025-07-17'], host='localhost')
2025-07-18 09:08:19,325 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'NisquallyMcKenna', '--data-date', '2025-07-17'], host='localhost')
2025-07-18 09:08:19,330 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'GreenwaterGreenwater', '--data-date', '2025-07-17'], host='localhost')
2025-07-18 09:08:19,335 INFO [manager] launching NextWorker(module='nowcast.workers.make_turbidity_file', args=[], host='localhost')
2025-07-18 09:08:19,340 INFO [manager] launching NextWorker(module='nowcast.workers.collect_NeahBay_ssh', args=['06'], host='localhost')
2025-07-18 09:08:19,343 INFO [manager] launching NextWorker(module='nowcast.workers.download_live_ocean', args=[], host='localhost')
2025-07-18 09:08:19,348 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['18', '2.5km'], host='localhost')
2025-07-18 09:08:19,351 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['18'], host='localhost')
2025-07-18 09:08:19,360 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2025-07-18 09:08:19,731 INFO [manager] launching NextWorker(module='nowcast.workers.grib_to_netcdf', args=['nowcast+'], host='localhost')
2025-07-18 09:08:21,167 INFO [collect_weather] running in process 1650094
2025-07-18 09:08:21,168 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-18 09:08:21,169 INFO [collect_weather] publishing log messages to tcp://*:5562
2025-07-18 09:08:21,171 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2025-07-18 09:08:21,177 INFO [collect_NeahBay_ssh] running in process 1650092
2025-07-18 09:08:21,178 INFO [collect_NeahBay_ssh] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-18 09:08:21,179 INFO [collect_NeahBay_ssh] publishing log messages to tcp://*:5563
2025-07-18 09:08:21,180 INFO [collect_NeahBay_ssh] connected to skookum.eos.ubc.ca port 5555
2025-07-18 09:08:21,181 INFO [collect_NeahBay_ssh] collecting Neah Bay ssh 06Z obs/forecast for 2025-07-18
2025-07-18 09:08:21,186 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/18/
2025-07-18 09:08:21,232 INFO [download_live_ocean] running in process 1650093
2025-07-18 09:08:21,233 INFO [download_live_ocean] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-18 09:08:21,234 INFO [download_live_ocean] publishing log messages to tcp://*:5564
2025-07-18 09:08:21,234 INFO [download_live_ocean] connected to skookum.eos.ubc.ca port 5555
2025-07-18 09:08:21,235 INFO [download_live_ocean] downloading Salish Sea western boundary day-averaged LiveOcean file for 2025-07-18
2025-07-18 09:08:21,706 INFO [collect_river_data] running in process 1650090
2025-07-18 09:08:21,708 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-18 09:08:21,708 INFO [collect_river_data] publishing log messages to tcp://*:5565
2025-07-18 09:08:21,709 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-07-18 09:08:21,710 INFO [collect_river_data] Collecting USGS GreenwaterGreenwater river data for 2025-07-17
2025-07-18 09:08:21,724 INFO [collect_river_data] running in process 1650089
2025-07-18 09:08:21,725 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-18 09:08:21,726 INFO [collect_river_data] publishing log messages to tcp://*:5566
2025-07-18 09:08:21,726 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-07-18 09:08:21,727 INFO [collect_river_data] Collecting USGS NisquallyMcKenna river data for 2025-07-17
2025-07-18 09:08:21,745 INFO [collect_river_data] running in process 1650086
2025-07-18 09:08:21,747 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-18 09:08:21,748 INFO [collect_river_data] publishing log messages to tcp://*:5567
2025-07-18 09:08:21,749 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-07-18 09:08:21,750 INFO [collect_river_data] Collecting USGS SkagitMountVernon river data for 2025-07-17
2025-07-18 09:08:21,756 INFO [collect_river_data] running in process 1650088
2025-07-18 09:08:21,758 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-18 09:08:21,759 INFO [collect_river_data] publishing log messages to tcp://*:5568
2025-07-18 09:08:21,760 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-07-18 09:08:21,761 INFO [collect_river_data] Collecting USGS SnohomishMonroe river data for 2025-07-17
2025-07-18 09:08:21,808 INFO [make_turbidity_file] running in process 1650091
2025-07-18 09:08:21,810 INFO [make_turbidity_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-18 09:08:21,812 INFO [make_turbidity_file] publishing log messages to tcp://*:5569
2025-07-18 09:08:21,813 INFO [make_turbidity_file] connected to skookum.eos.ubc.ca port 5555
2025-07-18 09:08:21,814 INFO [make_turbidity_file] Creating Fraser River turbidity forcing file for 2025-07-18
2025-07-18 09:08:21,915 INFO [download_live_ocean] downloaded 10206173 bytes from https://s3.kopah.uw.edu/liveocean-share/f2025.07.18/ubc.nc to /results/forcing/LiveOcean/downloaded/20250718/low_passed_UBC.nc
2025-07-18 09:08:21,916 INFO [download_live_ocean] Deflating in up to 1 concurrent sub-processes
2025-07-18 09:08:22,002 INFO [grib_to_netcdf] running in process 1650118
2025-07-18 09:08:22,004 INFO [grib_to_netcdf] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-18 09:08:22,004 INFO [grib_to_netcdf] publishing log messages to tcp://*:5570
2025-07-18 09:08:22,005 INFO [grib_to_netcdf] connected to skookum.eos.ubc.ca port 5555
2025-07-18 09:08:22,006 INFO [grib_to_netcdf] creating NEMO-atmos forcing files for 2025-07-18 nowcast and forecast runs
2025-07-18 09:08:22,127 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12089500&startDT=2025-07-17&endDT=2025-07-17 "HTTP/1.1 200 "
2025-07-18 09:08:22,150 INFO [collect_river_data] Appended USGS NisquallyMcKenna river average discharge for 2025-07-17 to: /results/forcing/rivers/observations/Nisqually_McKenna_flow
2025-07-18 09:08:22,151 INFO [collect_river_data] USGS NisquallyMcKenna river data collection for 2025-07-17 completed
2025-07-18 09:08:22,153 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-07-18 09:08:22,286 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12200500&startDT=2025-07-17&endDT=2025-07-17 "HTTP/1.1 200 "
2025-07-18 09:08:22,300 INFO [crop_gribs] running in process 1650095
2025-07-18 09:08:22,300 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-18 09:08:22,301 INFO [crop_gribs] publishing log messages to tcp://*:5571
2025-07-18 09:08:22,302 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2025-07-18 09:08:22,302 INFO [crop_gribs] cropping 2025-07-18 ECCC HRDPS 2.5km continental 18Z GRIB files to SalishSeaCast subdomain
2025-07-18 09:08:22,317 INFO [collect_river_data] Appended USGS SkagitMountVernon river average discharge for 2025-07-17 to: /results/forcing/rivers/observations/Skagit_MountVernon_flow
2025-07-18 09:08:22,318 INFO [collect_river_data] USGS SkagitMountVernon river data collection for 2025-07-17 completed
2025-07-18 09:08:22,319 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20250718/18/
2025-07-18 09:08:22,321 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-07-18 09:08:22,325 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12097500&startDT=2025-07-17&endDT=2025-07-17 "HTTP/1.1 200 "
2025-07-18 09:08:22,332 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12150800&startDT=2025-07-17&endDT=2025-07-17 "HTTP/1.1 200 "
2025-07-18 09:08:22,351 INFO [collect_river_data] Appended USGS SnohomishMonroe river average discharge for 2025-07-17 to: /results/forcing/rivers/observations/Snohomish_Monroe_flow
2025-07-18 09:08:22,352 INFO [collect_river_data] USGS SnohomishMonroe river data collection for 2025-07-17 completed
2025-07-18 09:08:22,355 INFO [collect_river_data] Appended USGS GreenwaterGreenwater river average discharge for 2025-07-17 to: /results/forcing/rivers/observations/Greenwater_Greenwater_flow
2025-07-18 09:08:22,356 INFO [collect_river_data] USGS GreenwaterGreenwater river data collection for 2025-07-17 completed
2025-07-18 09:08:22,444 INFO [collect_NeahBay_ssh] 2025-07-18T00:00:00-07:00 Neah Bay ssh 06Z obs/forecast data collection complete
2025-07-18 09:08:22,488 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-07-18 09:08:22,643 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-07-18 09:08:22,796 INFO [manager] checklist updated with [Neah Bay ssh data] items from collect_NeahBay_ssh worker
2025-07-18 09:08:22,942 INFO [manager] launching NextWorker(module='nowcast.workers.make_ssh_files', args=['nowcast', '--run-date', '2025-07-18'], host='localhost')
2025-07-18 09:08:22,953 INFO [download_live_ocean] netCDF4 deflated /results/forcing/LiveOcean/downloaded/20250718/low_passed_UBC.nc
2025-07-18 09:08:22,954 INFO [download_live_ocean] 2025-07-18 Live Ocean file for Salish Sea western boundary downloaded
2025-07-18 09:08:22,955 INFO [manager] checklist updated with [Live Ocean products] items from download_live_ocean worker
2025-07-18 09:08:23,278 INFO [manager] launching NextWorker(module='nowcast.workers.make_live_ocean_files', args=['--run-date', '2025-07-18'], host='localhost')
2025-07-18 09:08:24,446 INFO [make_turbidity_file] 2025-07-18 Fraser River turbidity file creation complete
2025-07-18 09:08:24,447 INFO [manager] checklist updated with [Fraser River turbidity] items from make_turbidity_file worker
2025-07-18 09:08:25,871 INFO [make_live_ocean_files] running in process 1650238
2025-07-18 09:08:25,872 INFO [make_live_ocean_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-18 09:08:25,872 INFO [make_live_ocean_files] publishing log messages to tcp://*:5563
2025-07-18 09:08:25,873 INFO [make_live_ocean_files] connected to skookum.eos.ubc.ca port 5555
2025-07-18 09:08:25,874 INFO [make_live_ocean_files] Creating T&S western boundary conditions file from 2025-07-18 Live Ocean run
2025-07-18 09:08:25,928 INFO [make_ssh_files] running in process 1650225
2025-07-18 09:08:25,929 INFO [make_ssh_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-18 09:08:25,930 INFO [make_ssh_files] publishing log messages to tcp://*:5564
2025-07-18 09:08:25,931 INFO [make_ssh_files] connected to skookum.eos.ubc.ca port 5555
2025-07-18 09:08:25,932 INFO [make_ssh_files] building nowcast sea surface height boundary conditions file(s) from 2025-07-18 NOAA Neah Bay 06Z observation and forecast values
2025-07-18 09:08:27,798 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2025m07d14.nc
2025-07-18 09:08:27,985 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2025m07d15.nc
2025-07-18 09:08:28,174 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2025m07d16.nc
2025-07-18 09:08:28,366 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2025m07d17.nc
2025-07-18 09:08:28,572 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2025m07d18.nc
2025-07-18 09:08:28,747 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2025m07d19.nc
2025-07-18 09:08:28,935 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2025m07d20.nc
2025-07-18 09:08:29,128 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2025m07d21.nc
2025-07-18 09:08:29,265 INFO [make_ssh_files] sea surface height boundary file for 2025-07-18 nowcast run created
2025-07-18 09:08:29,269 INFO [manager] checklist updated with [sea surface height forcing] items from make_ssh_files worker
2025-07-18 09:08:29,429 INFO [manager] launching NextWorker(module='nowcast.workers.make_runoff_file', args=['v202108'], host='localhost')
2025-07-18 09:08:29,434 INFO [manager] launching NextWorker(module='nowcast.workers.make_201702_runoff_file', args=[], host='localhost')
2025-07-18 09:08:30,754 INFO [make_201702_runoff_file] running in process 1650274
2025-07-18 09:08:30,755 INFO [make_201702_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-18 09:08:30,756 INFO [make_201702_runoff_file] publishing log messages to tcp://*:5564
2025-07-18 09:08:30,757 INFO [make_201702_runoff_file] connected to skookum.eos.ubc.ca port 5555
2025-07-18 09:08:30,801 INFO [make_201702_runoff_file] calculating runoff forcing for b201702 bathymetry
2025-07-18 09:08:30,927 INFO [make_201702_runoff_file] 2025-07-18 runoff file creation from Fraser at Hope and climatology elsewhere complete
2025-07-18 09:08:30,930 INFO [manager] checklist updated with [rivers forcing] items from make_201702_runoff_file worker
2025-07-18 09:08:31,232 INFO [make_runoff_file] running in process 1650272
2025-07-18 09:08:31,233 INFO [make_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-18 09:08:31,234 INFO [make_runoff_file] publishing log messages to tcp://*:5565
2025-07-18 09:08:31,235 INFO [make_runoff_file] connected to skookum.eos.ubc.ca port 5555
2025-07-18 09:08:31,238 INFO [make_runoff_file] calculating NEMO runoff forcing for v202108 bathymetry for 2025-07-17
2025-07-18 09:08:33,960 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/hrdps_y2025m07d18.nc
2025-07-18 09:08:33,963 INFO [make_runoff_file] stored NEMO runoff forcing for v202108 bathymetry for 2025-07-17: /results/forcing/rivers/R202108Dailies_y2025m07d17.nc
2025-07-18 09:08:33,964 INFO [make_runoff_file] 2025-07-17 runoff file creation completed
2025-07-18 09:08:33,966 INFO [manager] checklist updated with [rivers forcing] items from make_runoff_file worker
2025-07-18 09:08:41,592 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2025m07d19.nc
2025-07-18 09:08:45,907 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2025m07d20.nc
2025-07-18 09:08:45,926 INFO [grib_to_netcdf] 2025-07-18 NEMO-atmos forcing file for nowcast+ created
2025-07-18 09:08:45,929 INFO [manager] checklist updated with [weather forcing] items from grib_to_netcdf worker
2025-07-18 09:09:01,935 INFO [make_live_ocean_files] Stored T&S western boundary conditions file: /results/forcing/LiveOcean/boundary_conditions/LiveOcean_v201905_y2025m07d18.nc
2025-07-18 09:09:01,936 INFO [make_live_ocean_files] 2025-07-18 Live Ocean western boundary conditions files created
2025-07-18 09:09:01,938 INFO [manager] checklist updated with [Live Ocean boundary conditions] items from make_live_ocean_files worker
2025-07-18 09:09:02,257 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['weather'], host='localhost')
2025-07-18 09:09:02,258 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'nowcast+'], host='localhost')
2025-07-18 09:09:02,259 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'nowcast+'], host='localhost')
2025-07-18 09:09:02,260 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.graham', 'nowcast+'], host='localhost')
2025-07-18 09:09:02,261 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'nowcast+'], host='localhost')
2025-07-18 09:09:03,230 INFO [ping_erddap] running in process 1650320
2025-07-18 09:09:03,231 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-18 09:09:03,231 INFO [ping_erddap] publishing log messages to tcp://*:5563
2025-07-18 09:09:03,232 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2025-07-18 09:09:03,233 INFO [ping_erddap] weather ERDDAP dataset flag file(s) created
2025-07-18 09:09:03,235 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2025-07-18 09:09:03,406 INFO [upload_forcing] running in process 1650322
2025-07-18 09:09:03,407 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-18 09:09:03,408 INFO [upload_forcing] publishing log messages to tcp://*:5564
2025-07-18 09:09:03,409 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-07-18 09:09:03,410 INFO [upload_forcing] running in process 1650325
2025-07-18 09:09:03,411 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-18 09:09:03,412 INFO [upload_forcing] publishing log messages to tcp://*:5565
2025-07-18 09:09:03,413 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-07-18 09:09:03,449 INFO [upload_forcing] running in process 1650323
2025-07-18 09:09:03,451 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-18 09:09:03,452 INFO [upload_forcing] publishing log messages to tcp://*:5566
2025-07-18 09:09:03,452 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-07-18 09:09:03,466 INFO [upload_forcing] running in process 1650324
2025-07-18 09:09:03,467 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-18 09:09:03,468 INFO [upload_forcing] publishing log messages to tcp://*:5567
2025-07-18 09:09:03,468 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-07-18 09:09:05,732 INFO [upload_forcing] nowcast+ 2025-07-18 forcing files upload to orcinus-nowcast-agrif completed
2025-07-18 09:09:05,735 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-07-18 09:09:05,840 INFO [upload_forcing] nowcast+ 2025-07-18 forcing files upload to optimum-hindcast completed
2025-07-18 09:09:05,916 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-07-18 09:09:08,812 INFO [upload_forcing] nowcast+ 2025-07-18 forcing files upload to arbutus.cloud-nowcast completed
2025-07-18 09:09:08,817 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-07-18 09:09:08,980 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'nowcast+', '--run-date', '2025-07-18'], host='localhost')
2025-07-18 09:09:10,149 INFO [make_forcing_links] running in process 1650361
2025-07-18 09:09:10,150 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-18 09:09:10,151 INFO [make_forcing_links] publishing log messages to tcp://*:5563
2025-07-18 09:09:10,152 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2025-07-18 09:09:11,379 INFO [make_forcing_links] nowcast+ 2025-07-18 forcing file links on arbutus.cloud-nowcast created
2025-07-18 09:09:11,383 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2025-07-18 09:09:11,544 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2025-07-18'], host='arbutus.cloud-nowcast')
2025-07-18 09:09:13,508 INFO [run_NEMO] running in process 7174
2025-07-18 09:09:13,508 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-07-18 09:09:13,509 INFO [run_NEMO] publishing log messages to tcp://*:5556
2025-07-18 09:09:14,278 INFO [upload_forcing] nowcast+ 2025-07-18 forcing files upload to robot.graham completed
2025-07-18 09:09:14,283 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-07-18 09:09:16,098 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2025-07-18 09:09:16,272 WARNING [run_NEMO] There are uncommitted changes in /nemoShare/MEOPAR/nowcast-sys/SS-run-sets
2025-07-18 09:09:16,279 INFO [run_NEMO] nowcast: launching /nemoShare/MEOPAR/nowcast-sys/runs/18jul25nowcast_2025-07-18T090913.551109-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2025-07-18 09:09:16,281 INFO [run_NEMO] nowcast NEMO run for 2025-07-18 on arbutus.cloud-nowcast started
2025-07-18 09:09:16,284 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2025-07-18 09:09:16,285 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'nowcast'], host='arbutus.cloud-nowcast')
2025-07-18 09:09:18,202 INFO [watch_NEMO] running in process 7450
2025-07-18 09:09:18,203 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-07-18 09:09:18,204 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2025-07-18 09:09:18,205 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2025-07-18 09:09:18,270 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2025-07-18 09:14:18,271 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 18580800 = 2025-07-18 05:19:20 UTC, 22.2% complete
2025-07-18 09:19:18,273 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 18581334 = 2025-07-18 11:15:20 UTC, 46.9% complete
2025-07-18 09:24:18,279 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 18581861 = 2025-07-18 17:06:40 UTC, 71.3% complete
2025-07-18 09:29:18,276 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 18582366 = 2025-07-18 22:43:20 UTC, 94.7% complete
2025-07-18 09:34:18,279 INFO [watch_NEMO] nowcast NEMO run on arbutus.cloud-nowcast completed
2025-07-18 09:34:18,456 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2025-07-18 09:34:19,003 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'ssh', '--run-date', '2025-07-18'], host='localhost')
2025-07-18 09:34:19,038 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2025-07-18'], host='localhost')
2025-07-18 09:34:26,624 INFO [make_forcing_links] running in process 1650535
2025-07-18 09:34:26,625 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-18 09:34:26,626 INFO [make_forcing_links] publishing log messages to tcp://*:5563
2025-07-18 09:34:26,627 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2025-07-18 09:34:26,630 INFO [download_results] running in process 1650536
2025-07-18 09:34:26,632 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-18 09:34:26,634 INFO [download_results] publishing log messages to tcp://*:5564
2025-07-18 09:34:26,634 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2025-07-18 09:34:26,635 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/nowcast/18jul25 to /results/SalishSea/nowcast-blue.202111
2025-07-18 09:34:27,719 INFO [make_forcing_links] ssh 2025-07-18 forcing file links on arbutus.cloud-nowcast created
2025-07-18 09:34:27,722 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2025-07-18 09:34:28,177 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2025-07-18'], host='arbutus.cloud-nowcast')
2025-07-18 09:34:30,454 INFO [run_NEMO] running in process 8069
2025-07-18 09:34:30,477 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-07-18 09:34:30,496 INFO [run_NEMO] publishing log messages to tcp://*:5556
2025-07-18 09:34:30,510 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2025-07-18 09:34:31,320 WARNING [run_NEMO] There are uncommitted changes in /nemoShare/MEOPAR/nowcast-sys/SS-run-sets
2025-07-18 09:34:31,521 INFO [run_NEMO] forecast: launching /nemoShare/MEOPAR/nowcast-sys/runs/18jul25forecast_2025-07-18T093430.527022-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2025-07-18 09:34:31,543 INFO [run_NEMO] forecast NEMO run for 2025-07-18 on arbutus.cloud-nowcast started
2025-07-18 09:34:31,615 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2025-07-18 09:34:31,985 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'forecast'], host='arbutus.cloud-nowcast')
2025-07-18 09:34:34,183 INFO [watch_NEMO] running in process 8345
2025-07-18 09:34:34,185 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-07-18 09:34:34,186 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2025-07-18 09:34:34,187 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2025-07-18 09:34:34,255 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2025-07-18 09:36:43,403 INFO [download_results] nowcast 2025-07-18 results files from arbutus.cloud-nowcast downloaded
2025-07-18 09:36:43,454 INFO [manager] checklist updated with [results files] items from download_results worker
2025-07-18 09:36:43,738 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast', 'research', '--run-date', '2025-07-18'], host='localhost')
2025-07-18 09:36:43,747 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast', 'comparison', '--run-date', '2025-07-17'], host='localhost')
2025-07-18 09:36:43,751 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['nowcast', '--run-date', '2025-07-18'], host='localhost')
2025-07-18 09:36:54,893 INFO [make_CHS_currents_file] running in process 1650576
2025-07-18 09:36:54,894 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-18 09:36:54,896 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5563
2025-07-18 09:36:54,897 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555
2025-07-18 09:37:01,743 INFO [make_plots] running in process 1650575
2025-07-18 09:37:01,744 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-18 09:37:01,745 INFO [make_plots] publishing log messages to tcp://*:5564
2025-07-18 09:37:01,746 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2025-07-18 09:37:01,765 INFO [make_plots] running in process 1650573
2025-07-18 09:37:01,798 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-18 09:37:01,820 INFO [make_plots] publishing log messages to tcp://*:5565
2025-07-18 09:37:01,838 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2025-07-18 09:37:02,709 INFO [make_plots] preparing render list for 2025-07-17 NEMO nowcast-blue comparison figures
2025-07-18 09:37:02,766 INFO [make_plots] preparing render list for 2025-07-18 NEMO nowcast-blue research figures
2025-07-18 09:37:02,826 INFO [make_plots] added SH_wind figure to 2025-07-17 NEMO nowcast-blue comparison render list
2025-07-18 09:37:02,827 INFO [make_plots] added Compare_VENUS_East figure to 2025-07-17 NEMO nowcast-blue comparison render list
2025-07-18 09:37:02,828 INFO [make_plots] added Compare_VENUS_Central figure to 2025-07-17 NEMO nowcast-blue comparison render list
2025-07-18 09:37:02,828 INFO [make_plots] starting to render nemo nowcast comparison 17jul25 figures
2025-07-18 09:37:05,523 INFO [make_plots] added salinity_thalweg_and_surface figures to 2025-07-18 NEMO nowcast-blue research render list
2025-07-18 09:37:07,790 INFO [make_plots] added temperature_thalweg_and_surface figures to 2025-07-18 NEMO nowcast-blue research render list
2025-07-18 09:37:07,793 INFO [make_plots] added Currents_sections_and_surface figure to 2025-07-18 NEMO nowcast-blue research render list
2025-07-18 09:37:07,795 INFO [make_plots] added Currents_at_VENUS_Central figure to 2025-07-18 NEMO nowcast-blue research render list
2025-07-18 09:37:07,796 INFO [make_plots] added Currents_at_VENUS_East figure to 2025-07-18 NEMO nowcast-blue research render list
2025-07-18 09:37:07,797 INFO [make_plots] starting to render nemo nowcast research 18jul25 figures
2025-07-18 09:37:56,665 INFO [make_CHS_currents_file] Made CHS currents file for 2025-07-18 for nowcast
2025-07-18 09:37:56,678 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker
2025-07-18 09:39:27,595 INFO [make_plots] finished rendering nemo nowcast research 18jul25 figures
2025-07-18 09:39:27,686 INFO [make_plots] nemo research plots for 2025-07-18 nowcast completed
2025-07-18 09:39:27,762 INFO [manager] checklist updated with [plots] items from make_plots worker
2025-07-18 09:39:34,256 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 18582944 = 2025-07-19 05:08:40 UTC, 14.3% complete
2025-07-18 09:39:43,747 INFO [make_plots] finished rendering nemo nowcast comparison 17jul25 figures
2025-07-18 09:39:43,747 INFO [make_plots] nemo comparison plots for 2025-07-17 nowcast completed
2025-07-18 09:39:43,751 INFO [manager] checklist updated with [plots] items from make_plots worker
2025-07-18 09:44:34,257 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 18583441 = 2025-07-19 10:40:00 UTC, 29.6% complete
2025-07-18 09:49:34,263 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 18583935 = 2025-07-19 16:09:20 UTC, 44.9% complete
2025-07-18 09:54:34,259 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 18584436 = 2025-07-19 21:43:20 UTC, 60.4% complete
2025-07-18 09:59:34,261 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 18584938 = 2025-07-20 03:18:00 UTC, 75.9% complete
2025-07-18 10:04:34,262 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 18585434 = 2025-07-20 08:48:40 UTC, 91.2% complete
2025-07-18 10:09:34,266 INFO [watch_NEMO] forecast NEMO run on arbutus.cloud-nowcast completed
2025-07-18 10:09:34,274 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2025-07-18 10:09:35,232 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_wind_file', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2025-07-18'], host='arbutus.cloud-nowcast')
2025-07-18 10:09:35,245 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_current_file', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2025-07-18'], host='arbutus.cloud-nowcast')
2025-07-18 10:09:35,250 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'turbidity', '--run-date', '2025-07-18'], host='localhost')
2025-07-18 10:09:35,300 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'turbidity', '--run-date', '2025-07-18'], host='localhost')
2025-07-18 10:09:35,345 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.graham', 'turbidity', '--run-date', '2025-07-18'], host='localhost')
2025-07-18 10:09:35,453 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'turbidity', '--run-date', '2025-07-18'], host='localhost')
2025-07-18 10:09:35,497 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2025-07-18'], host='localhost')
2025-07-18 10:09:38,318 INFO [make_ww3_current_file] running in process 9448
2025-07-18 10:09:38,319 INFO [make_ww3_current_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-07-18 10:09:38,325 INFO [make_ww3_current_file] publishing log messages to tcp://*:5571
2025-07-18 10:09:38,338 INFO [make_ww3_wind_file] running in process 9447
2025-07-18 10:09:38,340 INFO [make_ww3_wind_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-07-18 10:09:38,343 INFO [make_ww3_wind_file] publishing log messages to tcp://*:5570
2025-07-18 10:09:38,381 INFO [make_ww3_wind_file] connected to skookum.eos.ubc.ca port 5555
2025-07-18 10:09:38,387 INFO [make_ww3_current_file] connected to skookum.eos.ubc.ca port 5555
2025-07-18 10:09:38,388 INFO [make_ww3_wind_file] Creating wwatch3 wind forcing file for 2025-07-18 forecast run
2025-07-18 10:09:38,390 INFO [make_ww3_current_file] Creating wwatch3 currents forcing file for 2025-07-18 forecast run
2025-07-18 10:09:45,779 INFO [upload_forcing] running in process 1651020
2025-07-18 10:09:45,779 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-18 10:09:45,809 INFO [upload_forcing] publishing log messages to tcp://*:5563
2025-07-18 10:09:45,815 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-07-18 10:09:45,816 INFO [upload_forcing] running in process 1651019
2025-07-18 10:09:45,816 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-18 10:09:45,818 INFO [upload_forcing] publishing log messages to tcp://*:5564
2025-07-18 10:09:45,819 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-07-18 10:09:45,886 INFO [upload_forcing] running in process 1651021
2025-07-18 10:09:45,909 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-18 10:09:45,969 INFO [upload_forcing] publishing log messages to tcp://*:5565
2025-07-18 10:09:46,052 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-07-18 10:09:46,181 INFO [upload_forcing] running in process 1651022
2025-07-18 10:09:46,285 INFO [download_results] running in process 1651023
2025-07-18 10:09:46,341 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-18 10:09:46,408 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-18 10:09:46,444 INFO [upload_forcing] publishing log messages to tcp://*:5566
2025-07-18 10:09:46,486 INFO [download_results] publishing log messages to tcp://*:5567
2025-07-18 10:09:46,493 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-07-18 10:09:46,498 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2025-07-18 10:09:46,499 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/forecast/18jul25 to /results/SalishSea/forecast.202111
2025-07-18 10:09:46,964 INFO [upload_forcing] turbidity 2025-07-18 forcing files upload to optimum-hindcast completed
2025-07-18 10:09:47,001 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-07-18 10:09:48,333 INFO [upload_forcing] turbidity 2025-07-18 forcing files upload to arbutus.cloud-nowcast completed
2025-07-18 10:09:48,427 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-07-18 10:09:48,761 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2025-07-18'], host='localhost')
2025-07-18 10:09:50,298 INFO [make_ww3_wind_file] wwatch3 wind forcing file created on arbutus.cloud-nowcast for 2025-07-18 forecast run
2025-07-18 10:09:50,677 INFO [manager] checklist updated with [WW3 wind forcing] items from make_ww3_wind_file worker
2025-07-18 10:09:51,297 INFO [upload_forcing] turbidity 2025-07-18 forcing files upload to robot.graham completed
2025-07-18 10:09:51,676 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-07-18 10:10:05,260 INFO [make_forcing_links] running in process 1651056
2025-07-18 10:10:05,262 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-18 10:10:05,315 INFO [make_forcing_links] publishing log messages to tcp://*:5564
2025-07-18 10:10:05,338 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2025-07-18 10:10:07,279 INFO [make_forcing_links] nowcast-green 2025-07-18 forcing file links on arbutus.cloud-nowcast created
2025-07-18 10:10:07,347 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2025-07-18 10:10:07,348 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2025-07-18'], host='arbutus.cloud-nowcast')
2025-07-18 10:10:09,649 INFO [run_NEMO] running in process 10087
2025-07-18 10:10:09,761 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-07-18 10:10:09,893 INFO [run_NEMO] publishing log messages to tcp://*:5556
2025-07-18 10:10:10,026 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2025-07-18 10:10:10,589 WARNING [run_NEMO] There are uncommitted changes in /nemoShare/MEOPAR/nowcast-sys/SS-run-sets
2025-07-18 10:10:11,021 INFO [run_NEMO] nowcast-green: launching /nemoShare/MEOPAR/nowcast-sys/runs/18jul25nowcast-green_2025-07-18T101009.690380-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2025-07-18 10:10:11,026 INFO [run_NEMO] nowcast-green NEMO run for 2025-07-18 on arbutus.cloud-nowcast started
2025-07-18 10:10:11,030 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2025-07-18 10:10:11,302 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'nowcast-green'], host='arbutus.cloud-nowcast')
2025-07-18 10:10:13,608 INFO [watch_NEMO] running in process 10366
2025-07-18 10:10:13,675 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-07-18 10:10:13,757 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2025-07-18 10:10:13,805 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2025-07-18 10:10:13,833 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2025-07-18 10:11:27,139 INFO [upload_forcing] turbidity 2025-07-18 forcing files upload to orcinus-nowcast-agrif completed
2025-07-18 10:11:27,312 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-07-18 10:11:27,527 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['orcinus-nowcast-agrif', 'nowcast-agrif', '--run-date', '2025-07-18'], host='localhost')
2025-07-18 10:11:27,961 INFO [download_results] forecast 2025-07-18 results files from arbutus.cloud-nowcast downloaded
2025-07-18 10:11:28,029 INFO [manager] checklist updated with [results files] items from download_results worker
2025-07-18 10:11:28,403 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['forecast', '--run-date', '2025-07-18'], host='localhost')
2025-07-18 10:11:37,981 INFO [make_forcing_links] running in process 1651078
2025-07-18 10:11:37,982 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-18 10:11:37,987 INFO [make_forcing_links] publishing log messages to tcp://*:5563
2025-07-18 10:11:37,989 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2025-07-18 10:11:39,026 INFO [make_forcing_links] nowcast-agrif 2025-07-18 forcing file links on orcinus-nowcast-agrif created
2025-07-18 10:11:39,029 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2025-07-18 10:11:39,353 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_agrif', args=['orcinus-nowcast-agrif', 'nowcast-agrif', '--run-date', '2025-07-18'], host='localhost')
2025-07-18 10:11:46,185 INFO [run_NEMO_agrif] running in process 1651102
2025-07-18 10:11:46,186 INFO [run_NEMO_agrif] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-18 10:11:46,284 INFO [run_NEMO_agrif] publishing log messages to tcp://*:5563
2025-07-18 10:11:46,310 INFO [run_NEMO_agrif] connected to skookum.eos.ubc.ca port 5555
2025-07-18 10:11:46,576 INFO [make_CHS_currents_file] running in process 1651080
2025-07-18 10:11:46,577 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-18 10:11:46,578 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5564
2025-07-18 10:11:46,579 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555
2025-07-18 10:11:47,450 CRITICAL [run_NEMO_agrif] unhandled exception:
Traceback (most recent call last):
  File "/SalishSeaCast/NEMO_Nowcast/nemo_nowcast/worker.py", line 387, in _do_work
    checklist = self.worker_func(
        self._parsed_args, self.config, self.tell_manager
    )
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/run_NEMO_agrif.py", line 109, in run_NEMO_agrif
    prev_run_namelists_info = _get_prev_run_namelists_info(
        sftp_client, host_name, run_date.shift(days=-1), config
    )
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/run_NEMO_agrif.py", line 155, in _get_prev_run_namelists_info
    sftp_client.get(os.fspath(prev_namelist_cfg), namelist_cfg.name)
    ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/paramiko/sftp_client.py", line 840, in get
    size = self.getfo(
        remotepath,
    ...<3 lines>...
        max_concurrent_prefetch_requests,
    )
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/paramiko/sftp_client.py", line 795, in getfo
    file_size = self.stat(remotepath).st_size
                ~~~~~~~~~^^^^^^^^^^^^
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/paramiko/sftp_client.py", line 493, in stat
    t, msg = self._request(CMD_STAT, path)
             ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/paramiko/sftp_client.py", line 857, in _request
    return self._read_response(num)
           ~~~~~~~~~~~~~~~~~~~^^^^^
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/paramiko/sftp_client.py", line 909, in _read_response
    self._convert_status(msg)
    ~~~~~~~~~~~~~~~~~~~~^^^^^
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/paramiko/sftp_client.py", line 938, in _convert_status
    raise IOError(errno.ENOENT, text)
FileNotFoundError: [Errno 2] No such file
2025-07-18 10:12:17,487 INFO [make_ww3_current_file] wwatch3 currents forcing file created on arbutus.cloud-nowcast for 2025-07-18 forecast run
2025-07-18 10:12:17,687 INFO [manager] checklist updated with [WW3 currents forcing] items from make_ww3_current_file worker
2025-07-18 10:12:18,173 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2025-07-18'], host='arbutus.cloud-nowcast')
2025-07-18 10:12:20,260 INFO [run_ww3] running in process 10476
2025-07-18 10:12:20,334 INFO [run_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-07-18 10:12:20,346 INFO [run_ww3] publishing log messages to tcp://*:5572
2025-07-18 10:12:20,347 INFO [run_ww3] connected to skookum.eos.ubc.ca port 5555
2025-07-18 10:12:20,350 INFO [run_ww3] Created run directory /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/nowcast_2025-07-18T101220.256642-0700
2025-07-18 10:12:20,351 INFO [run_ww3] nowcast: launching /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/nowcast_2025-07-18T101220.256642-0700/SoGWW3.sh on arbutus.cloud-nowcast
2025-07-18 10:12:20,353 INFO [run_ww3] nowcast WaveWatch3 run for 2025-07-18 on arbutus.cloud-nowcast started
2025-07-18 10:12:20,356 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker
2025-07-18 10:12:20,629 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'nowcast'], host='arbutus.cloud-nowcast')
2025-07-18 10:12:22,769 INFO [watch_ww3] running in process 10563
2025-07-18 10:12:22,770 INFO [watch_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-07-18 10:12:22,794 INFO [watch_ww3] publishing log messages to tcp://*:5573
2025-07-18 10:12:22,813 INFO [watch_ww3] connected to skookum.eos.ubc.ca port 5555
2025-07-18 10:12:22,990 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2025-07-18 10:13:13,800 INFO [make_CHS_currents_file] Made CHS currents file for 2025-07-18 for forecast
2025-07-18 10:13:13,877 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker
2025-07-18 10:13:14,333 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['nemo', 'forecast', '--run-date', '2025-07-18'], host='localhost')
2025-07-18 10:13:33,493 INFO [update_forecast_datasets] running in process 1651129
2025-07-18 10:13:33,494 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-18 10:13:33,512 INFO [update_forecast_datasets] publishing log messages to tcp://*:5563
2025-07-18 10:13:33,530 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2025-07-18 10:13:33,542 INFO [update_forecast_datasets] updating nemo forecast directory for forecast/18jul25 run
2025-07-18 10:13:34,392 INFO [update_forecast_datasets] updated nemo forecast directory for forecast/18jul25 run: /results/SalishSea/rolling-forecasts/nemo
2025-07-18 10:13:34,395 INFO [update_forecast_datasets] nemo 2025-07-18 forecast rolling forecast datasets updated
2025-07-18 10:13:34,411 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2025-07-18 10:13:34,695 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nemo-forecast'], host='localhost')
2025-07-18 10:13:34,703 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'forecast', 'publish', '--run-date', '2025-07-18'], host='localhost')
2025-07-18 10:13:34,720 INFO [manager] launching NextWorker(module='nowcast.workers.make_surface_current_tiles', args=['forecast', '--run-date', '2025-07-18'], host='localhost')
2025-07-18 10:13:41,304 INFO [ping_erddap] running in process 1651138
2025-07-18 10:13:41,304 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-18 10:13:41,308 INFO [ping_erddap] publishing log messages to tcp://*:5563
2025-07-18 10:13:41,320 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2025-07-18 10:13:41,365 INFO [ping_erddap] nemo-forecast ERDDAP dataset flag file(s) created
2025-07-18 10:13:41,400 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2025-07-18 10:13:48,315 INFO [make_surface_current_tiles] running in process 1651141
2025-07-18 10:13:48,317 INFO [make_surface_current_tiles] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-18 10:13:48,370 INFO [make_surface_current_tiles] publishing log messages to tcp://*:5563
2025-07-18 10:13:48,413 INFO [make_surface_current_tiles] connected to skookum.eos.ubc.ca port 5555
2025-07-18 10:13:48,452 INFO [make_surface_current_tiles] starting rendering of tiles for 2025-07-18 forecast in 6 processes into /results/nowcast-sys/figures/surface_currents/forecast/18jul25
2025-07-18 10:14:01,073 INFO [make_plots] running in process 1651140
2025-07-18 10:14:01,074 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-18 10:14:01,131 INFO [make_plots] publishing log messages to tcp://*:5564
2025-07-18 10:14:01,193 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2025-07-18 10:14:02,143 INFO [make_plots] preparing render list for 2025-07-18 NEMO forecast publish figures
2025-07-18 10:14:02,296 INFO [make_plots] added Website_thumbnail figure to 2025-07-18 NEMO forecast publish render list
2025-07-18 10:14:02,368 INFO [make_plots] added Threshold_website figure to 2025-07-18 NEMO forecast publish render list
2025-07-18 10:14:02,413 INFO [make_plots] added PA_tidal_predictions figure to 2025-07-18 NEMO forecast publish render list
2025-07-18 10:14:02,499 INFO [make_plots] added Boundary Bay figure to 2025-07-18 NEMO forecast publish render list
2025-07-18 10:14:02,561 INFO [make_plots] added Campbell River figure to 2025-07-18 NEMO forecast publish render list
2025-07-18 10:14:02,592 INFO [make_plots] added Cherry Point figure to 2025-07-18 NEMO forecast publish render list
2025-07-18 10:14:02,593 INFO [make_plots] added Friday Harbor figure to 2025-07-18 NEMO forecast publish render list
2025-07-18 10:14:02,595 INFO [make_plots] added Halfmoon Bay figure to 2025-07-18 NEMO forecast publish render list
2025-07-18 10:14:02,597 INFO [make_plots] added Nanaimo figure to 2025-07-18 NEMO forecast publish render list
2025-07-18 10:14:02,599 INFO [make_plots] added Neah Bay figure to 2025-07-18 NEMO forecast publish render list
2025-07-18 10:14:02,603 INFO [make_plots] added New Westminster figure to 2025-07-18 NEMO forecast publish render list
2025-07-18 10:14:02,605 INFO [make_plots] added Patricia Bay figure to 2025-07-18 NEMO forecast publish render list
2025-07-18 10:14:02,609 INFO [make_plots] added Point Atkinson figure to 2025-07-18 NEMO forecast publish render list
2025-07-18 10:14:02,610 INFO [make_plots] added Port Renfrew figure to 2025-07-18 NEMO forecast publish render list
2025-07-18 10:14:02,618 INFO [make_plots] added Sand Heads figure to 2025-07-18 NEMO forecast publish render list
2025-07-18 10:14:02,620 INFO [make_plots] added Sandy Cove figure to 2025-07-18 NEMO forecast publish render list
2025-07-18 10:14:02,621 INFO [make_plots] added Squamish figure to 2025-07-18 NEMO forecast publish render list
2025-07-18 10:14:02,622 INFO [make_plots] added Victoria figure to 2025-07-18 NEMO forecast publish render list
2025-07-18 10:14:02,623 INFO [make_plots] added Woodwards Landing figure to 2025-07-18 NEMO forecast publish render list
2025-07-18 10:14:02,625 INFO [make_plots] starting to render nemo forecast publish 18jul25 figures
2025-07-18 10:15:13,702 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18580485 = 2025-07-18 01:49:20 UTC, 7.6% complete
2025-07-18 10:17:22,809 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 99, 22.9% complete
2025-07-18 10:20:13,710 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18580673 = 2025-07-18 03:54:40 UTC, 16.3% complete
2025-07-18 10:22:22,813 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 201, 46.5% complete
2025-07-18 10:25:13,709 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18580857 = 2025-07-18 05:57:20 UTC, 24.8% complete
2025-07-18 10:27:22,814 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 309, 71.5% complete
2025-07-18 10:29:10,609 WARNING [make_plots] invalid station number for Boundary Bay station: None
2025-07-18 10:29:10,631 WARNING [make_plots] can't resolve a valid CHS station code for Boundary Bay
2025-07-18 10:29:54,406 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb996/data for station 08074 Campbell River from 2025-07-13 00:00:00Z to 2025-07-20 11:55:00Z
2025-07-18 10:30:13,710 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18581045 = 2025-07-18 08:02:40 UTC, 33.5% complete
2025-07-18 10:31:48,946 WARNING [make_plots] invalid station number for Halfmoon Bay station: None
2025-07-18 10:31:48,991 WARNING [make_plots] can't resolve a valid CHS station code for Halfmoon Bay
2025-07-18 10:32:22,811 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 417, 96.5% complete
2025-07-18 10:32:25,897 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb96d/data for station 07917 Nanaimo from 2025-07-13 00:00:00Z to 2025-07-20 11:55:00Z
2025-07-18 10:33:14,853 INFO [make_surface_current_tiles] starting PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast/18jul25
2025-07-18 10:33:36,787 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e13d0f4a073c4bbf93/data for station 07654 New Westminster from 2025-07-13 00:00:00Z to 2025-07-20 11:55:00Z
2025-07-18 10:34:16,563 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd2d/data for station 07277 Patricia Bay from 2025-07-13 00:00:00Z to 2025-07-20 11:55:00Z
2025-07-18 10:34:52,559 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94c/data for station 07795 Point Atkinson from 2025-07-13 00:00:00Z to 2025-07-20 11:55:00Z
2025-07-18 10:35:13,709 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18581230 = 2025-07-18 10:06:00 UTC, 42.1% complete
2025-07-18 10:35:28,820 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e23d0f4a073c4bc060/data for station 08525 Port Renfrew from 2025-07-13 00:00:00Z to 2025-07-20 11:55:00Z
2025-07-18 10:36:02,658 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb93c/data for station 07594 Sand Heads from 2025-07-13 00:00:00Z to 2025-07-20 11:55:00Z
2025-07-18 10:36:39,261 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc43b/data for station 07786 Sandy Cove from 2025-07-13 00:00:00Z to 2025-07-20 11:55:00Z
2025-07-18 10:37:17,728 WARNING [make_plots] invalid station number for Squamish station: None
2025-07-18 10:37:17,737 WARNING [make_plots] can't resolve a valid CHS station code for Squamish
2025-07-18 10:37:22,815 INFO [watch_ww3] nowcast WWATCH3 run on arbutus.cloud-nowcast completed
2025-07-18 10:37:22,843 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2025-07-18 10:37:24,172 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2025-07-18'], host='localhost')
2025-07-18 10:37:24,174 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2025-07-18'], host='arbutus.cloud-nowcast')
2025-07-18 10:37:26,161 INFO [run_ww3] running in process 10954
2025-07-18 10:37:26,180 INFO [run_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-07-18 10:37:26,199 INFO [run_ww3] publishing log messages to tcp://*:5572
2025-07-18 10:37:26,212 INFO [run_ww3] connected to skookum.eos.ubc.ca port 5555
2025-07-18 10:37:26,218 INFO [run_ww3] Created run directory /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast_2025-07-18T103726.155632-0700
2025-07-18 10:37:26,218 INFO [run_ww3] forecast: launching /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast_2025-07-18T103726.155632-0700/SoGWW3.sh on arbutus.cloud-nowcast
2025-07-18 10:37:26,220 INFO [run_ww3] forecast WaveWatch3 run for 2025-07-18 on arbutus.cloud-nowcast started
2025-07-18 10:37:26,221 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker
2025-07-18 10:37:26,549 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'forecast'], host='arbutus.cloud-nowcast')
2025-07-18 10:37:28,957 INFO [watch_ww3] running in process 11041
2025-07-18 10:37:29,002 INFO [watch_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-07-18 10:37:29,077 INFO [watch_ww3] publishing log messages to tcp://*:5573
2025-07-18 10:37:29,145 INFO [watch_ww3] connected to skookum.eos.ubc.ca port 5555
2025-07-18 10:37:29,190 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2025-07-18 10:37:34,202 INFO [download_wwatch3_results] running in process 1653056
2025-07-18 10:37:34,204 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-18 10:37:34,210 INFO [download_wwatch3_results] publishing log messages to tcp://*:5565
2025-07-18 10:37:34,212 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2025-07-18 10:37:53,458 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd1e/data for station 07120 Victoria from 2025-07-13 00:00:00Z to 2025-07-20 11:55:00Z
2025-07-18 10:38:09,705 INFO [download_wwatch3_results] nowcast 2025-07-18 results files from arbutus.cloud-nowcast downloaded
2025-07-18 10:38:09,713 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2025-07-18 10:38:32,597 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc42d/data for station 07610 Woodwards Landing from 2025-07-13 00:00:00Z to 2025-07-20 11:55:00Z
2025-07-18 10:39:13,366 INFO [make_plots] finished rendering nemo forecast publish 18jul25 figures
2025-07-18 10:39:13,371 INFO [make_plots] nemo publish plots for 2025-07-18 forecast completed
2025-07-18 10:39:13,559 INFO [manager] checklist updated with [plots] items from make_plots worker
2025-07-18 10:39:13,901 INFO [manager] launching NextWorker(module='nowcast.workers.make_feeds', args=['forecast', '--run-date', '2025-07-18'], host='localhost')
2025-07-18 10:39:44,557 INFO [make_feeds] running in process 1653082
2025-07-18 10:39:44,557 INFO [make_feeds] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-18 10:39:44,557 INFO [make_feeds] publishing log messages to tcp://*:5564
2025-07-18 10:39:44,558 INFO [make_feeds] connected to skookum.eos.ubc.ca port 5555
2025-07-18 10:40:13,709 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18581413 = 2025-07-18 12:08:00 UTC, 50.6% complete
2025-07-18 10:40:15,944 INFO [make_feeds] ATOM feeds for 2025-07-18 forecast run completed
2025-07-18 10:40:15,997 INFO [manager] checklist updated with [storm surge alert feeds] items from make_feeds worker
2025-07-18 10:42:28,997 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 93, 14.4% complete
2025-07-18 10:42:47,422 INFO [make_surface_current_tiles] finished PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast/18jul25
2025-07-18 10:42:47,468 INFO [make_surface_current_tiles] finished rendering of tiles for 2025-07-18 forecast into /results/nowcast-sys/figures/surface_currents/forecast/18jul25
2025-07-18 10:42:47,469 INFO [make_surface_current_tiles] surface current tile figures for 2025-07-18 forecast completed
2025-07-18 10:42:48,036 INFO [manager] checklist updated with [surface current tiles] items from make_surface_current_tiles worker
2025-07-18 10:45:13,706 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18581597 = 2025-07-18 14:10:40 UTC, 59.1% complete
2025-07-18 10:47:28,997 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 201, 31.0% complete
2025-07-18 10:50:13,706 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18581789 = 2025-07-18 16:18:40 UTC, 68.0% complete
2025-07-18 10:52:28,998 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 309, 47.7% complete
2025-07-18 10:55:13,707 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18581975 = 2025-07-18 18:22:40 UTC, 76.6% complete
2025-07-18 10:57:28,999 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 417, 64.4% complete
2025-07-18 11:00:13,709 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18582164 = 2025-07-18 20:28:40 UTC, 85.4% complete
2025-07-18 11:02:29,000 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 525, 81.0% complete
2025-07-18 11:05:13,710 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18582347 = 2025-07-18 22:30:40 UTC, 93.8% complete
2025-07-18 11:07:29,002 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 633, 97.7% complete
2025-07-18 11:10:13,712 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18582480 = 2025-07-18 23:59:20 UTC, 100.0% complete
2025-07-18 11:12:29,004 INFO [watch_ww3] forecast WWATCH3 run on arbutus.cloud-nowcast completed
2025-07-18 11:12:29,029 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2025-07-18 11:12:30,068 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2025-07-18'], host='localhost')
2025-07-18 11:12:36,185 INFO [download_wwatch3_results] running in process 1653446
2025-07-18 11:12:36,186 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-18 11:12:36,204 INFO [download_wwatch3_results] publishing log messages to tcp://*:5563
2025-07-18 11:12:36,221 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2025-07-18 11:13:18,272 INFO [download_wwatch3_results] forecast 2025-07-18 results files from arbutus.cloud-nowcast downloaded
2025-07-18 11:13:18,472 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2025-07-18 11:13:19,015 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['wwatch3', 'forecast', '--run-date', '2025-07-18'], host='localhost')
2025-07-18 11:13:25,376 INFO [update_forecast_datasets] running in process 1653455
2025-07-18 11:13:25,406 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-18 11:13:25,436 INFO [update_forecast_datasets] publishing log messages to tcp://*:5563
2025-07-18 11:13:25,463 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2025-07-18 11:13:25,481 INFO [update_forecast_datasets] updating wwatch3 most_recent_forecast directory from forecast/18jul25 run
2025-07-18 11:13:25,794 INFO [update_forecast_datasets] updated wwatch3 most_recent_forecast directory from forecast/18jul25 run
2025-07-18 11:13:25,795 INFO [update_forecast_datasets] updating wwatch3 forecast directory for forecast/18jul25 run
2025-07-18 11:13:26,183 INFO [update_forecast_datasets] updated wwatch3 forecast directory for forecast/18jul25 run: /results/SalishSea/rolling-forecasts/wwatch3
2025-07-18 11:13:26,235 INFO [update_forecast_datasets] wwatch3 2025-07-18 forecast rolling forecast datasets updated
2025-07-18 11:13:26,279 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2025-07-18 11:13:26,869 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['wwatch3-forecast'], host='localhost')
2025-07-18 11:13:33,395 INFO [ping_erddap] running in process 1653463
2025-07-18 11:13:33,509 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-18 11:13:33,590 INFO [ping_erddap] publishing log messages to tcp://*:5563
2025-07-18 11:13:33,696 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2025-07-18 11:13:33,846 INFO [ping_erddap] wwatch3-forecast ERDDAP dataset flag file(s) created
2025-07-18 11:13:33,939 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2025-07-18 11:13:34,008 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['wwatch3', 'forecast', 'publish', '--run-date', '2025-07-18'], host='localhost')
2025-07-18 11:13:53,451 INFO [make_plots] running in process 1653469
2025-07-18 11:13:53,452 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-18 11:13:53,453 INFO [make_plots] publishing log messages to tcp://*:5563
2025-07-18 11:13:53,454 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2025-07-18 11:13:53,455 INFO [make_plots] preparing render list for 2025-07-18 WaveWatch3 forecast publish figures
2025-07-18 11:13:53,456 INFO [make_plots] added Halibut Bank figure to 2025-07-18 WaveWatch3 forecast publish render list
2025-07-18 11:13:53,457 INFO [make_plots] added Sentry Shoal figure to 2025-07-18 WaveWatch3 forecast publish render list
2025-07-18 11:13:53,458 INFO [make_plots] starting to render wwatch3 forecast publish 18jul25 figures
2025-07-18 11:13:55,324 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46146 Halibut Bank
2025-07-18 11:14:18,352 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46131 Sentry Shoal
2025-07-18 11:14:27,840 INFO [make_plots] finished rendering wwatch3 forecast publish 18jul25 figures
2025-07-18 11:14:27,880 INFO [make_plots] wwatch3 publish plots for 2025-07-18 forecast completed
2025-07-18 11:14:27,972 INFO [manager] checklist updated with [plots] items from make_plots worker
2025-07-18 11:15:13,720 INFO [watch_NEMO] nowcast-green NEMO run on arbutus.cloud-nowcast completed
2025-07-18 11:15:13,723 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2025-07-18 11:15:14,461 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2025-07-18'], host='localhost')
2025-07-18 11:15:18,634 INFO [download_results] running in process 1653502
2025-07-18 11:15:18,635 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-18 11:15:18,636 INFO [download_results] publishing log messages to tcp://*:5563
2025-07-18 11:15:18,637 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2025-07-18 11:15:18,638 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/nowcast-green/18jul25 to /results2/SalishSea/nowcast-green.202111
2025-07-18 11:18:50,650 INFO [download_results] nowcast-green 2025-07-18 results files from arbutus.cloud-nowcast downloaded
2025-07-18 11:18:50,655 INFO [manager] checklist updated with [results files] items from download_results worker
2025-07-18 11:18:51,204 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast-green', 'research', '--run-date', '2025-07-18'], host='localhost')
2025-07-18 11:18:51,214 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nowcast-green'], host='localhost')
2025-07-18 11:18:51,252 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'physics', '--run-date', '2025-07-18'], host='localhost')
2025-07-18 11:18:51,277 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'biology', '--run-date', '2025-07-18'], host='localhost')
2025-07-18 11:18:51,292 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'chemistry', '--run-date', '2025-07-18'], host='localhost')
2025-07-18 11:18:57,558 INFO [ping_erddap] running in process 1653572
2025-07-18 11:18:57,559 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-18 11:18:57,572 INFO [ping_erddap] publishing log messages to tcp://*:5563
2025-07-18 11:18:57,588 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2025-07-18 11:18:57,593 INFO [ping_erddap] nowcast-green ERDDAP dataset flag file(s) created
2025-07-18 11:18:57,599 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2025-07-18 11:19:05,613 INFO [make_averaged_dataset] running in process 1653574
2025-07-18 11:19:05,617 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-18 11:19:05,619 INFO [make_averaged_dataset] publishing log messages to tcp://*:5563
2025-07-18 11:19:05,621 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555
2025-07-18 11:19:05,623 INFO [make_averaged_dataset] creating day-averaged dataset for 18-Jul-2025 biology
2025-07-18 11:19:05,629 INFO [make_averaged_dataset] running in process 1653573
2025-07-18 11:19:05,632 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-18 11:19:05,633 INFO [make_averaged_dataset] publishing log messages to tcp://*:5564
2025-07-18 11:19:05,634 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555
2025-07-18 11:19:05,635 INFO [make_averaged_dataset] creating day-averaged dataset for 18-Jul-2025 physics
2025-07-18 11:19:05,691 INFO [make_averaged_dataset] running in process 1653575
2025-07-18 11:19:05,694 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-18 11:19:05,695 INFO [make_averaged_dataset] publishing log messages to tcp://*:5565
2025-07-18 11:19:05,705 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555
2025-07-18 11:19:05,708 INFO [make_averaged_dataset] creating day-averaged dataset for 18-Jul-2025 chemistry
2025-07-18 11:19:10,425 INFO [make_plots] running in process 1653570
2025-07-18 11:19:10,429 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-18 11:19:10,437 INFO [make_plots] publishing log messages to tcp://*:5566
2025-07-18 11:19:10,445 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2025-07-18 11:19:11,131 INFO [make_plots] preparing render list for 2025-07-18 NEMO nowcast-green research figures
2025-07-18 11:19:23,152 INFO [make_plots] added temperature_thalweg_and_surface figures to 2025-07-18 NEMO nowcast-green research render list
2025-07-18 11:19:24,546 INFO [make_plots] added nitrate_thalweg_and_surface figures to 2025-07-18 NEMO nowcast-green research render list
2025-07-18 11:19:25,852 INFO [make_plots] added ammonium_thalweg_and_surface figures to 2025-07-18 NEMO nowcast-green research render list
2025-07-18 11:19:27,224 INFO [make_plots] added silicon_thalweg_and_surface figures to 2025-07-18 NEMO nowcast-green research render list
2025-07-18 11:19:28,621 INFO [make_plots] added dissolved_organic_nitrogen_thalweg_and_surface figures to 2025-07-18 NEMO nowcast-green research render list
2025-07-18 11:19:30,124 INFO [make_plots] added particulate_organic_nitrogen_thalweg_and_surface figures to 2025-07-18 NEMO nowcast-green research render list
2025-07-18 11:19:31,669 INFO [make_plots] added biogenic_silicon_thalweg_and_surface figures to 2025-07-18 NEMO nowcast-green research render list
2025-07-18 11:19:33,434 INFO [make_plots] added diatoms_thalweg_and_surface figures to 2025-07-18 NEMO nowcast-green research render list
2025-07-18 11:19:34,968 INFO [make_plots] added flagellates_thalweg_and_surface figures to 2025-07-18 NEMO nowcast-green research render list
2025-07-18 11:19:36,534 INFO [make_plots] added microzooplankton_thalweg_and_surface figures to 2025-07-18 NEMO nowcast-green research render list
2025-07-18 11:19:38,238 INFO [make_plots] added mesozooplankton_thalweg_and_surface figures to 2025-07-18 NEMO nowcast-green research render list
2025-07-18 11:19:39,680 INFO [make_plots] added turbidity_thalweg_and_surface figures to 2025-07-18 NEMO nowcast-green research render list
2025-07-18 11:19:42,122 INFO [make_plots] added temperature_salinity_timeseries figure to 2025-07-18 NEMO nowcast-green research render list
2025-07-18 11:19:42,147 INFO [make_plots] added nitrate_diatoms_timeseries figure to 2025-07-18 NEMO nowcast-green research render list
2025-07-18 11:19:42,148 INFO [make_plots] added diatoms_flagellates_timeseries figure to 2025-07-18 NEMO nowcast-green research render list
2025-07-18 11:19:42,150 INFO [make_plots] added z1_z2_zooplankton_timeseries figure to 2025-07-18 NEMO nowcast-green research render list
2025-07-18 11:19:42,152 INFO [make_plots] starting to render nemo nowcast-green research 18jul25 figures
2025-07-18 11:19:44,265 INFO [make_averaged_dataset] day-averaged dataset for 18-Jul-2025 physics created
2025-07-18 11:19:44,269 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker
2025-07-18 11:20:25,573 INFO [make_averaged_dataset] day-averaged dataset for 18-Jul-2025 chemistry created
2025-07-18 11:20:25,577 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker
2025-07-18 11:20:47,643 INFO [make_averaged_dataset] day-averaged dataset for 18-Jul-2025 biology created
2025-07-18 11:20:47,982 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker
2025-07-18 15:18:10,569 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/18/ to /results/forcing/atmospheric/continental2.5/GRIB/20250718/18/
2025-07-18 15:18:10,961 INFO [collect_weather] 2025-07-18 2.5km weather forecast 18 collection complete
2025-07-18 15:18:11,343 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2025-07-18 15:18:14,293 INFO [manager] launching NextWorker(module='nowcast.workers.download_weather', args=['00', '1km'], host='localhost')
2025-07-18 15:18:14,307 INFO [manager] launching NextWorker(module='nowcast.workers.download_weather', args=['12', '1km'], host='localhost')
2025-07-18 15:18:14,322 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['00', '2.5km'], host='localhost')
2025-07-18 15:18:14,372 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['00', '--fcst-date', '2025-07-19'], host='localhost')
2025-07-18 15:19:29,925 INFO [download_weather] running in process 1655819
2025-07-18 15:19:30,101 INFO [download_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-18 15:19:30,456 INFO [download_weather] publishing log messages to tcp://*:5562
2025-07-18 15:19:30,714 INFO [download_weather] connected to skookum.eos.ubc.ca port 5555
2025-07-18 15:19:31,081 INFO [download_weather] downloading 00 1 km forecast GRIB2 files for 20250718
2025-07-18 15:19:31,468 INFO [collect_weather] running in process 1655821
2025-07-18 15:19:31,668 INFO [download_weather] running in process 1655820
2025-07-18 15:19:31,800 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-18 15:19:31,897 INFO [download_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-18 15:19:32,006 INFO [collect_weather] publishing log messages to tcp://*:5563
2025-07-18 15:19:32,131 INFO [download_weather] publishing log messages to tcp://*:5564
2025-07-18 15:19:32,230 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2025-07-18 15:19:32,391 INFO [download_weather] connected to skookum.eos.ubc.ca port 5555
2025-07-18 15:19:32,766 INFO [download_weather] downloading 12 1 km forecast GRIB2 files for 20250718
2025-07-18 15:19:33,497 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/00/
2025-07-18 15:20:46,284 INFO [crop_gribs] running in process 1655822
2025-07-18 15:20:46,314 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-18 15:20:46,341 INFO [crop_gribs] publishing log messages to tcp://*:5565
2025-07-18 15:20:46,353 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2025-07-18 15:20:46,449 INFO [crop_gribs] cropping 2025-07-19 ECCC HRDPS 2.5km continental 00Z GRIB files to SalishSeaCast subdomain
2025-07-18 15:20:46,679 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20250719/00/
2025-07-18 15:23:58,349 INFO [download_weather] 2025-07-18 1km weather forecast 00 downloads complete
2025-07-18 15:23:58,564 INFO [manager] checklist updated with [weather forecast] items from download_weather worker
2025-07-18 15:24:07,857 INFO [download_weather] 2025-07-18 1km weather forecast 12 downloads complete
2025-07-18 15:24:08,001 INFO [manager] checklist updated with [weather forecast] items from download_weather worker
2025-07-18 15:37:35,723 CRITICAL [make_plots] unhandled exception:
Traceback (most recent call last):
  File "/SalishSeaCast/NEMO_Nowcast/nemo_nowcast/worker.py", line 387, in _do_work
    checklist = self.worker_func(
        self._parsed_args, self.config, self.tell_manager
    )
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 232, in make_plots
    checklist = _render_figures(
        config, model, run_type, plot_type, dmy, fig_functions, test_figure_id
    )
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 766, in _render_figures
    fig = _calc_figure(fig_func, args, kwargs)
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 833, in _calc_figure
    fig = fig_func(*args, **kwargs)
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/research/time_series_plots.py", line 80, in make_figure
    _plot_timeseries(axr, plot_data.right_var, right_variable, theme)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/research/time_series_plots.py", line 123, in _plot_timeseries
    ax.plot(plot_data.time, plot_data, color=theme.COLOURS["time series"][variable])
    ~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/matplotlib/axes/_axes.py", line 1777, in plot
    lines = [*self._get_lines(self, *args, data=data, **kwargs)]
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/matplotlib/axes/_base.py", line 297, in __call__
    yield from self._plot_args(
               ~~~~~~~~~~~~~~~^
        axes, this, kwargs, ambiguous_fmt_datakey=ambiguous_fmt_datakey,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        return_kwargs=return_kwargs
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/matplotlib/axes/_base.py", line 484, in _plot_args
    y = _check_1d(xy[1])
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/matplotlib/cbook.py", line 1351, in _check_1d
    x = _unpack_to_numpy(x)
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/matplotlib/cbook.py", line 2349, in _unpack_to_numpy
    return x.to_numpy()
           ~~~~~~~~~~^^
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/xarray/core/dataarray.py", line 831, in to_numpy
    return self.variable.to_numpy()
           ~~~~~~~~~~~~~~~~~~~~~~^^
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/xarray/namedarray/core.py", line 860, in to_numpy
    return to_numpy(self._data)
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/xarray/namedarray/pycompat.py", line 106, in to_numpy
    data = data.get_duck_array()  # type: ignore[no-untyped-call]
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/xarray/core/indexing.py", line 835, in get_duck_array
    self._ensure_cached()
    ~~~~~~~~~~~~~~~~~~~^^
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/xarray/core/indexing.py", line 832, in _ensure_cached
    self.array = as_indexable(self.array.get_duck_array())
                              ~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/xarray/core/indexing.py", line 789, in get_duck_array
    return self.array.get_duck_array()
           ~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/xarray/core/indexing.py", line 659, in get_duck_array
    array = array.get_duck_array()
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/xarray/coding/variables.py", line 81, in get_duck_array
    return self.func(self.array.get_duck_array())
                     ~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/xarray/core/indexing.py", line 652, in get_duck_array
    array = self.array[self.key]
            ~~~~~~~~~~^^^^^^^^^^
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/xarray/backends/netCDF4_.py", line 103, in __getitem__
    return indexing.explicit_indexing_adapter(
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        key, self.shape, indexing.IndexingSupport.OUTER, self._getitem
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/xarray/core/indexing.py", line 1013, in explicit_indexing_adapter
    result = raw_indexing_method(raw_key.tuple)
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/xarray/backends/netCDF4_.py", line 116, in _getitem
    array = getitem(original_array, key)
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/xarray/backends/common.py", line 256, in robust_getitem
    return array[key]
           ~~~~~^^^^^
  File "src/netCDF4/_netCDF4.pyx", line 5079, in netCDF4._netCDF4.Variable.__getitem__
  File "src/netCDF4/_netCDF4.pyx", line 6051, in netCDF4._netCDF4.Variable._get
  File "src/netCDF4/_netCDF4.pyx", line 2164, in netCDF4._netCDF4._ensure_nc_success
RuntimeError: NetCDF: Malformed or inaccessible DAP2 DATADDS or DAP4 DAP response
2025-07-18 17:08:23,761 INFO [crop_gribs] crop_gribs 18 has watched for 8h; retrying remaining 1 file(s)
2025-07-18 17:08:28,539 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20250718/18/
2025-07-18 17:08:28,546 INFO [crop_gribs] 2025-07-18 18 GRIBs cropping complete
2025-07-18 17:08:28,641 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2025-07-18 21:06:42,130 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/00/ to /results/forcing/atmospheric/continental2.5/GRIB/20250719/00/
2025-07-18 21:06:42,174 INFO [collect_weather] 2025-07-19 2.5km weather forecast 00 collection complete
2025-07-18 21:06:42,424 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2025-07-18 21:06:44,367 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['06', '2.5km'], host='localhost')
2025-07-18 21:06:44,479 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['06', '--fcst-date', '2025-07-19'], host='localhost')
2025-07-18 21:06:44,541 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20250719/00/
2025-07-18 21:06:44,557 INFO [crop_gribs] 2025-07-19 00 GRIBs cropping complete
2025-07-18 21:06:44,578 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2025-07-18 21:07:04,006 INFO [collect_weather] running in process 1658782
2025-07-18 21:07:04,006 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-18 21:07:04,008 INFO [collect_weather] publishing log messages to tcp://*:5562
2025-07-18 21:07:04,013 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2025-07-18 21:07:04,046 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/06/
2025-07-18 21:07:16,265 INFO [crop_gribs] running in process 1658784
2025-07-18 21:07:16,266 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-18 21:07:16,267 INFO [crop_gribs] publishing log messages to tcp://*:5563
2025-07-18 21:07:16,268 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2025-07-18 21:07:16,269 INFO [crop_gribs] cropping 2025-07-19 ECCC HRDPS 2.5km continental 06Z GRIB files to SalishSeaCast subdomain
2025-07-18 21:07:16,277 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20250719/06/
2025-07-19 03:16:33,599 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/06/ to /results/forcing/atmospheric/continental2.5/GRIB/20250719/06/
2025-07-19 03:16:33,636 INFO [collect_weather] 2025-07-19 2.5km weather forecast 06 collection complete
2025-07-19 03:16:33,747 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2025-07-19 03:16:35,402 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Capilano', '--data-date', '2025-07-18'], host='localhost')
2025-07-19 03:16:35,524 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'ChilliwackVedder', '--data-date', '2025-07-18'], host='localhost')
2025-07-19 03:16:35,570 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'ClowhomClowhomLake', '--data-date', '2025-07-18'], host='localhost')
2025-07-19 03:16:35,603 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Englishman', '--data-date', '2025-07-18'], host='localhost')
2025-07-19 03:16:35,624 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Fraser', '--data-date', '2025-07-18'], host='localhost')
2025-07-19 03:16:35,639 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'HomathkoMouth', '--data-date', '2025-07-18'], host='localhost')
2025-07-19 03:16:35,678 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'NicomeklLangley', '--data-date', '2025-07-18'], host='localhost')
2025-07-19 03:16:35,704 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'RobertsCreek', '--data-date', '2025-07-18'], host='localhost')
2025-07-19 03:16:35,711 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SalmonSayward', '--data-date', '2025-07-18'], host='localhost')
2025-07-19 03:16:35,754 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SanJuanPortRenfrew', '--data-date', '2025-07-18'], host='localhost')
2025-07-19 03:16:35,777 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SquamishBrackendale', '--data-date', '2025-07-18'], host='localhost')
2025-07-19 03:16:35,802 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaScotty', '--data-date', '2025-07-18'], host='localhost')
2025-07-19 03:16:35,815 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaBypass', '--data-date', '2025-07-18'], host='localhost')
2025-07-19 03:16:35,837 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaDiversion', '--data-date', '2025-07-18'], host='localhost')
2025-07-19 03:16:35,860 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ctd', args=['SCVIP'], host='localhost')
2025-07-19 03:16:35,877 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ctd', args=['SEVIP'], host='localhost')
2025-07-19 03:16:35,894 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ferry', args=['TWDP'], host='localhost')
2025-07-19 03:16:35,931 INFO [manager] launching NextWorker(module='nowcast.workers.get_vfpa_hadcp', args=['--data-date', '2025-07-18'], host='localhost')
2025-07-19 03:16:35,973 INFO [manager] launching NextWorker(module='nowcast.workers.collect_NeahBay_ssh', args=['00'], host='localhost')
2025-07-19 03:16:36,005 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['12', '2.5km'], host='localhost')
2025-07-19 03:16:36,038 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['12'], host='localhost')
2025-07-19 03:16:37,099 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20250719/06/
2025-07-19 03:16:37,154 INFO [crop_gribs] 2025-07-19 06 GRIBs cropping complete
2025-07-19 03:16:37,192 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2025-07-19 03:16:37,990 INFO [manager] launching NextWorker(module='nowcast.workers.grib_to_netcdf', args=['forecast2'], host='localhost')
2025-07-19 03:16:47,142 INFO [collect_NeahBay_ssh] running in process 1662251
2025-07-19 03:16:47,143 INFO [collect_NeahBay_ssh] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-19 03:16:47,144 INFO [collect_NeahBay_ssh] publishing log messages to tcp://*:5562
2025-07-19 03:16:47,154 INFO [collect_NeahBay_ssh] connected to skookum.eos.ubc.ca port 5555
2025-07-19 03:16:47,156 INFO [collect_NeahBay_ssh] collecting Neah Bay ssh 00Z obs/forecast for 2025-07-19
2025-07-19 03:16:47,171 INFO [collect_weather] running in process 1662252
2025-07-19 03:16:47,172 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-19 03:16:47,173 INFO [collect_weather] publishing log messages to tcp://*:5563
2025-07-19 03:16:47,175 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2025-07-19 03:16:47,365 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/12/
2025-07-19 03:16:48,513 INFO [collect_NeahBay_ssh] 2025-07-19T00:00:00-07:00 Neah Bay ssh 00Z obs/forecast data collection complete
2025-07-19 03:16:48,521 INFO [manager] checklist updated with [Neah Bay ssh data] items from collect_NeahBay_ssh worker
2025-07-19 03:16:49,422 INFO [manager] launching NextWorker(module='nowcast.workers.make_ssh_files', args=['forecast2', '--run-date', '2025-07-19'], host='localhost')
2025-07-19 03:16:51,545 INFO [collect_river_data] running in process 1662244
2025-07-19 03:16:51,606 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-19 03:16:51,647 INFO [collect_river_data] publishing log messages to tcp://*:5562
2025-07-19 03:16:51,665 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-07-19 03:16:51,671 INFO [collect_river_data] Collecting ECCC TheodosiaScotty river data for 2025-07-18
2025-07-19 03:16:51,732 INFO [collect_river_data] running in process 1662240
2025-07-19 03:16:51,767 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-19 03:16:51,796 INFO [collect_river_data] publishing log messages to tcp://*:5564
2025-07-19 03:16:51,836 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-07-19 03:16:51,866 INFO [collect_river_data] Collecting ECCC SalmonSayward river data for 2025-07-18
2025-07-19 03:16:51,929 INFO [collect_river_data] running in process 1662242
2025-07-19 03:16:51,981 INFO [collect_river_data] running in process 1662233
2025-07-19 03:16:51,988 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-19 03:16:51,996 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-19 03:16:52,003 INFO [collect_river_data] publishing log messages to tcp://*:5565
2025-07-19 03:16:52,010 INFO [collect_river_data] publishing log messages to tcp://*:5566
2025-07-19 03:16:52,014 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-07-19 03:16:52,016 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-07-19 03:16:52,022 INFO [collect_river_data] Collecting ECCC SanJuanPortRenfrew river data for 2025-07-18
2025-07-19 03:16:52,029 INFO [collect_river_data] Collecting ECCC ChilliwackVedder river data for 2025-07-18
2025-07-19 03:16:52,031 INFO [collect_river_data] running in process 1662239
2025-07-19 03:16:52,038 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-19 03:16:52,041 INFO [collect_river_data] publishing log messages to tcp://*:5567
2025-07-19 03:16:52,043 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-07-19 03:16:52,045 INFO [collect_river_data] Collecting ECCC RobertsCreek river data for 2025-07-18
2025-07-19 03:16:52,128 INFO [collect_river_data] Appended ECCC ChilliwackVedder river average discharge for 2025-07-18 to: /results/forcing/rivers/observations/Chilliwack_Vedder_flow
2025-07-19 03:16:52,129 INFO [collect_river_data] ECCC ChilliwackVedder river data collection for 2025-07-18 completed
2025-07-19 03:16:52,131 INFO [collect_river_data] Appended ECCC SalmonSayward river average discharge for 2025-07-18 to: /results/forcing/rivers/observations/Salmon_Sayward_flow
2025-07-19 03:16:52,132 INFO [collect_river_data] ECCC SalmonSayward river data collection for 2025-07-18 completed
2025-07-19 03:16:52,166 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-07-19 03:16:52,185 INFO [collect_river_data] Appended ECCC SanJuanPortRenfrew river average discharge for 2025-07-18 to: /results/forcing/rivers/observations/SanJuan_PortRenfrew_flow
2025-07-19 03:16:52,209 INFO [collect_river_data] ECCC SanJuanPortRenfrew river data collection for 2025-07-18 completed
2025-07-19 03:16:52,216 INFO [collect_river_data] Appended ECCC TheodosiaScotty river average discharge for 2025-07-18 to: /results/forcing/rivers/observations/Theodosia_Scotty_flow
2025-07-19 03:16:52,217 INFO [collect_river_data] Appended ECCC RobertsCreek river average discharge for 2025-07-18 to: /results/forcing/rivers/observations/RobertsCreek_flow
2025-07-19 03:16:52,219 INFO [collect_river_data] ECCC TheodosiaScotty river data collection for 2025-07-18 completed
2025-07-19 03:16:52,220 INFO [collect_river_data] ECCC RobertsCreek river data collection for 2025-07-18 completed
2025-07-19 03:16:52,469 INFO [collect_river_data] running in process 1662234
2025-07-19 03:16:52,497 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-19 03:16:52,500 INFO [collect_river_data] publishing log messages to tcp://*:5568
2025-07-19 03:16:52,501 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-07-19 03:16:52,503 INFO [collect_river_data] Collecting ECCC ClowhomClowhomLake river data for 2025-07-18
2025-07-19 03:16:52,597 INFO [collect_river_data] Appended ECCC ClowhomClowhomLake river average discharge for 2025-07-18 to: /results/forcing/rivers/observations/Clowhom_ClowhomLake_flow
2025-07-19 03:16:52,600 INFO [collect_river_data] ECCC ClowhomClowhomLake river data collection for 2025-07-18 completed
2025-07-19 03:16:52,723 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-07-19 03:16:52,738 INFO [collect_river_data] running in process 1662246
2025-07-19 03:16:52,742 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-19 03:16:52,747 INFO [collect_river_data] publishing log messages to tcp://*:5569
2025-07-19 03:16:52,758 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-07-19 03:16:52,760 INFO [collect_river_data] Collecting ECCC TheodosiaDiversion river data for 2025-07-18
2025-07-19 03:16:52,851 INFO [collect_river_data] Appended ECCC TheodosiaDiversion river average discharge for 2025-07-18 to: /results/forcing/rivers/observations/Theodosia_Diversion_flow
2025-07-19 03:16:52,852 INFO [collect_river_data] ECCC TheodosiaDiversion river data collection for 2025-07-18 completed
2025-07-19 03:16:52,915 INFO [grib_to_netcdf] running in process 1662254
2025-07-19 03:16:52,916 INFO [grib_to_netcdf] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-19 03:16:52,917 INFO [grib_to_netcdf] publishing log messages to tcp://*:5570
2025-07-19 03:16:52,918 INFO [grib_to_netcdf] connected to skookum.eos.ubc.ca port 5555
2025-07-19 03:16:52,919 INFO [grib_to_netcdf] creating NEMO-atmos forcing files for 2025-07-19 forecast2 run
2025-07-19 03:16:53,076 INFO [collect_river_data] running in process 1662235
2025-07-19 03:16:53,128 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-19 03:16:53,171 INFO [collect_river_data] publishing log messages to tcp://*:5571
2025-07-19 03:16:53,214 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-07-19 03:16:53,271 INFO [collect_river_data] Collecting ECCC Englishman river data for 2025-07-18
2025-07-19 03:16:53,343 INFO [collect_river_data] running in process 1662238
2025-07-19 03:16:53,402 INFO [collect_river_data] running in process 1662243
2025-07-19 03:16:53,449 INFO [collect_river_data] running in process 1662245
2025-07-19 03:16:53,494 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-19 03:16:53,558 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-19 03:16:53,630 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-19 03:16:53,669 INFO [collect_river_data] publishing log messages to tcp://*:5572
2025-07-19 03:16:53,707 INFO [collect_river_data] publishing log messages to tcp://*:5573
2025-07-19 03:16:53,719 INFO [collect_river_data] publishing log messages to tcp://*:5574
2025-07-19 03:16:53,722 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-07-19 03:16:53,727 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-07-19 03:16:53,728 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-07-19 03:16:53,731 INFO [collect_river_data] Collecting ECCC NicomeklLangley river data for 2025-07-18
2025-07-19 03:16:53,732 INFO [collect_river_data] Collecting ECCC SquamishBrackendale river data for 2025-07-18
2025-07-19 03:16:53,734 INFO [collect_river_data] Collecting ECCC TheodosiaBypass river data for 2025-07-18
2025-07-19 03:16:53,746 INFO [collect_river_data] Appended ECCC NicomeklLangley river average discharge for 2025-07-18 to: /results/forcing/rivers/observations/Nicomekl_Langley_flow
2025-07-19 03:16:53,747 INFO [collect_river_data] Appended ECCC SquamishBrackendale river average discharge for 2025-07-18 to: /results/forcing/rivers/observations/Squamish_Brackendale_flow
2025-07-19 03:16:53,753 INFO [collect_river_data] Appended ECCC TheodosiaBypass river average discharge for 2025-07-18 to: /results/forcing/rivers/observations/Theodosia_Bypass_flow
2025-07-19 03:16:53,755 INFO [collect_river_data] ECCC NicomeklLangley river data collection for 2025-07-18 completed
2025-07-19 03:16:53,756 INFO [collect_river_data] ECCC SquamishBrackendale river data collection for 2025-07-18 completed
2025-07-19 03:16:53,757 INFO [collect_river_data] ECCC TheodosiaBypass river data collection for 2025-07-18 completed
2025-07-19 03:16:53,759 INFO [collect_river_data] running in process 1662232
2025-07-19 03:16:53,762 INFO [collect_river_data] running in process 1662236
2025-07-19 03:16:53,764 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-19 03:16:53,766 INFO [collect_river_data] Appended ECCC Englishman river average discharge for 2025-07-18 to: /data/dlatorne/SOG-projects/SOG-forcing/ECget/Englishman_flow
2025-07-19 03:16:53,768 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-19 03:16:53,768 INFO [collect_river_data] publishing log messages to tcp://*:5575
2025-07-19 03:16:53,769 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-07-19 03:16:53,772 INFO [collect_river_data] publishing log messages to tcp://*:5576
2025-07-19 03:16:53,773 INFO [collect_river_data] ECCC Englishman river data collection for 2025-07-18 completed
2025-07-19 03:16:53,774 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-07-19 03:16:53,776 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-07-19 03:16:53,776 INFO [collect_river_data] Collecting ECCC Capilano river data for 2025-07-18
2025-07-19 03:16:53,777 INFO [collect_river_data] Collecting ECCC Fraser river data for 2025-07-18
2025-07-19 03:16:53,780 INFO [collect_river_data] Appended ECCC Capilano river average discharge for 2025-07-18 to: /opp/observations/rivers/Capilano/Caplilano_08GA010_day_avg_flow
2025-07-19 03:16:53,781 INFO [collect_river_data] Appended ECCC Fraser river average discharge for 2025-07-18 to: /data/dlatorne/SOG-projects/SOG-forcing/ECget/Fraser_flow
2025-07-19 03:16:53,783 INFO [collect_river_data] ECCC Capilano river data collection for 2025-07-18 completed
2025-07-19 03:16:53,784 INFO [collect_river_data] ECCC Fraser river data collection for 2025-07-18 completed
2025-07-19 03:16:53,838 INFO [collect_river_data] running in process 1662237
2025-07-19 03:16:53,860 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-19 03:16:53,897 INFO [collect_river_data] publishing log messages to tcp://*:5577
2025-07-19 03:16:53,933 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-07-19 03:16:53,944 INFO [collect_river_data] Collecting ECCC HomathkoMouth river data for 2025-07-18
2025-07-19 03:16:53,948 INFO [get_vfpa_hadcp] running in process 1662250
2025-07-19 03:16:53,950 INFO [get_vfpa_hadcp] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-19 03:16:53,951 INFO [get_vfpa_hadcp] publishing log messages to tcp://*:5564
2025-07-19 03:16:53,952 INFO [get_vfpa_hadcp] connected to skookum.eos.ubc.ca port 5555
2025-07-19 03:16:53,954 INFO [get_vfpa_hadcp] processing VFPA HADCP data from 2nd Narrows Rail Bridge for 2025-07-18
2025-07-19 03:16:54,001 INFO [collect_river_data] Appended ECCC HomathkoMouth river average discharge for 2025-07-18 to: /results/forcing/rivers/observations/Homathko_Mouth_flow
2025-07-19 03:16:54,003 INFO [collect_river_data] ECCC HomathkoMouth river data collection for 2025-07-18 completed
2025-07-19 03:16:54,104 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-07-19 03:16:54,826 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-07-19 03:16:55,383 INFO [crop_gribs] running in process 1662253
2025-07-19 03:16:55,389 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-19 03:16:55,391 INFO [crop_gribs] publishing log messages to tcp://*:5562
2025-07-19 03:16:55,392 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2025-07-19 03:16:55,393 INFO [crop_gribs] cropping 2025-07-19 ECCC HRDPS 2.5km continental 12Z GRIB files to SalishSeaCast subdomain
2025-07-19 03:16:55,401 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20250719/12/
2025-07-19 03:16:55,425 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-07-19 03:16:56,146 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-07-19 03:16:56,746 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-07-19 03:16:57,371 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-07-19 03:16:57,945 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-07-19 03:16:58,588 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-07-19 03:16:59,204 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-07-19 03:16:59,723 INFO [get_onc_ferry] running in process 1662249
2025-07-19 03:16:59,732 INFO [get_onc_ferry] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-19 03:16:59,733 INFO [get_onc_ferry] publishing log messages to tcp://*:5565
2025-07-19 03:16:59,745 INFO [get_onc_ferry] connected to skookum.eos.ubc.ca port 5555
2025-07-19 03:16:59,772 INFO [get_onc_ferry] requesting ONC TWDP.N1 NAV data for 2025-07-18
2025-07-19 03:16:59,799 INFO [get_onc_ctd] running in process 1662248
2025-07-19 03:16:59,810 INFO [get_onc_ctd] running in process 1662247
2025-07-19 03:16:59,811 INFO [get_onc_ctd] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-19 03:16:59,812 INFO [get_onc_ctd] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-19 03:16:59,813 INFO [get_onc_ctd] publishing log messages to tcp://*:5566
2025-07-19 03:16:59,815 INFO [get_onc_ctd] publishing log messages to tcp://*:5567
2025-07-19 03:16:59,816 INFO [get_onc_ctd] connected to skookum.eos.ubc.ca port 5555
2025-07-19 03:16:59,816 INFO [get_onc_ctd] connected to skookum.eos.ubc.ca port 5555
2025-07-19 03:16:59,817 INFO [get_onc_ctd] requesting ONC SEVIP CTD T&S data for 2025-07-18
2025-07-19 03:16:59,818 INFO [get_onc_ctd] requesting ONC SCVIP CTD T&S data for 2025-07-18
2025-07-19 03:16:59,822 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-07-19 03:17:00,375 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-07-19 03:17:10,234 INFO [get_vfpa_hadcp] added VFPA HADCP data from 2nd Narrows Rail Bridge for 2025-07-18 to /opp/observations/AISDATA/netcdf/VFPA_2ND_NARROWS_HADCP_2s_202507.nc
2025-07-19 03:17:10,235 INFO [get_vfpa_hadcp] VFPA HADCP observations added to 2025-07 netcdf file
2025-07-19 03:17:10,243 INFO [manager] checklist updated with [VFPA HADCP data] items from get_vfpa_hadcp worker
2025-07-19 03:17:10,853 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['VFPA-HADCP'], host='localhost')
2025-07-19 03:17:23,726 INFO [ping_erddap] running in process 1662602
2025-07-19 03:17:23,726 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-19 03:17:23,787 INFO [ping_erddap] publishing log messages to tcp://*:5564
2025-07-19 03:17:23,813 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2025-07-19 03:17:23,815 INFO [ping_erddap] VFPA-HADCP ERDDAP dataset flag file(s) created
2025-07-19 03:17:23,817 INFO [make_ssh_files] running in process 1662508
2025-07-19 03:17:23,819 INFO [make_ssh_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-19 03:17:23,821 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2025-07-19 03:17:23,823 INFO [make_ssh_files] publishing log messages to tcp://*:5568
2025-07-19 03:17:23,824 INFO [make_ssh_files] connected to skookum.eos.ubc.ca port 5555
2025-07-19 03:17:23,825 INFO [make_ssh_files] building forecast2 sea surface height boundary conditions file(s) from 2025-07-19 NOAA Neah Bay 00Z observation and forecast values
2025-07-19 03:17:27,545 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2025m07d20.nc
2025-07-19 03:17:28,398 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2025m07d14.nc
2025-07-19 03:17:28,809 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2025m07d15.nc
2025-07-19 03:17:29,153 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2025m07d16.nc
2025-07-19 03:17:29,505 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2025m07d17.nc
2025-07-19 03:17:29,826 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2025m07d18.nc
2025-07-19 03:17:30,305 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2025m07d19.nc
2025-07-19 03:17:30,632 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2025m07d20.nc
2025-07-19 03:17:30,977 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2025m07d21.nc
2025-07-19 03:17:31,264 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2025m07d22.nc
2025-07-19 03:17:31,895 INFO [make_ssh_files] sea surface height boundary file for 2025-07-19 forecast2 run created
2025-07-19 03:17:31,908 INFO [manager] checklist updated with [sea surface height forcing] items from make_ssh_files worker
2025-07-19 03:17:32,506 INFO [manager] launching NextWorker(module='nowcast.workers.make_runoff_file', args=['v202108'], host='localhost')
2025-07-19 03:17:32,569 INFO [manager] launching NextWorker(module='nowcast.workers.make_201702_runoff_file', args=[], host='localhost')
2025-07-19 03:17:33,197 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2025m07d21.nc
2025-07-19 03:17:33,213 INFO [grib_to_netcdf] 2025-07-19 NEMO-atmos forcing file for forecast2 created
2025-07-19 03:17:33,229 INFO [manager] checklist updated with [weather forcing] items from grib_to_netcdf worker
2025-07-19 03:17:35,985 INFO [get_onc_ctd] 2025-07-18 ONC SCVIP CTD T&S file created
2025-07-19 03:17:35,987 INFO [manager] checklist updated with [ONC CTD data] items from get_onc_ctd worker
2025-07-19 03:17:35,988 INFO [get_onc_ctd] 2025-07-18 ONC SEVIP CTD T&S file created
2025-07-19 03:17:36,080 INFO [get_onc_ferry] requesting ONC TWDP TSG data for 2025-07-18
2025-07-19 03:17:36,619 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['SCVIP-CTD'], host='localhost')
2025-07-19 03:17:36,783 INFO [manager] checklist updated with [ONC CTD data] items from get_onc_ctd worker
2025-07-19 03:17:37,460 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['SEVIP-CTD'], host='localhost')
2025-07-19 03:17:39,809 INFO [make_201702_runoff_file] running in process 1662640
2025-07-19 03:17:39,813 INFO [make_201702_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-19 03:17:39,816 INFO [make_201702_runoff_file] publishing log messages to tcp://*:5564
2025-07-19 03:17:39,820 INFO [make_201702_runoff_file] connected to skookum.eos.ubc.ca port 5555
2025-07-19 03:17:40,030 INFO [make_201702_runoff_file] calculating runoff forcing for b201702 bathymetry
2025-07-19 03:17:40,330 INFO [make_201702_runoff_file] 2025-07-19 runoff file creation from Fraser at Hope and climatology elsewhere complete
2025-07-19 03:17:40,602 INFO [manager] checklist updated with [rivers forcing] items from make_201702_runoff_file worker
2025-07-19 03:17:43,345 INFO [ping_erddap] running in process 1662666
2025-07-19 03:17:43,363 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-19 03:17:43,402 INFO [ping_erddap] publishing log messages to tcp://*:5564
2025-07-19 03:17:43,428 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2025-07-19 03:17:43,458 INFO [ping_erddap] SCVIP-CTD ERDDAP dataset flag file(s) created
2025-07-19 03:17:43,462 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2025-07-19 03:17:45,236 INFO [make_runoff_file] running in process 1662638
2025-07-19 03:17:45,266 INFO [make_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-19 03:17:45,301 INFO [make_runoff_file] publishing log messages to tcp://*:5564
2025-07-19 03:17:45,319 INFO [make_runoff_file] connected to skookum.eos.ubc.ca port 5555
2025-07-19 03:17:45,329 INFO [make_runoff_file] calculating NEMO runoff forcing for v202108 bathymetry for 2025-07-18
2025-07-19 03:17:45,569 INFO [ping_erddap] running in process 1662668
2025-07-19 03:17:45,629 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-19 03:17:45,682 INFO [ping_erddap] publishing log messages to tcp://*:5566
2025-07-19 03:17:45,739 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2025-07-19 03:17:45,795 INFO [ping_erddap] SEVIP-CTD ERDDAP dataset flag file(s) created
2025-07-19 03:17:45,799 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2025-07-19 03:17:47,567 ERROR [make_runoff_file] no 2025-07-18 discharge obs for SkagitMountVernon: patching
2025-07-19 03:17:50,029 ERROR [make_runoff_file] no 2025-07-18 discharge obs for SnohomishMonroe: patching
2025-07-19 03:17:50,489 ERROR [make_runoff_file] no 2025-07-18 discharge obs for NisquallyMcKenna: patching
2025-07-19 03:17:50,677 ERROR [make_runoff_file] no 2025-07-18 discharge obs for GreenwaterGreenwater: patching
2025-07-19 03:17:55,626 INFO [make_runoff_file] stored NEMO runoff forcing for v202108 bathymetry for 2025-07-18: /results/forcing/rivers/R202108Dailies_y2025m07d18.nc
2025-07-19 03:17:55,626 INFO [make_runoff_file] 2025-07-18 runoff file creation completed
2025-07-19 03:17:55,632 INFO [manager] checklist updated with [rivers forcing] items from make_runoff_file worker
2025-07-19 03:17:56,458 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'forecast2'], host='localhost')
2025-07-19 03:17:56,483 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'forecast2'], host='localhost')
2025-07-19 03:17:56,495 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.graham', 'forecast2'], host='localhost')
2025-07-19 03:17:56,513 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'forecast2'], host='localhost')
2025-07-19 03:18:04,600 INFO [upload_forcing] running in process 1662692
2025-07-19 03:18:04,600 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-19 03:18:04,696 INFO [upload_forcing] publishing log messages to tcp://*:5564
2025-07-19 03:18:04,803 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-07-19 03:18:05,006 INFO [upload_forcing] running in process 1662691
2025-07-19 03:18:05,232 INFO [upload_forcing] running in process 1662693
2025-07-19 03:18:05,445 INFO [upload_forcing] running in process 1662690
2025-07-19 03:18:05,531 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-19 03:18:05,584 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-19 03:18:05,625 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-19 03:18:05,638 INFO [upload_forcing] publishing log messages to tcp://*:5566
2025-07-19 03:18:05,640 INFO [upload_forcing] publishing log messages to tcp://*:5567
2025-07-19 03:18:05,641 INFO [upload_forcing] publishing log messages to tcp://*:5568
2025-07-19 03:18:05,642 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-07-19 03:18:05,644 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-07-19 03:18:05,650 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-07-19 03:18:06,835 INFO [upload_forcing] LiveOcean boundary conditions file not found; created symlink to /results/forcing/LiveOcean/boundary_conditions/LiveOcean_v201905_y2025m07d18.nc
2025-07-19 03:18:06,946 INFO [upload_forcing] forecast2 2025-07-19 forcing files upload to optimum-hindcast completed
2025-07-19 03:18:06,955 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-07-19 03:18:08,802 INFO [upload_forcing] forecast2 2025-07-19 forcing files upload to arbutus.cloud-nowcast completed
2025-07-19 03:18:08,863 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-07-19 03:18:09,510 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2025-07-19'], host='localhost')
2025-07-19 03:18:13,584 INFO [upload_forcing] forecast2 2025-07-19 forcing files upload to robot.graham completed
2025-07-19 03:18:13,630 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-07-19 03:18:16,270 INFO [make_forcing_links] running in process 1662720
2025-07-19 03:18:16,351 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-19 03:18:16,410 INFO [make_forcing_links] publishing log messages to tcp://*:5564
2025-07-19 03:18:16,490 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2025-07-19 03:18:16,658 INFO [get_onc_ferry] requesting ONC TWDP OXYSENSOR data for 2025-07-18
2025-07-19 03:18:17,670 INFO [make_forcing_links] forecast2 2025-07-19 forcing file links on arbutus.cloud-nowcast created
2025-07-19 03:18:17,922 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2025-07-19 03:18:18,197 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2025-07-19'], host='arbutus.cloud-nowcast')
2025-07-19 03:18:20,496 INFO [run_NEMO] running in process 25332
2025-07-19 03:18:20,497 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-07-19 03:18:20,609 INFO [run_NEMO] publishing log messages to tcp://*:5556
2025-07-19 03:18:20,686 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2025-07-19 03:18:21,262 WARNING [run_NEMO] There are uncommitted changes in /nemoShare/MEOPAR/nowcast-sys/SS-run-sets
2025-07-19 03:18:21,720 INFO [run_NEMO] forecast2: launching /nemoShare/MEOPAR/nowcast-sys/runs/18jul25forecast2_2025-07-19T031820.577285-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2025-07-19 03:18:21,721 INFO [run_NEMO] forecast2 NEMO run for 2025-07-19 on arbutus.cloud-nowcast started
2025-07-19 03:18:21,724 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2025-07-19 03:18:22,309 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'forecast2'], host='arbutus.cloud-nowcast')
2025-07-19 03:18:24,273 INFO [watch_NEMO] running in process 25608
2025-07-19 03:18:24,274 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-07-19 03:18:24,276 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2025-07-19 03:18:24,277 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2025-07-19 03:18:24,346 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2025-07-19 03:18:57,267 INFO [get_onc_ferry] requesting ONC TWDP TURBCHLFL data for 2025-07-18
2025-07-19 03:19:40,926 INFO [get_onc_ferry] requesting ONC TWDP CO2SENSOR data for 2025-07-18
2025-07-19 03:19:40,954 WARNING [get_onc_ferry] No ONC TWDP CO2SENSOR partial_pressure,co2 data for 2025-07-18; substituting empty dataset
2025-07-19 03:19:40,975 WARNING [get_onc_ferry] No ONC TWDP CO2SENSOR partial_pressure data for 2025-07-18; substituting empty dataset
2025-07-19 03:19:40,985 WARNING [get_onc_ferry] No ONC TWDP CO2SENSOR co2 data for 2025-07-18; substituting empty dataset
2025-07-19 03:19:40,995 INFO [get_onc_ferry] requesting ONC TWDP TEMPHUMID data for 2025-07-18
2025-07-19 03:19:41,033 WARNING [get_onc_ferry] No ONC TWDP TEMPHUMID air_temperature,rel_humidity data for 2025-07-18; substituting empty dataset
2025-07-19 03:19:41,057 WARNING [get_onc_ferry] No ONC TWDP TEMPHUMID air_temperature data for 2025-07-18; substituting empty dataset
2025-07-19 03:19:41,063 WARNING [get_onc_ferry] No ONC TWDP TEMPHUMID rel_humidity data for 2025-07-18; substituting empty dataset
2025-07-19 03:19:41,064 INFO [get_onc_ferry] requesting ONC TWDP BARPRESS data for 2025-07-18
2025-07-19 03:19:41,215 WARNING [get_onc_ferry] No ONC TWDP BARPRESS barometric_pressure data for 2025-07-18; substituting empty dataset
2025-07-19 03:19:41,217 WARNING [get_onc_ferry] No ONC TWDP BARPRESS barometric_pressure data for 2025-07-18; substituting empty dataset
2025-07-19 03:19:41,218 INFO [get_onc_ferry] requesting ONC TWDP PYRANOMETER data for 2025-07-18
2025-07-19 03:19:41,220 WARNING [get_onc_ferry] No ONC TWDP PYRANOMETER solar_radiation data for 2025-07-18; substituting empty dataset
2025-07-19 03:19:41,222 WARNING [get_onc_ferry] No ONC TWDP PYRANOMETER solar_radiation data for 2025-07-18; substituting empty dataset
2025-07-19 03:19:41,222 INFO [get_onc_ferry] requesting ONC TWDP PYRGEOMETER data for 2025-07-18
2025-07-19 03:19:41,225 WARNING [get_onc_ferry] No ONC TWDP PYRGEOMETER downward_radiation data for 2025-07-18; substituting empty dataset
2025-07-19 03:19:41,226 WARNING [get_onc_ferry] No ONC TWDP PYRGEOMETER downward_radiation data for 2025-07-18; substituting empty dataset
2025-07-19 03:19:41,678 WARNING [get_onc_ferry] ONC TWDP CO2SENSOR partial_pressure data for 2025-07-18 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2025-07-19 03:19:41,744 WARNING [get_onc_ferry] ONC TWDP CO2SENSOR co2 data for 2025-07-18 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2025-07-19 03:19:41,855 WARNING [get_onc_ferry] ONC TWDP TEMPHUMID air_temperature data for 2025-07-18 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2025-07-19 03:19:42,082 WARNING [get_onc_ferry] ONC TWDP TEMPHUMID rel_humidity data for 2025-07-18 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2025-07-19 03:19:42,278 WARNING [get_onc_ferry] ONC TWDP BARPRESS barometric_pressure data for 2025-07-18 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2025-07-19 03:19:42,374 WARNING [get_onc_ferry] ONC TWDP PYRANOMETER solar_radiation data for 2025-07-18 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2025-07-19 03:19:42,395 WARNING [get_onc_ferry] ONC TWDP PYRGEOMETER downward_radiation data for 2025-07-18 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2025-07-19 03:19:46,718 INFO [upload_forcing] forecast2 2025-07-19 forcing files upload to orcinus-nowcast-agrif completed
2025-07-19 03:19:46,732 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-07-19 03:21:38,013 INFO [get_onc_ferry] 2025-07-18 ONC TWDP ferry data file created
2025-07-19 03:21:38,267 INFO [manager] checklist updated with [ONC ferry data] items from get_onc_ferry worker
2025-07-19 03:21:38,879 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['TWDP-ferry'], host='localhost')
2025-07-19 03:21:47,310 INFO [ping_erddap] running in process 1662754
2025-07-19 03:21:47,311 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-19 03:21:47,339 INFO [ping_erddap] publishing log messages to tcp://*:5564
2025-07-19 03:21:47,378 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2025-07-19 03:21:47,397 INFO [ping_erddap] TWDP-ferry ERDDAP dataset flag file(s) created
2025-07-19 03:21:47,399 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2025-07-19 03:23:24,342 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 18585091 = 2025-07-20 05:00:00 UTC, 16.7% complete
2025-07-19 03:28:24,343 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 18585586 = 2025-07-20 10:30:00 UTC, 35.0% complete
2025-07-19 03:33:24,344 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 18586093 = 2025-07-20 16:08:00 UTC, 53.8% complete
2025-07-19 03:38:24,345 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 18586568 = 2025-07-20 21:24:40 UTC, 71.4% complete
2025-07-19 03:43:24,346 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 18587084 = 2025-07-21 03:08:40 UTC, 90.5% complete
2025-07-19 03:48:24,352 INFO [watch_NEMO] forecast2 NEMO run on arbutus.cloud-nowcast completed
2025-07-19 03:48:24,392 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2025-07-19 03:48:25,224 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_wind_file', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2025-07-19'], host='arbutus.cloud-nowcast')
2025-07-19 03:48:25,239 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_current_file', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2025-07-19'], host='arbutus.cloud-nowcast')
2025-07-19 03:48:25,247 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2025-07-18'], host='localhost')
2025-07-19 03:48:28,162 INFO [make_ww3_current_file] running in process 25981
2025-07-19 03:48:28,192 INFO [make_ww3_current_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-07-19 03:48:28,294 INFO [make_ww3_current_file] publishing log messages to tcp://*:5571
2025-07-19 03:48:28,390 INFO [make_ww3_current_file] connected to skookum.eos.ubc.ca port 5555
2025-07-19 03:48:28,473 INFO [make_ww3_current_file] Creating wwatch3 currents forcing file for 2025-07-19 forecast2 run
2025-07-19 03:48:28,575 INFO [make_ww3_wind_file] running in process 25983
2025-07-19 03:48:28,624 INFO [make_ww3_wind_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-07-19 03:48:28,677 INFO [make_ww3_wind_file] publishing log messages to tcp://*:5570
2025-07-19 03:48:28,725 INFO [make_ww3_wind_file] connected to skookum.eos.ubc.ca port 5555
2025-07-19 03:48:28,793 INFO [make_ww3_wind_file] Creating wwatch3 wind forcing file for 2025-07-19 forecast2 run
2025-07-19 03:48:35,004 INFO [download_results] running in process 1663015
2025-07-19 03:48:35,005 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-19 03:48:35,042 INFO [download_results] publishing log messages to tcp://*:5564
2025-07-19 03:48:35,096 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2025-07-19 03:48:35,136 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/forecast2/18jul25 to /results/SalishSea/forecast2.202111
2025-07-19 03:48:36,996 INFO [make_ww3_wind_file] wwatch3 wind forcing file created on arbutus.cloud-nowcast for 2025-07-19 forecast2 run
2025-07-19 03:48:37,368 INFO [manager] checklist updated with [WW3 wind forcing] items from make_ww3_wind_file worker
2025-07-19 03:49:56,041 INFO [download_results] forecast2 2025-07-18 results files from arbutus.cloud-nowcast downloaded
2025-07-19 03:49:56,072 INFO [manager] checklist updated with [results files] items from download_results worker
2025-07-19 03:49:56,761 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['forecast2', '--run-date', '2025-07-18'], host='localhost')
2025-07-19 03:50:11,937 INFO [make_CHS_currents_file] running in process 1663034
2025-07-19 03:50:11,938 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-19 03:50:12,003 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5564
2025-07-19 03:50:12,020 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555
2025-07-19 03:51:30,440 INFO [make_CHS_currents_file] Made CHS currents file for 2025-07-18 for forecast2
2025-07-19 03:51:30,538 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker
2025-07-19 03:51:31,152 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['nemo', 'forecast2', '--run-date', '2025-07-18'], host='localhost')
2025-07-19 03:51:37,106 INFO [update_forecast_datasets] running in process 1663061
2025-07-19 03:51:37,108 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-19 03:51:37,131 INFO [update_forecast_datasets] publishing log messages to tcp://*:5564
2025-07-19 03:51:37,149 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2025-07-19 03:51:37,167 INFO [update_forecast_datasets] updating nemo forecast directory for forecast2/18jul25 run
2025-07-19 03:52:05,278 INFO [make_ww3_current_file] wwatch3 currents forcing file created on arbutus.cloud-nowcast for 2025-07-19 forecast2 run
2025-07-19 03:52:05,392 INFO [manager] checklist updated with [WW3 currents forcing] items from make_ww3_current_file worker
2025-07-19 03:52:06,214 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2025-07-19'], host='arbutus.cloud-nowcast')
2025-07-19 03:52:08,244 INFO [run_ww3] running in process 26563
2025-07-19 03:52:08,295 INFO [run_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-07-19 03:52:08,339 INFO [run_ww3] publishing log messages to tcp://*:5572
2025-07-19 03:52:08,415 INFO [run_ww3] connected to skookum.eos.ubc.ca port 5555
2025-07-19 03:52:08,632 INFO [run_ww3] Created run directory /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast2_2025-07-19T035208.240856-0700
2025-07-19 03:52:08,653 INFO [run_ww3] forecast2: launching /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast2_2025-07-19T035208.240856-0700/SoGWW3.sh on arbutus.cloud-nowcast
2025-07-19 03:52:08,666 INFO [run_ww3] forecast2 WaveWatch3 run for 2025-07-19 on arbutus.cloud-nowcast started
2025-07-19 03:52:08,705 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker
2025-07-19 03:52:08,962 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'forecast2'], host='arbutus.cloud-nowcast')
2025-07-19 03:52:10,885 INFO [watch_ww3] running in process 26650
2025-07-19 03:52:10,886 INFO [watch_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-07-19 03:52:10,929 INFO [watch_ww3] publishing log messages to tcp://*:5573
2025-07-19 03:52:10,947 INFO [watch_ww3] connected to skookum.eos.ubc.ca port 5555
2025-07-19 03:52:10,979 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2025-07-19 03:57:07,096 INFO [update_forecast_datasets] updated nemo forecast directory for forecast2/18jul25 run: /results/SalishSea/rolling-forecasts/nemo
2025-07-19 03:57:07,097 INFO [update_forecast_datasets] nemo 2025-07-18 forecast2 rolling forecast datasets updated
2025-07-19 03:57:07,100 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2025-07-19 03:57:07,445 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nemo-forecast'], host='localhost')
2025-07-19 03:57:07,450 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'forecast2', 'publish', '--run-date', '2025-07-18'], host='localhost')
2025-07-19 03:57:07,455 INFO [manager] launching NextWorker(module='nowcast.workers.make_surface_current_tiles', args=['forecast2', '--run-date', '2025-07-18'], host='localhost')
2025-07-19 03:57:09,134 INFO [ping_erddap] running in process 1663162
2025-07-19 03:57:09,136 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-19 03:57:09,136 INFO [ping_erddap] publishing log messages to tcp://*:5564
2025-07-19 03:57:09,137 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2025-07-19 03:57:09,147 INFO [ping_erddap] nemo-forecast ERDDAP dataset flag file(s) created
2025-07-19 03:57:09,149 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2025-07-19 03:57:10,921 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 93, 17.2% complete
2025-07-19 03:57:10,941 INFO [make_surface_current_tiles] running in process 1663165
2025-07-19 03:57:10,943 INFO [make_surface_current_tiles] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-19 03:57:10,944 INFO [make_surface_current_tiles] publishing log messages to tcp://*:5564
2025-07-19 03:57:10,945 INFO [make_surface_current_tiles] connected to skookum.eos.ubc.ca port 5555
2025-07-19 03:57:11,052 INFO [make_surface_current_tiles] starting rendering of tiles for 2025-07-18 forecast2 in 6 processes into /results/nowcast-sys/figures/surface_currents/forecast2/18jul25
2025-07-19 03:57:13,168 INFO [make_plots] running in process 1663164
2025-07-19 03:57:13,170 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-19 03:57:13,171 INFO [make_plots] publishing log messages to tcp://*:5565
2025-07-19 03:57:13,182 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2025-07-19 03:57:13,703 INFO [make_plots] preparing render list for 2025-07-18 NEMO forecast2 publish figures
2025-07-19 03:57:13,829 INFO [make_plots] added Website_thumbnail figure to 2025-07-18 NEMO forecast2 publish render list
2025-07-19 03:57:13,830 INFO [make_plots] added Threshold_website figure to 2025-07-18 NEMO forecast2 publish render list
2025-07-19 03:57:13,831 INFO [make_plots] added PA_tidal_predictions figure to 2025-07-18 NEMO forecast2 publish render list
2025-07-19 03:57:13,832 INFO [make_plots] added Boundary Bay figure to 2025-07-18 NEMO forecast2 publish render list
2025-07-19 03:57:13,833 INFO [make_plots] added Campbell River figure to 2025-07-18 NEMO forecast2 publish render list
2025-07-19 03:57:13,834 INFO [make_plots] added Cherry Point figure to 2025-07-18 NEMO forecast2 publish render list
2025-07-19 03:57:13,835 INFO [make_plots] added Friday Harbor figure to 2025-07-18 NEMO forecast2 publish render list
2025-07-19 03:57:13,836 INFO [make_plots] added Halfmoon Bay figure to 2025-07-18 NEMO forecast2 publish render list
2025-07-19 03:57:13,836 INFO [make_plots] added Nanaimo figure to 2025-07-18 NEMO forecast2 publish render list
2025-07-19 03:57:13,837 INFO [make_plots] added Neah Bay figure to 2025-07-18 NEMO forecast2 publish render list
2025-07-19 03:57:13,838 INFO [make_plots] added New Westminster figure to 2025-07-18 NEMO forecast2 publish render list
2025-07-19 03:57:13,839 INFO [make_plots] added Patricia Bay figure to 2025-07-18 NEMO forecast2 publish render list
2025-07-19 03:57:13,840 INFO [make_plots] added Point Atkinson figure to 2025-07-18 NEMO forecast2 publish render list
2025-07-19 03:57:13,841 INFO [make_plots] added Port Renfrew figure to 2025-07-18 NEMO forecast2 publish render list
2025-07-19 03:57:13,842 INFO [make_plots] added Sand Heads figure to 2025-07-18 NEMO forecast2 publish render list
2025-07-19 03:57:13,842 INFO [make_plots] added Sandy Cove figure to 2025-07-18 NEMO forecast2 publish render list
2025-07-19 03:57:13,844 INFO [make_plots] added Squamish figure to 2025-07-18 NEMO forecast2 publish render list
2025-07-19 03:57:13,844 INFO [make_plots] added Victoria figure to 2025-07-18 NEMO forecast2 publish render list
2025-07-19 03:57:13,845 INFO [make_plots] added Woodwards Landing figure to 2025-07-18 NEMO forecast2 publish render list
2025-07-19 03:57:13,846 INFO [make_plots] starting to render nemo forecast2 publish 18jul25 figures
2025-07-19 04:02:10,922 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 201, 37.2% complete
2025-07-19 04:05:16,696 WARNING [make_plots] invalid station number for Boundary Bay station: None
2025-07-19 04:05:16,697 WARNING [make_plots] can't resolve a valid CHS station code for Boundary Bay
2025-07-19 04:05:44,634 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb996/data for station 08074 Campbell River from 2025-07-14 00:00:00Z to 2025-07-21 05:55:00Z
2025-07-19 04:06:59,678 WARNING [make_plots] invalid station number for Halfmoon Bay station: None
2025-07-19 04:06:59,679 WARNING [make_plots] can't resolve a valid CHS station code for Halfmoon Bay
2025-07-19 04:07:10,923 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 300, 55.6% complete
2025-07-19 04:07:23,076 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb96d/data for station 07917 Nanaimo from 2025-07-14 00:00:00Z to 2025-07-21 05:55:00Z
2025-07-19 04:08:06,734 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e13d0f4a073c4bbf93/data for station 07654 New Westminster from 2025-07-14 00:00:00Z to 2025-07-21 05:55:00Z
2025-07-19 04:08:26,988 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd2d/data for station 07277 Patricia Bay from 2025-07-14 00:00:00Z to 2025-07-21 05:55:00Z
2025-07-19 04:08:40,988 INFO [make_surface_current_tiles] starting PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast2/18jul25
2025-07-19 04:08:44,095 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94c/data for station 07795 Point Atkinson from 2025-07-14 00:00:00Z to 2025-07-21 05:55:00Z
2025-07-19 04:09:01,463 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e23d0f4a073c4bc060/data for station 08525 Port Renfrew from 2025-07-14 00:00:00Z to 2025-07-21 05:55:00Z
2025-07-19 04:09:18,421 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb93c/data for station 07594 Sand Heads from 2025-07-14 00:00:00Z to 2025-07-21 05:55:00Z
2025-07-19 04:09:35,883 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc43b/data for station 07786 Sandy Cove from 2025-07-14 00:00:00Z to 2025-07-21 05:55:00Z
2025-07-19 04:09:53,428 WARNING [make_plots] invalid station number for Squamish station: None
2025-07-19 04:09:53,429 WARNING [make_plots] can't resolve a valid CHS station code for Squamish
2025-07-19 04:10:10,123 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd1e/data for station 07120 Victoria from 2025-07-14 00:00:00Z to 2025-07-21 05:55:00Z
2025-07-19 04:10:27,029 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc42d/data for station 07610 Woodwards Landing from 2025-07-14 00:00:00Z to 2025-07-21 05:55:00Z
2025-07-19 04:10:44,596 INFO [make_plots] finished rendering nemo forecast2 publish 18jul25 figures
2025-07-19 04:10:44,599 INFO [make_plots] nemo publish plots for 2025-07-18 forecast2 completed
2025-07-19 04:10:44,612 INFO [manager] checklist updated with [plots] items from make_plots worker
2025-07-19 04:10:44,938 INFO [manager] launching NextWorker(module='nowcast.workers.make_feeds', args=['forecast2', '--run-date', '2025-07-18'], host='localhost')
2025-07-19 04:10:50,048 INFO [make_feeds] running in process 1665175
2025-07-19 04:10:50,049 INFO [make_feeds] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-19 04:10:50,050 INFO [make_feeds] publishing log messages to tcp://*:5565
2025-07-19 04:10:50,052 INFO [make_feeds] connected to skookum.eos.ubc.ca port 5555
2025-07-19 04:11:05,301 INFO [make_feeds] ATOM feeds for 2025-07-18 forecast2 run completed
2025-07-19 04:11:05,304 INFO [manager] checklist updated with [storm surge alert feeds] items from make_feeds worker
2025-07-19 04:11:05,595 INFO [manager] launching NextWorker(module='nemo_nowcast.workers.clear_checklist', args=[], host='localhost')
2025-07-19 04:11:06,872 INFO [clear_checklist] running in process 1665194
2025-07-19 04:11:06,874 INFO [clear_checklist] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-19 04:11:06,875 INFO [clear_checklist] publishing log messages to tcp://*:5565
2025-07-19 04:11:06,876 INFO [clear_checklist] connected to skookum.eos.ubc.ca port 5555
2025-07-19 04:11:06,877 INFO [clear_checklist] requesting that manager clear system state checklist
2025-07-19 04:11:07,046 INFO [manager] writing checklist to log file
2025-07-19 04:11:07,109 INFO [manager] checklist cleared
2025-07-19 04:11:07,113 INFO [clear_checklist] nowcast system checklist cleared
2025-07-19 04:11:07,116 INFO [manager] launching NextWorker(module='nemo_nowcast.workers.rotate_logs', args=[], host='localhost')
2025-07-19 04:11:08,333 INFO [rotate_logs] running in process 1665200
2025-07-19 04:11:08,334 INFO [rotate_logs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-19 04:11:08,335 INFO [rotate_logs] publishing log messages to tcp://*:5565
2025-07-19 04:11:08,336 INFO [rotate_logs] connected to skookum.eos.ubc.ca port 5555
2025-07-19 04:11:08,337 INFO [rotate_logs] rotating log files