2026-03-11 03:31:32,292 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/nowcast.log
2026-03-11 03:31:32,302 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/nowcast.debug.log
2026-03-11 03:31:32,307 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/checklist.log
2026-03-11 03:31:32,308 INFO [rotate_logs] log files rotated
2026-03-11 03:31:32,313 INFO [manager] checklist updated with [log rotation] items from rotate_logs worker
2026-03-11 03:33:51,313 INFO [make_surface_current_tiles] finished PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast2/10mar26
2026-03-11 03:33:51,336 INFO [make_surface_current_tiles] finished rendering of tiles for 2026-03-10 forecast2 into /results/nowcast-sys/figures/surface_currents/forecast2/10mar26
2026-03-11 03:33:51,337 INFO [make_surface_current_tiles] surface current tile figures for 2026-03-10 forecast2 completed
2026-03-11 03:33:51,629 INFO [manager] checklist updated with [surface current tiles] items from make_surface_current_tiles worker
2026-03-11 03:35:30,537 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 372, 68.9% complete
2026-03-11 03:40:30,539 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 471, 87.2% complete
2026-03-11 03:45:30,543 INFO [watch_ww3] forecast2 WWATCH3 run on arbutus.cloud-nowcast completed
2026-03-11 03:45:30,548 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2026-03-11 03:45:30,934 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2026-03-11'], host='localhost')
2026-03-11 03:45:32,304 INFO [download_wwatch3_results] running in process 1264680
2026-03-11 03:45:32,306 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-11 03:45:32,307 INFO [download_wwatch3_results] publishing log messages to tcp://*:5563
2026-03-11 03:45:32,308 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2026-03-11 03:46:08,768 INFO [download_wwatch3_results] forecast2 2026-03-11 results files from arbutus.cloud-nowcast downloaded
2026-03-11 03:46:08,772 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2026-03-11 03:46:08,920 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['wwatch3', 'forecast2', '--run-date', '2026-03-11'], host='localhost')
2026-03-11 03:46:10,198 INFO [update_forecast_datasets] running in process 1264688
2026-03-11 03:46:10,200 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-11 03:46:10,201 INFO [update_forecast_datasets] publishing log messages to tcp://*:5563
2026-03-11 03:46:10,202 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2026-03-11 03:46:10,203 INFO [update_forecast_datasets] updating wwatch3 most_recent_forecast directory from forecast2/11mar26 run
2026-03-11 03:46:10,207 INFO [update_forecast_datasets] updated wwatch3 most_recent_forecast directory from forecast2/11mar26 run
2026-03-11 03:46:10,208 INFO [update_forecast_datasets] updating wwatch3 forecast directory for forecast2/11mar26 run
2026-03-11 03:46:34,421 INFO [update_forecast_datasets] updated wwatch3 forecast directory for forecast2/11mar26 run: /results/SalishSea/rolling-forecasts/wwatch3
2026-03-11 03:46:34,421 INFO [update_forecast_datasets] wwatch3 2026-03-11 forecast2 rolling forecast datasets updated
2026-03-11 03:46:34,425 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2026-03-11 03:46:34,576 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['wwatch3-forecast'], host='localhost')
2026-03-11 03:46:35,989 INFO [ping_erddap] running in process 1264704
2026-03-11 03:46:36,154 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-11 03:46:36,156 INFO [ping_erddap] publishing log messages to tcp://*:5563
2026-03-11 03:46:36,157 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2026-03-11 03:46:36,159 INFO [ping_erddap] wwatch3-forecast ERDDAP dataset flag file(s) created
2026-03-11 03:46:36,162 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2026-03-11 03:46:36,163 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['wwatch3', 'forecast2', 'publish', '--run-date', '2026-03-11'], host='localhost')
2026-03-11 03:46:39,384 INFO [make_plots] running in process 1264709
2026-03-11 03:46:39,387 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-11 03:46:39,388 INFO [make_plots] publishing log messages to tcp://*:5563
2026-03-11 03:46:39,389 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2026-03-11 03:46:39,390 INFO [make_plots] preparing render list for 2026-03-11 WaveWatch3 forecast2 publish figures
2026-03-11 03:46:39,391 INFO [make_plots] added Halibut Bank figure to 2026-03-11 WaveWatch3 forecast2 publish render list
2026-03-11 03:46:39,392 INFO [make_plots] added Sentry Shoal figure to 2026-03-11 WaveWatch3 forecast2 publish render list
2026-03-11 03:46:39,393 INFO [make_plots] starting to render wwatch3 forecast2 publish 11mar26 figures
2026-03-11 03:46:39,992 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46146 Halibut Bank
2026-03-11 03:46:45,718 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46131 Sentry Shoal
2026-03-11 03:46:49,963 INFO [make_plots] finished rendering wwatch3 forecast2 publish 11mar26 figures
2026-03-11 03:46:49,964 INFO [make_plots] wwatch3 publish plots for 2026-03-11 forecast2 completed
2026-03-11 03:46:49,968 INFO [manager] checklist updated with [plots] items from make_plots worker
2026-03-11 08:42:22,841 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/12/ to /results/forcing/atmospheric/continental2.5/GRIB/20260311/12/
2026-03-11 08:42:22,842 INFO [collect_weather] 2026-03-11 2.5km weather forecast 12 collection complete
2026-03-11 08:42:22,844 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2026-03-11 08:42:23,274 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'SkagitMountVernon', '--data-date', '2026-03-10'], host='localhost')
2026-03-11 08:42:23,281 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'SnohomishMonroe', '--data-date', '2026-03-10'], host='localhost')
2026-03-11 08:42:23,287 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'NisquallyMcKenna', '--data-date', '2026-03-10'], host='localhost')
2026-03-11 08:42:23,291 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'GreenwaterGreenwater', '--data-date', '2026-03-10'], host='localhost')
2026-03-11 08:42:23,296 INFO [manager] launching NextWorker(module='nowcast.workers.make_turbidity_file', args=[], host='localhost')
2026-03-11 08:42:23,300 INFO [manager] launching NextWorker(module='nowcast.workers.collect_NeahBay_ssh', args=['06'], host='localhost')
2026-03-11 08:42:23,305 INFO [manager] launching NextWorker(module='nowcast.workers.download_live_ocean', args=[], host='localhost')
2026-03-11 08:42:23,310 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['18', '2.5km'], host='localhost')
2026-03-11 08:42:23,314 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['18'], host='localhost')
2026-03-11 08:42:25,029 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20260311/12/
2026-03-11 08:42:25,030 INFO [crop_gribs] 2026-03-11 12 GRIBs cropping complete
2026-03-11 08:42:25,031 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2026-03-11 08:42:25,400 INFO [manager] launching NextWorker(module='nowcast.workers.grib_to_netcdf', args=['nowcast+'], host='localhost')
2026-03-11 08:42:25,585 INFO [collect_weather] running in process 1268235
2026-03-11 08:42:25,585 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-11 08:42:25,586 INFO [collect_weather] publishing log messages to tcp://*:5562
2026-03-11 08:42:25,587 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2026-03-11 08:42:25,595 INFO [collect_NeahBay_ssh] running in process 1268233
2026-03-11 08:42:25,599 INFO [collect_NeahBay_ssh] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-11 08:42:25,599 INFO [collect_NeahBay_ssh] publishing log messages to tcp://*:5563
2026-03-11 08:42:25,600 INFO [collect_NeahBay_ssh] connected to skookum.eos.ubc.ca port 5555
2026-03-11 08:42:25,603 INFO [collect_NeahBay_ssh] collecting Neah Bay ssh 06Z obs/forecast for 2026-03-11
2026-03-11 08:42:25,606 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/18/
2026-03-11 08:42:25,827 INFO [download_live_ocean] running in process 1268234
2026-03-11 08:42:25,849 INFO [download_live_ocean] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-11 08:42:25,867 INFO [download_live_ocean] publishing log messages to tcp://*:5564
2026-03-11 08:42:25,878 INFO [download_live_ocean] connected to skookum.eos.ubc.ca port 5555
2026-03-11 08:42:25,880 INFO [download_live_ocean] downloading Salish Sea western boundary day-averaged LiveOcean file for 2026-03-11
2026-03-11 08:42:26,326 INFO [collect_river_data] running in process 1268229
2026-03-11 08:42:26,330 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-11 08:42:26,331 INFO [collect_river_data] publishing log messages to tcp://*:5565
2026-03-11 08:42:26,333 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-11 08:42:26,334 INFO [collect_river_data] Collecting USGS SnohomishMonroe river data for 2026-03-10
2026-03-11 08:42:26,335 INFO [download_live_ocean] downloaded 10160286 bytes from https://s3.kopah.uw.edu/liveocean-share/f2026.03.11/ubc.nc to /results/forcing/LiveOcean/downloaded/20260311/low_passed_UBC.nc
2026-03-11 08:42:26,335 INFO [download_live_ocean] Deflating in up to 1 concurrent sub-processes
2026-03-11 08:42:26,340 INFO [collect_river_data] running in process 1268227
2026-03-11 08:42:26,341 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-11 08:42:26,341 INFO [collect_river_data] publishing log messages to tcp://*:5566
2026-03-11 08:42:26,342 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-11 08:42:26,343 INFO [collect_river_data] Collecting USGS SkagitMountVernon river data for 2026-03-10
2026-03-11 08:42:26,377 INFO [collect_river_data] running in process 1268231
2026-03-11 08:42:26,378 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-11 08:42:26,378 INFO [collect_river_data] publishing log messages to tcp://*:5567
2026-03-11 08:42:26,379 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-11 08:42:26,380 INFO [collect_river_data] Collecting USGS GreenwaterGreenwater river data for 2026-03-10
2026-03-11 08:42:26,407 INFO [collect_river_data] running in process 1268230
2026-03-11 08:42:26,408 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-11 08:42:26,409 INFO [collect_river_data] publishing log messages to tcp://*:5568
2026-03-11 08:42:26,411 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-11 08:42:26,411 INFO [collect_river_data] Collecting USGS NisquallyMcKenna river data for 2026-03-10
2026-03-11 08:42:26,719 INFO [make_turbidity_file] running in process 1268232
2026-03-11 08:42:26,720 INFO [make_turbidity_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-11 08:42:26,720 INFO [make_turbidity_file] publishing log messages to tcp://*:5569
2026-03-11 08:42:26,721 INFO [make_turbidity_file] connected to skookum.eos.ubc.ca port 5555
2026-03-11 08:42:26,722 INFO [make_turbidity_file] Creating Fraser River turbidity forcing file for 2026-03-11
2026-03-11 08:42:26,926 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12150800&startDT=2026-03-10&endDT=2026-03-10 "HTTP/1.1 200 "
2026-03-11 08:42:26,928 CRITICAL [collect_river_data] SnohomishMonroe 2026-03-10 timeSeries is empty
2026-03-11 08:42:27,264 INFO [crop_gribs] running in process 1268236
2026-03-11 08:42:27,276 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12200500&startDT=2026-03-10&endDT=2026-03-10 "HTTP/1.1 200 "
2026-03-11 08:42:27,277 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12089500&startDT=2026-03-10&endDT=2026-03-10 "HTTP/1.1 200 "
2026-03-11 08:42:27,277 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12097500&startDT=2026-03-10&endDT=2026-03-10 "HTTP/1.1 200 "
2026-03-11 08:42:27,278 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-11 08:42:27,280 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-11 08:42:27,283 INFO [crop_gribs] publishing log messages to tcp://*:5570
2026-03-11 08:42:27,286 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2026-03-11 08:42:27,288 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-11 08:42:27,289 INFO [crop_gribs] cropping 2026-03-11 ECCC HRDPS 2.5km continental 18Z GRIB files to SalishSeaCast subdomain
2026-03-11 08:42:27,294 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20260311/18/
2026-03-11 08:42:27,296 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-11 08:42:27,299 INFO [collect_river_data] Appended USGS SkagitMountVernon river average discharge for 2026-03-10 to: /results/forcing/rivers/observations/Skagit_MountVernon_flow
2026-03-11 08:42:27,300 INFO [collect_river_data] Appended USGS NisquallyMcKenna river average discharge for 2026-03-10 to: /results/forcing/rivers/observations/Nisqually_McKenna_flow
2026-03-11 08:42:27,300 INFO [collect_river_data] Appended USGS GreenwaterGreenwater river average discharge for 2026-03-10 to: /results/forcing/rivers/observations/Greenwater_Greenwater_flow
2026-03-11 08:42:27,301 INFO [collect_river_data] USGS SkagitMountVernon river data collection for 2026-03-10 completed
2026-03-11 08:42:27,302 INFO [collect_river_data] USGS NisquallyMcKenna river data collection for 2026-03-10 completed
2026-03-11 08:42:27,302 INFO [collect_river_data] USGS GreenwaterGreenwater river data collection for 2026-03-10 completed
2026-03-11 08:42:27,353 INFO [download_live_ocean] netCDF4 deflated /results/forcing/LiveOcean/downloaded/20260311/low_passed_UBC.nc
2026-03-11 08:42:27,354 INFO [download_live_ocean] 2026-03-11 Live Ocean file for Salish Sea western boundary downloaded
2026-03-11 08:42:27,548 INFO [manager] checklist updated with [Live Ocean products] items from download_live_ocean worker
2026-03-11 08:42:27,565 CRITICAL [collect_river_data] Calculation of USGS SnohomishMonroe river average discharge for 2026-03-10 or appending it to SnohomishMonroe_flow file failed
2026-03-11 08:42:27,766 INFO [collect_NeahBay_ssh] 2026-03-11T00:00:00-07:00 Neah Bay ssh 06Z obs/forecast data collection complete
2026-03-11 08:42:27,877 INFO [manager] launching NextWorker(module='nowcast.workers.make_live_ocean_files', args=['--run-date', '2026-03-11'], host='localhost')
2026-03-11 08:42:27,897 INFO [grib_to_netcdf] running in process 1268321
2026-03-11 08:42:27,916 INFO [grib_to_netcdf] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-11 08:42:27,934 INFO [grib_to_netcdf] publishing log messages to tcp://*:5566
2026-03-11 08:42:27,952 INFO [grib_to_netcdf] connected to skookum.eos.ubc.ca port 5555
2026-03-11 08:42:27,964 INFO [grib_to_netcdf] creating NEMO-atmos forcing files for 2026-03-11 nowcast and forecast runs
2026-03-11 08:42:27,969 INFO [manager] checklist updated with [Neah Bay ssh data] items from collect_NeahBay_ssh worker
2026-03-11 08:42:28,165 INFO [manager] launching NextWorker(module='nowcast.workers.make_ssh_files', args=['nowcast', '--run-date', '2026-03-11'], host='localhost')
2026-03-11 08:42:29,561 INFO [make_turbidity_file] 2026-03-11 Fraser River turbidity file creation complete
2026-03-11 08:42:29,590 INFO [manager] checklist updated with [Fraser River turbidity] items from make_turbidity_file worker
2026-03-11 08:42:31,169 INFO [make_live_ocean_files] running in process 1268382
2026-03-11 08:42:31,170 INFO [make_live_ocean_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-11 08:42:31,171 INFO [make_live_ocean_files] publishing log messages to tcp://*:5563
2026-03-11 08:42:31,172 INFO [make_live_ocean_files] connected to skookum.eos.ubc.ca port 5555
2026-03-11 08:42:31,173 INFO [make_live_ocean_files] Creating T&S western boundary conditions file from 2026-03-11 Live Ocean run
2026-03-11 08:42:32,095 INFO [make_ssh_files] running in process 1268384
2026-03-11 08:42:32,118 INFO [make_ssh_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-11 08:42:32,136 INFO [make_ssh_files] publishing log messages to tcp://*:5564
2026-03-11 08:42:32,140 INFO [make_ssh_files] connected to skookum.eos.ubc.ca port 5555
2026-03-11 08:42:32,140 INFO [make_ssh_files] building nowcast sea surface height boundary conditions file(s) from 2026-03-11 NOAA Neah Bay 06Z observation and forecast values
2026-03-11 08:42:34,023 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2026m03d07.nc
2026-03-11 08:42:34,212 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2026m03d08.nc
2026-03-11 08:42:34,427 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2026m03d09.nc
2026-03-11 08:42:34,629 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2026m03d10.nc
2026-03-11 08:42:34,845 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2026m03d11.nc
2026-03-11 08:42:35,059 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2026m03d12.nc
2026-03-11 08:42:35,303 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2026m03d13.nc
2026-03-11 08:42:35,566 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2026m03d14.nc
2026-03-11 08:42:35,677 INFO [make_ssh_files] sea surface height boundary file for 2026-03-11 nowcast run created
2026-03-11 08:42:35,696 INFO [manager] checklist updated with [sea surface height forcing] items from make_ssh_files worker
2026-03-11 08:42:35,846 INFO [manager] launching NextWorker(module='nowcast.workers.make_runoff_file', args=['v202108'], host='localhost')
2026-03-11 08:42:35,851 INFO [manager] launching NextWorker(module='nowcast.workers.make_201702_runoff_file', args=[], host='localhost')
2026-03-11 08:42:37,771 INFO [make_201702_runoff_file] running in process 1268465
2026-03-11 08:42:37,773 INFO [make_201702_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-11 08:42:37,774 INFO [make_201702_runoff_file] publishing log messages to tcp://*:5564
2026-03-11 08:42:37,775 INFO [make_201702_runoff_file] connected to skookum.eos.ubc.ca port 5555
2026-03-11 08:42:37,802 INFO [make_201702_runoff_file] calculating runoff forcing for b201702 bathymetry
2026-03-11 08:42:37,898 INFO [make_201702_runoff_file] 2026-03-11 runoff file creation from Fraser at Hope and climatology elsewhere complete
2026-03-11 08:42:37,900 INFO [manager] checklist updated with [rivers forcing] items from make_201702_runoff_file worker
2026-03-11 08:42:38,620 INFO [make_runoff_file] running in process 1268463
2026-03-11 08:42:38,621 INFO [make_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-11 08:42:38,622 INFO [make_runoff_file] publishing log messages to tcp://*:5564
2026-03-11 08:42:38,623 INFO [make_runoff_file] connected to skookum.eos.ubc.ca port 5555
2026-03-11 08:42:38,624 INFO [make_runoff_file] calculating NEMO runoff forcing for v202108 bathymetry for 2026-03-10
2026-03-11 08:42:39,929 ERROR [make_runoff_file] no 2026-03-10 discharge obs for SnohomishMonroe: patching
2026-03-11 08:42:42,172 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/hrdps_y2026m03d11.nc
2026-03-11 08:42:42,570 INFO [make_runoff_file] stored NEMO runoff forcing for v202108 bathymetry for 2026-03-10: /results/forcing/rivers/R202108Dailies_y2026m03d10.nc
2026-03-11 08:42:42,571 INFO [make_runoff_file] 2026-03-10 runoff file creation completed
2026-03-11 08:42:42,573 INFO [manager] checklist updated with [rivers forcing] items from make_runoff_file worker
2026-03-11 08:42:52,235 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2026m03d12.nc
2026-03-11 08:42:57,829 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2026m03d13.nc
2026-03-11 08:42:57,844 INFO [grib_to_netcdf] 2026-03-11 NEMO-atmos forcing file for nowcast+ created
2026-03-11 08:42:57,847 INFO [manager] checklist updated with [weather forcing] items from grib_to_netcdf worker
2026-03-11 08:43:20,085 INFO [make_live_ocean_files] Stored T&S western boundary conditions file: /results/forcing/LiveOcean/boundary_conditions/LiveOcean_v201905_y2026m03d11.nc
2026-03-11 08:43:20,093 INFO [make_live_ocean_files] 2026-03-11 Live Ocean western boundary conditions files created
2026-03-11 08:43:20,107 INFO [manager] checklist updated with [Live Ocean boundary conditions] items from make_live_ocean_files worker
2026-03-11 08:43:20,262 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['weather'], host='localhost')
2026-03-11 08:43:20,266 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'nowcast+'], host='localhost')
2026-03-11 08:43:20,272 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'nowcast+'], host='localhost')
2026-03-11 08:43:20,276 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.nibi', 'nowcast+'], host='localhost')
2026-03-11 08:43:20,280 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'nowcast+'], host='localhost')
2026-03-11 08:43:21,760 INFO [ping_erddap] running in process 1268535
2026-03-11 08:43:21,760 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-11 08:43:21,761 INFO [ping_erddap] publishing log messages to tcp://*:5563
2026-03-11 08:43:21,762 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2026-03-11 08:43:21,763 INFO [ping_erddap] weather ERDDAP dataset flag file(s) created
2026-03-11 08:43:21,764 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2026-03-11 08:43:22,089 INFO [upload_forcing] running in process 1268540
2026-03-11 08:43:22,090 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-11 08:43:22,091 INFO [upload_forcing] publishing log messages to tcp://*:5564
2026-03-11 08:43:22,092 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-03-11 08:43:22,093 INFO [upload_forcing] running in process 1268538
2026-03-11 08:43:22,095 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-11 08:43:22,096 INFO [upload_forcing] publishing log messages to tcp://*:5565
2026-03-11 08:43:22,097 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-03-11 08:43:22,107 INFO [upload_forcing] running in process 1268537
2026-03-11 08:43:22,114 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-11 08:43:22,114 INFO [upload_forcing] publishing log messages to tcp://*:5566
2026-03-11 08:43:22,115 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-03-11 08:43:22,141 INFO [upload_forcing] running in process 1268539
2026-03-11 08:43:22,160 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-11 08:43:22,178 INFO [upload_forcing] publishing log messages to tcp://*:5567
2026-03-11 08:43:22,189 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-03-11 08:43:24,075 INFO [upload_forcing] nowcast+ 2026-03-11 forcing files upload to optimum-hindcast completed
2026-03-11 08:43:24,086 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-03-11 08:43:26,281 INFO [upload_forcing] nowcast+ 2026-03-11 forcing files upload to arbutus.cloud-nowcast completed
2026-03-11 08:43:26,286 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-03-11 08:43:26,438 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'nowcast+', '--run-date', '2026-03-11'], host='localhost')
2026-03-11 08:43:28,346 INFO [make_forcing_links] running in process 1268569
2026-03-11 08:43:28,347 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-11 08:43:28,348 INFO [make_forcing_links] publishing log messages to tcp://*:5563
2026-03-11 08:43:28,349 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2026-03-11 08:43:29,643 INFO [upload_forcing] nowcast+ 2026-03-11 forcing files upload to orcinus-nowcast-agrif completed
2026-03-11 08:43:29,647 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-03-11 08:43:29,797 INFO [make_forcing_links] nowcast+ 2026-03-11 forcing file links on arbutus.cloud-nowcast created
2026-03-11 08:43:29,849 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2026-03-11 08:43:30,000 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2026-03-11'], host='arbutus.cloud-nowcast')
2026-03-11 08:43:32,226 INFO [run_NEMO] running in process 29745
2026-03-11 08:43:32,252 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-11 08:43:32,270 INFO [run_NEMO] publishing log messages to tcp://*:5556
2026-03-11 08:43:32,289 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2026-03-11 08:43:32,995 WARNING [run_NEMO] There are uncommitted changes in /nemoShare/MEOPAR/nowcast-sys/SS-run-sets
2026-03-11 08:43:33,308 INFO [run_NEMO] nowcast: launching /nemoShare/MEOPAR/nowcast-sys/runs/11mar26nowcast_2026-03-11T084332.275728-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2026-03-11 08:43:33,338 INFO [run_NEMO] nowcast NEMO run for 2026-03-11 on arbutus.cloud-nowcast started
2026-03-11 08:43:33,373 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2026-03-11 08:43:33,513 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'nowcast'], host='arbutus.cloud-nowcast')
2026-03-11 08:43:35,538 INFO [watch_NEMO] running in process 30018
2026-03-11 08:43:35,560 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-11 08:43:35,563 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2026-03-11 08:43:35,565 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2026-03-11 08:43:35,607 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2026-03-11 08:43:36,207 INFO [upload_forcing] nowcast+ 2026-03-11 forcing files upload to robot.nibi completed
2026-03-11 08:43:36,212 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-03-11 08:48:35,608 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 19090607 = 2026-03-11 05:50:40 UTC, 24.4% complete
2026-03-11 08:53:35,609 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 19091174 = 2026-03-11 12:08:40 UTC, 50.6% complete
2026-03-11 08:58:35,609 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 19091714 = 2026-03-11 18:08:40 UTC, 75.6% complete
2026-03-11 09:03:35,610 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 19092240 = 2026-03-11 23:59:20 UTC, 100.0% complete
2026-03-11 09:08:35,615 INFO [watch_NEMO] nowcast NEMO run on arbutus.cloud-nowcast completed
2026-03-11 09:08:35,650 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2026-03-11 09:08:35,955 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'ssh', '--run-date', '2026-03-11'], host='localhost')
2026-03-11 09:08:35,963 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2026-03-11'], host='localhost')
2026-03-11 09:08:37,479 INFO [download_results] running in process 1268846
2026-03-11 09:08:37,480 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-11 09:08:37,481 INFO [download_results] publishing log messages to tcp://*:5563
2026-03-11 09:08:37,482 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2026-03-11 09:08:37,483 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/nowcast/11mar26 to /results/SalishSea/nowcast-blue.202111
2026-03-11 09:08:37,565 INFO [make_forcing_links] running in process 1268845
2026-03-11 09:08:37,590 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-11 09:08:37,603 INFO [make_forcing_links] publishing log messages to tcp://*:5564
2026-03-11 09:08:37,605 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2026-03-11 09:08:38,559 INFO [make_forcing_links] ssh 2026-03-11 forcing file links on arbutus.cloud-nowcast created
2026-03-11 09:08:38,664 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2026-03-11 09:08:38,753 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2026-03-11'], host='arbutus.cloud-nowcast')
2026-03-11 09:08:40,940 INFO [run_NEMO] running in process 30413
2026-03-11 09:08:40,941 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-11 09:08:40,944 INFO [run_NEMO] publishing log messages to tcp://*:5556
2026-03-11 09:08:40,972 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2026-03-11 09:08:41,737 WARNING [run_NEMO] There are uncommitted changes in /nemoShare/MEOPAR/nowcast-sys/SS-run-sets
2026-03-11 09:08:42,048 INFO [run_NEMO] forecast: launching /nemoShare/MEOPAR/nowcast-sys/runs/11mar26forecast_2026-03-11T090841.015765-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2026-03-11 09:08:42,063 INFO [run_NEMO] forecast NEMO run for 2026-03-11 on arbutus.cloud-nowcast started
2026-03-11 09:08:42,108 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2026-03-11 09:08:42,240 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'forecast'], host='arbutus.cloud-nowcast')
2026-03-11 09:08:44,297 INFO [watch_NEMO] running in process 30687
2026-03-11 09:08:44,298 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-11 09:08:44,299 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2026-03-11 09:08:44,300 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2026-03-11 09:08:44,395 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2026-03-11 09:09:17,272 INFO [download_results] nowcast 2026-03-11 results files from arbutus.cloud-nowcast downloaded
2026-03-11 09:09:17,280 INFO [manager] checklist updated with [results files] items from download_results worker
2026-03-11 09:09:17,426 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast', 'research', '--run-date', '2026-03-11'], host='localhost')
2026-03-11 09:09:17,480 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast', 'comparison', '--run-date', '2026-03-10'], host='localhost')
2026-03-11 09:09:17,488 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['nowcast', '--run-date', '2026-03-11'], host='localhost')
2026-03-11 09:09:20,793 INFO [make_CHS_currents_file] running in process 1268867
2026-03-11 09:09:20,793 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-11 09:09:20,794 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5563
2026-03-11 09:09:20,795 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555
2026-03-11 09:09:23,290 INFO [make_plots] running in process 1268866
2026-03-11 09:09:23,292 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-11 09:09:23,293 INFO [make_plots] publishing log messages to tcp://*:5564
2026-03-11 09:09:23,294 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2026-03-11 09:09:23,501 INFO [make_plots] running in process 1268864
2026-03-11 09:09:23,502 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-11 09:09:23,503 INFO [make_plots] publishing log messages to tcp://*:5565
2026-03-11 09:09:23,504 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2026-03-11 09:09:23,946 INFO [make_plots] preparing render list for 2026-03-10 NEMO nowcast-blue comparison figures
2026-03-11 09:09:24,081 INFO [make_plots] added SH_wind figure to 2026-03-10 NEMO nowcast-blue comparison render list
2026-03-11 09:09:24,082 INFO [make_plots] added Compare_VENUS_East figure to 2026-03-10 NEMO nowcast-blue comparison render list
2026-03-11 09:09:24,083 INFO [make_plots] added Compare_VENUS_Central figure to 2026-03-10 NEMO nowcast-blue comparison render list
2026-03-11 09:09:24,084 INFO [make_plots] starting to render nemo nowcast comparison 10mar26 figures
2026-03-11 09:09:24,142 INFO [make_plots] preparing render list for 2026-03-11 NEMO nowcast-blue research figures
2026-03-11 09:09:25,565 INFO [make_plots] added salinity_thalweg_and_surface figures to 2026-03-11 NEMO nowcast-blue research render list
2026-03-11 09:09:26,738 INFO [make_plots] added temperature_thalweg_and_surface figures to 2026-03-11 NEMO nowcast-blue research render list
2026-03-11 09:09:26,739 INFO [make_plots] added Currents_sections_and_surface figure to 2026-03-11 NEMO nowcast-blue research render list
2026-03-11 09:09:26,739 INFO [make_plots] added Currents_at_VENUS_Central figure to 2026-03-11 NEMO nowcast-blue research render list
2026-03-11 09:09:26,740 INFO [make_plots] added Currents_at_VENUS_East figure to 2026-03-11 NEMO nowcast-blue research render list
2026-03-11 09:09:26,741 INFO [make_plots] starting to render nemo nowcast research 11mar26 figures
2026-03-11 09:09:58,837 INFO [make_CHS_currents_file] Made CHS currents file for 2026-03-11 for nowcast
2026-03-11 09:09:58,839 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker
2026-03-11 09:10:35,502 INFO [make_plots] finished rendering nemo nowcast research 11mar26 figures
2026-03-11 09:10:35,504 INFO [make_plots] nemo research plots for 2026-03-11 nowcast completed
2026-03-11 09:10:35,522 INFO [manager] checklist updated with [plots] items from make_plots worker
2026-03-11 09:11:07,378 INFO [make_plots] finished rendering nemo nowcast comparison 10mar26 figures
2026-03-11 09:11:07,380 INFO [make_plots] nemo comparison plots for 2026-03-10 nowcast completed
2026-03-11 09:11:07,384 INFO [manager] checklist updated with [plots] items from make_plots worker
2026-03-11 09:13:44,388 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 19092723 = 2026-03-12 05:21:20 UTC, 14.9% complete
2026-03-11 09:18:44,389 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 19093266 = 2026-03-12 11:23:20 UTC, 31.6% complete
2026-03-11 09:23:44,390 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 19093818 = 2026-03-12 17:31:20 UTC, 48.7% complete
2026-03-11 09:28:44,391 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 19094371 = 2026-03-12 23:40:00 UTC, 65.8% complete
2026-03-11 09:33:44,392 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 19094947 = 2026-03-13 06:04:00 UTC, 83.5% complete
2026-03-11 09:38:44,394 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 19095477 = 2026-03-13 11:57:20 UTC, 99.9% complete
2026-03-11 09:43:44,399 INFO [watch_NEMO] forecast NEMO run on arbutus.cloud-nowcast completed
2026-03-11 09:43:44,402 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2026-03-11 09:43:44,736 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_wind_file', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2026-03-11'], host='arbutus.cloud-nowcast')
2026-03-11 09:43:44,743 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_current_file', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2026-03-11'], host='arbutus.cloud-nowcast')
2026-03-11 09:43:44,748 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'turbidity', '--run-date', '2026-03-11'], host='localhost')
2026-03-11 09:43:44,754 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'turbidity', '--run-date', '2026-03-11'], host='localhost')
2026-03-11 09:43:44,759 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.nibi', 'turbidity', '--run-date', '2026-03-11'], host='localhost')
2026-03-11 09:43:44,764 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'turbidity', '--run-date', '2026-03-11'], host='localhost')
2026-03-11 09:43:44,770 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2026-03-11'], host='localhost')
2026-03-11 09:43:46,178 INFO [download_results] running in process 1270937
2026-03-11 09:43:46,179 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-11 09:43:46,180 INFO [download_results] publishing log messages to tcp://*:5563
2026-03-11 09:43:46,181 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2026-03-11 09:43:46,182 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/forecast/11mar26 to /results/SalishSea/forecast.202111
2026-03-11 09:43:46,264 INFO [upload_forcing] running in process 1270936
2026-03-11 09:43:46,265 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-11 09:43:46,266 INFO [upload_forcing] publishing log messages to tcp://*:5564
2026-03-11 09:43:46,268 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-03-11 09:43:46,284 INFO [upload_forcing] running in process 1270933
2026-03-11 09:43:46,286 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-11 09:43:46,287 INFO [upload_forcing] publishing log messages to tcp://*:5565
2026-03-11 09:43:46,288 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-03-11 09:43:46,290 INFO [upload_forcing] running in process 1270935
2026-03-11 09:43:46,291 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-11 09:43:46,293 INFO [upload_forcing] publishing log messages to tcp://*:5566
2026-03-11 09:43:46,294 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-03-11 09:43:46,296 INFO [upload_forcing] running in process 1270934
2026-03-11 09:43:46,297 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-11 09:43:46,297 INFO [upload_forcing] publishing log messages to tcp://*:5567
2026-03-11 09:43:46,298 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-03-11 09:43:47,039 INFO [upload_forcing] turbidity 2026-03-11 forcing files upload to optimum-hindcast completed
2026-03-11 09:43:47,043 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-03-11 09:43:47,333 INFO [upload_forcing] turbidity 2026-03-11 forcing files upload to arbutus.cloud-nowcast completed
2026-03-11 09:43:47,337 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-03-11 09:43:47,499 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2026-03-11'], host='localhost')
2026-03-11 09:43:47,741 INFO [make_ww3_wind_file] running in process 31348
2026-03-11 09:43:47,742 INFO [make_ww3_wind_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-11 09:43:47,744 INFO [make_ww3_wind_file] publishing log messages to tcp://*:5570
2026-03-11 09:43:47,754 INFO [make_ww3_wind_file] connected to skookum.eos.ubc.ca port 5555
2026-03-11 09:43:47,756 INFO [make_ww3_wind_file] Creating wwatch3 wind forcing file for 2026-03-11 forecast run
2026-03-11 09:43:47,802 INFO [make_ww3_current_file] running in process 31347
2026-03-11 09:43:47,803 INFO [make_ww3_current_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-11 09:43:47,804 INFO [make_ww3_current_file] publishing log messages to tcp://*:5571
2026-03-11 09:43:47,805 INFO [make_ww3_current_file] connected to skookum.eos.ubc.ca port 5555
2026-03-11 09:43:47,806 INFO [make_ww3_current_file] Creating wwatch3 currents forcing file for 2026-03-11 forecast run
2026-03-11 09:43:48,466 INFO [upload_forcing] turbidity 2026-03-11 forcing files upload to orcinus-nowcast-agrif completed
2026-03-11 09:43:48,470 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-03-11 09:43:49,104 INFO [make_forcing_links] running in process 1270966
2026-03-11 09:43:49,105 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-11 09:43:49,107 INFO [make_forcing_links] publishing log messages to tcp://*:5564
2026-03-11 09:43:49,109 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2026-03-11 09:43:50,374 INFO [make_forcing_links] nowcast-green 2026-03-11 forcing file links on arbutus.cloud-nowcast created
2026-03-11 09:43:50,379 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2026-03-11 09:43:50,546 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2026-03-11'], host='arbutus.cloud-nowcast')
2026-03-11 09:43:53,298 INFO [upload_forcing] turbidity 2026-03-11 forcing files upload to robot.nibi completed
2026-03-11 09:43:53,302 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-03-11 09:43:54,570 INFO [run_NEMO] running in process 31960
2026-03-11 09:43:54,573 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-11 09:43:54,574 INFO [run_NEMO] publishing log messages to tcp://*:5556
2026-03-11 09:43:54,576 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2026-03-11 09:43:57,274 WARNING [run_NEMO] There are uncommitted changes in /nemoShare/MEOPAR/nowcast-sys/SS-run-sets
2026-03-11 09:43:58,060 INFO [run_NEMO] nowcast-green: launching /nemoShare/MEOPAR/nowcast-sys/runs/11mar26nowcast-green_2026-03-11T094354.683937-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2026-03-11 09:43:58,083 INFO [run_NEMO] nowcast-green NEMO run for 2026-03-11 on arbutus.cloud-nowcast started
2026-03-11 09:43:58,086 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2026-03-11 09:43:58,251 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'nowcast-green'], host='arbutus.cloud-nowcast')
2026-03-11 09:44:00,305 INFO [watch_NEMO] running in process 32257
2026-03-11 09:44:00,305 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-11 09:44:00,307 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2026-03-11 09:44:00,309 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2026-03-11 09:44:00,321 INFO [make_ww3_wind_file] wwatch3 wind forcing file created on arbutus.cloud-nowcast for 2026-03-11 forecast run
2026-03-11 09:44:00,326 INFO [manager] checklist updated with [WW3 wind forcing] items from make_ww3_wind_file worker
2026-03-11 09:44:00,376 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2026-03-11 09:44:33,180 INFO [download_results] forecast 2026-03-11 results files from arbutus.cloud-nowcast downloaded
2026-03-11 09:44:33,184 INFO [manager] checklist updated with [results files] items from download_results worker
2026-03-11 09:44:33,347 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['forecast', '--run-date', '2026-03-11'], host='localhost')
2026-03-11 09:44:35,867 INFO [make_CHS_currents_file] running in process 1270986
2026-03-11 09:44:35,868 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-11 09:44:35,869 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5563
2026-03-11 09:44:35,870 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555
2026-03-11 09:45:23,034 INFO [make_CHS_currents_file] Made CHS currents file for 2026-03-11 for forecast
2026-03-11 09:45:23,037 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker
2026-03-11 09:45:23,202 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['nemo', 'forecast', '--run-date', '2026-03-11'], host='localhost')
2026-03-11 09:45:24,375 INFO [update_forecast_datasets] running in process 1271019
2026-03-11 09:45:24,377 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-11 09:45:24,378 INFO [update_forecast_datasets] publishing log messages to tcp://*:5563
2026-03-11 09:45:24,380 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2026-03-11 09:45:24,381 INFO [update_forecast_datasets] updating nemo forecast directory for forecast/11mar26 run
2026-03-11 09:45:24,575 INFO [update_forecast_datasets] updated nemo forecast directory for forecast/11mar26 run: /results/SalishSea/rolling-forecasts/nemo
2026-03-11 09:45:24,576 INFO [update_forecast_datasets] nemo 2026-03-11 forecast rolling forecast datasets updated
2026-03-11 09:45:24,579 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2026-03-11 09:45:24,714 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nemo-forecast'], host='localhost')
2026-03-11 09:45:24,720 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'forecast', 'publish', '--run-date', '2026-03-11'], host='localhost')
2026-03-11 09:45:24,726 INFO [manager] launching NextWorker(module='nowcast.workers.make_surface_current_tiles', args=['forecast', '--run-date', '2026-03-11'], host='localhost')
2026-03-11 09:45:25,898 INFO [ping_erddap] running in process 1271025
2026-03-11 09:45:25,899 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-11 09:45:25,900 INFO [ping_erddap] publishing log messages to tcp://*:5563
2026-03-11 09:45:25,901 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2026-03-11 09:45:25,912 INFO [ping_erddap] nemo-forecast ERDDAP dataset flag file(s) created
2026-03-11 09:45:25,913 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2026-03-11 09:45:27,151 INFO [make_surface_current_tiles] running in process 1271028
2026-03-11 09:45:27,152 INFO [make_surface_current_tiles] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-11 09:45:27,153 INFO [make_surface_current_tiles] publishing log messages to tcp://*:5563
2026-03-11 09:45:27,154 INFO [make_surface_current_tiles] connected to skookum.eos.ubc.ca port 5555
2026-03-11 09:45:27,187 INFO [make_surface_current_tiles] starting rendering of tiles for 2026-03-11 forecast in 6 processes into /results/nowcast-sys/figures/surface_currents/forecast/11mar26
2026-03-11 09:45:28,287 INFO [make_plots] running in process 1271027
2026-03-11 09:45:28,290 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-11 09:45:28,292 INFO [make_plots] publishing log messages to tcp://*:5564
2026-03-11 09:45:28,293 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2026-03-11 09:45:28,827 INFO [make_plots] preparing render list for 2026-03-11 NEMO forecast publish figures
2026-03-11 09:45:29,280 INFO [make_plots] added Website_thumbnail figure to 2026-03-11 NEMO forecast publish render list
2026-03-11 09:45:29,288 INFO [make_plots] added Threshold_website figure to 2026-03-11 NEMO forecast publish render list
2026-03-11 09:45:29,288 INFO [make_plots] added PA_tidal_predictions figure to 2026-03-11 NEMO forecast publish render list
2026-03-11 09:45:29,289 INFO [make_plots] added Boundary Bay figure to 2026-03-11 NEMO forecast publish render list
2026-03-11 09:45:29,290 INFO [make_plots] added Campbell River figure to 2026-03-11 NEMO forecast publish render list
2026-03-11 09:45:29,291 INFO [make_plots] added Cherry Point figure to 2026-03-11 NEMO forecast publish render list
2026-03-11 09:45:29,292 INFO [make_plots] added Friday Harbor figure to 2026-03-11 NEMO forecast publish render list
2026-03-11 09:45:29,293 INFO [make_plots] added Halfmoon Bay figure to 2026-03-11 NEMO forecast publish render list
2026-03-11 09:45:29,294 INFO [make_plots] added Nanaimo figure to 2026-03-11 NEMO forecast publish render list
2026-03-11 09:45:29,295 INFO [make_plots] added Neah Bay figure to 2026-03-11 NEMO forecast publish render list
2026-03-11 09:45:29,296 INFO [make_plots] added New Westminster figure to 2026-03-11 NEMO forecast publish render list
2026-03-11 09:45:29,296 INFO [make_plots] added Patricia Bay figure to 2026-03-11 NEMO forecast publish render list
2026-03-11 09:45:29,297 INFO [make_plots] added Point Atkinson figure to 2026-03-11 NEMO forecast publish render list
2026-03-11 09:45:29,298 INFO [make_plots] added Port Renfrew figure to 2026-03-11 NEMO forecast publish render list
2026-03-11 09:45:29,299 INFO [make_plots] added Sand Heads figure to 2026-03-11 NEMO forecast publish render list
2026-03-11 09:45:29,301 INFO [make_plots] added Sandy Cove figure to 2026-03-11 NEMO forecast publish render list
2026-03-11 09:45:29,302 INFO [make_plots] added Squamish figure to 2026-03-11 NEMO forecast publish render list
2026-03-11 09:45:29,303 INFO [make_plots] added Victoria figure to 2026-03-11 NEMO forecast publish render list
2026-03-11 09:45:29,304 INFO [make_plots] added Woodwards Landing figure to 2026-03-11 NEMO forecast publish render list
2026-03-11 09:45:29,304 INFO [make_plots] starting to render nemo forecast publish 11mar26 figures
2026-03-11 09:46:26,771 INFO [make_ww3_current_file] wwatch3 currents forcing file created on arbutus.cloud-nowcast for 2026-03-11 forecast run
2026-03-11 09:46:26,774 INFO [manager] checklist updated with [WW3 currents forcing] items from make_ww3_current_file worker
2026-03-11 09:46:26,932 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2026-03-11'], host='arbutus.cloud-nowcast')
2026-03-11 09:46:29,149 INFO [run_ww3] running in process 32373
2026-03-11 09:46:29,160 INFO [run_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-11 09:46:29,160 INFO [run_ww3] publishing log messages to tcp://*:5572
2026-03-11 09:46:29,161 INFO [run_ww3] connected to skookum.eos.ubc.ca port 5555
2026-03-11 09:46:29,165 INFO [run_ww3] Created run directory /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/nowcast_2026-03-11T094629.150829-0700
2026-03-11 09:46:29,166 INFO [run_ww3] nowcast: launching /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/nowcast_2026-03-11T094629.150829-0700/SoGWW3.sh on arbutus.cloud-nowcast
2026-03-11 09:46:29,168 INFO [run_ww3] nowcast WaveWatch3 run for 2026-03-11 on arbutus.cloud-nowcast started
2026-03-11 09:46:29,169 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker
2026-03-11 09:46:29,471 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'nowcast'], host='arbutus.cloud-nowcast')
2026-03-11 09:46:31,434 INFO [watch_ww3] running in process 32460
2026-03-11 09:46:31,435 INFO [watch_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-11 09:46:31,436 INFO [watch_ww3] publishing log messages to tcp://*:5573
2026-03-11 09:46:31,436 INFO [watch_ww3] connected to skookum.eos.ubc.ca port 5555
2026-03-11 09:46:31,466 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2026-03-11 09:49:00,377 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19090255 = 2026-03-11 01:56:00 UTC, 8.1% complete
2026-03-11 09:51:31,466 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 93, 21.5% complete
2026-03-11 09:53:32,874 WARNING [make_plots] invalid station number for Boundary Bay station: None
2026-03-11 09:53:32,876 WARNING [make_plots] can't resolve a valid CHS station code for Boundary Bay
2026-03-11 09:53:56,038 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb996/data for station 08074 Campbell River from 2026-03-06 00:00:00Z to 2026-03-13 11:55:00Z
2026-03-11 09:54:00,378 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19090449 = 2026-03-11 04:05:20 UTC, 17.0% complete
2026-03-11 09:55:07,393 WARNING [make_plots] invalid station number for Halfmoon Bay station: None
2026-03-11 09:55:07,394 WARNING [make_plots] can't resolve a valid CHS station code for Halfmoon Bay
2026-03-11 09:55:30,942 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb96d/data for station 07917 Nanaimo from 2026-03-06 00:00:00Z to 2026-03-13 11:55:00Z
2026-03-11 09:56:07,916 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e13d0f4a073c4bbf93/data for station 07654 New Westminster from 2026-03-06 00:00:00Z to 2026-03-13 11:55:00Z
2026-03-11 09:56:09,140 INFO [make_surface_current_tiles] starting PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast/11mar26
2026-03-11 09:56:26,704 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd2d/data for station 07277 Patricia Bay from 2026-03-06 00:00:00Z to 2026-03-13 11:55:00Z
2026-03-11 09:56:31,467 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 192, 44.4% complete
2026-03-11 09:56:45,076 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94c/data for station 07795 Point Atkinson from 2026-03-06 00:00:00Z to 2026-03-13 11:55:00Z
2026-03-11 09:57:02,298 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e23d0f4a073c4bc060/data for station 08525 Port Renfrew from 2026-03-06 00:00:00Z to 2026-03-13 11:55:00Z
2026-03-11 09:57:20,551 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb93c/data for station 07594 Sand Heads from 2026-03-06 00:00:00Z to 2026-03-13 11:55:00Z
2026-03-11 09:57:37,796 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc43b/data for station 07786 Sandy Cove from 2026-03-06 00:00:00Z to 2026-03-13 11:55:00Z
2026-03-11 09:57:56,232 WARNING [make_plots] invalid station number for Squamish station: None
2026-03-11 09:57:56,232 WARNING [make_plots] can't resolve a valid CHS station code for Squamish
2026-03-11 09:58:13,193 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd1e/data for station 07120 Victoria from 2026-03-06 00:00:00Z to 2026-03-13 11:55:00Z
2026-03-11 09:58:31,618 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc42d/data for station 07610 Woodwards Landing from 2026-03-06 00:00:00Z to 2026-03-13 11:55:00Z
2026-03-11 09:58:49,341 INFO [make_plots] finished rendering nemo forecast publish 11mar26 figures
2026-03-11 09:58:49,343 INFO [make_plots] nemo publish plots for 2026-03-11 forecast completed
2026-03-11 09:58:49,352 INFO [manager] checklist updated with [plots] items from make_plots worker
2026-03-11 09:58:49,514 INFO [manager] launching NextWorker(module='nowcast.workers.make_feeds', args=['forecast', '--run-date', '2026-03-11'], host='localhost')
2026-03-11 09:58:54,732 INFO [make_feeds] running in process 1278054
2026-03-11 09:58:54,733 INFO [make_feeds] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-11 09:58:54,735 INFO [make_feeds] publishing log messages to tcp://*:5564
2026-03-11 09:58:54,736 INFO [make_feeds] connected to skookum.eos.ubc.ca port 5555
2026-03-11 09:59:00,378 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19090645 = 2026-03-11 06:16:00 UTC, 26.1% complete
2026-03-11 09:59:10,288 INFO [make_feeds] ATOM feeds for 2026-03-11 forecast run completed
2026-03-11 09:59:10,291 INFO [manager] checklist updated with [storm surge alert feeds] items from make_feeds worker
2026-03-11 10:01:11,720 INFO [make_surface_current_tiles] finished PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast/11mar26
2026-03-11 10:01:11,744 INFO [make_surface_current_tiles] finished rendering of tiles for 2026-03-11 forecast into /results/nowcast-sys/figures/surface_currents/forecast/11mar26
2026-03-11 10:01:11,746 INFO [make_surface_current_tiles] surface current tile figures for 2026-03-11 forecast completed
2026-03-11 10:01:12,067 INFO [manager] checklist updated with [surface current tiles] items from make_surface_current_tiles worker
2026-03-11 10:01:31,469 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 297, 68.8% complete
2026-03-11 10:04:00,383 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19090841 = 2026-03-11 08:26:40 UTC, 35.2% complete
2026-03-11 10:06:31,474 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 390, 90.3% complete
2026-03-11 10:09:00,387 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19091037 = 2026-03-11 10:37:20 UTC, 44.3% complete
2026-03-11 10:11:31,478 INFO [watch_ww3] nowcast WWATCH3 run on arbutus.cloud-nowcast completed
2026-03-11 10:11:31,480 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2026-03-11 10:11:32,015 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2026-03-11'], host='localhost')
2026-03-11 10:11:32,022 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2026-03-11'], host='arbutus.cloud-nowcast')
2026-03-11 10:11:33,359 INFO [download_wwatch3_results] running in process 1278464
2026-03-11 10:11:33,361 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-11 10:11:33,362 INFO [download_wwatch3_results] publishing log messages to tcp://*:5563
2026-03-11 10:11:33,363 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2026-03-11 10:11:34,065 INFO [run_ww3] running in process 32727
2026-03-11 10:11:34,069 INFO [run_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-11 10:11:34,070 INFO [run_ww3] publishing log messages to tcp://*:5572
2026-03-11 10:11:34,071 INFO [run_ww3] connected to skookum.eos.ubc.ca port 5555
2026-03-11 10:11:34,074 INFO [run_ww3] Created run directory /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast_2026-03-11T101134.061818-0700
2026-03-11 10:11:34,075 INFO [run_ww3] forecast: launching /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast_2026-03-11T101134.061818-0700/SoGWW3.sh on arbutus.cloud-nowcast
2026-03-11 10:11:34,086 INFO [run_ww3] forecast WaveWatch3 run for 2026-03-11 on arbutus.cloud-nowcast started
2026-03-11 10:11:34,088 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker
2026-03-11 10:11:34,400 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'forecast'], host='arbutus.cloud-nowcast')
2026-03-11 10:11:36,439 INFO [watch_ww3] running in process 414
2026-03-11 10:11:36,444 INFO [watch_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-11 10:11:36,445 INFO [watch_ww3] publishing log messages to tcp://*:5573
2026-03-11 10:11:36,446 INFO [watch_ww3] connected to skookum.eos.ubc.ca port 5555
2026-03-11 10:11:36,476 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2026-03-11 10:12:09,013 INFO [download_wwatch3_results] nowcast 2026-03-11 results files from arbutus.cloud-nowcast downloaded
2026-03-11 10:12:09,017 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2026-03-11 10:14:00,392 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19091232 = 2026-03-11 12:47:20 UTC, 53.3% complete
2026-03-11 10:16:36,479 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 84, 13.0% complete
2026-03-11 10:19:00,404 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19091427 = 2026-03-11 14:57:20 UTC, 62.3% complete
2026-03-11 10:21:36,481 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 180, 27.8% complete
2026-03-11 10:24:00,396 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19091622 = 2026-03-11 17:07:20 UTC, 71.4% complete
2026-03-11 10:26:36,483 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 273, 42.1% complete
2026-03-11 10:29:00,413 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19091816 = 2026-03-11 19:16:40 UTC, 80.4% complete
2026-03-11 10:31:36,484 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 378, 58.3% complete
2026-03-11 10:34:00,400 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19092011 = 2026-03-11 21:26:40 UTC, 89.4% complete
2026-03-11 10:36:36,482 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 480, 74.1% complete
2026-03-11 10:39:00,406 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19092207 = 2026-03-11 23:37:20 UTC, 98.5% complete
2026-03-11 10:41:36,483 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 585, 90.3% complete
2026-03-11 10:44:00,579 INFO [watch_NEMO] nowcast-green NEMO run on arbutus.cloud-nowcast completed
2026-03-11 10:44:00,583 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2026-03-11 10:44:01,057 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2026-03-11'], host='localhost')
2026-03-11 10:44:02,555 INFO [download_results] running in process 1278814
2026-03-11 10:44:02,556 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-11 10:44:02,557 INFO [download_results] publishing log messages to tcp://*:5563
2026-03-11 10:44:02,558 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2026-03-11 10:44:02,560 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/nowcast-green/11mar26 to /results2/SalishSea/nowcast-green.202111
2026-03-11 10:46:27,100 INFO [download_results] nowcast-green 2026-03-11 results files from arbutus.cloud-nowcast downloaded
2026-03-11 10:46:27,106 INFO [manager] checklist updated with [results files] items from download_results worker
2026-03-11 10:46:27,420 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast-green', 'research', '--run-date', '2026-03-11'], host='localhost')
2026-03-11 10:46:27,425 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nowcast-green'], host='localhost')
2026-03-11 10:46:27,431 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'physics', '--run-date', '2026-03-11'], host='localhost')
2026-03-11 10:46:27,436 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'biology', '--run-date', '2026-03-11'], host='localhost')
2026-03-11 10:46:27,441 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'chemistry', '--run-date', '2026-03-11'], host='localhost')
2026-03-11 10:46:29,005 INFO [ping_erddap] running in process 1278857
2026-03-11 10:46:29,006 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-11 10:46:29,007 INFO [ping_erddap] publishing log messages to tcp://*:5563
2026-03-11 10:46:29,008 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2026-03-11 10:46:29,014 INFO [ping_erddap] nowcast-green ERDDAP dataset flag file(s) created
2026-03-11 10:46:29,016 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2026-03-11 10:46:30,406 INFO [make_averaged_dataset] running in process 1278859
2026-03-11 10:46:30,408 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-11 10:46:30,409 INFO [make_averaged_dataset] publishing log messages to tcp://*:5563
2026-03-11 10:46:30,410 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555
2026-03-11 10:46:30,411 INFO [make_averaged_dataset] creating day-averaged dataset for 11-Mar-2026 biology
2026-03-11 10:46:30,412 INFO [make_averaged_dataset] running in process 1278860
2026-03-11 10:46:30,413 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-11 10:46:30,414 INFO [make_averaged_dataset] publishing log messages to tcp://*:5564
2026-03-11 10:46:30,415 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555
2026-03-11 10:46:30,416 INFO [make_averaged_dataset] creating day-averaged dataset for 11-Mar-2026 chemistry
2026-03-11 10:46:30,422 INFO [make_averaged_dataset] running in process 1278858
2026-03-11 10:46:30,423 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-11 10:46:30,423 INFO [make_averaged_dataset] publishing log messages to tcp://*:5565
2026-03-11 10:46:30,424 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555
2026-03-11 10:46:30,425 INFO [make_averaged_dataset] creating day-averaged dataset for 11-Mar-2026 physics
2026-03-11 10:46:31,257 INFO [make_plots] running in process 1278855
2026-03-11 10:46:31,258 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-11 10:46:31,259 INFO [make_plots] publishing log messages to tcp://*:5566
2026-03-11 10:46:31,260 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2026-03-11 10:46:31,646 INFO [make_plots] preparing render list for 2026-03-11 NEMO nowcast-green research figures
2026-03-11 10:46:33,450 INFO [make_plots] added temperature_thalweg_and_surface figures to 2026-03-11 NEMO nowcast-green research render list
2026-03-11 10:46:34,240 INFO [make_plots] added nitrate_thalweg_and_surface figures to 2026-03-11 NEMO nowcast-green research render list
2026-03-11 10:46:34,997 INFO [make_plots] added ammonium_thalweg_and_surface figures to 2026-03-11 NEMO nowcast-green research render list
2026-03-11 10:46:35,711 INFO [make_plots] added silicon_thalweg_and_surface figures to 2026-03-11 NEMO nowcast-green research render list
2026-03-11 10:46:36,464 INFO [make_plots] added dissolved_organic_nitrogen_thalweg_and_surface figures to 2026-03-11 NEMO nowcast-green research render list
2026-03-11 10:46:36,484 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2026-03-11 10:46:37,243 INFO [make_plots] added particulate_organic_nitrogen_thalweg_and_surface figures to 2026-03-11 NEMO nowcast-green research render list
2026-03-11 10:46:37,981 INFO [make_plots] added biogenic_silicon_thalweg_and_surface figures to 2026-03-11 NEMO nowcast-green research render list
2026-03-11 10:46:38,793 INFO [make_plots] added diatoms_thalweg_and_surface figures to 2026-03-11 NEMO nowcast-green research render list
2026-03-11 10:46:39,606 INFO [make_plots] added flagellates_thalweg_and_surface figures to 2026-03-11 NEMO nowcast-green research render list
2026-03-11 10:46:40,426 INFO [make_plots] added microzooplankton_thalweg_and_surface figures to 2026-03-11 NEMO nowcast-green research render list
2026-03-11 10:46:41,232 INFO [make_plots] added mesozooplankton_thalweg_and_surface figures to 2026-03-11 NEMO nowcast-green research render list
2026-03-11 10:46:41,976 INFO [make_plots] added turbidity_thalweg_and_surface figures to 2026-03-11 NEMO nowcast-green research render list
2026-03-11 10:46:43,573 INFO [make_plots] added temperature_salinity_timeseries figure to 2026-03-11 NEMO nowcast-green research render list
2026-03-11 10:46:43,573 INFO [make_plots] added nitrate_diatoms_timeseries figure to 2026-03-11 NEMO nowcast-green research render list
2026-03-11 10:46:43,574 INFO [make_plots] added diatoms_flagellates_timeseries figure to 2026-03-11 NEMO nowcast-green research render list
2026-03-11 10:46:43,575 INFO [make_plots] added z1_z2_zooplankton_timeseries figure to 2026-03-11 NEMO nowcast-green research render list
2026-03-11 10:46:43,577 INFO [make_plots] starting to render nemo nowcast-green research 11mar26 figures
2026-03-11 10:47:30,956 INFO [make_averaged_dataset] day-averaged dataset for 11-Mar-2026 chemistry created
2026-03-11 10:47:30,957 INFO [make_averaged_dataset] day-averaged dataset for 11-Mar-2026 physics created
2026-03-11 10:47:30,959 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker
2026-03-11 10:47:31,291 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker
2026-03-11 10:47:45,865 INFO [make_averaged_dataset] day-averaged dataset for 11-Mar-2026 biology created
2026-03-11 10:47:45,868 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker
2026-03-11 10:51:36,484 INFO [watch_ww3] forecast WWATCH3 run on arbutus.cloud-nowcast completed
2026-03-11 10:51:36,488 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2026-03-11 10:51:36,992 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2026-03-11'], host='localhost')
2026-03-11 10:51:38,499 INFO [download_wwatch3_results] running in process 1279037
2026-03-11 10:51:38,501 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-11 10:51:38,502 INFO [download_wwatch3_results] publishing log messages to tcp://*:5563
2026-03-11 10:51:38,503 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2026-03-11 10:52:15,697 INFO [download_wwatch3_results] forecast 2026-03-11 results files from arbutus.cloud-nowcast downloaded
2026-03-11 10:52:15,701 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2026-03-11 10:52:16,015 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['wwatch3', 'forecast', '--run-date', '2026-03-11'], host='localhost')
2026-03-11 10:52:17,156 INFO [update_forecast_datasets] running in process 1279048
2026-03-11 10:52:17,157 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-11 10:52:17,159 INFO [update_forecast_datasets] publishing log messages to tcp://*:5563
2026-03-11 10:52:17,160 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2026-03-11 10:52:17,161 INFO [update_forecast_datasets] updating wwatch3 most_recent_forecast directory from forecast/11mar26 run
2026-03-11 10:52:17,206 INFO [update_forecast_datasets] updated wwatch3 most_recent_forecast directory from forecast/11mar26 run
2026-03-11 10:52:17,207 INFO [update_forecast_datasets] updating wwatch3 forecast directory for forecast/11mar26 run
2026-03-11 10:52:17,326 INFO [update_forecast_datasets] updated wwatch3 forecast directory for forecast/11mar26 run: /results/SalishSea/rolling-forecasts/wwatch3
2026-03-11 10:52:17,328 INFO [update_forecast_datasets] wwatch3 2026-03-11 forecast rolling forecast datasets updated
2026-03-11 10:52:17,330 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2026-03-11 10:52:17,644 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['wwatch3-forecast'], host='localhost')
2026-03-11 10:52:18,922 INFO [ping_erddap] running in process 1279053
2026-03-11 10:52:18,925 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-11 10:52:18,926 INFO [ping_erddap] publishing log messages to tcp://*:5563
2026-03-11 10:52:18,927 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2026-03-11 10:52:18,928 INFO [ping_erddap] wwatch3-forecast ERDDAP dataset flag file(s) created
2026-03-11 10:52:18,931 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2026-03-11 10:52:19,244 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['wwatch3', 'forecast', 'publish', '--run-date', '2026-03-11'], host='localhost')
2026-03-11 10:52:23,013 INFO [make_plots] running in process 1279058
2026-03-11 10:52:23,015 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-11 10:52:23,016 INFO [make_plots] publishing log messages to tcp://*:5563
2026-03-11 10:52:23,017 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2026-03-11 10:52:23,018 INFO [make_plots] preparing render list for 2026-03-11 WaveWatch3 forecast publish figures
2026-03-11 10:52:23,019 INFO [make_plots] added Halibut Bank figure to 2026-03-11 WaveWatch3 forecast publish render list
2026-03-11 10:52:23,021 INFO [make_plots] added Sentry Shoal figure to 2026-03-11 WaveWatch3 forecast publish render list
2026-03-11 10:52:23,022 INFO [make_plots] starting to render wwatch3 forecast publish 11mar26 figures
2026-03-11 10:52:24,009 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46146 Halibut Bank
2026-03-11 10:52:29,729 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46131 Sentry Shoal
2026-03-11 10:52:34,629 INFO [make_plots] finished rendering wwatch3 forecast publish 11mar26 figures
2026-03-11 10:52:34,630 INFO [make_plots] wwatch3 publish plots for 2026-03-11 forecast completed
2026-03-11 10:52:34,634 INFO [manager] checklist updated with [plots] items from make_plots worker
2026-03-11 11:58:13,924 INFO [make_plots] finished rendering nemo nowcast-green research 11mar26 figures
2026-03-11 11:58:13,938 INFO [make_plots] nemo research plots for 2026-03-11 nowcast-green completed
2026-03-11 11:58:14,001 INFO [manager] checklist updated with [plots] items from make_plots worker
2026-03-11 14:45:19,236 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/18/ to /results/forcing/atmospheric/continental2.5/GRIB/20260311/18/
2026-03-11 14:45:19,237 INFO [collect_weather] 2026-03-11 2.5km weather forecast 18 collection complete
2026-03-11 14:45:19,241 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2026-03-11 14:45:19,879 INFO [manager] launching NextWorker(module='nowcast.workers.download_weather', args=['00', '1km'], host='localhost')
2026-03-11 14:45:19,891 INFO [manager] launching NextWorker(module='nowcast.workers.download_weather', args=['12', '1km'], host='localhost')
2026-03-11 14:45:19,897 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['00', '2.5km'], host='localhost')
2026-03-11 14:45:19,902 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['00', '--fcst-date', '2026-03-12'], host='localhost')
2026-03-11 14:45:20,225 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20260311/18/
2026-03-11 14:45:20,247 INFO [crop_gribs] 2026-03-11 18 GRIBs cropping complete
2026-03-11 14:45:20,282 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2026-03-11 14:45:22,260 INFO [download_weather] running in process 1282707
2026-03-11 14:45:22,261 INFO [download_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-11 14:45:22,262 INFO [download_weather] publishing log messages to tcp://*:5562
2026-03-11 14:45:22,263 INFO [download_weather] running in process 1282709
2026-03-11 14:45:22,264 INFO [download_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-11 14:45:22,265 INFO [download_weather] publishing log messages to tcp://*:5563
2026-03-11 14:45:22,266 INFO [download_weather] connected to skookum.eos.ubc.ca port 5555
2026-03-11 14:45:22,267 INFO [download_weather] downloading 12 1 km forecast GRIB2 files for 20260311
2026-03-11 14:45:22,268 INFO [download_weather] connected to skookum.eos.ubc.ca port 5555
2026-03-11 14:45:22,268 INFO [collect_weather] running in process 1282710
2026-03-11 14:45:22,269 INFO [download_weather] downloading 00 1 km forecast GRIB2 files for 20260311
2026-03-11 14:45:22,270 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-11 14:45:22,271 INFO [collect_weather] publishing log messages to tcp://*:5564
2026-03-11 14:45:22,271 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2026-03-11 14:45:22,334 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/00/
2026-03-11 14:45:23,860 INFO [crop_gribs] running in process 1282711
2026-03-11 14:45:23,861 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-11 14:45:23,862 INFO [crop_gribs] publishing log messages to tcp://*:5565
2026-03-11 14:45:23,863 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2026-03-11 14:45:23,865 INFO [crop_gribs] cropping 2026-03-12 ECCC HRDPS 2.5km continental 00Z GRIB files to SalishSeaCast subdomain
2026-03-11 14:45:23,878 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20260312/00/
2026-03-11 14:48:57,701 INFO [download_weather] 2026-03-11 1km weather forecast 12 downloads complete
2026-03-11 14:48:57,704 INFO [manager] checklist updated with [weather forecast] items from download_weather worker
2026-03-11 14:49:07,392 INFO [download_weather] 2026-03-11 1km weather forecast 00 downloads complete
2026-03-11 14:49:07,395 INFO [manager] checklist updated with [weather forecast] items from download_weather worker
2026-03-11 20:43:41,231 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/00/ to /results/forcing/atmospheric/continental2.5/GRIB/20260312/00/
2026-03-11 20:43:41,245 INFO [collect_weather] 2026-03-12 2.5km weather forecast 00 collection complete
2026-03-11 20:43:41,299 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2026-03-11 20:43:42,043 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['06', '2.5km'], host='localhost')
2026-03-11 20:43:42,059 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['06', '--fcst-date', '2026-03-12'], host='localhost')
2026-03-11 20:43:42,256 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20260312/00/
2026-03-11 20:43:42,260 INFO [crop_gribs] 2026-03-12 00 GRIBs cropping complete
2026-03-11 20:43:42,263 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2026-03-11 20:43:44,738 INFO [collect_weather] running in process 1287184
2026-03-11 20:43:44,739 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-11 20:43:44,740 INFO [collect_weather] publishing log messages to tcp://*:5562
2026-03-11 20:43:44,741 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2026-03-11 20:43:44,761 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/06/
2026-03-11 20:43:46,294 INFO [crop_gribs] running in process 1287185
2026-03-11 20:43:46,295 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-11 20:43:46,296 INFO [crop_gribs] publishing log messages to tcp://*:5563
2026-03-11 20:43:46,297 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2026-03-11 20:43:46,299 INFO [crop_gribs] cropping 2026-03-12 ECCC HRDPS 2.5km continental 06Z GRIB files to SalishSeaCast subdomain
2026-03-11 20:43:46,311 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20260312/06/
2026-03-12 02:42:05,727 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/06/ to /results/forcing/atmospheric/continental2.5/GRIB/20260312/06/
2026-03-12 02:42:05,757 INFO [collect_weather] 2026-03-12 2.5km weather forecast 06 collection complete
2026-03-12 02:42:05,794 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20260312/06/
2026-03-12 02:42:05,802 INFO [crop_gribs] 2026-03-12 06 GRIBs cropping complete
2026-03-12 02:42:05,837 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2026-03-12 02:42:06,660 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Capilano', '--data-date', '2026-03-11'], host='localhost')
2026-03-12 02:42:06,672 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'ChilliwackVedder', '--data-date', '2026-03-11'], host='localhost')
2026-03-12 02:42:06,678 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'ClowhomClowhomLake', '--data-date', '2026-03-11'], host='localhost')
2026-03-12 02:42:06,688 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Englishman', '--data-date', '2026-03-11'], host='localhost')
2026-03-12 02:42:06,693 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Fraser', '--data-date', '2026-03-11'], host='localhost')
2026-03-12 02:42:06,700 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'HomathkoMouth', '--data-date', '2026-03-11'], host='localhost')
2026-03-12 02:42:06,704 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'NicomeklLangley', '--data-date', '2026-03-11'], host='localhost')
2026-03-12 02:42:06,710 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'RobertsCreek', '--data-date', '2026-03-11'], host='localhost')
2026-03-12 02:42:06,714 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SalmonSayward', '--data-date', '2026-03-11'], host='localhost')
2026-03-12 02:42:06,720 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SanJuanPortRenfrew', '--data-date', '2026-03-11'], host='localhost')
2026-03-12 02:42:06,727 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SquamishBrackendale', '--data-date', '2026-03-11'], host='localhost')
2026-03-12 02:42:06,733 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaScotty', '--data-date', '2026-03-11'], host='localhost')
2026-03-12 02:42:06,743 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaBypass', '--data-date', '2026-03-11'], host='localhost')
2026-03-12 02:42:06,751 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaDiversion', '--data-date', '2026-03-11'], host='localhost')
2026-03-12 02:42:06,763 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ctd', args=['SCVIP'], host='localhost')
2026-03-12 02:42:06,768 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ctd', args=['SEVIP'], host='localhost')
2026-03-12 02:42:06,775 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ferry', args=['TWDP'], host='localhost')
2026-03-12 02:42:06,780 INFO [manager] launching NextWorker(module='nowcast.workers.get_vfpa_hadcp', args=['--data-date', '2026-03-11'], host='localhost')
2026-03-12 02:42:06,786 INFO [manager] launching NextWorker(module='nowcast.workers.collect_NeahBay_ssh', args=['00'], host='localhost')
2026-03-12 02:42:06,802 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['12', '2.5km'], host='localhost')
2026-03-12 02:42:06,811 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['12'], host='localhost')
2026-03-12 02:42:06,824 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2026-03-12 02:42:07,567 INFO [manager] launching NextWorker(module='nowcast.workers.grib_to_netcdf', args=['forecast2'], host='localhost')
2026-03-12 02:42:10,298 INFO [collect_NeahBay_ssh] running in process 1291299
2026-03-12 02:42:10,300 INFO [collect_NeahBay_ssh] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-12 02:42:10,301 INFO [collect_NeahBay_ssh] publishing log messages to tcp://*:5562
2026-03-12 02:42:10,302 INFO [collect_NeahBay_ssh] connected to skookum.eos.ubc.ca port 5555
2026-03-12 02:42:10,303 INFO [collect_NeahBay_ssh] collecting Neah Bay ssh 00Z obs/forecast for 2026-03-12
2026-03-12 02:42:10,314 INFO [collect_weather] running in process 1291300
2026-03-12 02:42:10,316 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-12 02:42:10,317 INFO [collect_weather] publishing log messages to tcp://*:5563
2026-03-12 02:42:10,317 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2026-03-12 02:42:10,346 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/12/
2026-03-12 02:42:11,588 INFO [collect_river_data] running in process 1291289
2026-03-12 02:42:11,589 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-12 02:42:11,590 INFO [collect_river_data] publishing log messages to tcp://*:5564
2026-03-12 02:42:11,591 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-12 02:42:11,593 INFO [collect_river_data] Collecting ECCC SanJuanPortRenfrew river data for 2026-03-11
2026-03-12 02:42:11,665 INFO [collect_river_data] running in process 1291292
2026-03-12 02:42:11,666 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-12 02:42:11,667 INFO [collect_river_data] publishing log messages to tcp://*:5565
2026-03-12 02:42:11,668 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-12 02:42:11,669 INFO [collect_river_data] Collecting ECCC TheodosiaScotty river data for 2026-03-11
2026-03-12 02:42:11,673 INFO [collect_river_data] running in process 1291284
2026-03-12 02:42:11,674 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-12 02:42:11,675 INFO [collect_river_data] publishing log messages to tcp://*:5566
2026-03-12 02:42:11,677 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-12 02:42:11,678 INFO [collect_river_data] Collecting ECCC HomathkoMouth river data for 2026-03-11
2026-03-12 02:42:11,686 INFO [collect_river_data] running in process 1291279
2026-03-12 02:42:11,687 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-12 02:42:11,687 INFO [collect_river_data] publishing log messages to tcp://*:5567
2026-03-12 02:42:11,688 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-12 02:42:11,689 INFO [collect_river_data] Collecting ECCC Capilano river data for 2026-03-11
2026-03-12 02:42:11,715 INFO [collect_river_data] running in process 1291291
2026-03-12 02:42:11,716 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-12 02:42:11,717 INFO [collect_river_data] publishing log messages to tcp://*:5568
2026-03-12 02:42:11,718 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-12 02:42:11,719 INFO [collect_river_data] Collecting ECCC SquamishBrackendale river data for 2026-03-11
2026-03-12 02:42:11,720 INFO [collect_river_data] Appended ECCC TheodosiaScotty river average discharge for 2026-03-11 to: /results/forcing/rivers/observations/Theodosia_Scotty_flow
2026-03-12 02:42:11,721 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-12 02:42:11,722 INFO [collect_river_data] ECCC TheodosiaScotty river data collection for 2026-03-11 completed
2026-03-12 02:42:11,726 INFO [collect_river_data] Appended ECCC SanJuanPortRenfrew river average discharge for 2026-03-11 to: /results/forcing/rivers/observations/SanJuan_PortRenfrew_flow
2026-03-12 02:42:11,727 INFO [collect_river_data] ECCC SanJuanPortRenfrew river data collection for 2026-03-11 completed
2026-03-12 02:42:11,730 INFO [collect_river_data] Appended ECCC HomathkoMouth river average discharge for 2026-03-11 to: /results/forcing/rivers/observations/Homathko_Mouth_flow
2026-03-12 02:42:11,730 INFO [collect_river_data] ECCC HomathkoMouth river data collection for 2026-03-11 completed
2026-03-12 02:42:11,741 INFO [collect_river_data] running in process 1291294
2026-03-12 02:42:11,743 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-12 02:42:11,744 INFO [collect_river_data] publishing log messages to tcp://*:5569
2026-03-12 02:42:11,745 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-12 02:42:11,748 INFO [collect_river_data] Collecting ECCC TheodosiaDiversion river data for 2026-03-11
2026-03-12 02:42:11,749 INFO [collect_river_data] Appended ECCC Capilano river average discharge for 2026-03-11 to: /opp/observations/rivers/Capilano/Caplilano_08GA010_day_avg_flow
2026-03-12 02:42:11,750 INFO [collect_river_data] ECCC Capilano river data collection for 2026-03-11 completed
2026-03-12 02:42:11,759 INFO [collect_river_data] Appended ECCC SquamishBrackendale river average discharge for 2026-03-11 to: /results/forcing/rivers/observations/Squamish_Brackendale_flow
2026-03-12 02:42:11,760 INFO [collect_river_data] ECCC SquamishBrackendale river data collection for 2026-03-11 completed
2026-03-12 02:42:11,773 INFO [collect_river_data] running in process 1291285
2026-03-12 02:42:11,774 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-12 02:42:11,775 INFO [collect_river_data] publishing log messages to tcp://*:5570
2026-03-12 02:42:11,776 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-12 02:42:11,776 INFO [collect_river_data] Collecting ECCC NicomeklLangley river data for 2026-03-11
2026-03-12 02:42:11,782 INFO [collect_river_data] Appended ECCC TheodosiaDiversion river average discharge for 2026-03-11 to: /results/forcing/rivers/observations/Theodosia_Diversion_flow
2026-03-12 02:42:11,784 INFO [collect_river_data] ECCC TheodosiaDiversion river data collection for 2026-03-11 completed
2026-03-12 02:42:11,795 INFO [collect_river_data] running in process 1291283
2026-03-12 02:42:11,796 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-12 02:42:11,797 INFO [collect_river_data] publishing log messages to tcp://*:5571
2026-03-12 02:42:11,799 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-12 02:42:11,800 INFO [collect_river_data] Collecting ECCC Fraser river data for 2026-03-11
2026-03-12 02:42:11,813 INFO [collect_river_data] running in process 1291286
2026-03-12 02:42:11,815 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-12 02:42:11,820 INFO [collect_river_data] publishing log messages to tcp://*:5572
2026-03-12 02:42:11,822 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-12 02:42:11,824 INFO [collect_river_data] Collecting ECCC RobertsCreek river data for 2026-03-11
2026-03-12 02:42:11,826 INFO [collect_river_data] running in process 1291293
2026-03-12 02:42:11,828 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-12 02:42:11,829 INFO [collect_river_data] publishing log messages to tcp://*:5573
2026-03-12 02:42:11,830 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-12 02:42:11,832 INFO [collect_river_data] Collecting ECCC TheodosiaBypass river data for 2026-03-11
2026-03-12 02:42:11,835 INFO [collect_river_data] running in process 1291288
2026-03-12 02:42:11,836 INFO [collect_river_data] Appended ECCC NicomeklLangley river average discharge for 2026-03-11 to: /results/forcing/rivers/observations/Nicomekl_Langley_flow
2026-03-12 02:42:11,836 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-12 02:42:11,838 INFO [collect_river_data] ECCC NicomeklLangley river data collection for 2026-03-11 completed
2026-03-12 02:42:11,839 INFO [collect_river_data] publishing log messages to tcp://*:5574
2026-03-12 02:42:11,841 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-12 02:42:11,842 INFO [collect_river_data] Collecting ECCC SalmonSayward river data for 2026-03-11
2026-03-12 02:42:11,855 INFO [collect_river_data] running in process 1291282
2026-03-12 02:42:11,856 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-12 02:42:11,857 INFO [collect_river_data] publishing log messages to tcp://*:5575
2026-03-12 02:42:11,857 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-12 02:42:11,859 INFO [collect_river_data] Collecting ECCC Englishman river data for 2026-03-11
2026-03-12 02:42:11,860 INFO [collect_river_data] running in process 1291281
2026-03-12 02:42:11,861 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-12 02:42:11,862 INFO [collect_river_data] publishing log messages to tcp://*:5576
2026-03-12 02:42:11,864 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-12 02:42:11,865 INFO [collect_river_data] Collecting ECCC ClowhomClowhomLake river data for 2026-03-11
2026-03-12 02:42:11,866 INFO [collect_river_data] running in process 1291280
2026-03-12 02:42:11,867 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-12 02:42:11,870 INFO [collect_river_data] publishing log messages to tcp://*:5577
2026-03-12 02:42:11,872 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-12 02:42:11,872 INFO [collect_river_data] Collecting ECCC ChilliwackVedder river data for 2026-03-11
2026-03-12 02:42:11,873 INFO [collect_river_data] Appended ECCC TheodosiaBypass river average discharge for 2026-03-11 to: /results/forcing/rivers/observations/Theodosia_Bypass_flow
2026-03-12 02:42:11,874 INFO [collect_river_data] ECCC TheodosiaBypass river data collection for 2026-03-11 completed
2026-03-12 02:42:11,877 INFO [collect_river_data] Appended ECCC RobertsCreek river average discharge for 2026-03-11 to: /results/forcing/rivers/observations/RobertsCreek_flow
2026-03-12 02:42:11,878 INFO [collect_river_data] ECCC RobertsCreek river data collection for 2026-03-11 completed
2026-03-12 02:42:11,885 INFO [collect_river_data] Appended ECCC Englishman river average discharge for 2026-03-11 to: /data/dlatorne/SOG-projects/SOG-forcing/ECget/Englishman_flow
2026-03-12 02:42:11,886 INFO [collect_river_data] ECCC Englishman river data collection for 2026-03-11 completed
2026-03-12 02:42:11,888 INFO [collect_river_data] Appended ECCC SalmonSayward river average discharge for 2026-03-11 to: /results/forcing/rivers/observations/Salmon_Sayward_flow
2026-03-12 02:42:11,888 INFO [collect_river_data] ECCC SalmonSayward river data collection for 2026-03-11 completed
2026-03-12 02:42:11,896 INFO [collect_river_data] Appended ECCC ClowhomClowhomLake river average discharge for 2026-03-11 to: /results/forcing/rivers/observations/Clowhom_ClowhomLake_flow
2026-03-12 02:42:11,897 INFO [collect_river_data] ECCC ClowhomClowhomLake river data collection for 2026-03-11 completed
2026-03-12 02:42:11,901 INFO [collect_NeahBay_ssh] 2026-03-12T00:00:00-07:00 Neah Bay ssh 00Z obs/forecast data collection complete
2026-03-12 02:42:11,902 INFO [collect_river_data] Appended ECCC ChilliwackVedder river average discharge for 2026-03-11 to: /results/forcing/rivers/observations/Chilliwack_Vedder_flow
2026-03-12 02:42:11,903 INFO [collect_river_data] ECCC ChilliwackVedder river data collection for 2026-03-11 completed
2026-03-12 02:42:11,909 INFO [grib_to_netcdf] running in process 1291349
2026-03-12 02:42:11,910 INFO [grib_to_netcdf] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-12 02:42:11,911 INFO [grib_to_netcdf] publishing log messages to tcp://*:5578
2026-03-12 02:42:11,912 INFO [grib_to_netcdf] connected to skookum.eos.ubc.ca port 5555
2026-03-12 02:42:11,913 INFO [grib_to_netcdf] creating NEMO-atmos forcing files for 2026-03-12 forecast2 run
2026-03-12 02:42:12,051 INFO [collect_river_data] Appended ECCC Fraser river average discharge for 2026-03-11 to: /data/dlatorne/SOG-projects/SOG-forcing/ECget/Fraser_flow
2026-03-12 02:42:12,053 INFO [collect_river_data] ECCC Fraser river data collection for 2026-03-11 completed
2026-03-12 02:42:12,183 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-12 02:42:12,232 INFO [get_vfpa_hadcp] running in process 1291298
2026-03-12 02:42:12,234 INFO [get_vfpa_hadcp] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-12 02:42:12,235 INFO [get_vfpa_hadcp] publishing log messages to tcp://*:5579
2026-03-12 02:42:12,236 INFO [get_vfpa_hadcp] connected to skookum.eos.ubc.ca port 5555
2026-03-12 02:42:12,237 INFO [get_vfpa_hadcp] processing VFPA HADCP data from 2nd Narrows Rail Bridge for 2026-03-11
2026-03-12 02:42:12,447 INFO [crop_gribs] running in process 1291301
2026-03-12 02:42:12,449 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-12 02:42:12,449 INFO [crop_gribs] publishing log messages to tcp://*:5590
2026-03-12 02:42:12,450 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2026-03-12 02:42:12,452 INFO [crop_gribs] cropping 2026-03-12 ECCC HRDPS 2.5km continental 12Z GRIB files to SalishSeaCast subdomain
2026-03-12 02:42:12,461 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20260312/12/
2026-03-12 02:42:12,578 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-12 02:42:13,046 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-12 02:42:13,273 INFO [get_onc_ctd] running in process 1291296
2026-03-12 02:42:13,273 INFO [get_onc_ctd] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-12 02:42:13,274 INFO [get_onc_ctd] publishing log messages to tcp://*:5564
2026-03-12 02:42:13,276 INFO [get_onc_ctd] connected to skookum.eos.ubc.ca port 5555
2026-03-12 02:42:13,276 INFO [get_onc_ctd] requesting ONC SEVIP CTD T&S data for 2026-03-11
2026-03-12 02:42:13,394 INFO [get_onc_ctd] running in process 1291295
2026-03-12 02:42:13,395 INFO [get_onc_ctd] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-12 02:42:13,396 INFO [get_onc_ctd] publishing log messages to tcp://*:5565
2026-03-12 02:42:13,397 INFO [get_onc_ctd] connected to skookum.eos.ubc.ca port 5555
2026-03-12 02:42:13,398 INFO [get_onc_ctd] requesting ONC SCVIP CTD T&S data for 2026-03-11
2026-03-12 02:42:13,403 INFO [get_onc_ferry] running in process 1291297
2026-03-12 02:42:13,404 INFO [get_onc_ferry] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-12 02:42:13,409 INFO [get_onc_ferry] publishing log messages to tcp://*:5566
2026-03-12 02:42:13,410 INFO [get_onc_ferry] connected to skookum.eos.ubc.ca port 5555
2026-03-12 02:42:13,410 INFO [get_onc_ferry] requesting ONC TWDP.N1 NAV data for 2026-03-11
2026-03-12 02:42:13,412 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-12 02:42:13,786 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-12 02:42:14,114 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-12 02:42:14,442 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-12 02:42:14,787 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-12 02:42:15,136 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-12 02:42:15,341 INFO [get_vfpa_hadcp] added VFPA HADCP data from 2nd Narrows Rail Bridge for 2026-03-11 to /opp/observations/AISDATA/netcdf/VFPA_2ND_NARROWS_HADCP_2s_202603.nc
2026-03-12 02:42:15,342 INFO [get_vfpa_hadcp] VFPA HADCP observations added to 2026-03 netcdf file
2026-03-12 02:42:15,487 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-12 02:42:15,811 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-12 02:42:16,137 INFO [manager] checklist updated with [Neah Bay ssh data] items from collect_NeahBay_ssh worker
2026-03-12 02:42:16,461 INFO [manager] launching NextWorker(module='nowcast.workers.make_ssh_files', args=['forecast2', '--run-date', '2026-03-12'], host='localhost')
2026-03-12 02:42:16,480 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-12 02:42:16,911 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-12 02:42:17,281 INFO [manager] checklist updated with [VFPA HADCP data] items from get_vfpa_hadcp worker
2026-03-12 02:42:17,698 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['VFPA-HADCP'], host='localhost')
2026-03-12 02:42:18,978 INFO [ping_erddap] running in process 1291798
2026-03-12 02:42:18,979 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-12 02:42:18,980 INFO [ping_erddap] publishing log messages to tcp://*:5562
2026-03-12 02:42:18,980 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2026-03-12 02:42:18,982 INFO [ping_erddap] VFPA-HADCP ERDDAP dataset flag file(s) created
2026-03-12 02:42:18,988 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2026-03-12 02:42:20,455 INFO [make_ssh_files] running in process 1291782
2026-03-12 02:42:20,456 INFO [make_ssh_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-12 02:42:20,457 INFO [make_ssh_files] publishing log messages to tcp://*:5562
2026-03-12 02:42:20,458 INFO [make_ssh_files] connected to skookum.eos.ubc.ca port 5555
2026-03-12 02:42:20,459 INFO [make_ssh_files] building forecast2 sea surface height boundary conditions file(s) from 2026-03-12 NOAA Neah Bay 00Z observation and forecast values
2026-03-12 02:42:22,861 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2026m03d07.nc
2026-03-12 02:42:23,093 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2026m03d08.nc
2026-03-12 02:42:23,290 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2026m03d09.nc
2026-03-12 02:42:23,516 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2026m03d10.nc
2026-03-12 02:42:23,694 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2026m03d11.nc
2026-03-12 02:42:23,973 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2026m03d12.nc
2026-03-12 02:42:24,233 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2026m03d13.nc
2026-03-12 02:42:24,477 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2026m03d14.nc
2026-03-12 02:42:24,602 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2026m03d13.nc
2026-03-12 02:42:24,688 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2026m03d15.nc
2026-03-12 02:42:25,014 INFO [make_ssh_files] sea surface height boundary file for 2026-03-12 forecast2 run created
2026-03-12 02:42:25,017 INFO [manager] checklist updated with [sea surface height forcing] items from make_ssh_files worker
2026-03-12 02:42:25,360 INFO [manager] launching NextWorker(module='nowcast.workers.make_runoff_file', args=['v202108'], host='localhost')
2026-03-12 02:42:25,367 INFO [manager] launching NextWorker(module='nowcast.workers.make_201702_runoff_file', args=[], host='localhost')
2026-03-12 02:42:26,767 INFO [make_201702_runoff_file] running in process 1291853
2026-03-12 02:42:26,768 INFO [make_201702_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-12 02:42:26,769 INFO [make_201702_runoff_file] publishing log messages to tcp://*:5562
2026-03-12 02:42:26,770 INFO [make_201702_runoff_file] connected to skookum.eos.ubc.ca port 5555
2026-03-12 02:42:26,812 INFO [make_201702_runoff_file] calculating runoff forcing for b201702 bathymetry
2026-03-12 02:42:26,956 INFO [make_201702_runoff_file] 2026-03-12 runoff file creation from Fraser at Hope and climatology elsewhere complete
2026-03-12 02:42:26,959 INFO [manager] checklist updated with [rivers forcing] items from make_201702_runoff_file worker
2026-03-12 02:42:27,692 INFO [make_runoff_file] running in process 1291852
2026-03-12 02:42:27,693 INFO [make_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-12 02:42:27,694 INFO [make_runoff_file] publishing log messages to tcp://*:5562
2026-03-12 02:42:27,695 INFO [make_runoff_file] connected to skookum.eos.ubc.ca port 5555
2026-03-12 02:42:27,699 INFO [make_runoff_file] calculating NEMO runoff forcing for v202108 bathymetry for 2026-03-11
2026-03-12 02:42:28,030 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2026m03d14.nc
2026-03-12 02:42:28,037 INFO [grib_to_netcdf] 2026-03-12 NEMO-atmos forcing file for forecast2 created
2026-03-12 02:42:28,039 INFO [manager] checklist updated with [weather forcing] items from grib_to_netcdf worker
2026-03-12 02:42:28,744 ERROR [make_runoff_file] no 2026-03-11 discharge obs for SkagitMountVernon: patching
2026-03-12 02:42:29,368 ERROR [make_runoff_file] no 2026-03-11 discharge obs for SnohomishMonroe: patching
2026-03-12 02:42:29,611 ERROR [make_runoff_file] no 2026-03-11 discharge obs for NisquallyMcKenna: patching
2026-03-12 02:42:29,738 ERROR [make_runoff_file] no 2026-03-11 discharge obs for GreenwaterGreenwater: patching
2026-03-12 02:42:31,236 INFO [make_runoff_file] stored NEMO runoff forcing for v202108 bathymetry for 2026-03-11: /results/forcing/rivers/R202108Dailies_y2026m03d11.nc
2026-03-12 02:42:31,237 INFO [make_runoff_file] 2026-03-11 runoff file creation completed
2026-03-12 02:42:31,239 INFO [manager] checklist updated with [rivers forcing] items from make_runoff_file worker
2026-03-12 02:42:31,582 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'forecast2'], host='localhost')
2026-03-12 02:42:31,584 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'forecast2'], host='localhost')
2026-03-12 02:42:31,586 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.nibi', 'forecast2'], host='localhost')
2026-03-12 02:42:31,592 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'forecast2'], host='localhost')
2026-03-12 02:42:32,188 INFO [get_onc_ctd] 2026-03-11 ONC SCVIP CTD T&S file created
2026-03-12 02:42:32,192 INFO [manager] checklist updated with [ONC CTD data] items from get_onc_ctd worker
2026-03-12 02:42:32,584 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['SCVIP-CTD'], host='localhost')
2026-03-12 02:42:32,794 INFO [get_onc_ctd] 2026-03-11 ONC SEVIP CTD T&S file created
2026-03-12 02:42:32,796 INFO [manager] checklist updated with [ONC CTD data] items from get_onc_ctd worker
2026-03-12 02:42:33,154 INFO [upload_forcing] running in process 1291900
2026-03-12 02:42:33,173 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-12 02:42:33,181 INFO [upload_forcing] publishing log messages to tcp://*:5562
2026-03-12 02:42:33,181 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-03-12 02:42:33,182 INFO [upload_forcing] running in process 1291899
2026-03-12 02:42:33,183 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-12 02:42:33,184 INFO [upload_forcing] publishing log messages to tcp://*:5565
2026-03-12 02:42:33,185 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-03-12 02:42:33,186 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['SEVIP-CTD'], host='localhost')
2026-03-12 02:42:33,197 INFO [upload_forcing] running in process 1291897
2026-03-12 02:42:33,197 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-12 02:42:33,198 INFO [upload_forcing] publishing log messages to tcp://*:5567
2026-03-12 02:42:33,199 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-03-12 02:42:33,221 INFO [upload_forcing] running in process 1291898
2026-03-12 02:42:33,223 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-12 02:42:33,224 INFO [upload_forcing] publishing log messages to tcp://*:5568
2026-03-12 02:42:33,225 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-03-12 02:42:33,837 INFO [ping_erddap] running in process 1291904
2026-03-12 02:42:33,838 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-12 02:42:33,839 INFO [ping_erddap] publishing log messages to tcp://*:5564
2026-03-12 02:42:33,840 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2026-03-12 02:42:33,842 INFO [ping_erddap] SCVIP-CTD ERDDAP dataset flag file(s) created
2026-03-12 02:42:33,844 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2026-03-12 02:42:34,141 INFO [get_onc_ferry] requesting ONC TWDP TSG data for 2026-03-11
2026-03-12 02:42:34,345 INFO [ping_erddap] running in process 1291921
2026-03-12 02:42:34,347 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-12 02:42:34,348 INFO [ping_erddap] publishing log messages to tcp://*:5569
2026-03-12 02:42:34,349 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2026-03-12 02:42:34,351 INFO [ping_erddap] SEVIP-CTD ERDDAP dataset flag file(s) created
2026-03-12 02:42:34,353 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2026-03-12 02:42:34,638 INFO [upload_forcing] LiveOcean boundary conditions file not found; created symlink to /results/forcing/LiveOcean/boundary_conditions/LiveOcean_v201905_y2026m03d11.nc
2026-03-12 02:42:34,755 INFO [upload_forcing] forecast2 2026-03-12 forcing files upload to optimum-hindcast completed
2026-03-12 02:42:34,759 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-03-12 02:42:36,004 INFO [upload_forcing] forecast2 2026-03-12 forcing files upload to arbutus.cloud-nowcast completed
2026-03-12 02:42:36,009 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-03-12 02:42:36,358 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2026-03-12'], host='localhost')
2026-03-12 02:42:37,698 INFO [make_forcing_links] running in process 1291945
2026-03-12 02:42:37,699 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-12 02:42:37,700 INFO [make_forcing_links] publishing log messages to tcp://*:5562
2026-03-12 02:42:37,701 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2026-03-12 02:42:38,814 INFO [make_forcing_links] forecast2 2026-03-12 forcing file links on arbutus.cloud-nowcast created
2026-03-12 02:42:38,821 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2026-03-12 02:42:39,486 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2026-03-12'], host='arbutus.cloud-nowcast')
2026-03-12 02:42:39,795 INFO [upload_forcing] forecast2 2026-03-12 forcing files upload to orcinus-nowcast-agrif completed
2026-03-12 02:42:39,799 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-03-12 02:42:41,603 INFO [run_NEMO] running in process 8594
2026-03-12 02:42:41,604 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-12 02:42:41,605 INFO [run_NEMO] publishing log messages to tcp://*:5556
2026-03-12 02:42:41,665 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2026-03-12 02:42:42,501 WARNING [run_NEMO] There are uncommitted changes in /nemoShare/MEOPAR/nowcast-sys/SS-run-sets
2026-03-12 02:42:42,828 INFO [run_NEMO] forecast2: launching /nemoShare/MEOPAR/nowcast-sys/runs/11mar26forecast2_2026-03-12T024241.715706-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2026-03-12 02:42:42,838 INFO [run_NEMO] forecast2 NEMO run for 2026-03-12 on arbutus.cloud-nowcast started
2026-03-12 02:42:42,840 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2026-03-12 02:42:43,214 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'forecast2'], host='arbutus.cloud-nowcast')
2026-03-12 02:42:45,249 INFO [watch_NEMO] running in process 8868
2026-03-12 02:42:45,250 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-12 02:42:45,251 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2026-03-12 02:42:45,253 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2026-03-12 02:42:45,317 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2026-03-12 02:42:46,005 INFO [upload_forcing] forecast2 2026-03-12 forcing files upload to robot.nibi completed
2026-03-12 02:42:46,009 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-03-12 02:42:58,064 INFO [get_onc_ferry] requesting ONC TWDP OXYSENSOR data for 2026-03-11
2026-03-12 02:43:21,565 INFO [get_onc_ferry] requesting ONC TWDP TURBCHLFL data for 2026-03-11
2026-03-12 02:43:43,377 INFO [get_onc_ferry] requesting ONC TWDP CO2SENSOR data for 2026-03-11
2026-03-12 02:43:43,455 WARNING [get_onc_ferry] No ONC TWDP CO2SENSOR partial_pressure,co2 data for 2026-03-11; substituting empty dataset
2026-03-12 02:43:43,462 WARNING [get_onc_ferry] No ONC TWDP CO2SENSOR partial_pressure data for 2026-03-11; substituting empty dataset
2026-03-12 02:43:43,465 WARNING [get_onc_ferry] No ONC TWDP CO2SENSOR co2 data for 2026-03-11; substituting empty dataset
2026-03-12 02:43:43,466 INFO [get_onc_ferry] requesting ONC TWDP TEMPHUMID data for 2026-03-11
2026-03-12 02:43:43,549 WARNING [get_onc_ferry] No ONC TWDP TEMPHUMID air_temperature,rel_humidity data for 2026-03-11; substituting empty dataset
2026-03-12 02:43:43,553 WARNING [get_onc_ferry] No ONC TWDP TEMPHUMID air_temperature data for 2026-03-11; substituting empty dataset
2026-03-12 02:43:43,555 WARNING [get_onc_ferry] No ONC TWDP TEMPHUMID rel_humidity data for 2026-03-11; substituting empty dataset
2026-03-12 02:43:43,555 INFO [get_onc_ferry] requesting ONC TWDP BARPRESS data for 2026-03-11
2026-03-12 02:43:43,618 WARNING [get_onc_ferry] No ONC TWDP BARPRESS barometric_pressure data for 2026-03-11; substituting empty dataset
2026-03-12 02:43:43,622 WARNING [get_onc_ferry] No ONC TWDP BARPRESS barometric_pressure data for 2026-03-11; substituting empty dataset
2026-03-12 02:43:43,624 INFO [get_onc_ferry] requesting ONC TWDP PYRANOMETER data for 2026-03-11
2026-03-12 02:43:43,688 WARNING [get_onc_ferry] No ONC TWDP PYRANOMETER solar_radiation data for 2026-03-11; substituting empty dataset
2026-03-12 02:43:43,692 WARNING [get_onc_ferry] No ONC TWDP PYRANOMETER solar_radiation data for 2026-03-11; substituting empty dataset
2026-03-12 02:43:43,693 INFO [get_onc_ferry] requesting ONC TWDP PYRGEOMETER data for 2026-03-11
2026-03-12 02:43:43,764 WARNING [get_onc_ferry] No ONC TWDP PYRGEOMETER downward_radiation data for 2026-03-11; substituting empty dataset
2026-03-12 02:43:43,768 WARNING [get_onc_ferry] No ONC TWDP PYRGEOMETER downward_radiation data for 2026-03-11; substituting empty dataset
2026-03-12 02:43:44,004 WARNING [get_onc_ferry] ONC TWDP CO2SENSOR partial_pressure data for 2026-03-11 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2026-03-12 02:43:44,021 WARNING [get_onc_ferry] ONC TWDP CO2SENSOR co2 data for 2026-03-11 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2026-03-12 02:43:44,040 WARNING [get_onc_ferry] ONC TWDP TEMPHUMID air_temperature data for 2026-03-11 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2026-03-12 02:43:44,060 WARNING [get_onc_ferry] ONC TWDP TEMPHUMID rel_humidity data for 2026-03-11 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2026-03-12 02:43:44,080 WARNING [get_onc_ferry] ONC TWDP BARPRESS barometric_pressure data for 2026-03-11 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2026-03-12 02:43:44,099 WARNING [get_onc_ferry] ONC TWDP PYRANOMETER solar_radiation data for 2026-03-11 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2026-03-12 02:43:44,115 WARNING [get_onc_ferry] ONC TWDP PYRGEOMETER downward_radiation data for 2026-03-11 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2026-03-12 02:44:21,040 INFO [get_onc_ferry] 2026-03-11 ONC TWDP ferry data file created
2026-03-12 02:44:21,043 INFO [manager] checklist updated with [ONC ferry data] items from get_onc_ferry worker
2026-03-12 02:44:21,323 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['TWDP-ferry'], host='localhost')
2026-03-12 02:44:22,624 INFO [ping_erddap] running in process 1292041
2026-03-12 02:44:22,626 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-12 02:44:22,627 INFO [ping_erddap] publishing log messages to tcp://*:5562
2026-03-12 02:44:22,628 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2026-03-12 02:44:22,630 INFO [ping_erddap] TWDP-ferry ERDDAP dataset flag file(s) created
2026-03-12 02:44:22,632 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2026-03-12 02:47:45,320 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 19094902 = 2026-03-13 05:34:00 UTC, 18.6% complete
2026-03-12 02:52:45,320 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 19095436 = 2026-03-13 11:30:00 UTC, 38.3% complete
2026-03-12 02:57:45,315 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 19096016 = 2026-03-13 17:56:40 UTC, 59.8% complete
2026-03-12 03:02:45,313 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 19096571 = 2026-03-14 00:06:40 UTC, 80.4% complete
2026-03-12 03:07:45,312 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 19097100 = 2026-03-14 05:59:20 UTC, 100.0% complete
2026-03-12 03:12:45,315 INFO [watch_NEMO] forecast2 NEMO run on arbutus.cloud-nowcast completed
2026-03-12 03:12:45,318 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2026-03-12 03:12:45,867 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_wind_file', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2026-03-12'], host='arbutus.cloud-nowcast')
2026-03-12 03:12:45,883 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_current_file', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2026-03-12'], host='arbutus.cloud-nowcast')
2026-03-12 03:12:45,888 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2026-03-11'], host='localhost')
2026-03-12 03:12:47,826 INFO [download_results] running in process 1292387
2026-03-12 03:12:47,827 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-12 03:12:47,828 INFO [download_results] publishing log messages to tcp://*:5562
2026-03-12 03:12:47,829 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2026-03-12 03:12:47,831 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/forecast2/11mar26 to /results/SalishSea/forecast2.202111
2026-03-12 03:12:48,594 INFO [make_ww3_current_file] running in process 9164
2026-03-12 03:12:48,596 INFO [make_ww3_current_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-12 03:12:48,597 INFO [make_ww3_current_file] publishing log messages to tcp://*:5571
2026-03-12 03:12:48,598 INFO [make_ww3_current_file] connected to skookum.eos.ubc.ca port 5555
2026-03-12 03:12:48,599 INFO [make_ww3_current_file] Creating wwatch3 currents forcing file for 2026-03-12 forecast2 run
2026-03-12 03:12:48,627 INFO [make_ww3_wind_file] running in process 9165
2026-03-12 03:12:48,629 INFO [make_ww3_wind_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-12 03:12:48,630 INFO [make_ww3_wind_file] publishing log messages to tcp://*:5570
2026-03-12 03:12:48,631 INFO [make_ww3_wind_file] connected to skookum.eos.ubc.ca port 5555
2026-03-12 03:12:48,632 INFO [make_ww3_wind_file] Creating wwatch3 wind forcing file for 2026-03-12 forecast2 run
2026-03-12 03:12:57,502 INFO [make_ww3_wind_file] wwatch3 wind forcing file created on arbutus.cloud-nowcast for 2026-03-12 forecast2 run
2026-03-12 03:12:57,506 INFO [manager] checklist updated with [WW3 wind forcing] items from make_ww3_wind_file worker
2026-03-12 03:13:19,771 INFO [download_results] forecast2 2026-03-11 results files from arbutus.cloud-nowcast downloaded
2026-03-12 03:13:19,775 INFO [manager] checklist updated with [results files] items from download_results worker
2026-03-12 03:13:20,119 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['forecast2', '--run-date', '2026-03-11'], host='localhost')
2026-03-12 03:13:23,062 INFO [make_CHS_currents_file] running in process 1292394
2026-03-12 03:13:23,063 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-12 03:13:23,064 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5562
2026-03-12 03:13:23,065 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555
2026-03-12 03:14:00,591 INFO [make_CHS_currents_file] Made CHS currents file for 2026-03-11 for forecast2
2026-03-12 03:14:00,594 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker
2026-03-12 03:14:00,945 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['nemo', 'forecast2', '--run-date', '2026-03-11'], host='localhost')
2026-03-12 03:14:02,210 INFO [update_forecast_datasets] running in process 1292427
2026-03-12 03:14:02,211 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-12 03:14:02,212 INFO [update_forecast_datasets] publishing log messages to tcp://*:5562
2026-03-12 03:14:02,213 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2026-03-12 03:14:02,214 INFO [update_forecast_datasets] updating nemo forecast directory for forecast2/11mar26 run
2026-03-12 03:16:23,999 INFO [make_ww3_current_file] wwatch3 currents forcing file created on arbutus.cloud-nowcast for 2026-03-12 forecast2 run
2026-03-12 03:16:24,004 INFO [manager] checklist updated with [WW3 currents forcing] items from make_ww3_current_file worker
2026-03-12 03:16:24,339 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2026-03-12'], host='arbutus.cloud-nowcast')
2026-03-12 03:16:26,361 INFO [run_ww3] running in process 9716
2026-03-12 03:16:26,362 INFO [run_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-12 03:16:26,363 INFO [run_ww3] publishing log messages to tcp://*:5572
2026-03-12 03:16:26,364 INFO [run_ww3] connected to skookum.eos.ubc.ca port 5555
2026-03-12 03:16:26,367 INFO [run_ww3] Created run directory /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast2_2026-03-12T031626.362957-0700
2026-03-12 03:16:26,368 INFO [run_ww3] forecast2: launching /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast2_2026-03-12T031626.362957-0700/SoGWW3.sh on arbutus.cloud-nowcast
2026-03-12 03:16:26,381 INFO [run_ww3] forecast2 WaveWatch3 run for 2026-03-12 on arbutus.cloud-nowcast started
2026-03-12 03:16:26,385 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker
2026-03-12 03:16:26,740 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'forecast2'], host='arbutus.cloud-nowcast')
2026-03-12 03:16:28,778 INFO [watch_ww3] running in process 9803
2026-03-12 03:16:28,779 INFO [watch_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-12 03:16:28,781 INFO [watch_ww3] publishing log messages to tcp://*:5573
2026-03-12 03:16:28,782 INFO [watch_ww3] connected to skookum.eos.ubc.ca port 5555
2026-03-12 03:16:28,813 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2026-03-12 03:18:49,381 INFO [update_forecast_datasets] updated nemo forecast directory for forecast2/11mar26 run: /results/SalishSea/rolling-forecasts/nemo
2026-03-12 03:18:49,382 INFO [update_forecast_datasets] nemo 2026-03-11 forecast2 rolling forecast datasets updated
2026-03-12 03:18:49,384 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2026-03-12 03:18:49,718 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nemo-forecast'], host='localhost')
2026-03-12 03:18:49,723 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'forecast2', 'publish', '--run-date', '2026-03-11'], host='localhost')
2026-03-12 03:18:49,728 INFO [manager] launching NextWorker(module='nowcast.workers.make_surface_current_tiles', args=['forecast2', '--run-date', '2026-03-11'], host='localhost')
2026-03-12 03:18:51,276 INFO [ping_erddap] running in process 1292605
2026-03-12 03:18:51,277 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-12 03:18:51,278 INFO [ping_erddap] publishing log messages to tcp://*:5562
2026-03-12 03:18:51,278 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2026-03-12 03:18:51,288 INFO [ping_erddap] nemo-forecast ERDDAP dataset flag file(s) created
2026-03-12 03:18:51,290 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2026-03-12 03:18:52,751 INFO [make_surface_current_tiles] running in process 1292608
2026-03-12 03:18:52,754 INFO [make_surface_current_tiles] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-12 03:18:52,756 INFO [make_surface_current_tiles] publishing log messages to tcp://*:5562
2026-03-12 03:18:52,757 INFO [make_surface_current_tiles] connected to skookum.eos.ubc.ca port 5555
2026-03-12 03:18:52,820 INFO [make_surface_current_tiles] starting rendering of tiles for 2026-03-11 forecast2 in 6 processes into /results/nowcast-sys/figures/surface_currents/forecast2/11mar26
2026-03-12 03:18:54,787 INFO [make_plots] running in process 1292607
2026-03-12 03:18:54,788 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-12 03:18:54,790 INFO [make_plots] publishing log messages to tcp://*:5564
2026-03-12 03:18:54,791 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2026-03-12 03:18:55,436 INFO [make_plots] preparing render list for 2026-03-11 NEMO forecast2 publish figures
2026-03-12 03:18:55,944 INFO [make_plots] added Website_thumbnail figure to 2026-03-11 NEMO forecast2 publish render list
2026-03-12 03:18:55,952 INFO [make_plots] added Threshold_website figure to 2026-03-11 NEMO forecast2 publish render list
2026-03-12 03:18:55,960 INFO [make_plots] added PA_tidal_predictions figure to 2026-03-11 NEMO forecast2 publish render list
2026-03-12 03:18:55,968 INFO [make_plots] added Boundary Bay figure to 2026-03-11 NEMO forecast2 publish render list
2026-03-12 03:18:55,976 INFO [make_plots] added Campbell River figure to 2026-03-11 NEMO forecast2 publish render list
2026-03-12 03:18:55,980 INFO [make_plots] added Cherry Point figure to 2026-03-11 NEMO forecast2 publish render list
2026-03-12 03:18:55,982 INFO [make_plots] added Friday Harbor figure to 2026-03-11 NEMO forecast2 publish render list
2026-03-12 03:18:55,983 INFO [make_plots] added Halfmoon Bay figure to 2026-03-11 NEMO forecast2 publish render list
2026-03-12 03:18:55,984 INFO [make_plots] added Nanaimo figure to 2026-03-11 NEMO forecast2 publish render list
2026-03-12 03:18:55,985 INFO [make_plots] added Neah Bay figure to 2026-03-11 NEMO forecast2 publish render list
2026-03-12 03:18:55,986 INFO [make_plots] added New Westminster figure to 2026-03-11 NEMO forecast2 publish render list
2026-03-12 03:18:55,987 INFO [make_plots] added Patricia Bay figure to 2026-03-11 NEMO forecast2 publish render list
2026-03-12 03:18:55,988 INFO [make_plots] added Point Atkinson figure to 2026-03-11 NEMO forecast2 publish render list
2026-03-12 03:18:55,990 INFO [make_plots] added Port Renfrew figure to 2026-03-11 NEMO forecast2 publish render list
2026-03-12 03:18:55,991 INFO [make_plots] added Sand Heads figure to 2026-03-11 NEMO forecast2 publish render list
2026-03-12 03:18:55,994 INFO [make_plots] added Sandy Cove figure to 2026-03-11 NEMO forecast2 publish render list
2026-03-12 03:18:55,995 INFO [make_plots] added Squamish figure to 2026-03-11 NEMO forecast2 publish render list
2026-03-12 03:18:55,997 INFO [make_plots] added Victoria figure to 2026-03-11 NEMO forecast2 publish render list
2026-03-12 03:18:55,997 INFO [make_plots] added Woodwards Landing figure to 2026-03-11 NEMO forecast2 publish render list
2026-03-12 03:18:55,998 INFO [make_plots] starting to render nemo forecast2 publish 11mar26 figures
2026-03-12 03:21:28,814 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 93, 17.2% complete
2026-03-12 03:26:28,816 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 198, 36.7% complete
2026-03-12 03:27:32,759 WARNING [make_plots] invalid station number for Boundary Bay station: None
2026-03-12 03:27:32,760 WARNING [make_plots] can't resolve a valid CHS station code for Boundary Bay
2026-03-12 03:28:00,479 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb996/data for station 08074 Campbell River from 2026-03-07 00:00:00Z to 2026-03-14 05:55:00Z
2026-03-12 03:29:05,638 WARNING [make_plots] invalid station number for Halfmoon Bay station: None
2026-03-12 03:29:05,639 WARNING [make_plots] can't resolve a valid CHS station code for Halfmoon Bay
2026-03-12 03:29:33,942 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb96d/data for station 07917 Nanaimo from 2026-03-07 00:00:00Z to 2026-03-14 05:55:00Z
2026-03-12 03:30:18,130 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e13d0f4a073c4bbf93/data for station 07654 New Westminster from 2026-03-07 00:00:00Z to 2026-03-14 05:55:00Z
2026-03-12 03:30:23,803 INFO [make_surface_current_tiles] starting PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast2/11mar26
2026-03-12 03:30:35,892 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd2d/data for station 07277 Patricia Bay from 2026-03-07 00:00:00Z to 2026-03-14 05:55:00Z
2026-03-12 03:30:53,516 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94c/data for station 07795 Point Atkinson from 2026-03-07 00:00:00Z to 2026-03-14 05:55:00Z
2026-03-12 03:31:10,542 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e23d0f4a073c4bc060/data for station 08525 Port Renfrew from 2026-03-07 00:00:00Z to 2026-03-14 05:55:00Z
2026-03-12 03:31:27,944 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb93c/data for station 07594 Sand Heads from 2026-03-07 00:00:00Z to 2026-03-14 05:55:00Z
2026-03-12 03:31:28,818 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 300, 55.6% complete
2026-03-12 03:31:45,211 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc43b/data for station 07786 Sandy Cove from 2026-03-07 00:00:00Z to 2026-03-14 05:55:00Z
2026-03-12 03:32:02,782 WARNING [make_plots] invalid station number for Squamish station: None
2026-03-12 03:32:02,783 WARNING [make_plots] can't resolve a valid CHS station code for Squamish
2026-03-12 03:32:19,588 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd1e/data for station 07120 Victoria from 2026-03-07 00:00:00Z to 2026-03-14 05:55:00Z
2026-03-12 03:32:37,045 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc42d/data for station 07610 Woodwards Landing from 2026-03-07 00:00:00Z to 2026-03-14 05:55:00Z
2026-03-12 03:32:53,905 INFO [make_plots] finished rendering nemo forecast2 publish 11mar26 figures
2026-03-12 03:32:53,907 INFO [make_plots] nemo publish plots for 2026-03-11 forecast2 completed
2026-03-12 03:32:53,915 INFO [manager] checklist updated with [plots] items from make_plots worker
2026-03-12 03:32:54,391 INFO [manager] launching NextWorker(module='nowcast.workers.make_feeds', args=['forecast2', '--run-date', '2026-03-11'], host='localhost')
2026-03-12 03:32:59,671 INFO [make_feeds] running in process 1297643
2026-03-12 03:32:59,673 INFO [make_feeds] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-12 03:32:59,675 INFO [make_feeds] publishing log messages to tcp://*:5564
2026-03-12 03:32:59,676 INFO [make_feeds] connected to skookum.eos.ubc.ca port 5555
2026-03-12 03:33:14,433 INFO [make_feeds] ATOM feeds for 2026-03-11 forecast2 run completed
2026-03-12 03:33:14,435 INFO [manager] checklist updated with [storm surge alert feeds] items from make_feeds worker
2026-03-12 03:33:14,729 INFO [manager] launching NextWorker(module='nemo_nowcast.workers.clear_checklist', args=[], host='localhost')
2026-03-12 03:33:15,940 INFO [clear_checklist] running in process 1297664
2026-03-12 03:33:15,942 INFO [clear_checklist] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-12 03:33:15,943 INFO [clear_checklist] publishing log messages to tcp://*:5564
2026-03-12 03:33:15,944 INFO [clear_checklist] connected to skookum.eos.ubc.ca port 5555
2026-03-12 03:33:15,945 INFO [clear_checklist] requesting that manager clear system state checklist
2026-03-12 03:33:15,948 INFO [manager] writing checklist to log file
2026-03-12 03:33:15,995 INFO [manager] checklist cleared
2026-03-12 03:33:15,997 INFO [clear_checklist] nowcast system checklist cleared
2026-03-12 03:33:15,999 INFO [manager] launching NextWorker(module='nemo_nowcast.workers.rotate_logs', args=[], host='localhost')
2026-03-12 03:33:17,083 INFO [rotate_logs] running in process 1297669
2026-03-12 03:33:17,084 INFO [rotate_logs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-12 03:33:17,085 INFO [rotate_logs] publishing log messages to tcp://*:5564
2026-03-12 03:33:17,086 INFO [rotate_logs] connected to skookum.eos.ubc.ca port 5555
2026-03-12 03:33:17,087 INFO [rotate_logs] rotating log files