2026-03-13 03:33:35,643 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/nowcast.log
2026-03-13 03:33:35,652 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/nowcast.debug.log
2026-03-13 03:33:35,658 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/checklist.log
2026-03-13 03:33:35,659 INFO [rotate_logs] log files rotated
2026-03-13 03:33:35,673 INFO [manager] checklist updated with [log rotation] items from rotate_logs worker
2026-03-13 03:35:15,447 INFO [make_surface_current_tiles] finished PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast2/12mar26
2026-03-13 03:35:15,471 INFO [make_surface_current_tiles] finished rendering of tiles for 2026-03-12 forecast2 into /results/nowcast-sys/figures/surface_currents/forecast2/12mar26
2026-03-13 03:35:15,472 INFO [make_surface_current_tiles] surface current tile figures for 2026-03-12 forecast2 completed
2026-03-13 03:35:15,945 INFO [manager] checklist updated with [surface current tiles] items from make_surface_current_tiles worker
2026-03-13 03:36:19,465 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 390, 72.2% complete
2026-03-13 03:41:19,467 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 495, 91.7% complete
2026-03-13 03:46:19,469 INFO [watch_ww3] forecast2 WWATCH3 run on arbutus.cloud-nowcast completed
2026-03-13 03:46:19,472 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2026-03-13 03:46:19,846 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2026-03-13'], host='localhost')
2026-03-13 03:46:21,687 INFO [download_wwatch3_results] running in process 1325568
2026-03-13 03:46:21,689 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-13 03:46:21,690 INFO [download_wwatch3_results] publishing log messages to tcp://*:5564
2026-03-13 03:46:21,691 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2026-03-13 03:46:58,085 INFO [download_wwatch3_results] forecast2 2026-03-13 results files from arbutus.cloud-nowcast downloaded
2026-03-13 03:46:58,096 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2026-03-13 03:46:58,256 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['wwatch3', 'forecast2', '--run-date', '2026-03-13'], host='localhost')
2026-03-13 03:46:59,568 INFO [update_forecast_datasets] running in process 1325585
2026-03-13 03:46:59,569 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-13 03:46:59,571 INFO [update_forecast_datasets] publishing log messages to tcp://*:5564
2026-03-13 03:46:59,572 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2026-03-13 03:46:59,572 INFO [update_forecast_datasets] updating wwatch3 most_recent_forecast directory from forecast2/13mar26 run
2026-03-13 03:46:59,601 INFO [update_forecast_datasets] updated wwatch3 most_recent_forecast directory from forecast2/13mar26 run
2026-03-13 03:46:59,602 INFO [update_forecast_datasets] updating wwatch3 forecast directory for forecast2/13mar26 run
2026-03-13 03:47:25,855 INFO [update_forecast_datasets] updated wwatch3 forecast directory for forecast2/13mar26 run: /results/SalishSea/rolling-forecasts/wwatch3
2026-03-13 03:47:25,856 INFO [update_forecast_datasets] wwatch3 2026-03-13 forecast2 rolling forecast datasets updated
2026-03-13 03:47:25,859 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2026-03-13 03:47:26,019 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['wwatch3-forecast'], host='localhost')
2026-03-13 03:47:27,249 INFO [ping_erddap] running in process 1325600
2026-03-13 03:47:27,251 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-13 03:47:27,252 INFO [ping_erddap] publishing log messages to tcp://*:5564
2026-03-13 03:47:27,253 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2026-03-13 03:47:27,254 INFO [ping_erddap] wwatch3-forecast ERDDAP dataset flag file(s) created
2026-03-13 03:47:27,256 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2026-03-13 03:47:27,405 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['wwatch3', 'forecast2', 'publish', '--run-date', '2026-03-13'], host='localhost')
2026-03-13 03:47:32,475 INFO [make_plots] running in process 1325608
2026-03-13 03:47:32,476 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-13 03:47:32,477 INFO [make_plots] publishing log messages to tcp://*:5564
2026-03-13 03:47:32,478 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2026-03-13 03:47:32,479 INFO [make_plots] preparing render list for 2026-03-13 WaveWatch3 forecast2 publish figures
2026-03-13 03:47:32,480 INFO [make_plots] added Halibut Bank figure to 2026-03-13 WaveWatch3 forecast2 publish render list
2026-03-13 03:47:32,480 INFO [make_plots] added Sentry Shoal figure to 2026-03-13 WaveWatch3 forecast2 publish render list
2026-03-13 03:47:32,481 INFO [make_plots] starting to render wwatch3 forecast2 publish 13mar26 figures
2026-03-13 03:47:33,332 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46146 Halibut Bank
2026-03-13 03:47:39,906 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46131 Sentry Shoal
2026-03-13 03:47:44,635 INFO [make_plots] finished rendering wwatch3 forecast2 publish 13mar26 figures
2026-03-13 03:47:44,637 INFO [make_plots] wwatch3 publish plots for 2026-03-13 forecast2 completed
2026-03-13 03:47:44,640 INFO [manager] checklist updated with [plots] items from make_plots worker
2026-03-13 08:44:29,086 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/12/ to /results/forcing/atmospheric/continental2.5/GRIB/20260313/12/
2026-03-13 08:44:29,114 INFO [collect_weather] 2026-03-13 2.5km weather forecast 12 collection complete
2026-03-13 08:44:29,185 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2026-03-13 08:44:29,734 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'SkagitMountVernon', '--data-date', '2026-03-12'], host='localhost')
2026-03-13 08:44:29,750 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'SnohomishMonroe', '--data-date', '2026-03-12'], host='localhost')
2026-03-13 08:44:29,761 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'NisquallyMcKenna', '--data-date', '2026-03-12'], host='localhost')
2026-03-13 08:44:29,765 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'GreenwaterGreenwater', '--data-date', '2026-03-12'], host='localhost')
2026-03-13 08:44:29,773 INFO [manager] launching NextWorker(module='nowcast.workers.make_turbidity_file', args=[], host='localhost')
2026-03-13 08:44:29,778 INFO [manager] launching NextWorker(module='nowcast.workers.collect_NeahBay_ssh', args=['06'], host='localhost')
2026-03-13 08:44:29,788 INFO [manager] launching NextWorker(module='nowcast.workers.download_live_ocean', args=[], host='localhost')
2026-03-13 08:44:29,803 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['18', '2.5km'], host='localhost')
2026-03-13 08:44:29,807 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['18'], host='localhost')
2026-03-13 08:44:31,741 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20260313/12/
2026-03-13 08:44:31,747 INFO [crop_gribs] 2026-03-13 12 GRIBs cropping complete
2026-03-13 08:44:31,757 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2026-03-13 08:44:32,107 INFO [manager] launching NextWorker(module='nowcast.workers.grib_to_netcdf', args=['nowcast+'], host='localhost')
2026-03-13 08:44:32,483 INFO [collect_weather] running in process 1329111
2026-03-13 08:44:32,484 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-13 08:44:32,486 INFO [collect_NeahBay_ssh] running in process 1329108
2026-03-13 08:44:32,489 INFO [collect_NeahBay_ssh] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-13 08:44:32,490 INFO [collect_NeahBay_ssh] publishing log messages to tcp://*:5563
2026-03-13 08:44:32,492 INFO [collect_NeahBay_ssh] connected to skookum.eos.ubc.ca port 5555
2026-03-13 08:44:32,494 INFO [collect_NeahBay_ssh] collecting Neah Bay ssh 06Z obs/forecast for 2026-03-13
2026-03-13 08:44:32,496 INFO [collect_weather] publishing log messages to tcp://*:5562
2026-03-13 08:44:32,497 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2026-03-13 08:44:32,522 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/18/
2026-03-13 08:44:32,533 INFO [download_live_ocean] running in process 1329110
2026-03-13 08:44:32,536 INFO [download_live_ocean] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-13 08:44:32,537 INFO [download_live_ocean] publishing log messages to tcp://*:5564
2026-03-13 08:44:32,538 INFO [download_live_ocean] connected to skookum.eos.ubc.ca port 5555
2026-03-13 08:44:32,540 INFO [download_live_ocean] downloading Salish Sea western boundary day-averaged LiveOcean file for 2026-03-13
2026-03-13 08:44:33,510 INFO [download_live_ocean] downloaded 10222917 bytes from https://s3.kopah.uw.edu/liveocean-share/f2026.03.13/ubc.nc to /results/forcing/LiveOcean/downloaded/20260313/low_passed_UBC.nc
2026-03-13 08:44:33,516 INFO [download_live_ocean] Deflating in up to 1 concurrent sub-processes
2026-03-13 08:44:33,877 INFO [collect_river_data] running in process 1329104
2026-03-13 08:44:33,878 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-13 08:44:33,879 INFO [collect_river_data] publishing log messages to tcp://*:5565
2026-03-13 08:44:33,880 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-13 08:44:33,881 INFO [collect_river_data] Collecting USGS SnohomishMonroe river data for 2026-03-12
2026-03-13 08:44:33,883 INFO [collect_river_data] running in process 1329105
2026-03-13 08:44:33,884 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-13 08:44:33,886 INFO [collect_river_data] publishing log messages to tcp://*:5566
2026-03-13 08:44:33,887 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-13 08:44:33,888 INFO [collect_river_data] Collecting USGS NisquallyMcKenna river data for 2026-03-12
2026-03-13 08:44:33,892 INFO [collect_river_data] running in process 1329106
2026-03-13 08:44:33,893 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-13 08:44:33,894 INFO [collect_river_data] publishing log messages to tcp://*:5567
2026-03-13 08:44:33,896 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-13 08:44:33,897 INFO [collect_river_data] Collecting USGS GreenwaterGreenwater river data for 2026-03-12
2026-03-13 08:44:33,902 INFO [collect_river_data] running in process 1329103
2026-03-13 08:44:33,903 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-13 08:44:33,905 INFO [collect_river_data] publishing log messages to tcp://*:5568
2026-03-13 08:44:33,906 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-13 08:44:33,908 INFO [collect_river_data] Collecting USGS SkagitMountVernon river data for 2026-03-12
2026-03-13 08:44:34,102 INFO [make_turbidity_file] running in process 1329107
2026-03-13 08:44:34,103 INFO [make_turbidity_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-13 08:44:34,104 INFO [make_turbidity_file] publishing log messages to tcp://*:5569
2026-03-13 08:44:34,105 INFO [make_turbidity_file] connected to skookum.eos.ubc.ca port 5555
2026-03-13 08:44:34,106 INFO [make_turbidity_file] Creating Fraser River turbidity forcing file for 2026-03-13
2026-03-13 08:44:34,180 INFO [collect_NeahBay_ssh] 2026-03-13T00:00:00-07:00 Neah Bay ssh 06Z obs/forecast data collection complete
2026-03-13 08:44:34,182 INFO [manager] checklist updated with [Neah Bay ssh data] items from collect_NeahBay_ssh worker
2026-03-13 08:44:34,374 INFO [manager] launching NextWorker(module='nowcast.workers.make_ssh_files', args=['nowcast', '--run-date', '2026-03-13'], host='localhost')
2026-03-13 08:44:34,463 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12089500&startDT=2026-03-12&endDT=2026-03-12 "HTTP/1.1 200 "
2026-03-13 08:44:34,469 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12097500&startDT=2026-03-12&endDT=2026-03-12 "HTTP/1.1 200 "
2026-03-13 08:44:34,470 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12150800&startDT=2026-03-12&endDT=2026-03-12 "HTTP/1.1 200 "
2026-03-13 08:44:34,549 INFO [download_live_ocean] netCDF4 deflated /results/forcing/LiveOcean/downloaded/20260313/low_passed_UBC.nc
2026-03-13 08:44:34,550 INFO [download_live_ocean] 2026-03-13 Live Ocean file for Salish Sea western boundary downloaded
2026-03-13 08:44:34,552 INFO [manager] checklist updated with [Live Ocean products] items from download_live_ocean worker
2026-03-13 08:44:34,564 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12200500&startDT=2026-03-12&endDT=2026-03-12 "HTTP/1.1 200 "
2026-03-13 08:44:34,584 INFO [crop_gribs] running in process 1329112
2026-03-13 08:44:34,586 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-13 08:44:34,587 INFO [crop_gribs] publishing log messages to tcp://*:5570
2026-03-13 08:44:34,588 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2026-03-13 08:44:34,589 INFO [crop_gribs] cropping 2026-03-13 ECCC HRDPS 2.5km continental 18Z GRIB files to SalishSeaCast subdomain
2026-03-13 08:44:34,604 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20260313/18/
2026-03-13 08:44:34,622 INFO [collect_river_data] Appended USGS SnohomishMonroe river average discharge for 2026-03-12 to: /results/forcing/rivers/observations/Snohomish_Monroe_flow
2026-03-13 08:44:34,623 INFO [collect_river_data] USGS SnohomishMonroe river data collection for 2026-03-12 completed
2026-03-13 08:44:34,626 INFO [collect_river_data] Appended USGS SkagitMountVernon river average discharge for 2026-03-12 to: /results/forcing/rivers/observations/Skagit_MountVernon_flow
2026-03-13 08:44:34,627 INFO [collect_river_data] Appended USGS NisquallyMcKenna river average discharge for 2026-03-12 to: /results/forcing/rivers/observations/Nisqually_McKenna_flow
2026-03-13 08:44:34,628 INFO [collect_river_data] USGS SkagitMountVernon river data collection for 2026-03-12 completed
2026-03-13 08:44:34,630 INFO [collect_river_data] USGS NisquallyMcKenna river data collection for 2026-03-12 completed
2026-03-13 08:44:34,642 INFO [grib_to_netcdf] running in process 1329196
2026-03-13 08:44:34,645 INFO [grib_to_netcdf] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-13 08:44:34,645 INFO [grib_to_netcdf] publishing log messages to tcp://*:5563
2026-03-13 08:44:34,647 INFO [grib_to_netcdf] connected to skookum.eos.ubc.ca port 5555
2026-03-13 08:44:34,648 INFO [grib_to_netcdf] creating NEMO-atmos forcing files for 2026-03-13 nowcast and forecast runs
2026-03-13 08:44:34,673 INFO [collect_river_data] Appended USGS GreenwaterGreenwater river average discharge for 2026-03-12 to: /results/forcing/rivers/observations/Greenwater_Greenwater_flow
2026-03-13 08:44:34,674 INFO [collect_river_data] USGS GreenwaterGreenwater river data collection for 2026-03-12 completed
2026-03-13 08:44:34,935 INFO [manager] launching NextWorker(module='nowcast.workers.make_live_ocean_files', args=['--run-date', '2026-03-13'], host='localhost')
2026-03-13 08:44:34,943 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-13 08:44:35,217 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-13 08:44:35,436 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-13 08:44:35,645 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-13 08:44:37,217 INFO [make_turbidity_file] 2026-03-13 Fraser River turbidity file creation complete
2026-03-13 08:44:37,220 INFO [manager] checklist updated with [Fraser River turbidity] items from make_turbidity_file worker
2026-03-13 08:44:38,208 INFO [make_live_ocean_files] running in process 1329262
2026-03-13 08:44:38,209 INFO [make_live_ocean_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-13 08:44:38,210 INFO [make_live_ocean_files] publishing log messages to tcp://*:5564
2026-03-13 08:44:38,211 INFO [make_live_ocean_files] connected to skookum.eos.ubc.ca port 5555
2026-03-13 08:44:38,212 INFO [make_live_ocean_files] Creating T&S western boundary conditions file from 2026-03-13 Live Ocean run
2026-03-13 08:44:38,241 INFO [make_ssh_files] running in process 1329244
2026-03-13 08:44:38,243 INFO [make_ssh_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-13 08:44:38,244 INFO [make_ssh_files] publishing log messages to tcp://*:5565
2026-03-13 08:44:38,245 INFO [make_ssh_files] connected to skookum.eos.ubc.ca port 5555
2026-03-13 08:44:38,246 INFO [make_ssh_files] building nowcast sea surface height boundary conditions file(s) from 2026-03-13 NOAA Neah Bay 06Z observation and forecast values
2026-03-13 08:44:40,422 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2026m03d09.nc
2026-03-13 08:44:40,639 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2026m03d10.nc
2026-03-13 08:44:40,860 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2026m03d11.nc
2026-03-13 08:44:41,063 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2026m03d12.nc
2026-03-13 08:44:41,301 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2026m03d13.nc
2026-03-13 08:44:41,570 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2026m03d14.nc
2026-03-13 08:44:41,819 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2026m03d15.nc
2026-03-13 08:44:42,081 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2026m03d16.nc
2026-03-13 08:44:42,322 INFO [make_ssh_files] sea surface height boundary file for 2026-03-13 nowcast run created
2026-03-13 08:44:42,489 INFO [manager] checklist updated with [sea surface height forcing] items from make_ssh_files worker
2026-03-13 08:44:42,645 INFO [manager] launching NextWorker(module='nowcast.workers.make_runoff_file', args=['v202108'], host='localhost')
2026-03-13 08:44:42,654 INFO [manager] launching NextWorker(module='nowcast.workers.make_201702_runoff_file', args=[], host='localhost')
2026-03-13 08:44:44,269 INFO [make_201702_runoff_file] running in process 1329333
2026-03-13 08:44:44,270 INFO [make_201702_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-13 08:44:44,271 INFO [make_201702_runoff_file] publishing log messages to tcp://*:5565
2026-03-13 08:44:44,272 INFO [make_201702_runoff_file] connected to skookum.eos.ubc.ca port 5555
2026-03-13 08:44:44,311 INFO [make_201702_runoff_file] calculating runoff forcing for b201702 bathymetry
2026-03-13 08:44:44,463 INFO [make_201702_runoff_file] 2026-03-13 runoff file creation from Fraser at Hope and climatology elsewhere complete
2026-03-13 08:44:44,465 INFO [manager] checklist updated with [rivers forcing] items from make_201702_runoff_file worker
2026-03-13 08:44:44,913 INFO [make_runoff_file] running in process 1329331
2026-03-13 08:44:44,914 INFO [make_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-13 08:44:44,915 INFO [make_runoff_file] publishing log messages to tcp://*:5565
2026-03-13 08:44:44,917 INFO [make_runoff_file] connected to skookum.eos.ubc.ca port 5555
2026-03-13 08:44:44,919 INFO [make_runoff_file] calculating NEMO runoff forcing for v202108 bathymetry for 2026-03-12
2026-03-13 08:44:48,329 INFO [make_runoff_file] stored NEMO runoff forcing for v202108 bathymetry for 2026-03-12: /results/forcing/rivers/R202108Dailies_y2026m03d12.nc
2026-03-13 08:44:48,330 INFO [make_runoff_file] 2026-03-12 runoff file creation completed
2026-03-13 08:44:48,332 INFO [manager] checklist updated with [rivers forcing] items from make_runoff_file worker
2026-03-13 08:44:50,692 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/hrdps_y2026m03d13.nc
2026-03-13 08:45:01,773 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2026m03d14.nc
2026-03-13 08:45:07,503 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2026m03d15.nc
2026-03-13 08:45:07,518 INFO [grib_to_netcdf] 2026-03-13 NEMO-atmos forcing file for nowcast+ created
2026-03-13 08:45:07,520 INFO [manager] checklist updated with [weather forcing] items from grib_to_netcdf worker
2026-03-13 08:45:26,119 INFO [make_live_ocean_files] Stored T&S western boundary conditions file: /results/forcing/LiveOcean/boundary_conditions/LiveOcean_v201905_y2026m03d13.nc
2026-03-13 08:45:26,120 INFO [make_live_ocean_files] 2026-03-13 Live Ocean western boundary conditions files created
2026-03-13 08:45:26,122 INFO [manager] checklist updated with [Live Ocean boundary conditions] items from make_live_ocean_files worker
2026-03-13 08:45:26,283 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['weather'], host='localhost')
2026-03-13 08:45:26,289 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'nowcast+'], host='localhost')
2026-03-13 08:45:26,293 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'nowcast+'], host='localhost')
2026-03-13 08:45:26,299 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.nibi', 'nowcast+'], host='localhost')
2026-03-13 08:45:26,306 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'nowcast+'], host='localhost')
2026-03-13 08:45:28,186 INFO [ping_erddap] running in process 1329410
2026-03-13 08:45:28,188 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-13 08:45:28,189 INFO [ping_erddap] publishing log messages to tcp://*:5563
2026-03-13 08:45:28,190 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2026-03-13 08:45:28,192 INFO [ping_erddap] weather ERDDAP dataset flag file(s) created
2026-03-13 08:45:28,196 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2026-03-13 08:45:28,263 INFO [upload_forcing] running in process 1329414
2026-03-13 08:45:28,264 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-13 08:45:28,266 INFO [upload_forcing] publishing log messages to tcp://*:5564
2026-03-13 08:45:28,267 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-03-13 08:45:28,270 INFO [upload_forcing] running in process 1329412
2026-03-13 08:45:28,271 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-13 08:45:28,273 INFO [upload_forcing] publishing log messages to tcp://*:5565
2026-03-13 08:45:28,274 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-03-13 08:45:28,286 INFO [upload_forcing] running in process 1329413
2026-03-13 08:45:28,287 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-13 08:45:28,288 INFO [upload_forcing] publishing log messages to tcp://*:5566
2026-03-13 08:45:28,289 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-03-13 08:45:28,303 INFO [upload_forcing] running in process 1329415
2026-03-13 08:45:28,304 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-13 08:45:28,305 INFO [upload_forcing] publishing log messages to tcp://*:5567
2026-03-13 08:45:28,306 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-03-13 08:45:30,231 INFO [upload_forcing] nowcast+ 2026-03-13 forcing files upload to optimum-hindcast completed
2026-03-13 08:45:30,235 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-03-13 08:45:32,110 INFO [upload_forcing] nowcast+ 2026-03-13 forcing files upload to arbutus.cloud-nowcast completed
2026-03-13 08:45:32,115 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-03-13 08:45:32,269 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'nowcast+', '--run-date', '2026-03-13'], host='localhost')
2026-03-13 08:45:34,147 INFO [make_forcing_links] running in process 1329450
2026-03-13 08:45:34,148 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-13 08:45:34,149 INFO [make_forcing_links] publishing log messages to tcp://*:5563
2026-03-13 08:45:34,150 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2026-03-13 08:45:35,675 INFO [upload_forcing] nowcast+ 2026-03-13 forcing files upload to orcinus-nowcast-agrif completed
2026-03-13 08:45:35,678 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-03-13 08:45:36,667 INFO [make_forcing_links] nowcast+ 2026-03-13 forcing file links on arbutus.cloud-nowcast created
2026-03-13 08:45:36,674 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2026-03-13 08:45:36,830 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2026-03-13'], host='arbutus.cloud-nowcast')
2026-03-13 08:45:38,851 INFO [upload_forcing] nowcast+ 2026-03-13 forcing files upload to robot.nibi completed
2026-03-13 08:45:38,856 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-03-13 08:45:38,904 INFO [run_NEMO] running in process 32501
2026-03-13 08:45:38,905 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-13 08:45:38,906 INFO [run_NEMO] publishing log messages to tcp://*:5556
2026-03-13 08:45:38,966 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2026-03-13 08:45:39,653 WARNING [run_NEMO] There are uncommitted changes in /nemoShare/MEOPAR/nowcast-sys/SS-run-sets
2026-03-13 08:45:39,961 INFO [run_NEMO] nowcast: launching /nemoShare/MEOPAR/nowcast-sys/runs/13mar26nowcast_2026-03-13T084538.945596-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2026-03-13 08:45:39,975 INFO [run_NEMO] nowcast NEMO run for 2026-03-13 on arbutus.cloud-nowcast started
2026-03-13 08:45:39,979 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2026-03-13 08:45:40,134 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'nowcast'], host='arbutus.cloud-nowcast')
2026-03-13 08:45:42,144 INFO [watch_NEMO] running in process 310
2026-03-13 08:45:42,146 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-13 08:45:42,147 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2026-03-13 08:45:42,148 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2026-03-13 08:45:42,280 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2026-03-13 08:50:42,283 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 19094864 = 2026-03-13 05:08:40 UTC, 21.4% complete
2026-03-13 08:55:42,285 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 19095394 = 2026-03-13 11:02:00 UTC, 46.0% complete
2026-03-13 09:00:42,282 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 19095968 = 2026-03-13 17:24:40 UTC, 72.6% complete
2026-03-13 09:05:42,279 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 19096535 = 2026-03-13 23:42:40 UTC, 98.8% complete
2026-03-13 09:10:42,281 INFO [watch_NEMO] nowcast NEMO run on arbutus.cloud-nowcast completed
2026-03-13 09:10:42,285 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2026-03-13 09:10:42,791 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'ssh', '--run-date', '2026-03-13'], host='localhost')
2026-03-13 09:10:42,801 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2026-03-13'], host='localhost')
2026-03-13 09:10:44,857 INFO [download_results] running in process 1329718
2026-03-13 09:10:44,858 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-13 09:10:44,859 INFO [download_results] publishing log messages to tcp://*:5563
2026-03-13 09:10:44,861 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2026-03-13 09:10:44,861 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/nowcast/13mar26 to /results/SalishSea/nowcast-blue.202111
2026-03-13 09:10:44,941 INFO [make_forcing_links] running in process 1329717
2026-03-13 09:10:44,942 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-13 09:10:44,943 INFO [make_forcing_links] publishing log messages to tcp://*:5564
2026-03-13 09:10:44,943 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2026-03-13 09:10:45,904 INFO [make_forcing_links] ssh 2026-03-13 forcing file links on arbutus.cloud-nowcast created
2026-03-13 09:10:45,912 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2026-03-13 09:10:46,093 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2026-03-13'], host='arbutus.cloud-nowcast')
2026-03-13 09:10:48,196 INFO [run_NEMO] running in process 833
2026-03-13 09:10:48,197 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-13 09:10:48,198 INFO [run_NEMO] publishing log messages to tcp://*:5556
2026-03-13 09:10:48,199 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2026-03-13 09:10:48,957 WARNING [run_NEMO] There are uncommitted changes in /nemoShare/MEOPAR/nowcast-sys/SS-run-sets
2026-03-13 09:10:49,273 INFO [run_NEMO] forecast: launching /nemoShare/MEOPAR/nowcast-sys/runs/13mar26forecast_2026-03-13T091048.267200-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2026-03-13 09:10:49,286 INFO [run_NEMO] forecast NEMO run for 2026-03-13 on arbutus.cloud-nowcast started
2026-03-13 09:10:49,290 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2026-03-13 09:10:49,457 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'forecast'], host='arbutus.cloud-nowcast')
2026-03-13 09:10:51,487 INFO [watch_NEMO] running in process 1124
2026-03-13 09:10:51,489 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-13 09:10:51,490 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2026-03-13 09:10:51,491 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2026-03-13 09:10:51,559 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2026-03-13 09:11:20,488 INFO [download_results] nowcast 2026-03-13 results files from arbutus.cloud-nowcast downloaded
2026-03-13 09:11:20,492 INFO [manager] checklist updated with [results files] items from download_results worker
2026-03-13 09:11:20,658 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast', 'research', '--run-date', '2026-03-13'], host='localhost')
2026-03-13 09:11:20,663 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast', 'comparison', '--run-date', '2026-03-12'], host='localhost')
2026-03-13 09:11:20,669 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['nowcast', '--run-date', '2026-03-13'], host='localhost')
2026-03-13 09:11:24,545 INFO [make_CHS_currents_file] running in process 1329740
2026-03-13 09:11:24,548 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-13 09:11:24,550 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5563
2026-03-13 09:11:24,553 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555
2026-03-13 09:11:26,791 INFO [make_plots] running in process 1329739
2026-03-13 09:11:26,792 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-13 09:11:26,793 INFO [make_plots] publishing log messages to tcp://*:5564
2026-03-13 09:11:26,794 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2026-03-13 09:11:26,795 INFO [make_plots] running in process 1329737
2026-03-13 09:11:26,797 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-13 09:11:26,809 INFO [make_plots] publishing log messages to tcp://*:5565
2026-03-13 09:11:26,810 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2026-03-13 09:11:28,295 INFO [make_plots] preparing render list for 2026-03-13 NEMO nowcast-blue research figures
2026-03-13 09:11:28,296 INFO [make_plots] preparing render list for 2026-03-12 NEMO nowcast-blue comparison figures
2026-03-13 09:11:28,404 INFO [make_plots] added SH_wind figure to 2026-03-12 NEMO nowcast-blue comparison render list
2026-03-13 09:11:28,405 INFO [make_plots] added Compare_VENUS_East figure to 2026-03-12 NEMO nowcast-blue comparison render list
2026-03-13 09:11:28,406 INFO [make_plots] added Compare_VENUS_Central figure to 2026-03-12 NEMO nowcast-blue comparison render list
2026-03-13 09:11:28,407 INFO [make_plots] starting to render nemo nowcast comparison 12mar26 figures
2026-03-13 09:11:29,697 INFO [make_plots] added salinity_thalweg_and_surface figures to 2026-03-13 NEMO nowcast-blue research render list
2026-03-13 09:11:30,744 INFO [make_plots] added temperature_thalweg_and_surface figures to 2026-03-13 NEMO nowcast-blue research render list
2026-03-13 09:11:30,747 INFO [make_plots] added Currents_sections_and_surface figure to 2026-03-13 NEMO nowcast-blue research render list
2026-03-13 09:11:30,748 INFO [make_plots] added Currents_at_VENUS_Central figure to 2026-03-13 NEMO nowcast-blue research render list
2026-03-13 09:11:30,749 INFO [make_plots] added Currents_at_VENUS_East figure to 2026-03-13 NEMO nowcast-blue research render list
2026-03-13 09:11:30,750 INFO [make_plots] starting to render nemo nowcast research 13mar26 figures
2026-03-13 09:12:00,680 INFO [make_CHS_currents_file] Made CHS currents file for 2026-03-13 for nowcast
2026-03-13 09:12:00,683 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker
2026-03-13 09:12:34,875 INFO [make_plots] finished rendering nemo nowcast research 13mar26 figures
2026-03-13 09:12:34,877 INFO [make_plots] nemo research plots for 2026-03-13 nowcast completed
2026-03-13 09:12:34,891 INFO [manager] checklist updated with [plots] items from make_plots worker
2026-03-13 09:13:05,377 INFO [make_plots] finished rendering nemo nowcast comparison 12mar26 figures
2026-03-13 09:13:05,380 INFO [make_plots] nemo comparison plots for 2026-03-12 nowcast completed
2026-03-13 09:13:05,384 INFO [manager] checklist updated with [plots] items from make_plots worker
2026-03-13 09:15:51,556 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 19097052 = 2026-03-14 05:27:20 UTC, 15.2% complete
2026-03-13 09:20:51,554 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 19097576 = 2026-03-14 11:16:40 UTC, 31.3% complete
2026-03-13 09:25:51,556 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 19098104 = 2026-03-14 17:08:40 UTC, 47.6% complete
2026-03-13 09:30:51,560 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 19098662 = 2026-03-14 23:20:40 UTC, 64.9% complete
2026-03-13 09:35:51,563 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 19099194 = 2026-03-15 05:15:20 UTC, 81.3% complete
2026-03-13 09:40:51,565 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 19099775 = 2026-03-15 11:42:40 UTC, 99.2% complete
2026-03-13 09:45:51,570 INFO [watch_NEMO] forecast NEMO run on arbutus.cloud-nowcast completed
2026-03-13 09:45:51,578 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2026-03-13 09:45:51,979 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_wind_file', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2026-03-13'], host='arbutus.cloud-nowcast')
2026-03-13 09:45:51,993 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_current_file', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2026-03-13'], host='arbutus.cloud-nowcast')
2026-03-13 09:45:51,997 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'turbidity', '--run-date', '2026-03-13'], host='localhost')
2026-03-13 09:45:52,006 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'turbidity', '--run-date', '2026-03-13'], host='localhost')
2026-03-13 09:45:52,012 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.nibi', 'turbidity', '--run-date', '2026-03-13'], host='localhost')
2026-03-13 09:45:52,022 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'turbidity', '--run-date', '2026-03-13'], host='localhost')
2026-03-13 09:45:52,027 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2026-03-13'], host='localhost')
2026-03-13 09:45:54,134 INFO [download_results] running in process 1330355
2026-03-13 09:45:54,135 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-13 09:45:54,136 INFO [download_results] publishing log messages to tcp://*:5563
2026-03-13 09:45:54,137 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2026-03-13 09:45:54,138 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/forecast/13mar26 to /results/SalishSea/forecast.202111
2026-03-13 09:45:54,191 INFO [upload_forcing] running in process 1330354
2026-03-13 09:45:54,192 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-13 09:45:54,193 INFO [upload_forcing] publishing log messages to tcp://*:5564
2026-03-13 09:45:54,194 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-03-13 09:45:54,203 INFO [upload_forcing] running in process 1330351
2026-03-13 09:45:54,204 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-13 09:45:54,205 INFO [upload_forcing] publishing log messages to tcp://*:5565
2026-03-13 09:45:54,206 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-03-13 09:45:54,208 INFO [upload_forcing] running in process 1330353
2026-03-13 09:45:54,209 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-13 09:45:54,210 INFO [upload_forcing] publishing log messages to tcp://*:5566
2026-03-13 09:45:54,211 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-03-13 09:45:54,213 INFO [upload_forcing] running in process 1330352
2026-03-13 09:45:54,214 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-13 09:45:54,215 INFO [upload_forcing] publishing log messages to tcp://*:5567
2026-03-13 09:45:54,216 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-03-13 09:45:54,844 INFO [make_ww3_current_file] running in process 2106
2026-03-13 09:45:54,845 INFO [make_ww3_current_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-13 09:45:54,855 INFO [make_ww3_current_file] publishing log messages to tcp://*:5571
2026-03-13 09:45:54,873 INFO [make_ww3_current_file] connected to skookum.eos.ubc.ca port 5555
2026-03-13 09:45:54,891 INFO [make_ww3_current_file] Creating wwatch3 currents forcing file for 2026-03-13 forecast run
2026-03-13 09:45:54,951 INFO [upload_forcing] turbidity 2026-03-13 forcing files upload to optimum-hindcast completed
2026-03-13 09:45:54,957 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-03-13 09:45:55,285 INFO [upload_forcing] turbidity 2026-03-13 forcing files upload to arbutus.cloud-nowcast completed
2026-03-13 09:45:55,289 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-03-13 09:45:55,452 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2026-03-13'], host='localhost')
2026-03-13 09:45:56,435 INFO [upload_forcing] turbidity 2026-03-13 forcing files upload to orcinus-nowcast-agrif completed
2026-03-13 09:45:56,439 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-03-13 09:45:57,668 INFO [make_forcing_links] running in process 1330387
2026-03-13 09:45:57,669 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-13 09:45:57,670 INFO [make_forcing_links] publishing log messages to tcp://*:5564
2026-03-13 09:45:57,671 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2026-03-13 09:45:58,642 INFO [upload_forcing] turbidity 2026-03-13 forcing files upload to robot.nibi completed
2026-03-13 09:45:58,646 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-03-13 09:46:00,512 INFO [make_forcing_links] nowcast-green 2026-03-13 forcing file links on arbutus.cloud-nowcast created
2026-03-13 09:46:00,516 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2026-03-13 09:46:00,680 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2026-03-13'], host='arbutus.cloud-nowcast')
2026-03-13 09:46:05,516 INFO [run_NEMO] running in process 2748
2026-03-13 09:46:05,516 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-13 09:46:05,518 INFO [run_NEMO] publishing log messages to tcp://*:5556
2026-03-13 09:46:05,519 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2026-03-13 09:46:06,113 INFO [manager] checklist updated with [WW3 wind forcing] items from make_ww3_wind_file worker
2026-03-13 09:46:06,117 INFO [make_ww3_wind_file] wwatch3 wind forcing file created on arbutus.cloud-nowcast for 2026-03-13 forecast run
2026-03-13 09:46:06,342 WARNING [run_NEMO] There are uncommitted changes in /nemoShare/MEOPAR/nowcast-sys/SS-run-sets
2026-03-13 09:46:06,680 INFO [run_NEMO] nowcast-green: launching /nemoShare/MEOPAR/nowcast-sys/runs/13mar26nowcast-green_2026-03-13T094605.546524-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2026-03-13 09:46:06,700 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2026-03-13 09:46:06,702 INFO [run_NEMO] nowcast-green NEMO run for 2026-03-13 on arbutus.cloud-nowcast started
2026-03-13 09:46:06,883 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'nowcast-green'], host='arbutus.cloud-nowcast')
2026-03-13 09:46:09,001 INFO [watch_NEMO] running in process 3025
2026-03-13 09:46:09,003 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-13 09:46:09,004 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2026-03-13 09:46:09,005 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2026-03-13 09:46:09,088 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2026-03-13 09:46:55,395 INFO [download_results] forecast 2026-03-13 results files from arbutus.cloud-nowcast downloaded
2026-03-13 09:46:55,560 INFO [manager] checklist updated with [results files] items from download_results worker
2026-03-13 09:46:55,726 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['forecast', '--run-date', '2026-03-13'], host='localhost')
2026-03-13 09:47:00,147 INFO [make_CHS_currents_file] running in process 1330409
2026-03-13 09:47:00,148 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-13 09:47:00,150 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5563
2026-03-13 09:47:00,151 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555
2026-03-13 09:47:49,183 INFO [make_CHS_currents_file] Made CHS currents file for 2026-03-13 for forecast
2026-03-13 09:47:49,186 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker
2026-03-13 09:47:49,355 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['nemo', 'forecast', '--run-date', '2026-03-13'], host='localhost')
2026-03-13 09:47:51,194 INFO [update_forecast_datasets] running in process 1330445
2026-03-13 09:47:51,195 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-13 09:47:51,196 INFO [update_forecast_datasets] publishing log messages to tcp://*:5563
2026-03-13 09:47:51,197 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2026-03-13 09:47:51,198 INFO [update_forecast_datasets] updating nemo forecast directory for forecast/13mar26 run
2026-03-13 09:47:51,492 INFO [update_forecast_datasets] updated nemo forecast directory for forecast/13mar26 run: /results/SalishSea/rolling-forecasts/nemo
2026-03-13 09:47:51,493 INFO [update_forecast_datasets] nemo 2026-03-13 forecast rolling forecast datasets updated
2026-03-13 09:47:51,495 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2026-03-13 09:47:51,659 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nemo-forecast'], host='localhost')
2026-03-13 09:47:51,665 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'forecast', 'publish', '--run-date', '2026-03-13'], host='localhost')
2026-03-13 09:47:51,671 INFO [manager] launching NextWorker(module='nowcast.workers.make_surface_current_tiles', args=['forecast', '--run-date', '2026-03-13'], host='localhost')
2026-03-13 09:47:53,374 INFO [ping_erddap] running in process 1330452
2026-03-13 09:47:53,375 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-13 09:47:53,376 INFO [ping_erddap] publishing log messages to tcp://*:5563
2026-03-13 09:47:53,377 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2026-03-13 09:47:53,391 INFO [ping_erddap] nemo-forecast ERDDAP dataset flag file(s) created
2026-03-13 09:47:53,393 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2026-03-13 09:47:56,088 INFO [make_surface_current_tiles] running in process 1330455
2026-03-13 09:47:56,092 INFO [make_surface_current_tiles] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-13 09:47:56,093 INFO [make_surface_current_tiles] publishing log messages to tcp://*:5563
2026-03-13 09:47:56,096 INFO [make_surface_current_tiles] connected to skookum.eos.ubc.ca port 5555
2026-03-13 09:47:56,154 INFO [make_surface_current_tiles] starting rendering of tiles for 2026-03-13 forecast in 6 processes into /results/nowcast-sys/figures/surface_currents/forecast/13mar26
2026-03-13 09:47:58,928 INFO [make_plots] running in process 1330454
2026-03-13 09:47:58,929 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-13 09:47:58,930 INFO [make_plots] publishing log messages to tcp://*:5564
2026-03-13 09:47:58,931 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2026-03-13 09:47:59,880 INFO [make_plots] preparing render list for 2026-03-13 NEMO forecast publish figures
2026-03-13 09:48:00,968 INFO [make_plots] added Website_thumbnail figure to 2026-03-13 NEMO forecast publish render list
2026-03-13 09:48:00,968 INFO [make_plots] added Threshold_website figure to 2026-03-13 NEMO forecast publish render list
2026-03-13 09:48:00,969 INFO [make_plots] added PA_tidal_predictions figure to 2026-03-13 NEMO forecast publish render list
2026-03-13 09:48:00,970 INFO [make_plots] added Boundary Bay figure to 2026-03-13 NEMO forecast publish render list
2026-03-13 09:48:00,971 INFO [make_plots] added Campbell River figure to 2026-03-13 NEMO forecast publish render list
2026-03-13 09:48:00,972 INFO [make_plots] added Cherry Point figure to 2026-03-13 NEMO forecast publish render list
2026-03-13 09:48:00,973 INFO [make_plots] added Friday Harbor figure to 2026-03-13 NEMO forecast publish render list
2026-03-13 09:48:00,974 INFO [make_plots] added Halfmoon Bay figure to 2026-03-13 NEMO forecast publish render list
2026-03-13 09:48:00,975 INFO [make_plots] added Nanaimo figure to 2026-03-13 NEMO forecast publish render list
2026-03-13 09:48:00,976 INFO [make_plots] added Neah Bay figure to 2026-03-13 NEMO forecast publish render list
2026-03-13 09:48:00,977 INFO [make_plots] added New Westminster figure to 2026-03-13 NEMO forecast publish render list
2026-03-13 09:48:00,978 INFO [make_plots] added Patricia Bay figure to 2026-03-13 NEMO forecast publish render list
2026-03-13 09:48:00,979 INFO [make_plots] added Point Atkinson figure to 2026-03-13 NEMO forecast publish render list
2026-03-13 09:48:00,980 INFO [make_plots] added Port Renfrew figure to 2026-03-13 NEMO forecast publish render list
2026-03-13 09:48:00,981 INFO [make_plots] added Sand Heads figure to 2026-03-13 NEMO forecast publish render list
2026-03-13 09:48:00,983 INFO [make_plots] added Sandy Cove figure to 2026-03-13 NEMO forecast publish render list
2026-03-13 09:48:00,984 INFO [make_plots] added Squamish figure to 2026-03-13 NEMO forecast publish render list
2026-03-13 09:48:00,985 INFO [make_plots] added Victoria figure to 2026-03-13 NEMO forecast publish render list
2026-03-13 09:48:00,986 INFO [make_plots] added Woodwards Landing figure to 2026-03-13 NEMO forecast publish render list
2026-03-13 09:48:00,987 INFO [make_plots] starting to render nemo forecast publish 13mar26 figures
2026-03-13 09:48:38,311 INFO [make_ww3_current_file] wwatch3 currents forcing file created on arbutus.cloud-nowcast for 2026-03-13 forecast run
2026-03-13 09:48:38,315 INFO [manager] checklist updated with [WW3 currents forcing] items from make_ww3_current_file worker
2026-03-13 09:48:38,634 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2026-03-13'], host='arbutus.cloud-nowcast')
2026-03-13 09:48:40,676 INFO [run_ww3] running in process 3157
2026-03-13 09:48:40,678 INFO [run_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-13 09:48:40,678 INFO [run_ww3] publishing log messages to tcp://*:5572
2026-03-13 09:48:40,679 INFO [run_ww3] connected to skookum.eos.ubc.ca port 5555
2026-03-13 09:48:40,683 INFO [run_ww3] Created run directory /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/nowcast_2026-03-13T094840.675489-0700
2026-03-13 09:48:40,684 INFO [run_ww3] nowcast: launching /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/nowcast_2026-03-13T094840.675489-0700/SoGWW3.sh on arbutus.cloud-nowcast
2026-03-13 09:48:40,702 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker
2026-03-13 09:48:40,704 INFO [run_ww3] nowcast WaveWatch3 run for 2026-03-13 on arbutus.cloud-nowcast started
2026-03-13 09:48:40,979 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'nowcast'], host='arbutus.cloud-nowcast')
2026-03-13 09:48:43,053 INFO [watch_ww3] running in process 3244
2026-03-13 09:48:43,054 INFO [watch_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-13 09:48:43,055 INFO [watch_ww3] publishing log messages to tcp://*:5573
2026-03-13 09:48:43,056 INFO [watch_ww3] connected to skookum.eos.ubc.ca port 5555
2026-03-13 09:48:43,094 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2026-03-13 09:51:09,089 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19094575 = 2026-03-13 01:56:00 UTC, 8.1% complete
2026-03-13 09:53:43,087 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 93, 21.5% complete
2026-03-13 09:56:09,091 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19094770 = 2026-03-13 04:06:00 UTC, 17.1% complete
2026-03-13 09:58:30,759 WARNING [make_plots] invalid station number for Boundary Bay station: None
2026-03-13 09:58:30,760 WARNING [make_plots] can't resolve a valid CHS station code for Boundary Bay
2026-03-13 09:58:43,089 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 201, 46.5% complete
2026-03-13 09:58:53,346 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb996/data for station 08074 Campbell River from 2026-03-08 00:00:00Z to 2026-03-15 11:55:00Z
2026-03-13 09:59:48,034 INFO [make_surface_current_tiles] starting PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast/13mar26
2026-03-13 09:59:58,725 WARNING [make_plots] invalid station number for Halfmoon Bay station: None
2026-03-13 09:59:58,725 WARNING [make_plots] can't resolve a valid CHS station code for Halfmoon Bay
2026-03-13 10:00:17,130 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb96d/data for station 07917 Nanaimo from 2026-03-08 00:00:00Z to 2026-03-15 11:55:00Z
2026-03-13 10:00:55,987 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e13d0f4a073c4bbf93/data for station 07654 New Westminster from 2026-03-08 00:00:00Z to 2026-03-15 11:55:00Z
2026-03-13 10:01:09,093 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19094966 = 2026-03-13 06:16:40 UTC, 26.2% complete
2026-03-13 10:01:17,000 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd2d/data for station 07277 Patricia Bay from 2026-03-08 00:00:00Z to 2026-03-15 11:55:00Z
2026-03-13 10:01:36,053 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94c/data for station 07795 Point Atkinson from 2026-03-08 00:00:00Z to 2026-03-15 11:55:00Z
2026-03-13 10:01:55,602 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e23d0f4a073c4bc060/data for station 08525 Port Renfrew from 2026-03-08 00:00:00Z to 2026-03-15 11:55:00Z
2026-03-13 10:02:13,394 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb93c/data for station 07594 Sand Heads from 2026-03-08 00:00:00Z to 2026-03-15 11:55:00Z
2026-03-13 10:02:31,473 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc43b/data for station 07786 Sandy Cove from 2026-03-08 00:00:00Z to 2026-03-15 11:55:00Z
2026-03-13 10:02:49,922 WARNING [make_plots] invalid station number for Squamish station: None
2026-03-13 10:02:49,923 WARNING [make_plots] can't resolve a valid CHS station code for Squamish
2026-03-13 10:03:08,847 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd1e/data for station 07120 Victoria from 2026-03-08 00:00:00Z to 2026-03-15 11:55:00Z
2026-03-13 10:03:27,901 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc42d/data for station 07610 Woodwards Landing from 2026-03-08 00:00:00Z to 2026-03-15 11:55:00Z
2026-03-13 10:03:43,090 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 306, 70.8% complete
2026-03-13 10:03:46,834 INFO [make_plots] finished rendering nemo forecast publish 13mar26 figures
2026-03-13 10:03:46,837 INFO [make_plots] nemo publish plots for 2026-03-13 forecast completed
2026-03-13 10:03:46,845 INFO [manager] checklist updated with [plots] items from make_plots worker
2026-03-13 10:03:47,018 INFO [manager] launching NextWorker(module='nowcast.workers.make_feeds', args=['forecast', '--run-date', '2026-03-13'], host='localhost')
2026-03-13 10:03:52,487 INFO [make_feeds] running in process 1335179
2026-03-13 10:03:52,488 INFO [make_feeds] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-13 10:03:52,489 INFO [make_feeds] publishing log messages to tcp://*:5564
2026-03-13 10:03:52,490 INFO [make_feeds] connected to skookum.eos.ubc.ca port 5555
2026-03-13 10:04:09,018 INFO [make_feeds] ATOM feeds for 2026-03-13 forecast run completed
2026-03-13 10:04:09,022 INFO [manager] checklist updated with [storm surge alert feeds] items from make_feeds worker
2026-03-13 10:05:31,788 INFO [make_surface_current_tiles] finished PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast/13mar26
2026-03-13 10:05:31,814 INFO [make_surface_current_tiles] finished rendering of tiles for 2026-03-13 forecast into /results/nowcast-sys/figures/surface_currents/forecast/13mar26
2026-03-13 10:05:31,815 INFO [make_surface_current_tiles] surface current tile figures for 2026-03-13 forecast completed
2026-03-13 10:05:32,140 INFO [manager] checklist updated with [surface current tiles] items from make_surface_current_tiles worker
2026-03-13 10:06:09,094 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19095161 = 2026-03-13 08:26:40 UTC, 35.2% complete
2026-03-13 10:08:43,092 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 408, 94.4% complete
2026-03-13 10:11:09,095 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19095357 = 2026-03-13 10:37:20 UTC, 44.3% complete
2026-03-13 10:13:43,092 INFO [watch_ww3] nowcast WWATCH3 run on arbutus.cloud-nowcast completed
2026-03-13 10:13:43,096 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2026-03-13 10:13:43,600 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2026-03-13'], host='localhost')
2026-03-13 10:13:43,608 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2026-03-13'], host='arbutus.cloud-nowcast')
2026-03-13 10:13:45,423 INFO [download_wwatch3_results] running in process 1335351
2026-03-13 10:13:45,424 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-13 10:13:45,425 INFO [download_wwatch3_results] publishing log messages to tcp://*:5563
2026-03-13 10:13:45,425 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2026-03-13 10:13:45,717 INFO [run_ww3] running in process 3777
2026-03-13 10:13:45,718 INFO [run_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-13 10:13:45,719 INFO [run_ww3] publishing log messages to tcp://*:5572
2026-03-13 10:13:45,720 INFO [run_ww3] connected to skookum.eos.ubc.ca port 5555
2026-03-13 10:13:45,724 INFO [run_ww3] Created run directory /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast_2026-03-13T101345.716267-0700
2026-03-13 10:13:45,725 INFO [run_ww3] forecast: launching /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast_2026-03-13T101345.716267-0700/SoGWW3.sh on arbutus.cloud-nowcast
2026-03-13 10:13:45,738 INFO [run_ww3] forecast WaveWatch3 run for 2026-03-13 on arbutus.cloud-nowcast started
2026-03-13 10:13:45,743 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker
2026-03-13 10:13:46,069 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'forecast'], host='arbutus.cloud-nowcast')
2026-03-13 10:13:48,091 INFO [watch_ww3] running in process 3934
2026-03-13 10:13:48,094 INFO [watch_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-13 10:13:48,095 INFO [watch_ww3] publishing log messages to tcp://*:5573
2026-03-13 10:13:48,096 INFO [watch_ww3] connected to skookum.eos.ubc.ca port 5555
2026-03-13 10:13:48,294 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2026-03-13 10:14:22,117 INFO [download_wwatch3_results] nowcast 2026-03-13 results files from arbutus.cloud-nowcast downloaded
2026-03-13 10:14:22,120 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2026-03-13 10:16:09,098 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19095552 = 2026-03-13 12:47:20 UTC, 53.3% complete
2026-03-13 10:18:48,295 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 93, 14.4% complete
2026-03-13 10:21:09,100 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19095747 = 2026-03-13 14:57:20 UTC, 62.3% complete
2026-03-13 10:23:48,298 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 192, 29.6% complete
2026-03-13 10:26:09,100 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19095942 = 2026-03-13 17:07:20 UTC, 71.4% complete
2026-03-13 10:28:48,299 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 291, 44.9% complete
2026-03-13 10:31:09,102 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19096137 = 2026-03-13 19:17:20 UTC, 80.4% complete
2026-03-13 10:33:48,300 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 390, 60.2% complete
2026-03-13 10:36:09,103 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19096332 = 2026-03-13 21:27:20 UTC, 89.4% complete
2026-03-13 10:38:48,301 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 498, 76.9% complete
2026-03-13 10:41:09,104 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19096527 = 2026-03-13 23:37:20 UTC, 98.5% complete
2026-03-13 10:43:48,342 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 603, 93.1% complete
2026-03-13 10:46:09,112 INFO [watch_NEMO] nowcast-green NEMO run on arbutus.cloud-nowcast completed
2026-03-13 10:46:09,116 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2026-03-13 10:46:09,738 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2026-03-13'], host='localhost')
2026-03-13 10:46:13,421 INFO [download_results] running in process 1335741
2026-03-13 10:46:13,444 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-13 10:46:13,450 INFO [download_results] publishing log messages to tcp://*:5563
2026-03-13 10:46:13,450 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2026-03-13 10:46:13,452 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/nowcast-green/13mar26 to /results2/SalishSea/nowcast-green.202111
2026-03-13 10:48:38,076 INFO [download_results] nowcast-green 2026-03-13 results files from arbutus.cloud-nowcast downloaded
2026-03-13 10:48:38,081 INFO [manager] checklist updated with [results files] items from download_results worker
2026-03-13 10:48:38,547 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast-green', 'research', '--run-date', '2026-03-13'], host='localhost')
2026-03-13 10:48:38,597 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nowcast-green'], host='localhost')
2026-03-13 10:48:38,604 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'physics', '--run-date', '2026-03-13'], host='localhost')
2026-03-13 10:48:38,612 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'biology', '--run-date', '2026-03-13'], host='localhost')
2026-03-13 10:48:38,617 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'chemistry', '--run-date', '2026-03-13'], host='localhost')
2026-03-13 10:48:41,217 INFO [ping_erddap] running in process 1335782
2026-03-13 10:48:41,219 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-13 10:48:41,220 INFO [ping_erddap] publishing log messages to tcp://*:5563
2026-03-13 10:48:41,221 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2026-03-13 10:48:41,226 INFO [ping_erddap] nowcast-green ERDDAP dataset flag file(s) created
2026-03-13 10:48:41,227 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2026-03-13 10:48:44,992 INFO [make_averaged_dataset] running in process 1335784
2026-03-13 10:48:45,017 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-13 10:48:45,030 INFO [make_averaged_dataset] publishing log messages to tcp://*:5563
2026-03-13 10:48:45,031 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555
2026-03-13 10:48:45,032 INFO [make_averaged_dataset] creating day-averaged dataset for 13-Mar-2026 biology
2026-03-13 10:48:45,033 INFO [make_averaged_dataset] running in process 1335785
2026-03-13 10:48:45,034 INFO [make_averaged_dataset] running in process 1335783
2026-03-13 10:48:45,035 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-13 10:48:45,036 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-13 10:48:45,037 INFO [make_averaged_dataset] publishing log messages to tcp://*:5564
2026-03-13 10:48:45,037 INFO [make_averaged_dataset] publishing log messages to tcp://*:5565
2026-03-13 10:48:45,038 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555
2026-03-13 10:48:45,040 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555
2026-03-13 10:48:45,041 INFO [make_averaged_dataset] creating day-averaged dataset for 13-Mar-2026 chemistry
2026-03-13 10:48:45,042 INFO [make_averaged_dataset] creating day-averaged dataset for 13-Mar-2026 physics
2026-03-13 10:48:47,431 INFO [make_plots] running in process 1335780
2026-03-13 10:48:47,433 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-13 10:48:47,435 INFO [make_plots] publishing log messages to tcp://*:5566
2026-03-13 10:48:47,436 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2026-03-13 10:48:48,152 INFO [make_plots] preparing render list for 2026-03-13 NEMO nowcast-green research figures
2026-03-13 10:48:48,304 INFO [watch_ww3] forecast WWATCH3 run on arbutus.cloud-nowcast completed
2026-03-13 10:48:48,329 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2026-03-13 10:48:48,843 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2026-03-13'], host='localhost')
2026-03-13 10:48:51,273 INFO [make_plots] added temperature_thalweg_and_surface figures to 2026-03-13 NEMO nowcast-green research render list
2026-03-13 10:48:51,647 INFO [download_wwatch3_results] running in process 1335883
2026-03-13 10:48:51,649 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-13 10:48:51,650 INFO [download_wwatch3_results] publishing log messages to tcp://*:5567
2026-03-13 10:48:51,651 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2026-03-13 10:48:54,827 INFO [make_plots] added nitrate_thalweg_and_surface figures to 2026-03-13 NEMO nowcast-green research render list
2026-03-13 10:48:55,623 INFO [make_plots] added ammonium_thalweg_and_surface figures to 2026-03-13 NEMO nowcast-green research render list
2026-03-13 10:48:56,424 INFO [make_plots] added silicon_thalweg_and_surface figures to 2026-03-13 NEMO nowcast-green research render list
2026-03-13 10:48:57,398 INFO [make_plots] added dissolved_organic_nitrogen_thalweg_and_surface figures to 2026-03-13 NEMO nowcast-green research render list
2026-03-13 10:48:58,297 INFO [make_plots] added particulate_organic_nitrogen_thalweg_and_surface figures to 2026-03-13 NEMO nowcast-green research render list
2026-03-13 10:48:59,300 INFO [make_plots] added biogenic_silicon_thalweg_and_surface figures to 2026-03-13 NEMO nowcast-green research render list
2026-03-13 10:49:00,977 INFO [make_plots] added diatoms_thalweg_and_surface figures to 2026-03-13 NEMO nowcast-green research render list
2026-03-13 10:49:02,069 INFO [make_plots] added flagellates_thalweg_and_surface figures to 2026-03-13 NEMO nowcast-green research render list
2026-03-13 10:49:02,990 INFO [make_plots] added microzooplankton_thalweg_and_surface figures to 2026-03-13 NEMO nowcast-green research render list
2026-03-13 10:49:03,936 INFO [make_plots] added mesozooplankton_thalweg_and_surface figures to 2026-03-13 NEMO nowcast-green research render list
2026-03-13 10:49:04,877 INFO [make_plots] added turbidity_thalweg_and_surface figures to 2026-03-13 NEMO nowcast-green research render list
2026-03-13 10:49:06,738 INFO [make_plots] added temperature_salinity_timeseries figure to 2026-03-13 NEMO nowcast-green research render list
2026-03-13 10:49:06,761 INFO [make_plots] added nitrate_diatoms_timeseries figure to 2026-03-13 NEMO nowcast-green research render list
2026-03-13 10:49:06,762 INFO [make_plots] added diatoms_flagellates_timeseries figure to 2026-03-13 NEMO nowcast-green research render list
2026-03-13 10:49:06,763 INFO [make_plots] added z1_z2_zooplankton_timeseries figure to 2026-03-13 NEMO nowcast-green research render list
2026-03-13 10:49:06,764 INFO [make_plots] starting to render nemo nowcast-green research 13mar26 figures
2026-03-13 10:49:33,074 INFO [download_wwatch3_results] forecast 2026-03-13 results files from arbutus.cloud-nowcast downloaded
2026-03-13 10:49:33,077 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2026-03-13 10:49:33,416 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['wwatch3', 'forecast', '--run-date', '2026-03-13'], host='localhost')
2026-03-13 10:49:35,235 INFO [update_forecast_datasets] running in process 1335929
2026-03-13 10:49:35,237 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-13 10:49:35,238 INFO [update_forecast_datasets] publishing log messages to tcp://*:5567
2026-03-13 10:49:35,239 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2026-03-13 10:49:35,239 INFO [update_forecast_datasets] updating wwatch3 most_recent_forecast directory from forecast/13mar26 run
2026-03-13 10:49:35,332 INFO [update_forecast_datasets] updated wwatch3 most_recent_forecast directory from forecast/13mar26 run
2026-03-13 10:49:35,333 INFO [update_forecast_datasets] updating wwatch3 forecast directory for forecast/13mar26 run
2026-03-13 10:49:37,560 INFO [update_forecast_datasets] updated wwatch3 forecast directory for forecast/13mar26 run: /results/SalishSea/rolling-forecasts/wwatch3
2026-03-13 10:49:37,578 INFO [update_forecast_datasets] wwatch3 2026-03-13 forecast rolling forecast datasets updated
2026-03-13 10:49:37,600 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2026-03-13 10:49:38,096 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['wwatch3-forecast'], host='localhost')
2026-03-13 10:49:40,062 INFO [ping_erddap] running in process 1335934
2026-03-13 10:49:40,063 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-13 10:49:40,063 INFO [ping_erddap] publishing log messages to tcp://*:5567
2026-03-13 10:49:40,064 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2026-03-13 10:49:40,083 INFO [ping_erddap] wwatch3-forecast ERDDAP dataset flag file(s) created
2026-03-13 10:49:40,177 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2026-03-13 10:49:40,436 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['wwatch3', 'forecast', 'publish', '--run-date', '2026-03-13'], host='localhost')
2026-03-13 10:49:40,698 INFO [make_averaged_dataset] day-averaged dataset for 13-Mar-2026 chemistry created
2026-03-13 10:49:40,700 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker
2026-03-13 10:49:46,609 INFO [make_averaged_dataset] day-averaged dataset for 13-Mar-2026 physics created
2026-03-13 10:49:46,611 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker
2026-03-13 10:49:49,177 INFO [make_plots] running in process 1335939
2026-03-13 10:49:49,221 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-13 10:49:49,238 INFO [make_plots] publishing log messages to tcp://*:5564
2026-03-13 10:49:49,256 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2026-03-13 10:49:49,274 INFO [make_plots] preparing render list for 2026-03-13 WaveWatch3 forecast publish figures
2026-03-13 10:49:49,278 INFO [make_plots] added Halibut Bank figure to 2026-03-13 WaveWatch3 forecast publish render list
2026-03-13 10:49:49,279 INFO [make_plots] added Sentry Shoal figure to 2026-03-13 WaveWatch3 forecast publish render list
2026-03-13 10:49:49,279 INFO [make_plots] starting to render wwatch3 forecast publish 13mar26 figures
2026-03-13 10:49:50,605 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46146 Halibut Bank
2026-03-13 10:49:55,696 INFO [make_averaged_dataset] day-averaged dataset for 13-Mar-2026 biology created
2026-03-13 10:49:55,699 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker
2026-03-13 10:49:58,751 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46131 Sentry Shoal
2026-03-13 10:50:03,689 INFO [make_plots] finished rendering wwatch3 forecast publish 13mar26 figures
2026-03-13 10:50:03,699 INFO [make_plots] wwatch3 publish plots for 2026-03-13 forecast completed
2026-03-13 10:50:03,741 INFO [manager] checklist updated with [plots] items from make_plots worker
2026-03-13 12:07:29,643 INFO [make_plots] finished rendering nemo nowcast-green research 13mar26 figures
2026-03-13 12:07:29,811 INFO [make_plots] nemo research plots for 2026-03-13 nowcast-green completed
2026-03-13 12:07:29,914 INFO [manager] checklist updated with [plots] items from make_plots worker
2026-03-13 14:43:41,929 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/18/ to /results/forcing/atmospheric/continental2.5/GRIB/20260313/18/
2026-03-13 14:43:41,964 INFO [collect_weather] 2026-03-13 2.5km weather forecast 18 collection complete
2026-03-13 14:43:42,004 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2026-03-13 14:43:42,094 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20260313/18/
2026-03-13 14:43:42,104 INFO [crop_gribs] 2026-03-13 18 GRIBs cropping complete
2026-03-13 14:43:42,758 INFO [manager] launching NextWorker(module='nowcast.workers.download_weather', args=['00', '1km'], host='localhost')
2026-03-13 14:43:42,767 INFO [manager] launching NextWorker(module='nowcast.workers.download_weather', args=['12', '1km'], host='localhost')
2026-03-13 14:43:42,772 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['00', '2.5km'], host='localhost')
2026-03-13 14:43:42,780 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['00', '--fcst-date', '2026-03-14'], host='localhost')
2026-03-13 14:43:42,790 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2026-03-13 14:43:46,274 INFO [collect_weather] running in process 1342940
2026-03-13 14:43:46,280 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-13 14:43:46,297 INFO [collect_weather] publishing log messages to tcp://*:5562
2026-03-13 14:43:46,323 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2026-03-13 14:43:46,333 INFO [download_weather] running in process 1342939
2026-03-13 14:43:46,343 INFO [download_weather] running in process 1342938
2026-03-13 14:43:46,356 INFO [download_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-13 14:43:46,357 INFO [download_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-13 14:43:46,358 INFO [download_weather] publishing log messages to tcp://*:5563
2026-03-13 14:43:46,360 INFO [download_weather] publishing log messages to tcp://*:5564
2026-03-13 14:43:46,362 INFO [download_weather] connected to skookum.eos.ubc.ca port 5555
2026-03-13 14:43:46,363 INFO [download_weather] connected to skookum.eos.ubc.ca port 5555
2026-03-13 14:43:46,365 INFO [download_weather] downloading 12 1 km forecast GRIB2 files for 20260313
2026-03-13 14:43:46,366 INFO [download_weather] downloading 00 1 km forecast GRIB2 files for 20260313
2026-03-13 14:43:46,426 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/00/
2026-03-13 14:43:48,842 INFO [crop_gribs] running in process 1342941
2026-03-13 14:43:48,843 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-13 14:43:48,844 INFO [crop_gribs] publishing log messages to tcp://*:5565
2026-03-13 14:43:48,846 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2026-03-13 14:43:48,847 INFO [crop_gribs] cropping 2026-03-14 ECCC HRDPS 2.5km continental 00Z GRIB files to SalishSeaCast subdomain
2026-03-13 14:43:48,863 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20260314/00/
2026-03-13 14:47:21,700 INFO [download_weather] 2026-03-13 1km weather forecast 00 downloads complete
2026-03-13 14:47:21,704 INFO [manager] checklist updated with [weather forecast] items from download_weather worker
2026-03-13 14:47:22,964 INFO [download_weather] 2026-03-13 1km weather forecast 12 downloads complete
2026-03-13 14:47:22,966 INFO [manager] checklist updated with [weather forecast] items from download_weather worker
2026-03-13 20:44:45,828 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/00/ to /results/forcing/atmospheric/continental2.5/GRIB/20260314/00/
2026-03-13 20:44:45,855 INFO [collect_weather] 2026-03-14 2.5km weather forecast 00 collection complete
2026-03-13 20:44:45,955 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2026-03-13 20:44:46,617 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20260314/00/
2026-03-13 20:44:46,624 INFO [crop_gribs] 2026-03-14 00 GRIBs cropping complete
2026-03-13 20:44:46,766 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['06', '2.5km'], host='localhost')
2026-03-13 20:44:46,781 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['06', '--fcst-date', '2026-03-14'], host='localhost')
2026-03-13 20:44:46,788 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2026-03-13 20:44:49,647 INFO [collect_weather] running in process 1347836
2026-03-13 20:44:49,648 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-13 20:44:49,649 INFO [collect_weather] publishing log messages to tcp://*:5562
2026-03-13 20:44:49,649 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2026-03-13 20:44:49,695 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/06/
2026-03-13 20:44:51,042 INFO [crop_gribs] running in process 1347837
2026-03-13 20:44:51,043 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-13 20:44:51,044 INFO [crop_gribs] publishing log messages to tcp://*:5563
2026-03-13 20:44:51,045 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2026-03-13 20:44:51,046 INFO [crop_gribs] cropping 2026-03-14 ECCC HRDPS 2.5km continental 06Z GRIB files to SalishSeaCast subdomain
2026-03-13 20:44:51,059 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20260314/06/
2026-03-14 02:40:19,480 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/06/ to /results/forcing/atmospheric/continental2.5/GRIB/20260314/06/
2026-03-14 02:40:19,481 INFO [collect_weather] 2026-03-14 2.5km weather forecast 06 collection complete
2026-03-14 02:40:19,486 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2026-03-14 02:40:19,760 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20260314/06/
2026-03-14 02:40:19,762 INFO [crop_gribs] 2026-03-14 06 GRIBs cropping complete
2026-03-14 02:40:20,327 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Capilano', '--data-date', '2026-03-13'], host='localhost')
2026-03-14 02:40:20,344 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'ChilliwackVedder', '--data-date', '2026-03-13'], host='localhost')
2026-03-14 02:40:20,351 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'ClowhomClowhomLake', '--data-date', '2026-03-13'], host='localhost')
2026-03-14 02:40:20,358 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Englishman', '--data-date', '2026-03-13'], host='localhost')
2026-03-14 02:40:20,369 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Fraser', '--data-date', '2026-03-13'], host='localhost')
2026-03-14 02:40:20,373 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'HomathkoMouth', '--data-date', '2026-03-13'], host='localhost')
2026-03-14 02:40:20,379 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'NicomeklLangley', '--data-date', '2026-03-13'], host='localhost')
2026-03-14 02:40:20,385 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'RobertsCreek', '--data-date', '2026-03-13'], host='localhost')
2026-03-14 02:40:20,391 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SalmonSayward', '--data-date', '2026-03-13'], host='localhost')
2026-03-14 02:40:20,402 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SanJuanPortRenfrew', '--data-date', '2026-03-13'], host='localhost')
2026-03-14 02:40:20,410 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SquamishBrackendale', '--data-date', '2026-03-13'], host='localhost')
2026-03-14 02:40:20,416 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaScotty', '--data-date', '2026-03-13'], host='localhost')
2026-03-14 02:40:20,424 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaBypass', '--data-date', '2026-03-13'], host='localhost')
2026-03-14 02:40:20,436 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaDiversion', '--data-date', '2026-03-13'], host='localhost')
2026-03-14 02:40:20,440 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ctd', args=['SCVIP'], host='localhost')
2026-03-14 02:40:20,453 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ctd', args=['SEVIP'], host='localhost')
2026-03-14 02:40:20,462 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ferry', args=['TWDP'], host='localhost')
2026-03-14 02:40:20,472 INFO [manager] launching NextWorker(module='nowcast.workers.get_vfpa_hadcp', args=['--data-date', '2026-03-13'], host='localhost')
2026-03-14 02:40:20,491 INFO [manager] launching NextWorker(module='nowcast.workers.collect_NeahBay_ssh', args=['00'], host='localhost')
2026-03-14 02:40:20,497 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['12', '2.5km'], host='localhost')
2026-03-14 02:40:20,509 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['12'], host='localhost')
2026-03-14 02:40:20,523 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2026-03-14 02:40:21,175 INFO [manager] launching NextWorker(module='nowcast.workers.grib_to_netcdf', args=['forecast2'], host='localhost')
2026-03-14 02:40:24,128 INFO [collect_weather] running in process 1351779
2026-03-14 02:40:24,128 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-14 02:40:24,131 INFO [collect_weather] publishing log messages to tcp://*:5562
2026-03-14 02:40:24,132 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2026-03-14 02:40:24,135 INFO [collect_NeahBay_ssh] running in process 1351778
2026-03-14 02:40:24,137 INFO [collect_NeahBay_ssh] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-14 02:40:24,138 INFO [collect_NeahBay_ssh] publishing log messages to tcp://*:5563
2026-03-14 02:40:24,140 INFO [collect_NeahBay_ssh] connected to skookum.eos.ubc.ca port 5555
2026-03-14 02:40:24,140 INFO [collect_NeahBay_ssh] collecting Neah Bay ssh 00Z obs/forecast for 2026-03-14
2026-03-14 02:40:24,168 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/12/
2026-03-14 02:40:25,526 INFO [collect_river_data] running in process 1351767
2026-03-14 02:40:25,527 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-14 02:40:25,528 INFO [collect_river_data] publishing log messages to tcp://*:5564
2026-03-14 02:40:25,530 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-14 02:40:25,531 INFO [collect_river_data] Collecting ECCC RobertsCreek river data for 2026-03-13
2026-03-14 02:40:25,571 INFO [collect_river_data] running in process 1351766
2026-03-14 02:40:25,575 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-14 02:40:25,577 INFO [collect_river_data] publishing log messages to tcp://*:5565
2026-03-14 02:40:25,577 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-14 02:40:25,579 INFO [collect_river_data] Collecting ECCC NicomeklLangley river data for 2026-03-13
2026-03-14 02:40:25,583 INFO [collect_river_data] running in process 1351764
2026-03-14 02:40:25,584 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-14 02:40:25,586 INFO [collect_river_data] publishing log messages to tcp://*:5566
2026-03-14 02:40:25,588 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-14 02:40:25,589 INFO [collect_river_data] Collecting ECCC Fraser river data for 2026-03-13
2026-03-14 02:40:25,603 INFO [collect_river_data] running in process 1351761
2026-03-14 02:40:25,604 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-14 02:40:25,605 INFO [collect_river_data] publishing log messages to tcp://*:5567
2026-03-14 02:40:25,606 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-14 02:40:25,610 INFO [collect_river_data] Collecting ECCC ClowhomClowhomLake river data for 2026-03-13
2026-03-14 02:40:25,649 INFO [collect_river_data] running in process 1351769
2026-03-14 02:40:25,650 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-14 02:40:25,651 INFO [collect_river_data] publishing log messages to tcp://*:5568
2026-03-14 02:40:25,651 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-14 02:40:25,653 INFO [collect_river_data] Collecting ECCC SanJuanPortRenfrew river data for 2026-03-13
2026-03-14 02:40:25,662 INFO [collect_river_data] Appended ECCC Fraser river average discharge for 2026-03-13 to: /data/dlatorne/SOG-projects/SOG-forcing/ECget/Fraser_flow
2026-03-14 02:40:25,663 INFO [collect_river_data] ECCC Fraser river data collection for 2026-03-13 completed
2026-03-14 02:40:25,665 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-14 02:40:25,668 INFO [collect_river_data] Appended ECCC RobertsCreek river average discharge for 2026-03-13 to: /results/forcing/rivers/observations/RobertsCreek_flow
2026-03-14 02:40:25,669 INFO [collect_river_data] ECCC RobertsCreek river data collection for 2026-03-13 completed
2026-03-14 02:40:25,671 INFO [collect_river_data] Appended ECCC NicomeklLangley river average discharge for 2026-03-13 to: /results/forcing/rivers/observations/Nicomekl_Langley_flow
2026-03-14 02:40:25,673 INFO [collect_river_data] ECCC NicomeklLangley river data collection for 2026-03-13 completed
2026-03-14 02:40:25,675 INFO [collect_river_data] Appended ECCC ClowhomClowhomLake river average discharge for 2026-03-13 to: /results/forcing/rivers/observations/Clowhom_ClowhomLake_flow
2026-03-14 02:40:25,675 INFO [collect_river_data] ECCC ClowhomClowhomLake river data collection for 2026-03-13 completed
2026-03-14 02:40:25,677 INFO [collect_river_data] running in process 1351765
2026-03-14 02:40:25,678 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-14 02:40:25,679 INFO [collect_river_data] publishing log messages to tcp://*:5569
2026-03-14 02:40:25,680 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-14 02:40:25,681 INFO [collect_river_data] Collecting ECCC HomathkoMouth river data for 2026-03-13
2026-03-14 02:40:25,701 INFO [collect_river_data] Appended ECCC SanJuanPortRenfrew river average discharge for 2026-03-13 to: /results/forcing/rivers/observations/SanJuan_PortRenfrew_flow
2026-03-14 02:40:25,701 INFO [collect_river_data] ECCC SanJuanPortRenfrew river data collection for 2026-03-13 completed
2026-03-14 02:40:25,714 INFO [collect_river_data] Appended ECCC HomathkoMouth river average discharge for 2026-03-13 to: /results/forcing/rivers/observations/Homathko_Mouth_flow
2026-03-14 02:40:25,715 INFO [collect_river_data] ECCC HomathkoMouth river data collection for 2026-03-13 completed
2026-03-14 02:40:25,757 INFO [collect_river_data] running in process 1351772
2026-03-14 02:40:25,758 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-14 02:40:25,759 INFO [collect_river_data] publishing log messages to tcp://*:5570
2026-03-14 02:40:25,760 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-14 02:40:25,761 INFO [collect_river_data] Collecting ECCC TheodosiaBypass river data for 2026-03-13
2026-03-14 02:40:25,764 INFO [collect_river_data] running in process 1351763
2026-03-14 02:40:25,765 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-14 02:40:25,767 INFO [collect_river_data] publishing log messages to tcp://*:5571
2026-03-14 02:40:25,768 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-14 02:40:25,769 INFO [collect_river_data] Collecting ECCC Englishman river data for 2026-03-13
2026-03-14 02:40:25,770 INFO [collect_river_data] running in process 1351771
2026-03-14 02:40:25,770 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-14 02:40:25,772 INFO [collect_river_data] publishing log messages to tcp://*:5572
2026-03-14 02:40:25,774 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-14 02:40:25,774 INFO [collect_river_data] Collecting ECCC TheodosiaScotty river data for 2026-03-13
2026-03-14 02:40:25,775 INFO [collect_river_data] running in process 1351759
2026-03-14 02:40:25,777 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-14 02:40:25,778 INFO [collect_river_data] publishing log messages to tcp://*:5573
2026-03-14 02:40:25,778 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-14 02:40:25,779 INFO [collect_river_data] Collecting ECCC Capilano river data for 2026-03-13
2026-03-14 02:40:25,785 INFO [collect_river_data] running in process 1351760
2026-03-14 02:40:25,788 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-14 02:40:25,789 INFO [collect_river_data] publishing log messages to tcp://*:5574
2026-03-14 02:40:25,790 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-14 02:40:25,791 INFO [collect_river_data] Collecting ECCC ChilliwackVedder river data for 2026-03-13
2026-03-14 02:40:25,793 INFO [collect_river_data] running in process 1351768
2026-03-14 02:40:25,794 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-14 02:40:25,795 INFO [collect_river_data] publishing log messages to tcp://*:5575
2026-03-14 02:40:25,796 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-14 02:40:25,798 INFO [collect_river_data] Collecting ECCC SalmonSayward river data for 2026-03-13
2026-03-14 02:40:25,804 INFO [collect_river_data] Appended ECCC TheodosiaBypass river average discharge for 2026-03-13 to: /results/forcing/rivers/observations/Theodosia_Bypass_flow
2026-03-14 02:40:25,805 INFO [collect_river_data] ECCC TheodosiaBypass river data collection for 2026-03-13 completed
2026-03-14 02:40:25,807 INFO [collect_NeahBay_ssh] 2026-03-14T00:00:00-07:00 Neah Bay ssh 00Z obs/forecast data collection complete
2026-03-14 02:40:25,815 INFO [collect_river_data] Appended ECCC TheodosiaScotty river average discharge for 2026-03-13 to: /results/forcing/rivers/observations/Theodosia_Scotty_flow
2026-03-14 02:40:25,820 INFO [collect_river_data] ECCC TheodosiaScotty river data collection for 2026-03-13 completed
2026-03-14 02:40:25,822 INFO [grib_to_netcdf] running in process 1351828
2026-03-14 02:40:25,824 INFO [grib_to_netcdf] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-14 02:40:25,824 INFO [collect_river_data] Appended ECCC Englishman river average discharge for 2026-03-13 to: /data/dlatorne/SOG-projects/SOG-forcing/ECget/Englishman_flow
2026-03-14 02:40:25,825 INFO [grib_to_netcdf] publishing log messages to tcp://*:5576
2026-03-14 02:40:25,827 INFO [collect_river_data] ECCC Englishman river data collection for 2026-03-13 completed
2026-03-14 02:40:25,828 INFO [grib_to_netcdf] connected to skookum.eos.ubc.ca port 5555
2026-03-14 02:40:25,829 INFO [grib_to_netcdf] creating NEMO-atmos forcing files for 2026-03-14 forecast2 run
2026-03-14 02:40:25,831 INFO [collect_river_data] Appended ECCC ChilliwackVedder river average discharge for 2026-03-13 to: /results/forcing/rivers/observations/Chilliwack_Vedder_flow
2026-03-14 02:40:25,832 INFO [collect_river_data] ECCC ChilliwackVedder river data collection for 2026-03-13 completed
2026-03-14 02:40:25,833 INFO [collect_river_data] running in process 1351773
2026-03-14 02:40:25,835 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-14 02:40:25,836 INFO [collect_river_data] publishing log messages to tcp://*:5577
2026-03-14 02:40:25,837 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-14 02:40:25,837 INFO [collect_river_data] Collecting ECCC TheodosiaDiversion river data for 2026-03-13
2026-03-14 02:40:25,838 INFO [collect_river_data] Appended ECCC SalmonSayward river average discharge for 2026-03-13 to: /results/forcing/rivers/observations/Salmon_Sayward_flow
2026-03-14 02:40:25,839 INFO [collect_river_data] ECCC SalmonSayward river data collection for 2026-03-13 completed
2026-03-14 02:40:25,855 INFO [collect_river_data] running in process 1351770
2026-03-14 02:40:25,856 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-14 02:40:25,857 INFO [collect_river_data] publishing log messages to tcp://*:5578
2026-03-14 02:40:25,859 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-14 02:40:25,860 INFO [collect_river_data] Collecting ECCC SquamishBrackendale river data for 2026-03-13
2026-03-14 02:40:25,870 INFO [collect_river_data] Appended ECCC Capilano river average discharge for 2026-03-13 to: /opp/observations/rivers/Capilano/Caplilano_08GA010_day_avg_flow
2026-03-14 02:40:25,871 INFO [collect_river_data] ECCC Capilano river data collection for 2026-03-13 completed
2026-03-14 02:40:25,874 INFO [collect_river_data] Appended ECCC TheodosiaDiversion river average discharge for 2026-03-13 to: /results/forcing/rivers/observations/Theodosia_Diversion_flow
2026-03-14 02:40:25,876 INFO [collect_river_data] ECCC TheodosiaDiversion river data collection for 2026-03-13 completed
2026-03-14 02:40:25,900 INFO [collect_river_data] Appended ECCC SquamishBrackendale river average discharge for 2026-03-13 to: /results/forcing/rivers/observations/Squamish_Brackendale_flow
2026-03-14 02:40:25,901 INFO [collect_river_data] ECCC SquamishBrackendale river data collection for 2026-03-13 completed
2026-03-14 02:40:26,058 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-14 02:40:26,225 INFO [get_vfpa_hadcp] running in process 1351777
2026-03-14 02:40:26,226 INFO [get_vfpa_hadcp] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-14 02:40:26,229 INFO [get_vfpa_hadcp] publishing log messages to tcp://*:5579
2026-03-14 02:40:26,232 INFO [get_vfpa_hadcp] connected to skookum.eos.ubc.ca port 5555
2026-03-14 02:40:26,233 INFO [get_vfpa_hadcp] processing VFPA HADCP data from 2nd Narrows Rail Bridge for 2026-03-13
2026-03-14 02:40:26,451 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-14 02:40:26,458 INFO [crop_gribs] running in process 1351780
2026-03-14 02:40:26,459 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-14 02:40:26,460 INFO [crop_gribs] publishing log messages to tcp://*:5566
2026-03-14 02:40:26,461 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2026-03-14 02:40:26,462 INFO [crop_gribs] cropping 2026-03-14 ECCC HRDPS 2.5km continental 12Z GRIB files to SalishSeaCast subdomain
2026-03-14 02:40:26,475 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20260314/12/
2026-03-14 02:40:27,071 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-14 02:40:27,386 INFO [get_onc_ctd] running in process 1351774
2026-03-14 02:40:27,388 INFO [get_onc_ctd] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-14 02:40:27,389 INFO [get_onc_ctd] publishing log messages to tcp://*:5564
2026-03-14 02:40:27,390 INFO [get_onc_ctd] connected to skookum.eos.ubc.ca port 5555
2026-03-14 02:40:27,391 INFO [get_onc_ctd] requesting ONC SCVIP CTD T&S data for 2026-03-13
2026-03-14 02:40:27,398 INFO [get_onc_ctd] running in process 1351775
2026-03-14 02:40:27,399 INFO [get_onc_ctd] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-14 02:40:27,401 INFO [get_onc_ctd] publishing log messages to tcp://*:5565
2026-03-14 02:40:27,402 INFO [get_onc_ctd] connected to skookum.eos.ubc.ca port 5555
2026-03-14 02:40:27,403 INFO [get_onc_ctd] requesting ONC SEVIP CTD T&S data for 2026-03-13
2026-03-14 02:40:27,484 INFO [get_onc_ferry] running in process 1351776
2026-03-14 02:40:27,508 INFO [get_onc_ferry] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-14 02:40:27,512 INFO [get_onc_ferry] publishing log messages to tcp://*:5590
2026-03-14 02:40:27,513 INFO [get_onc_ferry] connected to skookum.eos.ubc.ca port 5555
2026-03-14 02:40:27,514 INFO [get_onc_ferry] requesting ONC TWDP.N1 NAV data for 2026-03-13
2026-03-14 02:40:27,519 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-14 02:40:27,892 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-14 02:40:28,245 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-14 02:40:28,627 INFO [manager] checklist updated with [Neah Bay ssh data] items from collect_NeahBay_ssh worker
2026-03-14 02:40:28,999 INFO [manager] launching NextWorker(module='nowcast.workers.make_ssh_files', args=['forecast2', '--run-date', '2026-03-14'], host='localhost')
2026-03-14 02:40:29,006 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-14 02:40:29,117 INFO [get_vfpa_hadcp] added VFPA HADCP data from 2nd Narrows Rail Bridge for 2026-03-13 to /opp/observations/AISDATA/netcdf/VFPA_2ND_NARROWS_HADCP_2s_202603.nc
2026-03-14 02:40:29,118 INFO [get_vfpa_hadcp] VFPA HADCP observations added to 2026-03 netcdf file
2026-03-14 02:40:29,487 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-14 02:40:29,817 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-14 02:40:30,162 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-14 02:40:30,564 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-14 02:40:30,903 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-14 02:40:31,276 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-14 02:40:31,620 INFO [manager] checklist updated with [VFPA HADCP data] items from get_vfpa_hadcp worker
2026-03-14 02:40:32,243 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['VFPA-HADCP'], host='localhost')
2026-03-14 02:40:33,451 INFO [make_ssh_files] running in process 1352252
2026-03-14 02:40:33,453 INFO [make_ssh_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-14 02:40:33,454 INFO [make_ssh_files] publishing log messages to tcp://*:5563
2026-03-14 02:40:33,455 INFO [make_ssh_files] connected to skookum.eos.ubc.ca port 5555
2026-03-14 02:40:33,456 INFO [make_ssh_files] building forecast2 sea surface height boundary conditions file(s) from 2026-03-14 NOAA Neah Bay 00Z observation and forecast values
2026-03-14 02:40:33,622 INFO [ping_erddap] running in process 1352282
2026-03-14 02:40:33,623 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-14 02:40:33,624 INFO [ping_erddap] publishing log messages to tcp://*:5567
2026-03-14 02:40:33,627 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2026-03-14 02:40:33,629 INFO [ping_erddap] VFPA-HADCP ERDDAP dataset flag file(s) created
2026-03-14 02:40:33,632 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2026-03-14 02:40:36,066 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2026m03d09.nc
2026-03-14 02:40:36,304 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2026m03d10.nc
2026-03-14 02:40:36,512 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2026m03d11.nc
2026-03-14 02:40:36,748 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2026m03d12.nc
2026-03-14 02:40:36,976 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2026m03d13.nc
2026-03-14 02:40:37,202 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2026m03d14.nc
2026-03-14 02:40:37,502 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2026m03d15.nc
2026-03-14 02:40:37,733 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2026m03d16.nc
2026-03-14 02:40:37,948 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2026m03d17.nc
2026-03-14 02:40:38,142 INFO [make_ssh_files] sea surface height boundary file for 2026-03-14 forecast2 run created
2026-03-14 02:40:38,145 INFO [manager] checklist updated with [sea surface height forcing] items from make_ssh_files worker
2026-03-14 02:40:38,592 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2026m03d15.nc
2026-03-14 02:40:38,675 INFO [manager] launching NextWorker(module='nowcast.workers.make_runoff_file', args=['v202108'], host='localhost')
2026-03-14 02:40:38,679 INFO [manager] launching NextWorker(module='nowcast.workers.make_201702_runoff_file', args=[], host='localhost')
2026-03-14 02:40:40,259 INFO [make_201702_runoff_file] running in process 1352337
2026-03-14 02:40:40,260 INFO [make_201702_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-14 02:40:40,261 INFO [make_201702_runoff_file] publishing log messages to tcp://*:5563
2026-03-14 02:40:40,262 INFO [make_201702_runoff_file] connected to skookum.eos.ubc.ca port 5555
2026-03-14 02:40:40,305 INFO [make_201702_runoff_file] calculating runoff forcing for b201702 bathymetry
2026-03-14 02:40:40,461 INFO [make_201702_runoff_file] 2026-03-14 runoff file creation from Fraser at Hope and climatology elsewhere complete
2026-03-14 02:40:40,463 INFO [manager] checklist updated with [rivers forcing] items from make_201702_runoff_file worker
2026-03-14 02:40:40,934 INFO [make_runoff_file] running in process 1352335
2026-03-14 02:40:40,935 INFO [make_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-14 02:40:40,939 INFO [make_runoff_file] publishing log messages to tcp://*:5567
2026-03-14 02:40:40,941 INFO [make_runoff_file] connected to skookum.eos.ubc.ca port 5555
2026-03-14 02:40:40,941 INFO [make_runoff_file] calculating NEMO runoff forcing for v202108 bathymetry for 2026-03-13
2026-03-14 02:40:42,134 ERROR [make_runoff_file] no 2026-03-13 discharge obs for SkagitMountVernon: patching
2026-03-14 02:40:42,203 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2026m03d16.nc
2026-03-14 02:40:42,209 INFO [grib_to_netcdf] 2026-03-14 NEMO-atmos forcing file for forecast2 created
2026-03-14 02:40:42,211 INFO [manager] checklist updated with [weather forcing] items from grib_to_netcdf worker
2026-03-14 02:40:42,923 ERROR [make_runoff_file] no 2026-03-13 discharge obs for SnohomishMonroe: patching
2026-03-14 02:40:43,163 ERROR [make_runoff_file] no 2026-03-13 discharge obs for NisquallyMcKenna: patching
2026-03-14 02:40:43,277 ERROR [make_runoff_file] no 2026-03-13 discharge obs for GreenwaterGreenwater: patching
2026-03-14 02:40:44,810 INFO [make_runoff_file] stored NEMO runoff forcing for v202108 bathymetry for 2026-03-13: /results/forcing/rivers/R202108Dailies_y2026m03d13.nc
2026-03-14 02:40:44,811 INFO [make_runoff_file] 2026-03-13 runoff file creation completed
2026-03-14 02:40:44,813 INFO [manager] checklist updated with [rivers forcing] items from make_runoff_file worker
2026-03-14 02:40:45,318 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'forecast2'], host='localhost')
2026-03-14 02:40:45,319 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'forecast2'], host='localhost')
2026-03-14 02:40:45,320 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.nibi', 'forecast2'], host='localhost')
2026-03-14 02:40:45,322 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'forecast2'], host='localhost')
2026-03-14 02:40:46,820 INFO [upload_forcing] running in process 1352379
2026-03-14 02:40:46,821 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-14 02:40:46,822 INFO [upload_forcing] publishing log messages to tcp://*:5563
2026-03-14 02:40:46,823 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-03-14 02:40:46,827 INFO [get_onc_ctd] 2026-03-13 ONC SCVIP CTD T&S file created
2026-03-14 02:40:46,830 INFO [manager] checklist updated with [ONC CTD data] items from get_onc_ctd worker
2026-03-14 02:40:46,848 INFO [upload_forcing] running in process 1352377
2026-03-14 02:40:46,849 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-14 02:40:46,849 INFO [upload_forcing] publishing log messages to tcp://*:5567
2026-03-14 02:40:46,850 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-03-14 02:40:46,904 INFO [upload_forcing] running in process 1352378
2026-03-14 02:40:46,905 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-14 02:40:46,906 INFO [upload_forcing] publishing log messages to tcp://*:5568
2026-03-14 02:40:46,907 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-03-14 02:40:46,916 INFO [upload_forcing] running in process 1352376
2026-03-14 02:40:46,918 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-14 02:40:46,919 INFO [upload_forcing] publishing log messages to tcp://*:5569
2026-03-14 02:40:46,920 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-03-14 02:40:47,242 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['SCVIP-CTD'], host='localhost')
2026-03-14 02:40:47,749 INFO [get_onc_ctd] 2026-03-13 ONC SEVIP CTD T&S file created
2026-03-14 02:40:47,753 INFO [manager] checklist updated with [ONC CTD data] items from get_onc_ctd worker
2026-03-14 02:40:48,180 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['SEVIP-CTD'], host='localhost')
2026-03-14 02:40:48,349 INFO [get_onc_ferry] requesting ONC TWDP TSG data for 2026-03-13
2026-03-14 02:40:48,493 INFO [ping_erddap] running in process 1352404
2026-03-14 02:40:48,494 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-14 02:40:48,496 INFO [ping_erddap] publishing log messages to tcp://*:5564
2026-03-14 02:40:48,499 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2026-03-14 02:40:48,501 INFO [ping_erddap] SCVIP-CTD ERDDAP dataset flag file(s) created
2026-03-14 02:40:48,502 INFO [upload_forcing] LiveOcean boundary conditions file not found; created symlink to /results/forcing/LiveOcean/boundary_conditions/LiveOcean_v201905_y2026m03d13.nc
2026-03-14 02:40:48,503 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2026-03-14 02:40:48,548 INFO [upload_forcing] forecast2 2026-03-14 forcing files upload to optimum-hindcast completed
2026-03-14 02:40:48,910 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-03-14 02:40:49,463 INFO [ping_erddap] running in process 1352411
2026-03-14 02:40:49,464 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-14 02:40:49,465 INFO [ping_erddap] publishing log messages to tcp://*:5564
2026-03-14 02:40:49,467 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2026-03-14 02:40:49,468 INFO [ping_erddap] SEVIP-CTD ERDDAP dataset flag file(s) created
2026-03-14 02:40:49,471 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2026-03-14 02:40:51,375 INFO [upload_forcing] forecast2 2026-03-14 forcing files upload to arbutus.cloud-nowcast completed
2026-03-14 02:40:51,379 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-03-14 02:40:51,726 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2026-03-14'], host='localhost')
2026-03-14 02:40:53,230 INFO [make_forcing_links] running in process 1352423
2026-03-14 02:40:53,231 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-14 02:40:53,232 INFO [make_forcing_links] publishing log messages to tcp://*:5563
2026-03-14 02:40:53,233 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2026-03-14 02:40:53,838 INFO [upload_forcing] forecast2 2026-03-14 forcing files upload to orcinus-nowcast-agrif completed
2026-03-14 02:40:53,841 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-03-14 02:40:54,395 INFO [make_forcing_links] forecast2 2026-03-14 forcing file links on arbutus.cloud-nowcast created
2026-03-14 02:40:54,401 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2026-03-14 02:40:54,751 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2026-03-14'], host='arbutus.cloud-nowcast')
2026-03-14 02:40:55,030 INFO [upload_forcing] forecast2 2026-03-14 forcing files upload to robot.nibi completed
2026-03-14 02:40:55,035 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-03-14 02:40:56,869 INFO [run_NEMO] running in process 13237
2026-03-14 02:40:56,870 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-14 02:40:56,872 INFO [run_NEMO] publishing log messages to tcp://*:5556
2026-03-14 02:40:57,012 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2026-03-14 02:40:57,793 WARNING [run_NEMO] There are uncommitted changes in /nemoShare/MEOPAR/nowcast-sys/SS-run-sets
2026-03-14 02:40:58,095 INFO [run_NEMO] forecast2: launching /nemoShare/MEOPAR/nowcast-sys/runs/13mar26forecast2_2026-03-14T024057.104676-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2026-03-14 02:40:58,110 INFO [run_NEMO] forecast2 NEMO run for 2026-03-14 on arbutus.cloud-nowcast started
2026-03-14 02:40:58,118 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2026-03-14 02:40:58,457 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'forecast2'], host='arbutus.cloud-nowcast')
2026-03-14 02:41:00,558 INFO [watch_NEMO] running in process 13510
2026-03-14 02:41:00,561 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-14 02:41:00,562 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2026-03-14 02:41:00,563 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2026-03-14 02:41:00,648 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2026-03-14 02:41:11,398 INFO [get_onc_ferry] requesting ONC TWDP OXYSENSOR data for 2026-03-13
2026-03-14 02:41:33,597 INFO [get_onc_ferry] requesting ONC TWDP TURBCHLFL data for 2026-03-13
2026-03-14 02:41:55,238 INFO [get_onc_ferry] requesting ONC TWDP CO2SENSOR data for 2026-03-13
2026-03-14 02:41:55,308 WARNING [get_onc_ferry] No ONC TWDP CO2SENSOR partial_pressure,co2 data for 2026-03-13; substituting empty dataset
2026-03-14 02:41:55,314 WARNING [get_onc_ferry] No ONC TWDP CO2SENSOR partial_pressure data for 2026-03-13; substituting empty dataset
2026-03-14 02:41:55,316 WARNING [get_onc_ferry] No ONC TWDP CO2SENSOR co2 data for 2026-03-13; substituting empty dataset
2026-03-14 02:41:55,317 INFO [get_onc_ferry] requesting ONC TWDP TEMPHUMID data for 2026-03-13
2026-03-14 02:41:55,383 WARNING [get_onc_ferry] No ONC TWDP TEMPHUMID air_temperature,rel_humidity data for 2026-03-13; substituting empty dataset
2026-03-14 02:41:55,388 WARNING [get_onc_ferry] No ONC TWDP TEMPHUMID air_temperature data for 2026-03-13; substituting empty dataset
2026-03-14 02:41:55,390 WARNING [get_onc_ferry] No ONC TWDP TEMPHUMID rel_humidity data for 2026-03-13; substituting empty dataset
2026-03-14 02:41:55,391 INFO [get_onc_ferry] requesting ONC TWDP BARPRESS data for 2026-03-13
2026-03-14 02:41:55,462 WARNING [get_onc_ferry] No ONC TWDP BARPRESS barometric_pressure data for 2026-03-13; substituting empty dataset
2026-03-14 02:41:55,466 WARNING [get_onc_ferry] No ONC TWDP BARPRESS barometric_pressure data for 2026-03-13; substituting empty dataset
2026-03-14 02:41:55,467 INFO [get_onc_ferry] requesting ONC TWDP PYRANOMETER data for 2026-03-13
2026-03-14 02:41:55,535 WARNING [get_onc_ferry] No ONC TWDP PYRANOMETER solar_radiation data for 2026-03-13; substituting empty dataset
2026-03-14 02:41:55,538 WARNING [get_onc_ferry] No ONC TWDP PYRANOMETER solar_radiation data for 2026-03-13; substituting empty dataset
2026-03-14 02:41:55,539 INFO [get_onc_ferry] requesting ONC TWDP PYRGEOMETER data for 2026-03-13
2026-03-14 02:41:55,607 WARNING [get_onc_ferry] No ONC TWDP PYRGEOMETER downward_radiation data for 2026-03-13; substituting empty dataset
2026-03-14 02:41:55,610 WARNING [get_onc_ferry] No ONC TWDP PYRGEOMETER downward_radiation data for 2026-03-13; substituting empty dataset
2026-03-14 02:41:55,913 WARNING [get_onc_ferry] ONC TWDP CO2SENSOR partial_pressure data for 2026-03-13 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2026-03-14 02:41:55,914 WARNING [get_onc_ferry] ONC TWDP CO2SENSOR co2 data for 2026-03-13 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2026-03-14 02:41:55,916 WARNING [get_onc_ferry] ONC TWDP TEMPHUMID air_temperature data for 2026-03-13 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2026-03-14 02:41:55,917 WARNING [get_onc_ferry] ONC TWDP TEMPHUMID rel_humidity data for 2026-03-13 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2026-03-14 02:41:55,919 WARNING [get_onc_ferry] ONC TWDP BARPRESS barometric_pressure data for 2026-03-13 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2026-03-14 02:41:55,922 WARNING [get_onc_ferry] ONC TWDP PYRANOMETER solar_radiation data for 2026-03-13 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2026-03-14 02:41:55,925 WARNING [get_onc_ferry] ONC TWDP PYRGEOMETER downward_radiation data for 2026-03-13 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2026-03-14 02:42:33,396 INFO [get_onc_ferry] 2026-03-13 ONC TWDP ferry data file created
2026-03-14 02:42:33,399 INFO [manager] checklist updated with [ONC ferry data] items from get_onc_ferry worker
2026-03-14 02:42:33,893 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['TWDP-ferry'], host='localhost')
2026-03-14 02:42:35,070 INFO [ping_erddap] running in process 1352451
2026-03-14 02:42:35,071 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-14 02:42:35,073 INFO [ping_erddap] publishing log messages to tcp://*:5563
2026-03-14 02:42:35,074 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2026-03-14 02:42:35,075 INFO [ping_erddap] TWDP-ferry ERDDAP dataset flag file(s) created
2026-03-14 02:42:35,077 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2026-03-14 02:46:00,646 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 19099207 = 2026-03-15 05:24:00 UTC, 18.0% complete
2026-03-14 02:51:00,646 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 19099747 = 2026-03-15 11:24:00 UTC, 38.0% complete
2026-03-14 02:56:00,647 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 19100325 = 2026-03-15 17:49:20 UTC, 59.4% complete
2026-03-14 03:01:00,648 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 19100872 = 2026-03-15 23:54:00 UTC, 79.7% complete
2026-03-14 03:06:00,652 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 19101420 = 2026-03-16 05:59:20 UTC, 100.0% complete
2026-03-14 03:11:00,657 INFO [watch_NEMO] forecast2 NEMO run on arbutus.cloud-nowcast completed
2026-03-14 03:11:00,660 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2026-03-14 03:11:01,250 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_wind_file', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2026-03-14'], host='arbutus.cloud-nowcast')
2026-03-14 03:11:01,263 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_current_file', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2026-03-14'], host='arbutus.cloud-nowcast')
2026-03-14 03:11:01,267 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2026-03-13'], host='localhost')
2026-03-14 03:11:02,477 INFO [download_results] running in process 1352759
2026-03-14 03:11:02,478 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-14 03:11:02,479 INFO [download_results] publishing log messages to tcp://*:5563
2026-03-14 03:11:02,480 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2026-03-14 03:11:02,481 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/forecast2/13mar26 to /results/SalishSea/forecast2.202111
2026-03-14 03:11:04,023 INFO [make_ww3_current_file] running in process 13873
2026-03-14 03:11:04,024 INFO [make_ww3_current_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-14 03:11:04,025 INFO [make_ww3_current_file] publishing log messages to tcp://*:5571
2026-03-14 03:11:04,027 INFO [make_ww3_current_file] connected to skookum.eos.ubc.ca port 5555
2026-03-14 03:11:04,028 INFO [make_ww3_current_file] Creating wwatch3 currents forcing file for 2026-03-14 forecast2 run
2026-03-14 03:11:05,066 INFO [make_ww3_wind_file] running in process 13960
2026-03-14 03:11:05,068 INFO [make_ww3_wind_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-14 03:11:05,069 INFO [make_ww3_wind_file] publishing log messages to tcp://*:5570
2026-03-14 03:11:05,070 INFO [make_ww3_wind_file] connected to skookum.eos.ubc.ca port 5555
2026-03-14 03:11:05,071 INFO [make_ww3_wind_file] Creating wwatch3 wind forcing file for 2026-03-14 forecast2 run
2026-03-14 03:11:14,874 INFO [make_ww3_wind_file] wwatch3 wind forcing file created on arbutus.cloud-nowcast for 2026-03-14 forecast2 run
2026-03-14 03:11:14,878 INFO [manager] checklist updated with [WW3 wind forcing] items from make_ww3_wind_file worker
2026-03-14 03:11:47,529 INFO [download_results] forecast2 2026-03-13 results files from arbutus.cloud-nowcast downloaded
2026-03-14 03:11:47,533 INFO [manager] checklist updated with [results files] items from download_results worker
2026-03-14 03:11:47,878 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['forecast2', '--run-date', '2026-03-13'], host='localhost')
2026-03-14 03:11:49,855 INFO [make_CHS_currents_file] running in process 1352773
2026-03-14 03:11:49,857 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-14 03:11:49,858 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5563
2026-03-14 03:11:49,859 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555
2026-03-14 03:12:25,759 INFO [make_CHS_currents_file] Made CHS currents file for 2026-03-13 for forecast2
2026-03-14 03:12:25,762 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker
2026-03-14 03:12:26,102 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['nemo', 'forecast2', '--run-date', '2026-03-13'], host='localhost')
2026-03-14 03:12:27,371 INFO [update_forecast_datasets] running in process 1352799
2026-03-14 03:12:27,372 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-14 03:12:27,373 INFO [update_forecast_datasets] publishing log messages to tcp://*:5563
2026-03-14 03:12:27,375 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2026-03-14 03:12:27,376 INFO [update_forecast_datasets] updating nemo forecast directory for forecast2/13mar26 run
2026-03-14 03:14:41,688 INFO [make_ww3_current_file] wwatch3 currents forcing file created on arbutus.cloud-nowcast for 2026-03-14 forecast2 run
2026-03-14 03:14:41,692 INFO [manager] checklist updated with [WW3 currents forcing] items from make_ww3_current_file worker
2026-03-14 03:14:42,037 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2026-03-14'], host='arbutus.cloud-nowcast')
2026-03-14 03:14:44,307 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker
2026-03-14 03:14:44,484 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'forecast2'], host='arbutus.cloud-nowcast')
2026-03-14 03:14:47,881 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2026-03-14 03:16:43,848 INFO [update_forecast_datasets] updated nemo forecast directory for forecast2/13mar26 run: /results/SalishSea/rolling-forecasts/nemo
2026-03-14 03:16:43,849 INFO [update_forecast_datasets] nemo 2026-03-13 forecast2 rolling forecast datasets updated
2026-03-14 03:16:43,852 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2026-03-14 03:16:44,182 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nemo-forecast'], host='localhost')
2026-03-14 03:16:44,186 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'forecast2', 'publish', '--run-date', '2026-03-13'], host='localhost')
2026-03-14 03:16:44,192 INFO [manager] launching NextWorker(module='nowcast.workers.make_surface_current_tiles', args=['forecast2', '--run-date', '2026-03-13'], host='localhost')
2026-03-14 03:16:45,464 INFO [ping_erddap] running in process 1352904
2026-03-14 03:16:45,465 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-14 03:16:45,466 INFO [ping_erddap] publishing log messages to tcp://*:5563
2026-03-14 03:16:45,467 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2026-03-14 03:16:45,476 INFO [ping_erddap] nemo-forecast ERDDAP dataset flag file(s) created
2026-03-14 03:16:45,478 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2026-03-14 03:16:46,819 INFO [make_surface_current_tiles] running in process 1352907
2026-03-14 03:16:46,820 INFO [make_surface_current_tiles] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-14 03:16:46,821 INFO [make_surface_current_tiles] publishing log messages to tcp://*:5563
2026-03-14 03:16:46,822 INFO [make_surface_current_tiles] connected to skookum.eos.ubc.ca port 5555
2026-03-14 03:16:46,864 INFO [make_surface_current_tiles] starting rendering of tiles for 2026-03-13 forecast2 in 6 processes into /results/nowcast-sys/figures/surface_currents/forecast2/13mar26
2026-03-14 03:16:47,903 INFO [make_plots] running in process 1352906
2026-03-14 03:16:47,904 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-14 03:16:47,905 INFO [make_plots] publishing log messages to tcp://*:5564
2026-03-14 03:16:47,906 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2026-03-14 03:16:48,520 INFO [make_plots] preparing render list for 2026-03-13 NEMO forecast2 publish figures
2026-03-14 03:16:48,976 INFO [make_plots] added Website_thumbnail figure to 2026-03-13 NEMO forecast2 publish render list
2026-03-14 03:16:48,976 INFO [make_plots] added Threshold_website figure to 2026-03-13 NEMO forecast2 publish render list
2026-03-14 03:16:48,977 INFO [make_plots] added PA_tidal_predictions figure to 2026-03-13 NEMO forecast2 publish render list
2026-03-14 03:16:48,978 INFO [make_plots] added Boundary Bay figure to 2026-03-13 NEMO forecast2 publish render list
2026-03-14 03:16:48,979 INFO [make_plots] added Campbell River figure to 2026-03-13 NEMO forecast2 publish render list
2026-03-14 03:16:48,979 INFO [make_plots] added Cherry Point figure to 2026-03-13 NEMO forecast2 publish render list
2026-03-14 03:16:48,980 INFO [make_plots] added Friday Harbor figure to 2026-03-13 NEMO forecast2 publish render list
2026-03-14 03:16:48,981 INFO [make_plots] added Halfmoon Bay figure to 2026-03-13 NEMO forecast2 publish render list
2026-03-14 03:16:48,982 INFO [make_plots] added Nanaimo figure to 2026-03-13 NEMO forecast2 publish render list
2026-03-14 03:16:48,982 INFO [make_plots] added Neah Bay figure to 2026-03-13 NEMO forecast2 publish render list
2026-03-14 03:16:48,983 INFO [make_plots] added New Westminster figure to 2026-03-13 NEMO forecast2 publish render list
2026-03-14 03:16:48,984 INFO [make_plots] added Patricia Bay figure to 2026-03-13 NEMO forecast2 publish render list
2026-03-14 03:16:48,984 INFO [make_plots] added Point Atkinson figure to 2026-03-13 NEMO forecast2 publish render list
2026-03-14 03:16:48,985 INFO [make_plots] added Port Renfrew figure to 2026-03-13 NEMO forecast2 publish render list
2026-03-14 03:16:48,985 INFO [make_plots] added Sand Heads figure to 2026-03-13 NEMO forecast2 publish render list
2026-03-14 03:16:48,986 INFO [make_plots] added Sandy Cove figure to 2026-03-13 NEMO forecast2 publish render list
2026-03-14 03:16:48,987 INFO [make_plots] added Squamish figure to 2026-03-13 NEMO forecast2 publish render list
2026-03-14 03:16:48,987 INFO [make_plots] added Victoria figure to 2026-03-13 NEMO forecast2 publish render list
2026-03-14 03:16:48,988 INFO [make_plots] added Woodwards Landing figure to 2026-03-13 NEMO forecast2 publish render list
2026-03-14 03:16:48,989 INFO [make_plots] starting to render nemo forecast2 publish 13mar26 figures
2026-03-14 03:19:47,743 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 93, 17.2% complete
2026-03-14 03:24:26,325 WARNING [make_plots] invalid station number for Boundary Bay station: None
2026-03-14 03:24:26,326 WARNING [make_plots] can't resolve a valid CHS station code for Boundary Bay
2026-03-14 03:24:47,743 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 201, 37.2% complete
2026-03-14 03:24:49,790 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb996/data for station 08074 Campbell River from 2026-03-09 00:00:00Z to 2026-03-16 05:55:00Z
2026-03-14 03:25:58,647 WARNING [make_plots] invalid station number for Halfmoon Bay station: None
2026-03-14 03:25:58,648 WARNING [make_plots] can't resolve a valid CHS station code for Halfmoon Bay
2026-03-14 03:26:20,587 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb96d/data for station 07917 Nanaimo from 2026-03-09 00:00:00Z to 2026-03-16 05:55:00Z
2026-03-14 03:27:01,263 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e13d0f4a073c4bbf93/data for station 07654 New Westminster from 2026-03-09 00:00:00Z to 2026-03-16 05:55:00Z
2026-03-14 03:27:23,812 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd2d/data for station 07277 Patricia Bay from 2026-03-09 00:00:00Z to 2026-03-16 05:55:00Z
2026-03-14 03:27:44,231 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94c/data for station 07795 Point Atkinson from 2026-03-09 00:00:00Z to 2026-03-16 05:55:00Z
2026-03-14 03:27:58,785 INFO [make_surface_current_tiles] starting PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast2/13mar26
2026-03-14 03:28:00,748 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e23d0f4a073c4bc060/data for station 08525 Port Renfrew from 2026-03-09 00:00:00Z to 2026-03-16 05:55:00Z
2026-03-14 03:28:16,159 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb93c/data for station 07594 Sand Heads from 2026-03-09 00:00:00Z to 2026-03-16 05:55:00Z
2026-03-14 03:28:33,562 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc43b/data for station 07786 Sandy Cove from 2026-03-09 00:00:00Z to 2026-03-16 05:55:00Z
2026-03-14 03:28:52,305 WARNING [make_plots] invalid station number for Squamish station: None
2026-03-14 03:28:52,306 WARNING [make_plots] can't resolve a valid CHS station code for Squamish
2026-03-14 03:29:09,415 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd1e/data for station 07120 Victoria from 2026-03-09 00:00:00Z to 2026-03-16 05:55:00Z
2026-03-14 03:29:25,919 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc42d/data for station 07610 Woodwards Landing from 2026-03-09 00:00:00Z to 2026-03-16 05:55:00Z
2026-03-14 03:29:42,268 INFO [make_plots] finished rendering nemo forecast2 publish 13mar26 figures
2026-03-14 03:29:42,270 INFO [make_plots] nemo publish plots for 2026-03-13 forecast2 completed
2026-03-14 03:29:42,279 INFO [manager] checklist updated with [plots] items from make_plots worker
2026-03-14 03:29:42,793 INFO [manager] launching NextWorker(module='nowcast.workers.make_feeds', args=['forecast2', '--run-date', '2026-03-13'], host='localhost')
2026-03-14 03:29:46,044 INFO [make_feeds] running in process 1357846
2026-03-14 03:29:46,045 INFO [make_feeds] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-14 03:29:46,046 INFO [make_feeds] publishing log messages to tcp://*:5564
2026-03-14 03:29:46,047 INFO [make_feeds] connected to skookum.eos.ubc.ca port 5555
2026-03-14 03:29:47,744 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 300, 55.6% complete
2026-03-14 03:30:00,944 INFO [make_feeds] ATOM feeds for 2026-03-13 forecast2 run completed
2026-03-14 03:30:00,947 INFO [manager] checklist updated with [storm surge alert feeds] items from make_feeds worker
2026-03-14 03:30:01,269 INFO [manager] launching NextWorker(module='nemo_nowcast.workers.clear_checklist', args=[], host='localhost')
2026-03-14 03:30:02,331 INFO [clear_checklist] running in process 1357868
2026-03-14 03:30:02,332 INFO [clear_checklist] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-14 03:30:02,333 INFO [clear_checklist] publishing log messages to tcp://*:5564
2026-03-14 03:30:02,334 INFO [clear_checklist] connected to skookum.eos.ubc.ca port 5555
2026-03-14 03:30:02,335 INFO [clear_checklist] requesting that manager clear system state checklist
2026-03-14 03:30:02,337 INFO [manager] writing checklist to log file
2026-03-14 03:30:02,385 INFO [manager] checklist cleared
2026-03-14 03:30:02,387 INFO [clear_checklist] nowcast system checklist cleared
2026-03-14 03:30:02,389 INFO [manager] launching NextWorker(module='nemo_nowcast.workers.rotate_logs', args=[], host='localhost')
2026-03-14 03:30:03,559 INFO [rotate_logs] running in process 1357874
2026-03-14 03:30:03,560 INFO [rotate_logs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-14 03:30:03,561 INFO [rotate_logs] publishing log messages to tcp://*:5564
2026-03-14 03:30:03,562 INFO [rotate_logs] connected to skookum.eos.ubc.ca port 5555
2026-03-14 03:30:03,563 INFO [rotate_logs] rotating log files