2026-03-12 03:33:17,095 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/nowcast.log
2026-03-12 03:33:17,104 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/nowcast.debug.log
2026-03-12 03:33:17,110 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/checklist.log
2026-03-12 03:33:17,111 INFO [rotate_logs] log files rotated
2026-03-12 03:33:17,125 INFO [manager] checklist updated with [log rotation] items from rotate_logs worker
2026-03-12 03:35:12,932 INFO [make_surface_current_tiles] finished PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast2/11mar26
2026-03-12 03:35:12,956 INFO [make_surface_current_tiles] finished rendering of tiles for 2026-03-11 forecast2 into /results/nowcast-sys/figures/surface_currents/forecast2/11mar26
2026-03-12 03:35:12,957 INFO [make_surface_current_tiles] surface current tile figures for 2026-03-11 forecast2 completed
2026-03-12 03:35:13,233 INFO [manager] checklist updated with [surface current tiles] items from make_surface_current_tiles worker
2026-03-12 03:36:28,820 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 405, 75.0% complete
2026-03-12 03:41:28,821 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 507, 93.9% complete
2026-03-12 03:46:28,822 INFO [watch_ww3] forecast2 WWATCH3 run on arbutus.cloud-nowcast completed
2026-03-12 03:46:28,825 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2026-03-12 03:46:29,146 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2026-03-12'], host='localhost')
2026-03-12 03:46:30,479 INFO [download_wwatch3_results] running in process 1297818
2026-03-12 03:46:30,481 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-12 03:46:30,482 INFO [download_wwatch3_results] publishing log messages to tcp://*:5562
2026-03-12 03:46:30,483 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2026-03-12 03:47:05,311 INFO [download_wwatch3_results] forecast2 2026-03-12 results files from arbutus.cloud-nowcast downloaded
2026-03-12 03:47:05,315 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2026-03-12 03:47:05,467 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['wwatch3', 'forecast2', '--run-date', '2026-03-12'], host='localhost')
2026-03-12 03:47:06,767 INFO [update_forecast_datasets] running in process 1297832
2026-03-12 03:47:06,768 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-12 03:47:06,770 INFO [update_forecast_datasets] publishing log messages to tcp://*:5562
2026-03-12 03:47:06,771 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2026-03-12 03:47:06,772 INFO [update_forecast_datasets] updating wwatch3 most_recent_forecast directory from forecast2/12mar26 run
2026-03-12 03:47:06,805 INFO [update_forecast_datasets] updated wwatch3 most_recent_forecast directory from forecast2/12mar26 run
2026-03-12 03:47:06,806 INFO [update_forecast_datasets] updating wwatch3 forecast directory for forecast2/12mar26 run
2026-03-12 03:47:32,177 INFO [update_forecast_datasets] updated wwatch3 forecast directory for forecast2/12mar26 run: /results/SalishSea/rolling-forecasts/wwatch3
2026-03-12 03:47:32,178 INFO [update_forecast_datasets] wwatch3 2026-03-12 forecast2 rolling forecast datasets updated
2026-03-12 03:47:32,181 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2026-03-12 03:47:32,332 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['wwatch3-forecast'], host='localhost')
2026-03-12 03:47:33,570 INFO [ping_erddap] running in process 1297849
2026-03-12 03:47:33,572 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-12 03:47:33,573 INFO [ping_erddap] publishing log messages to tcp://*:5562
2026-03-12 03:47:33,574 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2026-03-12 03:47:33,576 INFO [ping_erddap] wwatch3-forecast ERDDAP dataset flag file(s) created
2026-03-12 03:47:33,579 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2026-03-12 03:47:33,725 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['wwatch3', 'forecast2', 'publish', '--run-date', '2026-03-12'], host='localhost')
2026-03-12 03:47:37,300 INFO [make_plots] running in process 1297854
2026-03-12 03:47:37,301 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-12 03:47:37,302 INFO [make_plots] publishing log messages to tcp://*:5562
2026-03-12 03:47:37,304 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2026-03-12 03:47:37,305 INFO [make_plots] preparing render list for 2026-03-12 WaveWatch3 forecast2 publish figures
2026-03-12 03:47:37,306 INFO [make_plots] added Halibut Bank figure to 2026-03-12 WaveWatch3 forecast2 publish render list
2026-03-12 03:47:37,307 INFO [make_plots] added Sentry Shoal figure to 2026-03-12 WaveWatch3 forecast2 publish render list
2026-03-12 03:47:37,308 INFO [make_plots] starting to render wwatch3 forecast2 publish 12mar26 figures
2026-03-12 03:47:38,161 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46146 Halibut Bank
2026-03-12 03:47:43,596 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46131 Sentry Shoal
2026-03-12 03:47:48,101 INFO [make_plots] finished rendering wwatch3 forecast2 publish 12mar26 figures
2026-03-12 03:47:48,102 INFO [make_plots] wwatch3 publish plots for 2026-03-12 forecast2 completed
2026-03-12 03:47:48,107 INFO [manager] checklist updated with [plots] items from make_plots worker
2026-03-12 08:43:05,393 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/12/ to /results/forcing/atmospheric/continental2.5/GRIB/20260312/12/
2026-03-12 08:43:05,425 INFO [collect_weather] 2026-03-12 2.5km weather forecast 12 collection complete
2026-03-12 08:43:05,510 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2026-03-12 08:43:06,053 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'SkagitMountVernon', '--data-date', '2026-03-11'], host='localhost')
2026-03-12 08:43:06,067 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'SnohomishMonroe', '--data-date', '2026-03-11'], host='localhost')
2026-03-12 08:43:06,072 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'NisquallyMcKenna', '--data-date', '2026-03-11'], host='localhost')
2026-03-12 08:43:06,079 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'GreenwaterGreenwater', '--data-date', '2026-03-11'], host='localhost')
2026-03-12 08:43:06,088 INFO [manager] launching NextWorker(module='nowcast.workers.make_turbidity_file', args=[], host='localhost')
2026-03-12 08:43:06,094 INFO [manager] launching NextWorker(module='nowcast.workers.collect_NeahBay_ssh', args=['06'], host='localhost')
2026-03-12 08:43:06,099 INFO [manager] launching NextWorker(module='nowcast.workers.download_live_ocean', args=[], host='localhost')
2026-03-12 08:43:06,105 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['18', '2.5km'], host='localhost')
2026-03-12 08:43:06,113 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['18'], host='localhost')
2026-03-12 08:43:06,721 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20260312/12/
2026-03-12 08:43:06,760 INFO [crop_gribs] 2026-03-12 12 GRIBs cropping complete
2026-03-12 08:43:06,844 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2026-03-12 08:43:07,085 INFO [manager] launching NextWorker(module='nowcast.workers.grib_to_netcdf', args=['nowcast+'], host='localhost')
2026-03-12 08:43:09,795 INFO [download_live_ocean] running in process 1301356
2026-03-12 08:43:09,796 INFO [download_live_ocean] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-12 08:43:09,799 INFO [download_live_ocean] publishing log messages to tcp://*:5562
2026-03-12 08:43:09,800 INFO [download_live_ocean] connected to skookum.eos.ubc.ca port 5555
2026-03-12 08:43:09,800 INFO [download_live_ocean] downloading Salish Sea western boundary day-averaged LiveOcean file for 2026-03-12
2026-03-12 08:43:09,801 INFO [collect_NeahBay_ssh] running in process 1301355
2026-03-12 08:43:09,802 INFO [collect_NeahBay_ssh] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-12 08:43:09,805 INFO [collect_NeahBay_ssh] publishing log messages to tcp://*:5563
2026-03-12 08:43:09,807 INFO [collect_NeahBay_ssh] connected to skookum.eos.ubc.ca port 5555
2026-03-12 08:43:09,809 INFO [collect_NeahBay_ssh] collecting Neah Bay ssh 06Z obs/forecast for 2026-03-12
2026-03-12 08:43:09,812 INFO [collect_weather] running in process 1301357
2026-03-12 08:43:09,814 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-12 08:43:09,815 INFO [collect_weather] publishing log messages to tcp://*:5564
2026-03-12 08:43:09,816 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2026-03-12 08:43:09,870 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/18/
2026-03-12 08:43:10,405 INFO [download_live_ocean] downloaded 10215788 bytes from https://s3.kopah.uw.edu/liveocean-share/f2026.03.12/ubc.nc to /results/forcing/LiveOcean/downloaded/20260312/low_passed_UBC.nc
2026-03-12 08:43:10,407 INFO [download_live_ocean] Deflating in up to 1 concurrent sub-processes
2026-03-12 08:43:10,900 INFO [collect_river_data] running in process 1301353
2026-03-12 08:43:10,901 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-12 08:43:10,902 INFO [collect_river_data] publishing log messages to tcp://*:5565
2026-03-12 08:43:10,903 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-12 08:43:10,904 INFO [collect_river_data] Collecting USGS GreenwaterGreenwater river data for 2026-03-11
2026-03-12 08:43:10,907 INFO [collect_river_data] running in process 1301351
2026-03-12 08:43:10,909 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-12 08:43:10,910 INFO [collect_river_data] publishing log messages to tcp://*:5566
2026-03-12 08:43:10,911 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-12 08:43:10,917 INFO [collect_river_data] Collecting USGS SnohomishMonroe river data for 2026-03-11
2026-03-12 08:43:10,955 INFO [collect_river_data] running in process 1301350
2026-03-12 08:43:10,957 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-12 08:43:10,958 INFO [collect_river_data] publishing log messages to tcp://*:5567
2026-03-12 08:43:10,959 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-12 08:43:10,963 INFO [collect_river_data] Collecting USGS SkagitMountVernon river data for 2026-03-11
2026-03-12 08:43:10,968 INFO [collect_river_data] running in process 1301352
2026-03-12 08:43:10,968 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-12 08:43:10,969 INFO [collect_river_data] publishing log messages to tcp://*:5568
2026-03-12 08:43:10,970 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-12 08:43:10,971 INFO [collect_river_data] Collecting USGS NisquallyMcKenna river data for 2026-03-11
2026-03-12 08:43:11,014 INFO [grib_to_netcdf] running in process 1301362
2026-03-12 08:43:11,016 INFO [grib_to_netcdf] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-12 08:43:11,019 INFO [grib_to_netcdf] publishing log messages to tcp://*:5569
2026-03-12 08:43:11,020 INFO [grib_to_netcdf] connected to skookum.eos.ubc.ca port 5555
2026-03-12 08:43:11,021 INFO [grib_to_netcdf] creating NEMO-atmos forcing files for 2026-03-12 nowcast and forecast runs
2026-03-12 08:43:11,304 INFO [make_turbidity_file] running in process 1301354
2026-03-12 08:43:11,305 INFO [make_turbidity_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-12 08:43:11,306 INFO [make_turbidity_file] publishing log messages to tcp://*:5570
2026-03-12 08:43:11,308 INFO [make_turbidity_file] connected to skookum.eos.ubc.ca port 5555
2026-03-12 08:43:11,309 INFO [make_turbidity_file] Creating Fraser River turbidity forcing file for 2026-03-12
2026-03-12 08:43:11,522 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12089500&startDT=2026-03-11&endDT=2026-03-11 "HTTP/1.1 200 "
2026-03-12 08:43:11,527 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12200500&startDT=2026-03-11&endDT=2026-03-11 "HTTP/1.1 200 "
2026-03-12 08:43:11,531 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12150800&startDT=2026-03-11&endDT=2026-03-11 "HTTP/1.1 200 "
2026-03-12 08:43:11,532 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12097500&startDT=2026-03-11&endDT=2026-03-11 "HTTP/1.1 200 "
2026-03-12 08:43:11,693 INFO [download_live_ocean] netCDF4 deflated /results/forcing/LiveOcean/downloaded/20260312/low_passed_UBC.nc
2026-03-12 08:43:11,695 INFO [download_live_ocean] 2026-03-12 Live Ocean file for Salish Sea western boundary downloaded
2026-03-12 08:43:11,697 INFO [manager] checklist updated with [Live Ocean products] items from download_live_ocean worker
2026-03-12 08:43:11,874 INFO [crop_gribs] running in process 1301359
2026-03-12 08:43:11,874 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-12 08:43:11,875 INFO [crop_gribs] publishing log messages to tcp://*:5571
2026-03-12 08:43:11,877 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2026-03-12 08:43:11,878 INFO [crop_gribs] cropping 2026-03-12 ECCC HRDPS 2.5km continental 18Z GRIB files to SalishSeaCast subdomain
2026-03-12 08:43:11,892 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20260312/18/
2026-03-12 08:43:11,909 INFO [collect_river_data] Appended USGS GreenwaterGreenwater river average discharge for 2026-03-11 to: /results/forcing/rivers/observations/Greenwater_Greenwater_flow
2026-03-12 08:43:11,911 INFO [collect_river_data] USGS GreenwaterGreenwater river data collection for 2026-03-11 completed
2026-03-12 08:43:11,937 INFO [collect_river_data] Appended USGS SnohomishMonroe river average discharge for 2026-03-11 to: /results/forcing/rivers/observations/Snohomish_Monroe_flow
2026-03-12 08:43:11,939 INFO [collect_river_data] USGS SnohomishMonroe river data collection for 2026-03-11 completed
2026-03-12 08:43:11,959 INFO [collect_river_data] Appended USGS SkagitMountVernon river average discharge for 2026-03-11 to: /results/forcing/rivers/observations/Skagit_MountVernon_flow
2026-03-12 08:43:11,960 INFO [collect_river_data] USGS SkagitMountVernon river data collection for 2026-03-11 completed
2026-03-12 08:43:12,012 INFO [collect_river_data] Appended USGS NisquallyMcKenna river average discharge for 2026-03-11 to: /results/forcing/rivers/observations/Nisqually_McKenna_flow
2026-03-12 08:43:12,013 INFO [collect_river_data] USGS NisquallyMcKenna river data collection for 2026-03-11 completed
2026-03-12 08:43:12,030 INFO [collect_NeahBay_ssh] 2026-03-12T00:00:00-07:00 Neah Bay ssh 06Z obs/forecast data collection complete
2026-03-12 08:43:12,045 INFO [manager] launching NextWorker(module='nowcast.workers.make_live_ocean_files', args=['--run-date', '2026-03-12'], host='localhost')
2026-03-12 08:43:12,054 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-12 08:43:12,259 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-12 08:43:12,537 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-12 08:43:12,814 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-12 08:43:13,004 INFO [manager] checklist updated with [Neah Bay ssh data] items from collect_NeahBay_ssh worker
2026-03-12 08:43:13,280 INFO [manager] launching NextWorker(module='nowcast.workers.make_ssh_files', args=['nowcast', '--run-date', '2026-03-12'], host='localhost')
2026-03-12 08:43:15,362 INFO [make_turbidity_file] 2026-03-12 Fraser River turbidity file creation complete
2026-03-12 08:43:15,370 INFO [manager] checklist updated with [Fraser River turbidity] items from make_turbidity_file worker
2026-03-12 08:43:16,071 INFO [make_live_ocean_files] running in process 1301496
2026-03-12 08:43:16,073 INFO [make_live_ocean_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-12 08:43:16,073 INFO [make_live_ocean_files] publishing log messages to tcp://*:5562
2026-03-12 08:43:16,074 INFO [make_live_ocean_files] connected to skookum.eos.ubc.ca port 5555
2026-03-12 08:43:16,075 INFO [make_live_ocean_files] Creating T&S western boundary conditions file from 2026-03-12 Live Ocean run
2026-03-12 08:43:17,025 INFO [make_ssh_files] running in process 1301513
2026-03-12 08:43:17,026 INFO [make_ssh_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-12 08:43:17,027 INFO [make_ssh_files] publishing log messages to tcp://*:5563
2026-03-12 08:43:17,028 INFO [make_ssh_files] connected to skookum.eos.ubc.ca port 5555
2026-03-12 08:43:17,029 INFO [make_ssh_files] building nowcast sea surface height boundary conditions file(s) from 2026-03-12 NOAA Neah Bay 06Z observation and forecast values
2026-03-12 08:43:19,107 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2026m03d08.nc
2026-03-12 08:43:19,319 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2026m03d09.nc
2026-03-12 08:43:19,511 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2026m03d10.nc
2026-03-12 08:43:19,714 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2026m03d11.nc
2026-03-12 08:43:19,997 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2026m03d12.nc
2026-03-12 08:43:20,210 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2026m03d13.nc
2026-03-12 08:43:20,445 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2026m03d14.nc
2026-03-12 08:43:20,699 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2026m03d15.nc
2026-03-12 08:43:20,867 INFO [make_ssh_files] sea surface height boundary file for 2026-03-12 nowcast run created
2026-03-12 08:43:20,875 INFO [manager] checklist updated with [sea surface height forcing] items from make_ssh_files worker
2026-03-12 08:43:21,034 INFO [manager] launching NextWorker(module='nowcast.workers.make_runoff_file', args=['v202108'], host='localhost')
2026-03-12 08:43:21,038 INFO [manager] launching NextWorker(module='nowcast.workers.make_201702_runoff_file', args=[], host='localhost')
2026-03-12 08:43:22,510 INFO [make_201702_runoff_file] running in process 1301579
2026-03-12 08:43:22,512 INFO [make_201702_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-12 08:43:22,513 INFO [make_201702_runoff_file] publishing log messages to tcp://*:5563
2026-03-12 08:43:22,514 INFO [make_201702_runoff_file] connected to skookum.eos.ubc.ca port 5555
2026-03-12 08:43:22,543 INFO [make_201702_runoff_file] calculating runoff forcing for b201702 bathymetry
2026-03-12 08:43:22,675 INFO [make_201702_runoff_file] 2026-03-12 runoff file creation from Fraser at Hope and climatology elsewhere complete
2026-03-12 08:43:22,677 INFO [manager] checklist updated with [rivers forcing] items from make_201702_runoff_file worker
2026-03-12 08:43:23,120 INFO [make_runoff_file] running in process 1301577
2026-03-12 08:43:23,121 INFO [make_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-12 08:43:23,122 INFO [make_runoff_file] publishing log messages to tcp://*:5563
2026-03-12 08:43:23,123 INFO [make_runoff_file] connected to skookum.eos.ubc.ca port 5555
2026-03-12 08:43:23,125 INFO [make_runoff_file] calculating NEMO runoff forcing for v202108 bathymetry for 2026-03-11
2026-03-12 08:43:26,158 INFO [make_runoff_file] stored NEMO runoff forcing for v202108 bathymetry for 2026-03-11: /results/forcing/rivers/R202108Dailies_y2026m03d11.nc
2026-03-12 08:43:26,159 INFO [make_runoff_file] 2026-03-11 runoff file creation completed
2026-03-12 08:43:26,161 INFO [manager] checklist updated with [rivers forcing] items from make_runoff_file worker
2026-03-12 08:43:27,361 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/hrdps_y2026m03d12.nc
2026-03-12 08:43:45,441 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2026m03d13.nc
2026-03-12 08:43:51,344 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2026m03d14.nc
2026-03-12 08:43:51,361 INFO [grib_to_netcdf] 2026-03-12 NEMO-atmos forcing file for nowcast+ created
2026-03-12 08:43:51,364 INFO [manager] checklist updated with [weather forcing] items from grib_to_netcdf worker
2026-03-12 08:44:00,737 INFO [make_live_ocean_files] Stored T&S western boundary conditions file: /results/forcing/LiveOcean/boundary_conditions/LiveOcean_v201905_y2026m03d12.nc
2026-03-12 08:44:00,738 INFO [make_live_ocean_files] 2026-03-12 Live Ocean western boundary conditions files created
2026-03-12 08:44:00,740 INFO [manager] checklist updated with [Live Ocean boundary conditions] items from make_live_ocean_files worker
2026-03-12 08:44:00,889 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['weather'], host='localhost')
2026-03-12 08:44:00,891 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'nowcast+'], host='localhost')
2026-03-12 08:44:00,895 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'nowcast+'], host='localhost')
2026-03-12 08:44:00,899 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.nibi', 'nowcast+'], host='localhost')
2026-03-12 08:44:00,903 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'nowcast+'], host='localhost')
2026-03-12 08:44:02,270 INFO [ping_erddap] running in process 1301653
2026-03-12 08:44:02,272 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-12 08:44:02,273 INFO [ping_erddap] publishing log messages to tcp://*:5562
2026-03-12 08:44:02,276 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2026-03-12 08:44:02,278 INFO [ping_erddap] weather ERDDAP dataset flag file(s) created
2026-03-12 08:44:02,281 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2026-03-12 08:44:02,485 INFO [upload_forcing] running in process 1301658
2026-03-12 08:44:02,487 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-12 08:44:02,488 INFO [upload_forcing] publishing log messages to tcp://*:5563
2026-03-12 08:44:02,489 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-03-12 08:44:02,493 INFO [upload_forcing] running in process 1301656
2026-03-12 08:44:02,494 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-12 08:44:02,495 INFO [upload_forcing] publishing log messages to tcp://*:5565
2026-03-12 08:44:02,497 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-03-12 08:44:02,503 INFO [upload_forcing] running in process 1301657
2026-03-12 08:44:02,504 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-12 08:44:02,505 INFO [upload_forcing] publishing log messages to tcp://*:5566
2026-03-12 08:44:02,506 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-03-12 08:44:02,552 INFO [upload_forcing] running in process 1301655
2026-03-12 08:44:02,554 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-12 08:44:02,555 INFO [upload_forcing] publishing log messages to tcp://*:5567
2026-03-12 08:44:02,556 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-03-12 08:44:04,399 INFO [upload_forcing] nowcast+ 2026-03-12 forcing files upload to optimum-hindcast completed
2026-03-12 08:44:04,403 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-03-12 08:44:06,332 INFO [upload_forcing] nowcast+ 2026-03-12 forcing files upload to arbutus.cloud-nowcast completed
2026-03-12 08:44:06,336 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-03-12 08:44:06,500 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'nowcast+', '--run-date', '2026-03-12'], host='localhost')
2026-03-12 08:44:07,914 INFO [make_forcing_links] running in process 1301687
2026-03-12 08:44:07,915 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-12 08:44:07,916 INFO [make_forcing_links] publishing log messages to tcp://*:5562
2026-03-12 08:44:07,917 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2026-03-12 08:44:09,072 INFO [make_forcing_links] nowcast+ 2026-03-12 forcing file links on arbutus.cloud-nowcast created
2026-03-12 08:44:09,080 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2026-03-12 08:44:09,240 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2026-03-12'], host='arbutus.cloud-nowcast')
2026-03-12 08:44:09,540 INFO [upload_forcing] nowcast+ 2026-03-12 forcing files upload to orcinus-nowcast-agrif completed
2026-03-12 08:44:09,544 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-03-12 08:44:11,340 INFO [run_NEMO] running in process 12587
2026-03-12 08:44:11,341 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-12 08:44:11,342 INFO [run_NEMO] publishing log messages to tcp://*:5556
2026-03-12 08:44:11,355 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2026-03-12 08:44:12,096 WARNING [run_NEMO] There are uncommitted changes in /nemoShare/MEOPAR/nowcast-sys/SS-run-sets
2026-03-12 08:44:12,404 INFO [run_NEMO] nowcast: launching /nemoShare/MEOPAR/nowcast-sys/runs/12mar26nowcast_2026-03-12T084411.386989-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2026-03-12 08:44:12,418 INFO [run_NEMO] nowcast NEMO run for 2026-03-12 on arbutus.cloud-nowcast started
2026-03-12 08:44:12,423 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2026-03-12 08:44:12,576 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'nowcast'], host='arbutus.cloud-nowcast')
2026-03-12 08:44:14,559 INFO [watch_NEMO] running in process 12861
2026-03-12 08:44:14,560 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-12 08:44:14,561 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2026-03-12 08:44:14,563 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2026-03-12 08:44:14,622 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2026-03-12 08:44:14,821 INFO [upload_forcing] nowcast+ 2026-03-12 forcing files upload to robot.nibi completed
2026-03-12 08:44:14,826 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-03-12 08:49:14,623 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 19092727 = 2026-03-12 05:24:00 UTC, 22.5% complete
2026-03-12 08:54:14,623 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 19093251 = 2026-03-12 11:13:20 UTC, 46.8% complete
2026-03-12 08:59:14,623 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 19093813 = 2026-03-12 17:28:00 UTC, 72.8% complete
2026-03-12 09:04:14,625 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 19094367 = 2026-03-12 23:37:20 UTC, 98.5% complete
2026-03-12 09:09:14,628 INFO [watch_NEMO] nowcast NEMO run on arbutus.cloud-nowcast completed
2026-03-12 09:09:14,635 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2026-03-12 09:09:14,988 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'ssh', '--run-date', '2026-03-12'], host='localhost')
2026-03-12 09:09:15,003 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2026-03-12'], host='localhost')
2026-03-12 09:09:17,000 INFO [download_results] running in process 1301980
2026-03-12 09:09:17,002 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-12 09:09:17,003 INFO [download_results] publishing log messages to tcp://*:5562
2026-03-12 09:09:17,004 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2026-03-12 09:09:17,005 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/nowcast/12mar26 to /results/SalishSea/nowcast-blue.202111
2026-03-12 09:09:17,058 INFO [make_forcing_links] running in process 1301979
2026-03-12 09:09:17,059 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-12 09:09:17,061 INFO [make_forcing_links] publishing log messages to tcp://*:5563
2026-03-12 09:09:17,062 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2026-03-12 09:09:18,077 INFO [make_forcing_links] ssh 2026-03-12 forcing file links on arbutus.cloud-nowcast created
2026-03-12 09:09:18,085 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2026-03-12 09:09:18,244 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2026-03-12'], host='arbutus.cloud-nowcast')
2026-03-12 09:09:20,369 INFO [run_NEMO] running in process 13263
2026-03-12 09:09:20,370 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-12 09:09:20,371 INFO [run_NEMO] publishing log messages to tcp://*:5556
2026-03-12 09:09:20,372 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2026-03-12 09:09:21,123 WARNING [run_NEMO] There are uncommitted changes in /nemoShare/MEOPAR/nowcast-sys/SS-run-sets
2026-03-12 09:09:21,438 INFO [run_NEMO] forecast: launching /nemoShare/MEOPAR/nowcast-sys/runs/12mar26forecast_2026-03-12T090920.439493-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2026-03-12 09:09:21,452 INFO [run_NEMO] forecast NEMO run for 2026-03-12 on arbutus.cloud-nowcast started
2026-03-12 09:09:21,456 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2026-03-12 09:09:21,614 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'forecast'], host='arbutus.cloud-nowcast')
2026-03-12 09:09:23,820 INFO [watch_NEMO] running in process 13536
2026-03-12 09:09:23,821 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-12 09:09:23,823 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2026-03-12 09:09:23,824 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2026-03-12 09:09:23,928 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2026-03-12 09:09:47,807 INFO [download_results] nowcast 2026-03-12 results files from arbutus.cloud-nowcast downloaded
2026-03-12 09:09:47,814 INFO [manager] checklist updated with [results files] items from download_results worker
2026-03-12 09:09:47,995 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast', 'research', '--run-date', '2026-03-12'], host='localhost')
2026-03-12 09:09:48,001 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast', 'comparison', '--run-date', '2026-03-11'], host='localhost')
2026-03-12 09:09:48,006 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['nowcast', '--run-date', '2026-03-12'], host='localhost')
2026-03-12 09:09:52,748 INFO [make_CHS_currents_file] running in process 1302007
2026-03-12 09:09:52,756 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-12 09:09:52,757 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5562
2026-03-12 09:09:52,758 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555
2026-03-12 09:09:55,335 INFO [make_plots] running in process 1302006
2026-03-12 09:09:55,336 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-12 09:09:55,338 INFO [make_plots] publishing log messages to tcp://*:5563
2026-03-12 09:09:55,339 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2026-03-12 09:09:55,342 INFO [make_plots] running in process 1302004
2026-03-12 09:09:55,342 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-12 09:09:55,344 INFO [make_plots] publishing log messages to tcp://*:5565
2026-03-12 09:09:55,348 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2026-03-12 09:09:55,836 INFO [make_plots] preparing render list for 2026-03-12 NEMO nowcast-blue research figures
2026-03-12 09:09:55,838 INFO [make_plots] preparing render list for 2026-03-11 NEMO nowcast-blue comparison figures
2026-03-12 09:09:55,946 INFO [make_plots] added SH_wind figure to 2026-03-11 NEMO nowcast-blue comparison render list
2026-03-12 09:09:55,947 INFO [make_plots] added Compare_VENUS_East figure to 2026-03-11 NEMO nowcast-blue comparison render list
2026-03-12 09:09:55,949 INFO [make_plots] added Compare_VENUS_Central figure to 2026-03-11 NEMO nowcast-blue comparison render list
2026-03-12 09:09:55,950 INFO [make_plots] starting to render nemo nowcast comparison 11mar26 figures
2026-03-12 09:09:57,325 INFO [make_plots] added salinity_thalweg_and_surface figures to 2026-03-12 NEMO nowcast-blue research render list
2026-03-12 09:09:58,329 INFO [make_plots] added temperature_thalweg_and_surface figures to 2026-03-12 NEMO nowcast-blue research render list
2026-03-12 09:09:58,329 INFO [make_plots] added Currents_sections_and_surface figure to 2026-03-12 NEMO nowcast-blue research render list
2026-03-12 09:09:58,330 INFO [make_plots] added Currents_at_VENUS_Central figure to 2026-03-12 NEMO nowcast-blue research render list
2026-03-12 09:09:58,331 INFO [make_plots] added Currents_at_VENUS_East figure to 2026-03-12 NEMO nowcast-blue research render list
2026-03-12 09:09:58,331 INFO [make_plots] starting to render nemo nowcast research 12mar26 figures
2026-03-12 09:10:31,260 INFO [make_CHS_currents_file] Made CHS currents file for 2026-03-12 for nowcast
2026-03-12 09:10:31,263 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker
2026-03-12 09:11:06,995 INFO [make_plots] finished rendering nemo nowcast research 12mar26 figures
2026-03-12 09:11:06,998 INFO [make_plots] nemo research plots for 2026-03-12 nowcast completed
2026-03-12 09:11:07,019 INFO [manager] checklist updated with [plots] items from make_plots worker
2026-03-12 09:11:28,017 INFO [make_plots] finished rendering nemo nowcast comparison 11mar26 figures
2026-03-12 09:11:28,019 INFO [make_plots] nemo comparison plots for 2026-03-11 nowcast completed
2026-03-12 09:11:28,024 INFO [manager] checklist updated with [plots] items from make_plots worker
2026-03-12 09:14:23,927 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 19094905 = 2026-03-13 05:36:00 UTC, 15.6% complete
2026-03-12 09:19:23,929 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 19095416 = 2026-03-13 11:16:40 UTC, 31.3% complete
2026-03-12 09:24:23,931 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 19095951 = 2026-03-13 17:13:20 UTC, 47.9% complete
2026-03-12 09:29:23,933 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 19096474 = 2026-03-13 23:02:00 UTC, 64.0% complete
2026-03-12 09:34:23,936 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 19097010 = 2026-03-14 04:59:20 UTC, 80.5% complete
2026-03-12 09:39:23,937 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 19097517 = 2026-03-14 10:37:20 UTC, 96.2% complete
2026-03-12 09:44:23,942 INFO [watch_NEMO] forecast NEMO run on arbutus.cloud-nowcast completed
2026-03-12 09:44:23,950 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2026-03-12 09:44:24,322 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_wind_file', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2026-03-12'], host='arbutus.cloud-nowcast')
2026-03-12 09:44:24,342 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_current_file', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2026-03-12'], host='arbutus.cloud-nowcast')
2026-03-12 09:44:24,345 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'turbidity', '--run-date', '2026-03-12'], host='localhost')
2026-03-12 09:44:24,355 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'turbidity', '--run-date', '2026-03-12'], host='localhost')
2026-03-12 09:44:24,360 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.nibi', 'turbidity', '--run-date', '2026-03-12'], host='localhost')
2026-03-12 09:44:24,367 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'turbidity', '--run-date', '2026-03-12'], host='localhost')
2026-03-12 09:44:24,375 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2026-03-12'], host='localhost')
2026-03-12 09:44:26,488 INFO [download_results] running in process 1302424
2026-03-12 09:44:26,490 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-12 09:44:26,491 INFO [download_results] publishing log messages to tcp://*:5562
2026-03-12 09:44:26,492 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2026-03-12 09:44:26,493 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/forecast/12mar26 to /results/SalishSea/forecast.202111
2026-03-12 09:44:26,558 INFO [upload_forcing] running in process 1302421
2026-03-12 09:44:26,559 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-12 09:44:26,560 INFO [upload_forcing] publishing log messages to tcp://*:5563
2026-03-12 09:44:26,561 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-03-12 09:44:26,562 INFO [upload_forcing] running in process 1302422
2026-03-12 09:44:26,563 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-12 09:44:26,564 INFO [upload_forcing] publishing log messages to tcp://*:5565
2026-03-12 09:44:26,566 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-03-12 09:44:26,646 INFO [upload_forcing] running in process 1302419
2026-03-12 09:44:26,647 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-12 09:44:26,648 INFO [upload_forcing] publishing log messages to tcp://*:5566
2026-03-12 09:44:26,650 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-03-12 09:44:26,652 INFO [upload_forcing] running in process 1302420
2026-03-12 09:44:26,653 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-12 09:44:26,654 INFO [upload_forcing] publishing log messages to tcp://*:5567
2026-03-12 09:44:26,655 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-03-12 09:44:27,184 INFO [make_ww3_wind_file] running in process 14260
2026-03-12 09:44:27,185 INFO [make_ww3_wind_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-12 09:44:27,186 INFO [make_ww3_wind_file] publishing log messages to tcp://*:5570
2026-03-12 09:44:27,198 INFO [make_ww3_wind_file] connected to skookum.eos.ubc.ca port 5555
2026-03-12 09:44:27,199 INFO [make_ww3_wind_file] Creating wwatch3 wind forcing file for 2026-03-12 forecast run
2026-03-12 09:44:27,222 INFO [make_ww3_current_file] running in process 14261
2026-03-12 09:44:27,224 INFO [make_ww3_current_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-12 09:44:27,225 INFO [make_ww3_current_file] publishing log messages to tcp://*:5571
2026-03-12 09:44:27,226 INFO [make_ww3_current_file] connected to skookum.eos.ubc.ca port 5555
2026-03-12 09:44:27,227 INFO [make_ww3_current_file] Creating wwatch3 currents forcing file for 2026-03-12 forecast run
2026-03-12 09:44:27,320 INFO [upload_forcing] turbidity 2026-03-12 forcing files upload to optimum-hindcast completed
2026-03-12 09:44:27,325 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-03-12 09:44:27,703 INFO [upload_forcing] turbidity 2026-03-12 forcing files upload to arbutus.cloud-nowcast completed
2026-03-12 09:44:27,708 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-03-12 09:44:27,867 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2026-03-12'], host='localhost')
2026-03-12 09:44:28,935 INFO [upload_forcing] turbidity 2026-03-12 forcing files upload to orcinus-nowcast-agrif completed
2026-03-12 09:44:28,939 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-03-12 09:44:31,672 INFO [upload_forcing] turbidity 2026-03-12 forcing files upload to robot.nibi completed
2026-03-12 09:44:31,679 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-03-12 09:44:32,003 INFO [make_forcing_links] running in process 1302459
2026-03-12 09:44:32,037 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-12 09:44:32,059 INFO [make_forcing_links] publishing log messages to tcp://*:5565
2026-03-12 09:44:32,072 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2026-03-12 09:44:34,723 INFO [make_forcing_links] nowcast-green 2026-03-12 forcing file links on arbutus.cloud-nowcast created
2026-03-12 09:44:34,728 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2026-03-12 09:44:34,900 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2026-03-12'], host='arbutus.cloud-nowcast')
2026-03-12 09:44:38,655 INFO [run_NEMO] running in process 14903
2026-03-12 09:44:38,656 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-12 09:44:38,658 INFO [run_NEMO] publishing log messages to tcp://*:5556
2026-03-12 09:44:38,659 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2026-03-12 09:44:38,895 INFO [make_ww3_wind_file] wwatch3 wind forcing file created on arbutus.cloud-nowcast for 2026-03-12 forecast run
2026-03-12 09:44:38,900 INFO [manager] checklist updated with [WW3 wind forcing] items from make_ww3_wind_file worker
2026-03-12 09:44:39,490 WARNING [run_NEMO] There are uncommitted changes in /nemoShare/MEOPAR/nowcast-sys/SS-run-sets
2026-03-12 09:44:39,860 INFO [run_NEMO] nowcast-green: launching /nemoShare/MEOPAR/nowcast-sys/runs/12mar26nowcast-green_2026-03-12T094438.706547-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2026-03-12 09:44:39,885 INFO [run_NEMO] nowcast-green NEMO run for 2026-03-12 on arbutus.cloud-nowcast started
2026-03-12 09:44:39,892 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2026-03-12 09:44:40,053 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'nowcast-green'], host='arbutus.cloud-nowcast')
2026-03-12 09:44:42,373 INFO [watch_NEMO] running in process 15181
2026-03-12 09:44:42,377 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-12 09:44:42,382 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2026-03-12 09:44:42,387 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2026-03-12 09:44:42,451 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2026-03-12 09:45:15,683 INFO [download_results] forecast 2026-03-12 results files from arbutus.cloud-nowcast downloaded
2026-03-12 09:45:15,687 INFO [manager] checklist updated with [results files] items from download_results worker
2026-03-12 09:45:15,857 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['forecast', '--run-date', '2026-03-12'], host='localhost')
2026-03-12 09:45:21,399 INFO [make_CHS_currents_file] running in process 1302479
2026-03-12 09:45:21,400 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-12 09:45:21,402 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5562
2026-03-12 09:45:21,403 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555
2026-03-12 09:46:14,574 INFO [make_CHS_currents_file] Made CHS currents file for 2026-03-12 for forecast
2026-03-12 09:46:14,578 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker
2026-03-12 09:46:14,736 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['nemo', 'forecast', '--run-date', '2026-03-12'], host='localhost')
2026-03-12 09:46:16,845 INFO [update_forecast_datasets] running in process 1302513
2026-03-12 09:46:16,846 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-12 09:46:16,847 INFO [update_forecast_datasets] publishing log messages to tcp://*:5562
2026-03-12 09:46:16,848 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2026-03-12 09:46:16,849 INFO [update_forecast_datasets] updating nemo forecast directory for forecast/12mar26 run
2026-03-12 09:46:19,696 INFO [update_forecast_datasets] updated nemo forecast directory for forecast/12mar26 run: /results/SalishSea/rolling-forecasts/nemo
2026-03-12 09:46:19,697 INFO [update_forecast_datasets] nemo 2026-03-12 forecast rolling forecast datasets updated
2026-03-12 09:46:19,701 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2026-03-12 09:46:19,886 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nemo-forecast'], host='localhost')
2026-03-12 09:46:19,889 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'forecast', 'publish', '--run-date', '2026-03-12'], host='localhost')
2026-03-12 09:46:19,895 INFO [manager] launching NextWorker(module='nowcast.workers.make_surface_current_tiles', args=['forecast', '--run-date', '2026-03-12'], host='localhost')
2026-03-12 09:46:21,634 INFO [ping_erddap] running in process 1302518
2026-03-12 09:46:21,636 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-12 09:46:21,638 INFO [ping_erddap] publishing log messages to tcp://*:5562
2026-03-12 09:46:21,639 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2026-03-12 09:46:21,654 INFO [ping_erddap] nemo-forecast ERDDAP dataset flag file(s) created
2026-03-12 09:46:21,657 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2026-03-12 09:46:24,788 INFO [make_surface_current_tiles] running in process 1302521
2026-03-12 09:46:24,815 INFO [make_surface_current_tiles] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-12 09:46:24,817 INFO [make_surface_current_tiles] publishing log messages to tcp://*:5562
2026-03-12 09:46:24,818 INFO [make_surface_current_tiles] connected to skookum.eos.ubc.ca port 5555
2026-03-12 09:46:24,852 INFO [make_surface_current_tiles] starting rendering of tiles for 2026-03-12 forecast in 6 processes into /results/nowcast-sys/figures/surface_currents/forecast/12mar26
2026-03-12 09:46:27,217 INFO [make_plots] running in process 1302520
2026-03-12 09:46:27,241 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-12 09:46:27,264 INFO [make_plots] publishing log messages to tcp://*:5563
2026-03-12 09:46:27,265 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2026-03-12 09:46:27,955 INFO [make_plots] preparing render list for 2026-03-12 NEMO forecast publish figures
2026-03-12 09:46:28,507 INFO [make_plots] added Website_thumbnail figure to 2026-03-12 NEMO forecast publish render list
2026-03-12 09:46:28,511 INFO [make_plots] added Threshold_website figure to 2026-03-12 NEMO forecast publish render list
2026-03-12 09:46:28,512 INFO [make_plots] added PA_tidal_predictions figure to 2026-03-12 NEMO forecast publish render list
2026-03-12 09:46:28,513 INFO [make_plots] added Boundary Bay figure to 2026-03-12 NEMO forecast publish render list
2026-03-12 09:46:28,514 INFO [make_plots] added Campbell River figure to 2026-03-12 NEMO forecast publish render list
2026-03-12 09:46:28,516 INFO [make_plots] added Cherry Point figure to 2026-03-12 NEMO forecast publish render list
2026-03-12 09:46:28,518 INFO [make_plots] added Friday Harbor figure to 2026-03-12 NEMO forecast publish render list
2026-03-12 09:46:28,519 INFO [make_plots] added Halfmoon Bay figure to 2026-03-12 NEMO forecast publish render list
2026-03-12 09:46:28,520 INFO [make_plots] added Nanaimo figure to 2026-03-12 NEMO forecast publish render list
2026-03-12 09:46:28,521 INFO [make_plots] added Neah Bay figure to 2026-03-12 NEMO forecast publish render list
2026-03-12 09:46:28,522 INFO [make_plots] added New Westminster figure to 2026-03-12 NEMO forecast publish render list
2026-03-12 09:46:28,523 INFO [make_plots] added Patricia Bay figure to 2026-03-12 NEMO forecast publish render list
2026-03-12 09:46:28,524 INFO [make_plots] added Point Atkinson figure to 2026-03-12 NEMO forecast publish render list
2026-03-12 09:46:28,525 INFO [make_plots] added Port Renfrew figure to 2026-03-12 NEMO forecast publish render list
2026-03-12 09:46:28,526 INFO [make_plots] added Sand Heads figure to 2026-03-12 NEMO forecast publish render list
2026-03-12 09:46:28,527 INFO [make_plots] added Sandy Cove figure to 2026-03-12 NEMO forecast publish render list
2026-03-12 09:46:28,528 INFO [make_plots] added Squamish figure to 2026-03-12 NEMO forecast publish render list
2026-03-12 09:46:28,529 INFO [make_plots] added Victoria figure to 2026-03-12 NEMO forecast publish render list
2026-03-12 09:46:28,530 INFO [make_plots] added Woodwards Landing figure to 2026-03-12 NEMO forecast publish render list
2026-03-12 09:46:28,531 INFO [make_plots] starting to render nemo forecast publish 12mar26 figures
2026-03-12 09:47:04,183 INFO [make_ww3_current_file] wwatch3 currents forcing file created on arbutus.cloud-nowcast for 2026-03-12 forecast run
2026-03-12 09:47:04,188 INFO [manager] checklist updated with [WW3 currents forcing] items from make_ww3_current_file worker
2026-03-12 09:47:04,405 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2026-03-12'], host='arbutus.cloud-nowcast')
2026-03-12 09:47:06,589 INFO [run_ww3] running in process 15286
2026-03-12 09:47:06,609 INFO [run_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-12 09:47:06,615 INFO [run_ww3] publishing log messages to tcp://*:5572
2026-03-12 09:47:06,616 INFO [run_ww3] connected to skookum.eos.ubc.ca port 5555
2026-03-12 09:47:06,620 INFO [run_ww3] Created run directory /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/nowcast_2026-03-12T094706.588004-0700
2026-03-12 09:47:06,621 INFO [run_ww3] nowcast: launching /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/nowcast_2026-03-12T094706.588004-0700/SoGWW3.sh on arbutus.cloud-nowcast
2026-03-12 09:47:06,623 INFO [run_ww3] nowcast WaveWatch3 run for 2026-03-12 on arbutus.cloud-nowcast started
2026-03-12 09:47:06,624 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker
2026-03-12 09:47:06,918 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'nowcast'], host='arbutus.cloud-nowcast')
2026-03-12 09:47:08,954 INFO [watch_ww3] running in process 15373
2026-03-12 09:47:08,955 INFO [watch_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-12 09:47:08,956 INFO [watch_ww3] publishing log messages to tcp://*:5573
2026-03-12 09:47:08,957 INFO [watch_ww3] connected to skookum.eos.ubc.ca port 5555
2026-03-12 09:47:08,987 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2026-03-12 09:49:42,451 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19092418 = 2026-03-12 01:58:00 UTC, 8.2% complete
2026-03-12 09:52:08,988 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 93, 21.5% complete
2026-03-12 09:54:42,452 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19092615 = 2026-03-12 04:09:20 UTC, 17.3% complete
2026-03-12 09:56:32,643 WARNING [make_plots] invalid station number for Boundary Bay station: None
2026-03-12 09:56:32,644 WARNING [make_plots] can't resolve a valid CHS station code for Boundary Bay
2026-03-12 09:56:59,440 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb996/data for station 08074 Campbell River from 2026-03-07 00:00:00Z to 2026-03-14 11:55:00Z
2026-03-12 09:57:08,988 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 183, 42.4% complete
2026-03-12 09:57:57,075 INFO [make_surface_current_tiles] starting PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast/12mar26
2026-03-12 09:58:11,665 WARNING [make_plots] invalid station number for Halfmoon Bay station: None
2026-03-12 09:58:11,666 WARNING [make_plots] can't resolve a valid CHS station code for Halfmoon Bay
2026-03-12 09:58:29,064 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb96d/data for station 07917 Nanaimo from 2026-03-07 00:00:00Z to 2026-03-14 11:55:00Z
2026-03-12 09:59:06,066 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e13d0f4a073c4bbf93/data for station 07654 New Westminster from 2026-03-07 00:00:00Z to 2026-03-14 11:55:00Z
2026-03-12 09:59:24,713 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd2d/data for station 07277 Patricia Bay from 2026-03-07 00:00:00Z to 2026-03-14 11:55:00Z
2026-03-12 09:59:42,452 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19092814 = 2026-03-12 06:22:00 UTC, 26.5% complete
2026-03-12 09:59:44,076 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94c/data for station 07795 Point Atkinson from 2026-03-07 00:00:00Z to 2026-03-14 11:55:00Z
2026-03-12 10:00:01,962 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e23d0f4a073c4bc060/data for station 08525 Port Renfrew from 2026-03-07 00:00:00Z to 2026-03-14 11:55:00Z
2026-03-12 10:00:21,678 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb93c/data for station 07594 Sand Heads from 2026-03-07 00:00:00Z to 2026-03-14 11:55:00Z
2026-03-12 10:00:39,388 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc43b/data for station 07786 Sandy Cove from 2026-03-07 00:00:00Z to 2026-03-14 11:55:00Z
2026-03-12 10:00:59,327 WARNING [make_plots] invalid station number for Squamish station: None
2026-03-12 10:00:59,327 WARNING [make_plots] can't resolve a valid CHS station code for Squamish
2026-03-12 10:01:16,270 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd1e/data for station 07120 Victoria from 2026-03-07 00:00:00Z to 2026-03-14 11:55:00Z
2026-03-12 10:01:34,449 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc42d/data for station 07610 Woodwards Landing from 2026-03-07 00:00:00Z to 2026-03-14 11:55:00Z
2026-03-12 10:01:52,965 INFO [make_plots] finished rendering nemo forecast publish 12mar26 figures
2026-03-12 10:01:52,966 INFO [make_plots] nemo publish plots for 2026-03-12 forecast completed
2026-03-12 10:01:53,134 INFO [manager] checklist updated with [plots] items from make_plots worker
2026-03-12 10:01:53,284 INFO [manager] launching NextWorker(module='nowcast.workers.make_feeds', args=['forecast', '--run-date', '2026-03-12'], host='localhost')
2026-03-12 10:01:59,955 INFO [make_feeds] running in process 1307214
2026-03-12 10:01:59,957 INFO [make_feeds] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-12 10:01:59,958 INFO [make_feeds] publishing log messages to tcp://*:5563
2026-03-12 10:01:59,960 INFO [make_feeds] connected to skookum.eos.ubc.ca port 5555
2026-03-12 10:02:08,989 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 288, 66.7% complete
2026-03-12 10:02:16,195 INFO [make_feeds] ATOM feeds for 2026-03-12 forecast run completed
2026-03-12 10:02:16,199 INFO [manager] checklist updated with [storm surge alert feeds] items from make_feeds worker
2026-03-12 10:03:47,654 INFO [make_surface_current_tiles] finished PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast/12mar26
2026-03-12 10:03:47,692 INFO [make_surface_current_tiles] finished rendering of tiles for 2026-03-12 forecast into /results/nowcast-sys/figures/surface_currents/forecast/12mar26
2026-03-12 10:03:47,717 INFO [make_surface_current_tiles] surface current tile figures for 2026-03-12 forecast completed
2026-03-12 10:03:48,041 INFO [manager] checklist updated with [surface current tiles] items from make_surface_current_tiles worker
2026-03-12 10:04:42,453 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19093012 = 2026-03-12 08:34:00 UTC, 35.7% complete
2026-03-12 10:07:08,990 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 390, 90.3% complete
2026-03-12 10:09:42,454 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19093210 = 2026-03-12 10:46:00 UTC, 44.9% complete
2026-03-12 10:12:08,992 INFO [watch_ww3] nowcast WWATCH3 run on arbutus.cloud-nowcast completed
2026-03-12 10:12:08,996 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2026-03-12 10:12:09,676 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2026-03-12'], host='localhost')
2026-03-12 10:12:09,683 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2026-03-12'], host='arbutus.cloud-nowcast')
2026-03-12 10:12:11,713 INFO [run_ww3] running in process 15597
2026-03-12 10:12:11,734 INFO [run_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-12 10:12:11,746 INFO [run_ww3] publishing log messages to tcp://*:5572
2026-03-12 10:12:11,748 INFO [run_ww3] connected to skookum.eos.ubc.ca port 5555
2026-03-12 10:12:11,751 INFO [run_ww3] Created run directory /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast_2026-03-12T101211.714374-0700
2026-03-12 10:12:11,752 INFO [run_ww3] forecast: launching /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast_2026-03-12T101211.714374-0700/SoGWW3.sh on arbutus.cloud-nowcast
2026-03-12 10:12:11,754 INFO [run_ww3] forecast WaveWatch3 run for 2026-03-12 on arbutus.cloud-nowcast started
2026-03-12 10:12:11,755 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker
2026-03-12 10:12:12,010 INFO [download_wwatch3_results] running in process 1307336
2026-03-12 10:12:12,032 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-12 10:12:12,044 INFO [download_wwatch3_results] publishing log messages to tcp://*:5562
2026-03-12 10:12:12,045 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2026-03-12 10:12:12,082 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'forecast'], host='arbutus.cloud-nowcast')
2026-03-12 10:12:14,056 INFO [watch_ww3] running in process 15752
2026-03-12 10:12:14,059 INFO [watch_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-12 10:12:14,061 INFO [watch_ww3] publishing log messages to tcp://*:5573
2026-03-12 10:12:14,062 INFO [watch_ww3] connected to skookum.eos.ubc.ca port 5555
2026-03-12 10:12:14,099 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2026-03-12 10:12:45,458 INFO [download_wwatch3_results] nowcast 2026-03-12 results files from arbutus.cloud-nowcast downloaded
2026-03-12 10:12:45,463 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2026-03-12 10:14:42,456 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19093407 = 2026-03-12 12:57:20 UTC, 54.0% complete
2026-03-12 10:17:14,094 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 93, 14.4% complete
2026-03-12 10:19:42,459 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19093605 = 2026-03-12 15:09:20 UTC, 63.2% complete
2026-03-12 10:22:14,096 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 192, 29.6% complete
2026-03-12 10:24:42,461 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19093802 = 2026-03-12 17:20:40 UTC, 72.3% complete
2026-03-12 10:27:14,098 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 300, 46.3% complete
2026-03-12 10:29:42,461 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19093999 = 2026-03-12 19:32:00 UTC, 81.4% complete
2026-03-12 10:32:14,098 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 405, 62.5% complete
2026-03-12 10:34:42,462 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19094195 = 2026-03-12 21:42:40 UTC, 90.5% complete
2026-03-12 10:37:14,099 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 507, 78.2% complete
2026-03-12 10:39:42,463 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19094392 = 2026-03-12 23:54:00 UTC, 99.6% complete
2026-03-12 10:42:14,100 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 606, 93.5% complete
2026-03-12 10:44:42,471 INFO [watch_NEMO] nowcast-green NEMO run on arbutus.cloud-nowcast completed
2026-03-12 10:44:42,475 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2026-03-12 10:44:42,903 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2026-03-12'], host='localhost')
2026-03-12 10:44:44,983 INFO [download_results] running in process 1308604
2026-03-12 10:44:44,984 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-12 10:44:44,985 INFO [download_results] publishing log messages to tcp://*:5562
2026-03-12 10:44:44,986 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2026-03-12 10:44:44,987 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/nowcast-green/12mar26 to /results2/SalishSea/nowcast-green.202111
2026-03-12 10:47:04,206 INFO [download_results] nowcast-green 2026-03-12 results files from arbutus.cloud-nowcast downloaded
2026-03-12 10:47:04,213 INFO [manager] checklist updated with [results files] items from download_results worker
2026-03-12 10:47:04,691 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast-green', 'research', '--run-date', '2026-03-12'], host='localhost')
2026-03-12 10:47:04,699 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nowcast-green'], host='localhost')
2026-03-12 10:47:04,704 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'physics', '--run-date', '2026-03-12'], host='localhost')
2026-03-12 10:47:04,710 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'biology', '--run-date', '2026-03-12'], host='localhost')
2026-03-12 10:47:04,714 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'chemistry', '--run-date', '2026-03-12'], host='localhost')
2026-03-12 10:47:06,549 INFO [ping_erddap] running in process 1308646
2026-03-12 10:47:06,550 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-12 10:47:06,550 INFO [ping_erddap] publishing log messages to tcp://*:5562
2026-03-12 10:47:06,551 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2026-03-12 10:47:06,558 INFO [ping_erddap] nowcast-green ERDDAP dataset flag file(s) created
2026-03-12 10:47:06,559 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2026-03-12 10:47:09,237 INFO [make_averaged_dataset] running in process 1308649
2026-03-12 10:47:09,240 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-12 10:47:09,241 INFO [make_averaged_dataset] publishing log messages to tcp://*:5562
2026-03-12 10:47:09,242 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555
2026-03-12 10:47:09,243 INFO [make_averaged_dataset] creating day-averaged dataset for 12-Mar-2026 chemistry
2026-03-12 10:47:09,246 INFO [make_averaged_dataset] running in process 1308647
2026-03-12 10:47:09,248 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-12 10:47:09,249 INFO [make_averaged_dataset] publishing log messages to tcp://*:5563
2026-03-12 10:47:09,252 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555
2026-03-12 10:47:09,253 INFO [make_averaged_dataset] creating day-averaged dataset for 12-Mar-2026 physics
2026-03-12 10:47:09,264 INFO [make_averaged_dataset] running in process 1308648
2026-03-12 10:47:09,266 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-12 10:47:09,269 INFO [make_averaged_dataset] publishing log messages to tcp://*:5565
2026-03-12 10:47:09,270 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555
2026-03-12 10:47:09,270 INFO [make_averaged_dataset] creating day-averaged dataset for 12-Mar-2026 biology
2026-03-12 10:47:10,710 INFO [make_plots] running in process 1308644
2026-03-12 10:47:10,711 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-12 10:47:10,712 INFO [make_plots] publishing log messages to tcp://*:5566
2026-03-12 10:47:10,715 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2026-03-12 10:47:11,245 INFO [make_plots] preparing render list for 2026-03-12 NEMO nowcast-green research figures
2026-03-12 10:47:13,864 INFO [make_plots] added temperature_thalweg_and_surface figures to 2026-03-12 NEMO nowcast-green research render list
2026-03-12 10:47:14,101 INFO [watch_ww3] forecast WWATCH3 run on arbutus.cloud-nowcast completed
2026-03-12 10:47:14,142 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2026-03-12 10:47:14,645 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2026-03-12'], host='localhost')
2026-03-12 10:47:14,730 INFO [make_plots] added nitrate_thalweg_and_surface figures to 2026-03-12 NEMO nowcast-green research render list
2026-03-12 10:47:15,554 INFO [make_plots] added ammonium_thalweg_and_surface figures to 2026-03-12 NEMO nowcast-green research render list
2026-03-12 10:47:15,971 INFO [download_wwatch3_results] running in process 1308773
2026-03-12 10:47:15,972 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-12 10:47:15,973 INFO [download_wwatch3_results] publishing log messages to tcp://*:5567
2026-03-12 10:47:15,974 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2026-03-12 10:47:16,340 INFO [make_plots] added silicon_thalweg_and_surface figures to 2026-03-12 NEMO nowcast-green research render list
2026-03-12 10:47:17,128 INFO [make_plots] added dissolved_organic_nitrogen_thalweg_and_surface figures to 2026-03-12 NEMO nowcast-green research render list
2026-03-12 10:47:17,920 INFO [make_plots] added particulate_organic_nitrogen_thalweg_and_surface figures to 2026-03-12 NEMO nowcast-green research render list
2026-03-12 10:47:18,746 INFO [make_plots] added biogenic_silicon_thalweg_and_surface figures to 2026-03-12 NEMO nowcast-green research render list
2026-03-12 10:47:19,670 INFO [make_plots] added diatoms_thalweg_and_surface figures to 2026-03-12 NEMO nowcast-green research render list
2026-03-12 10:47:20,587 INFO [make_plots] added flagellates_thalweg_and_surface figures to 2026-03-12 NEMO nowcast-green research render list
2026-03-12 10:47:21,465 INFO [make_plots] added microzooplankton_thalweg_and_surface figures to 2026-03-12 NEMO nowcast-green research render list
2026-03-12 10:47:22,398 INFO [make_plots] added mesozooplankton_thalweg_and_surface figures to 2026-03-12 NEMO nowcast-green research render list
2026-03-12 10:47:23,425 INFO [make_plots] added turbidity_thalweg_and_surface figures to 2026-03-12 NEMO nowcast-green research render list
2026-03-12 10:47:24,576 INFO [make_plots] added temperature_salinity_timeseries figure to 2026-03-12 NEMO nowcast-green research render list
2026-03-12 10:47:24,577 INFO [make_plots] added nitrate_diatoms_timeseries figure to 2026-03-12 NEMO nowcast-green research render list
2026-03-12 10:47:24,578 INFO [make_plots] added diatoms_flagellates_timeseries figure to 2026-03-12 NEMO nowcast-green research render list
2026-03-12 10:47:24,578 INFO [make_plots] added z1_z2_zooplankton_timeseries figure to 2026-03-12 NEMO nowcast-green research render list
2026-03-12 10:47:24,579 INFO [make_plots] starting to render nemo nowcast-green research 12mar26 figures
2026-03-12 10:47:42,477 INFO [make_averaged_dataset] day-averaged dataset for 12-Mar-2026 physics created
2026-03-12 10:47:42,505 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker
2026-03-12 10:47:59,814 INFO [download_wwatch3_results] forecast 2026-03-12 results files from arbutus.cloud-nowcast downloaded
2026-03-12 10:47:59,817 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2026-03-12 10:48:00,124 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['wwatch3', 'forecast', '--run-date', '2026-03-12'], host='localhost')
2026-03-12 10:48:02,118 INFO [update_forecast_datasets] running in process 1308792
2026-03-12 10:48:02,119 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-12 10:48:02,119 INFO [update_forecast_datasets] publishing log messages to tcp://*:5563
2026-03-12 10:48:02,120 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2026-03-12 10:48:02,121 INFO [update_forecast_datasets] updating wwatch3 most_recent_forecast directory from forecast/12mar26 run
2026-03-12 10:48:02,124 INFO [update_forecast_datasets] updated wwatch3 most_recent_forecast directory from forecast/12mar26 run
2026-03-12 10:48:02,125 INFO [update_forecast_datasets] updating wwatch3 forecast directory for forecast/12mar26 run
2026-03-12 10:48:02,546 INFO [update_forecast_datasets] updated wwatch3 forecast directory for forecast/12mar26 run: /results/SalishSea/rolling-forecasts/wwatch3
2026-03-12 10:48:02,547 INFO [update_forecast_datasets] wwatch3 2026-03-12 forecast rolling forecast datasets updated
2026-03-12 10:48:02,550 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2026-03-12 10:48:02,862 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['wwatch3-forecast'], host='localhost')
2026-03-12 10:48:04,617 INFO [ping_erddap] running in process 1308797
2026-03-12 10:48:04,641 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-12 10:48:04,649 INFO [ping_erddap] publishing log messages to tcp://*:5563
2026-03-12 10:48:04,650 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2026-03-12 10:48:04,703 INFO [ping_erddap] wwatch3-forecast ERDDAP dataset flag file(s) created
2026-03-12 10:48:04,706 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2026-03-12 10:48:05,005 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['wwatch3', 'forecast', 'publish', '--run-date', '2026-03-12'], host='localhost')
2026-03-12 10:48:07,379 INFO [make_averaged_dataset] day-averaged dataset for 12-Mar-2026 chemistry created
2026-03-12 10:48:07,424 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker
2026-03-12 10:48:10,215 INFO [make_plots] running in process 1308803
2026-03-12 10:48:10,233 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-12 10:48:10,236 INFO [make_plots] publishing log messages to tcp://*:5562
2026-03-12 10:48:10,237 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2026-03-12 10:48:10,237 INFO [make_plots] preparing render list for 2026-03-12 WaveWatch3 forecast publish figures
2026-03-12 10:48:10,238 INFO [make_plots] added Halibut Bank figure to 2026-03-12 WaveWatch3 forecast publish render list
2026-03-12 10:48:10,239 INFO [make_plots] added Sentry Shoal figure to 2026-03-12 WaveWatch3 forecast publish render list
2026-03-12 10:48:10,240 INFO [make_plots] starting to render wwatch3 forecast publish 12mar26 figures
2026-03-12 10:48:11,369 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46146 Halibut Bank
2026-03-12 10:48:18,700 INFO [make_averaged_dataset] day-averaged dataset for 12-Mar-2026 biology created
2026-03-12 10:48:18,703 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker
2026-03-12 10:48:21,513 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46131 Sentry Shoal
2026-03-12 10:48:26,318 INFO [make_plots] finished rendering wwatch3 forecast publish 12mar26 figures
2026-03-12 10:48:26,319 INFO [make_plots] wwatch3 publish plots for 2026-03-12 forecast completed
2026-03-12 10:48:26,322 INFO [manager] checklist updated with [plots] items from make_plots worker
2026-03-12 11:59:32,844 INFO [make_plots] finished rendering nemo nowcast-green research 12mar26 figures
2026-03-12 11:59:32,911 INFO [make_plots] nemo research plots for 2026-03-12 nowcast-green completed
2026-03-12 11:59:32,980 INFO [manager] checklist updated with [plots] items from make_plots worker
2026-03-12 14:42:28,205 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/18/ to /results/forcing/atmospheric/continental2.5/GRIB/20260312/18/
2026-03-12 14:42:28,232 INFO [collect_weather] 2026-03-12 2.5km weather forecast 18 collection complete
2026-03-12 14:42:28,255 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2026-03-12 14:42:28,850 INFO [manager] launching NextWorker(module='nowcast.workers.download_weather', args=['00', '1km'], host='localhost')
2026-03-12 14:42:28,861 INFO [manager] launching NextWorker(module='nowcast.workers.download_weather', args=['12', '1km'], host='localhost')
2026-03-12 14:42:28,867 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['00', '2.5km'], host='localhost')
2026-03-12 14:42:28,877 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['00', '--fcst-date', '2026-03-13'], host='localhost')
2026-03-12 14:42:29,665 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20260312/18/
2026-03-12 14:42:29,673 INFO [crop_gribs] 2026-03-12 18 GRIBs cropping complete
2026-03-12 14:42:29,680 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2026-03-12 14:42:31,508 INFO [download_weather] running in process 1311329
2026-03-12 14:42:31,509 INFO [download_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-12 14:42:31,509 INFO [download_weather] publishing log messages to tcp://*:5562
2026-03-12 14:42:31,510 INFO [download_weather] connected to skookum.eos.ubc.ca port 5555
2026-03-12 14:42:31,511 INFO [download_weather] downloading 12 1 km forecast GRIB2 files for 20260312
2026-03-12 14:42:31,512 INFO [download_weather] running in process 1311328
2026-03-12 14:42:31,513 INFO [download_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-12 14:42:31,513 INFO [download_weather] publishing log messages to tcp://*:5563
2026-03-12 14:42:31,514 INFO [download_weather] connected to skookum.eos.ubc.ca port 5555
2026-03-12 14:42:31,515 INFO [download_weather] downloading 00 1 km forecast GRIB2 files for 20260312
2026-03-12 14:42:31,516 INFO [collect_weather] running in process 1311330
2026-03-12 14:42:31,517 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-12 14:42:31,518 INFO [collect_weather] publishing log messages to tcp://*:5564
2026-03-12 14:42:31,518 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2026-03-12 14:42:31,854 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/00/
2026-03-12 14:42:33,616 INFO [crop_gribs] running in process 1311331
2026-03-12 14:42:33,620 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-12 14:42:33,621 INFO [crop_gribs] publishing log messages to tcp://*:5565
2026-03-12 14:42:33,622 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2026-03-12 14:42:33,623 INFO [crop_gribs] cropping 2026-03-13 ECCC HRDPS 2.5km continental 00Z GRIB files to SalishSeaCast subdomain
2026-03-12 14:42:33,632 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20260313/00/
2026-03-12 14:46:10,874 INFO [download_weather] 2026-03-12 1km weather forecast 00 downloads complete
2026-03-12 14:46:10,877 INFO [manager] checklist updated with [weather forecast] items from download_weather worker
2026-03-12 14:46:19,649 INFO [download_weather] 2026-03-12 1km weather forecast 12 downloads complete
2026-03-12 14:46:19,651 INFO [manager] checklist updated with [weather forecast] items from download_weather worker
2026-03-12 20:46:01,886 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/00/ to /results/forcing/atmospheric/continental2.5/GRIB/20260313/00/
2026-03-12 20:46:01,914 INFO [collect_weather] 2026-03-13 2.5km weather forecast 00 collection complete
2026-03-12 20:46:01,947 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2026-03-12 20:46:02,445 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20260313/00/
2026-03-12 20:46:02,452 INFO [crop_gribs] 2026-03-13 00 GRIBs cropping complete
2026-03-12 20:46:02,552 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['06', '2.5km'], host='localhost')
2026-03-12 20:46:02,562 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['06', '--fcst-date', '2026-03-13'], host='localhost')
2026-03-12 20:46:02,577 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2026-03-12 20:46:05,365 INFO [collect_weather] running in process 1315145
2026-03-12 20:46:05,366 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-12 20:46:05,367 INFO [collect_weather] publishing log messages to tcp://*:5562
2026-03-12 20:46:05,368 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2026-03-12 20:46:05,440 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/06/
2026-03-12 20:46:06,997 INFO [crop_gribs] running in process 1315146
2026-03-12 20:46:06,998 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-12 20:46:06,999 INFO [crop_gribs] publishing log messages to tcp://*:5563
2026-03-12 20:46:07,000 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2026-03-12 20:46:07,000 INFO [crop_gribs] cropping 2026-03-13 ECCC HRDPS 2.5km continental 06Z GRIB files to SalishSeaCast subdomain
2026-03-12 20:46:07,009 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20260313/06/
2026-03-13 02:41:48,835 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/06/ to /results/forcing/atmospheric/continental2.5/GRIB/20260313/06/
2026-03-13 02:41:48,867 INFO [collect_weather] 2026-03-13 2.5km weather forecast 06 collection complete
2026-03-13 02:41:48,968 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2026-03-13 02:41:49,245 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20260313/06/
2026-03-13 02:41:49,250 INFO [crop_gribs] 2026-03-13 06 GRIBs cropping complete
2026-03-13 02:41:49,824 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Capilano', '--data-date', '2026-03-12'], host='localhost')
2026-03-13 02:41:49,842 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'ChilliwackVedder', '--data-date', '2026-03-12'], host='localhost')
2026-03-13 02:41:49,847 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'ClowhomClowhomLake', '--data-date', '2026-03-12'], host='localhost')
2026-03-13 02:41:49,857 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Englishman', '--data-date', '2026-03-12'], host='localhost')
2026-03-13 02:41:49,863 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Fraser', '--data-date', '2026-03-12'], host='localhost')
2026-03-13 02:41:49,869 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'HomathkoMouth', '--data-date', '2026-03-12'], host='localhost')
2026-03-13 02:41:49,873 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'NicomeklLangley', '--data-date', '2026-03-12'], host='localhost')
2026-03-13 02:41:49,878 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'RobertsCreek', '--data-date', '2026-03-12'], host='localhost')
2026-03-13 02:41:49,884 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SalmonSayward', '--data-date', '2026-03-12'], host='localhost')
2026-03-13 02:41:49,888 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SanJuanPortRenfrew', '--data-date', '2026-03-12'], host='localhost')
2026-03-13 02:41:49,895 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SquamishBrackendale', '--data-date', '2026-03-12'], host='localhost')
2026-03-13 02:41:49,904 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaScotty', '--data-date', '2026-03-12'], host='localhost')
2026-03-13 02:41:49,908 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaBypass', '--data-date', '2026-03-12'], host='localhost')
2026-03-13 02:41:49,916 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaDiversion', '--data-date', '2026-03-12'], host='localhost')
2026-03-13 02:41:49,927 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ctd', args=['SCVIP'], host='localhost')
2026-03-13 02:41:49,937 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ctd', args=['SEVIP'], host='localhost')
2026-03-13 02:41:49,950 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ferry', args=['TWDP'], host='localhost')
2026-03-13 02:41:49,957 INFO [manager] launching NextWorker(module='nowcast.workers.get_vfpa_hadcp', args=['--data-date', '2026-03-12'], host='localhost')
2026-03-13 02:41:49,977 INFO [manager] launching NextWorker(module='nowcast.workers.collect_NeahBay_ssh', args=['00'], host='localhost')
2026-03-13 02:41:49,991 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['12', '2.5km'], host='localhost')
2026-03-13 02:41:50,002 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['12'], host='localhost')
2026-03-13 02:41:50,021 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2026-03-13 02:41:50,915 INFO [manager] launching NextWorker(module='nowcast.workers.grib_to_netcdf', args=['forecast2'], host='localhost')
2026-03-13 02:41:53,770 INFO [collect_NeahBay_ssh] running in process 1319174
2026-03-13 02:41:53,771 INFO [collect_NeahBay_ssh] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-13 02:41:53,773 INFO [collect_NeahBay_ssh] publishing log messages to tcp://*:5562
2026-03-13 02:41:53,775 INFO [collect_NeahBay_ssh] connected to skookum.eos.ubc.ca port 5555
2026-03-13 02:41:53,778 INFO [collect_NeahBay_ssh] collecting Neah Bay ssh 00Z obs/forecast for 2026-03-13
2026-03-13 02:41:53,779 INFO [collect_weather] running in process 1319175
2026-03-13 02:41:53,780 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-13 02:41:53,781 INFO [collect_weather] publishing log messages to tcp://*:5563
2026-03-13 02:41:53,782 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2026-03-13 02:41:53,862 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/12/
2026-03-13 02:41:55,236 INFO [collect_NeahBay_ssh] 2026-03-13T00:00:00-07:00 Neah Bay ssh 00Z obs/forecast data collection complete
2026-03-13 02:41:55,240 INFO [manager] checklist updated with [Neah Bay ssh data] items from collect_NeahBay_ssh worker
2026-03-13 02:41:55,442 INFO [collect_river_data] running in process 1319155
2026-03-13 02:41:55,443 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-13 02:41:55,443 INFO [collect_river_data] publishing log messages to tcp://*:5564
2026-03-13 02:41:55,444 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-13 02:41:55,445 INFO [collect_river_data] Collecting ECCC Capilano river data for 2026-03-12
2026-03-13 02:41:55,449 INFO [collect_river_data] running in process 1319169
2026-03-13 02:41:55,451 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-13 02:41:55,452 INFO [collect_river_data] publishing log messages to tcp://*:5565
2026-03-13 02:41:55,453 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-13 02:41:55,454 INFO [collect_river_data] Collecting ECCC TheodosiaDiversion river data for 2026-03-12
2026-03-13 02:41:55,564 INFO [collect_river_data] Appended ECCC Capilano river average discharge for 2026-03-12 to: /opp/observations/rivers/Capilano/Caplilano_08GA010_day_avg_flow
2026-03-13 02:41:55,565 INFO [collect_river_data] ECCC Capilano river data collection for 2026-03-12 completed
2026-03-13 02:41:55,580 INFO [collect_river_data] running in process 1319168
2026-03-13 02:41:55,582 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-13 02:41:55,583 INFO [collect_river_data] publishing log messages to tcp://*:5566
2026-03-13 02:41:55,584 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-13 02:41:55,585 INFO [collect_river_data] Collecting ECCC TheodosiaBypass river data for 2026-03-12
2026-03-13 02:41:55,600 INFO [collect_river_data] Appended ECCC TheodosiaDiversion river average discharge for 2026-03-12 to: /results/forcing/rivers/observations/Theodosia_Diversion_flow
2026-03-13 02:41:55,601 INFO [collect_river_data] ECCC TheodosiaDiversion river data collection for 2026-03-12 completed
2026-03-13 02:41:55,604 INFO [collect_river_data] running in process 1319157
2026-03-13 02:41:55,605 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-13 02:41:55,606 INFO [collect_river_data] publishing log messages to tcp://*:5567
2026-03-13 02:41:55,607 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-13 02:41:55,608 INFO [collect_river_data] Collecting ECCC ClowhomClowhomLake river data for 2026-03-12
2026-03-13 02:41:55,611 INFO [collect_river_data] running in process 1319156
2026-03-13 02:41:55,612 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-13 02:41:55,613 INFO [collect_river_data] publishing log messages to tcp://*:5568
2026-03-13 02:41:55,614 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-13 02:41:55,615 INFO [collect_river_data] Collecting ECCC ChilliwackVedder river data for 2026-03-12
2026-03-13 02:41:55,622 INFO [collect_river_data] Appended ECCC TheodosiaBypass river average discharge for 2026-03-12 to: /results/forcing/rivers/observations/Theodosia_Bypass_flow
2026-03-13 02:41:55,623 INFO [collect_river_data] ECCC TheodosiaBypass river data collection for 2026-03-12 completed
2026-03-13 02:41:55,648 INFO [collect_river_data] Appended ECCC ClowhomClowhomLake river average discharge for 2026-03-12 to: /results/forcing/rivers/observations/Clowhom_ClowhomLake_flow
2026-03-13 02:41:55,649 INFO [collect_river_data] ECCC ClowhomClowhomLake river data collection for 2026-03-12 completed
2026-03-13 02:41:55,667 INFO [collect_river_data] running in process 1319159
2026-03-13 02:41:55,668 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-13 02:41:55,669 INFO [collect_river_data] publishing log messages to tcp://*:5569
2026-03-13 02:41:55,670 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-13 02:41:55,671 INFO [collect_river_data] Collecting ECCC Fraser river data for 2026-03-12
2026-03-13 02:41:55,689 INFO [collect_river_data] running in process 1319167
2026-03-13 02:41:55,690 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-13 02:41:55,694 INFO [collect_river_data] publishing log messages to tcp://*:5570
2026-03-13 02:41:55,699 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-13 02:41:55,700 INFO [collect_river_data] Collecting ECCC TheodosiaScotty river data for 2026-03-12
2026-03-13 02:41:55,702 INFO [collect_river_data] Appended ECCC ChilliwackVedder river average discharge for 2026-03-12 to: /results/forcing/rivers/observations/Chilliwack_Vedder_flow
2026-03-13 02:41:55,703 INFO [collect_river_data] ECCC ChilliwackVedder river data collection for 2026-03-12 completed
2026-03-13 02:41:55,718 INFO [collect_river_data] running in process 1319160
2026-03-13 02:41:55,719 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-13 02:41:55,720 INFO [collect_river_data] publishing log messages to tcp://*:5571
2026-03-13 02:41:55,722 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-13 02:41:55,723 INFO [collect_river_data] Collecting ECCC HomathkoMouth river data for 2026-03-12
2026-03-13 02:41:55,753 INFO [collect_river_data] Appended ECCC TheodosiaScotty river average discharge for 2026-03-12 to: /results/forcing/rivers/observations/Theodosia_Scotty_flow
2026-03-13 02:41:55,754 INFO [collect_river_data] ECCC TheodosiaScotty river data collection for 2026-03-12 completed
2026-03-13 02:41:55,757 INFO [collect_river_data] running in process 1319158
2026-03-13 02:41:55,758 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-13 02:41:55,758 INFO [collect_river_data] publishing log messages to tcp://*:5572
2026-03-13 02:41:55,759 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-13 02:41:55,760 INFO [collect_river_data] Collecting ECCC Englishman river data for 2026-03-12
2026-03-13 02:41:55,761 INFO [collect_river_data] Appended ECCC HomathkoMouth river average discharge for 2026-03-12 to: /results/forcing/rivers/observations/Homathko_Mouth_flow
2026-03-13 02:41:55,762 INFO [collect_river_data] ECCC HomathkoMouth river data collection for 2026-03-12 completed
2026-03-13 02:41:55,765 INFO [collect_river_data] running in process 1319166
2026-03-13 02:41:55,766 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-13 02:41:55,766 INFO [collect_river_data] publishing log messages to tcp://*:5573
2026-03-13 02:41:55,767 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-13 02:41:55,768 INFO [collect_river_data] Collecting ECCC SquamishBrackendale river data for 2026-03-12
2026-03-13 02:41:55,770 INFO [collect_river_data] running in process 1319161
2026-03-13 02:41:55,771 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-13 02:41:55,771 INFO [collect_river_data] publishing log messages to tcp://*:5574
2026-03-13 02:41:55,773 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-13 02:41:55,774 INFO [collect_river_data] Collecting ECCC NicomeklLangley river data for 2026-03-12
2026-03-13 02:41:55,774 INFO [collect_river_data] running in process 1319163
2026-03-13 02:41:55,776 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-13 02:41:55,777 INFO [collect_river_data] publishing log messages to tcp://*:5575
2026-03-13 02:41:55,778 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-13 02:41:55,780 INFO [collect_river_data] Collecting ECCC RobertsCreek river data for 2026-03-12
2026-03-13 02:41:55,787 INFO [collect_river_data] running in process 1319164
2026-03-13 02:41:55,788 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-13 02:41:55,789 INFO [collect_river_data] publishing log messages to tcp://*:5576
2026-03-13 02:41:55,790 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-13 02:41:55,790 INFO [collect_river_data] Collecting ECCC SalmonSayward river data for 2026-03-12
2026-03-13 02:41:55,804 INFO [collect_river_data] Appended ECCC SquamishBrackendale river average discharge for 2026-03-12 to: /results/forcing/rivers/observations/Squamish_Brackendale_flow
2026-03-13 02:41:55,806 INFO [collect_river_data] ECCC SquamishBrackendale river data collection for 2026-03-12 completed
2026-03-13 02:41:55,812 INFO [collect_river_data] running in process 1319165
2026-03-13 02:41:55,814 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-13 02:41:55,815 INFO [collect_river_data] publishing log messages to tcp://*:5577
2026-03-13 02:41:55,816 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-13 02:41:55,816 INFO [collect_river_data] Collecting ECCC SanJuanPortRenfrew river data for 2026-03-12
2026-03-13 02:41:55,817 INFO [manager] launching NextWorker(module='nowcast.workers.make_ssh_files', args=['forecast2', '--run-date', '2026-03-13'], host='localhost')
2026-03-13 02:41:55,820 INFO [collect_river_data] Appended ECCC RobertsCreek river average discharge for 2026-03-12 to: /results/forcing/rivers/observations/RobertsCreek_flow
2026-03-13 02:41:55,822 INFO [collect_river_data] ECCC RobertsCreek river data collection for 2026-03-12 completed
2026-03-13 02:41:55,824 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-13 02:41:55,826 INFO [collect_river_data] Appended ECCC NicomeklLangley river average discharge for 2026-03-12 to: /results/forcing/rivers/observations/Nicomekl_Langley_flow
2026-03-13 02:41:55,827 INFO [collect_river_data] Appended ECCC SalmonSayward river average discharge for 2026-03-12 to: /results/forcing/rivers/observations/Salmon_Sayward_flow
2026-03-13 02:41:55,830 INFO [collect_river_data] ECCC NicomeklLangley river data collection for 2026-03-12 completed
2026-03-13 02:41:55,830 INFO [collect_river_data] ECCC SalmonSayward river data collection for 2026-03-12 completed
2026-03-13 02:41:55,855 INFO [collect_river_data] Appended ECCC SanJuanPortRenfrew river average discharge for 2026-03-12 to: /results/forcing/rivers/observations/SanJuan_PortRenfrew_flow
2026-03-13 02:41:55,865 INFO [collect_river_data] ECCC SanJuanPortRenfrew river data collection for 2026-03-12 completed
2026-03-13 02:41:55,937 INFO [collect_river_data] Appended ECCC Fraser river average discharge for 2026-03-12 to: /data/dlatorne/SOG-projects/SOG-forcing/ECget/Fraser_flow
2026-03-13 02:41:55,938 INFO [collect_river_data] ECCC Fraser river data collection for 2026-03-12 completed
2026-03-13 02:41:55,990 INFO [grib_to_netcdf] running in process 1319235
2026-03-13 02:41:55,992 INFO [grib_to_netcdf] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-13 02:41:55,993 INFO [grib_to_netcdf] publishing log messages to tcp://*:5578
2026-03-13 02:41:55,994 INFO [grib_to_netcdf] connected to skookum.eos.ubc.ca port 5555
2026-03-13 02:41:55,995 INFO [grib_to_netcdf] creating NEMO-atmos forcing files for 2026-03-13 forecast2 run
2026-03-13 02:41:56,003 INFO [collect_river_data] Appended ECCC Englishman river average discharge for 2026-03-12 to: /data/dlatorne/SOG-projects/SOG-forcing/ECget/Englishman_flow
2026-03-13 02:41:56,005 INFO [collect_river_data] ECCC Englishman river data collection for 2026-03-12 completed
2026-03-13 02:41:56,047 INFO [get_vfpa_hadcp] running in process 1319173
2026-03-13 02:41:56,049 INFO [get_vfpa_hadcp] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-13 02:41:56,050 INFO [get_vfpa_hadcp] publishing log messages to tcp://*:5579
2026-03-13 02:41:56,050 INFO [get_vfpa_hadcp] connected to skookum.eos.ubc.ca port 5555
2026-03-13 02:41:56,051 INFO [get_vfpa_hadcp] processing VFPA HADCP data from 2nd Narrows Rail Bridge for 2026-03-12
2026-03-13 02:41:56,208 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-13 02:41:56,280 INFO [crop_gribs] running in process 1319176
2026-03-13 02:41:56,281 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-13 02:41:56,282 INFO [crop_gribs] publishing log messages to tcp://*:5562
2026-03-13 02:41:56,284 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2026-03-13 02:41:56,285 INFO [crop_gribs] cropping 2026-03-13 ECCC HRDPS 2.5km continental 12Z GRIB files to SalishSeaCast subdomain
2026-03-13 02:41:56,297 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20260313/12/
2026-03-13 02:41:56,696 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-13 02:41:57,189 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-13 02:41:57,228 INFO [get_onc_ctd] running in process 1319171
2026-03-13 02:41:57,248 INFO [get_onc_ctd] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-13 02:41:57,249 INFO [get_onc_ctd] publishing log messages to tcp://*:5564
2026-03-13 02:41:57,250 INFO [get_onc_ctd] connected to skookum.eos.ubc.ca port 5555
2026-03-13 02:41:57,251 INFO [get_onc_ctd] requesting ONC SEVIP CTD T&S data for 2026-03-12
2026-03-13 02:41:57,344 INFO [get_onc_ferry] running in process 1319172
2026-03-13 02:41:57,347 INFO [get_onc_ferry] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-13 02:41:57,348 INFO [get_onc_ferry] publishing log messages to tcp://*:5565
2026-03-13 02:41:57,348 INFO [get_onc_ferry] connected to skookum.eos.ubc.ca port 5555
2026-03-13 02:41:57,349 INFO [get_onc_ferry] requesting ONC TWDP.N1 NAV data for 2026-03-12
2026-03-13 02:41:57,368 INFO [get_onc_ctd] running in process 1319170
2026-03-13 02:41:57,369 INFO [get_onc_ctd] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-13 02:41:57,372 INFO [get_onc_ctd] publishing log messages to tcp://*:5590
2026-03-13 02:41:57,373 INFO [get_onc_ctd] connected to skookum.eos.ubc.ca port 5555
2026-03-13 02:41:57,374 INFO [get_onc_ctd] requesting ONC SCVIP CTD T&S data for 2026-03-12
2026-03-13 02:41:57,601 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-13 02:41:57,949 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-13 02:41:58,431 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-13 02:41:58,834 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-13 02:41:59,242 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-13 02:41:59,273 INFO [get_vfpa_hadcp] added VFPA HADCP data from 2nd Narrows Rail Bridge for 2026-03-12 to /opp/observations/AISDATA/netcdf/VFPA_2ND_NARROWS_HADCP_2s_202603.nc
2026-03-13 02:41:59,274 INFO [get_vfpa_hadcp] VFPA HADCP observations added to 2026-03 netcdf file
2026-03-13 02:41:59,614 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-13 02:42:00,024 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-13 02:42:00,435 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-13 02:42:00,825 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-13 02:42:01,187 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-13 02:42:01,720 INFO [make_ssh_files] running in process 1319481
2026-03-13 02:42:01,721 INFO [make_ssh_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-13 02:42:01,722 INFO [make_ssh_files] publishing log messages to tcp://*:5566
2026-03-13 02:42:01,724 INFO [make_ssh_files] connected to skookum.eos.ubc.ca port 5555
2026-03-13 02:42:01,725 INFO [make_ssh_files] building forecast2 sea surface height boundary conditions file(s) from 2026-03-13 NOAA Neah Bay 00Z observation and forecast values
2026-03-13 02:42:01,827 INFO [manager] checklist updated with [VFPA HADCP data] items from get_vfpa_hadcp worker
2026-03-13 02:42:02,390 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['VFPA-HADCP'], host='localhost')
2026-03-13 02:42:04,051 INFO [ping_erddap] running in process 1319675
2026-03-13 02:42:04,052 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-13 02:42:04,053 INFO [ping_erddap] publishing log messages to tcp://*:5567
2026-03-13 02:42:04,054 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2026-03-13 02:42:04,055 INFO [ping_erddap] VFPA-HADCP ERDDAP dataset flag file(s) created
2026-03-13 02:42:04,057 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2026m03d08.nc
2026-03-13 02:42:04,058 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2026-03-13 02:42:04,287 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2026m03d09.nc
2026-03-13 02:42:04,543 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2026m03d10.nc
2026-03-13 02:42:04,826 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2026m03d11.nc
2026-03-13 02:42:05,051 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2026m03d12.nc
2026-03-13 02:42:05,343 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2026m03d13.nc
2026-03-13 02:42:05,547 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2026m03d14.nc
2026-03-13 02:42:05,854 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2026m03d15.nc
2026-03-13 02:42:06,109 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2026m03d16.nc
2026-03-13 02:42:06,388 INFO [make_ssh_files] sea surface height boundary file for 2026-03-13 forecast2 run created
2026-03-13 02:42:06,393 INFO [manager] checklist updated with [sea surface height forcing] items from make_ssh_files worker
2026-03-13 02:42:06,844 INFO [manager] launching NextWorker(module='nowcast.workers.make_runoff_file', args=['v202108'], host='localhost')
2026-03-13 02:42:06,850 INFO [manager] launching NextWorker(module='nowcast.workers.make_201702_runoff_file', args=[], host='localhost')
2026-03-13 02:42:08,424 INFO [make_201702_runoff_file] running in process 1319712
2026-03-13 02:42:08,426 INFO [make_201702_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-13 02:42:08,426 INFO [make_201702_runoff_file] publishing log messages to tcp://*:5566
2026-03-13 02:42:08,428 INFO [make_201702_runoff_file] connected to skookum.eos.ubc.ca port 5555
2026-03-13 02:42:08,464 INFO [make_201702_runoff_file] calculating runoff forcing for b201702 bathymetry
2026-03-13 02:42:08,602 INFO [make_201702_runoff_file] 2026-03-13 runoff file creation from Fraser at Hope and climatology elsewhere complete
2026-03-13 02:42:08,629 INFO [manager] checklist updated with [rivers forcing] items from make_201702_runoff_file worker
2026-03-13 02:42:09,277 INFO [make_runoff_file] running in process 1319711
2026-03-13 02:42:09,278 INFO [make_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-13 02:42:09,279 INFO [make_runoff_file] publishing log messages to tcp://*:5566
2026-03-13 02:42:09,280 INFO [make_runoff_file] connected to skookum.eos.ubc.ca port 5555
2026-03-13 02:42:09,283 INFO [make_runoff_file] calculating NEMO runoff forcing for v202108 bathymetry for 2026-03-12
2026-03-13 02:42:10,662 ERROR [make_runoff_file] no 2026-03-12 discharge obs for SkagitMountVernon: patching
2026-03-13 02:42:11,438 ERROR [make_runoff_file] no 2026-03-12 discharge obs for SnohomishMonroe: patching
2026-03-13 02:42:11,813 ERROR [make_runoff_file] no 2026-03-12 discharge obs for NisquallyMcKenna: patching
2026-03-13 02:42:11,965 ERROR [make_runoff_file] no 2026-03-12 discharge obs for GreenwaterGreenwater: patching
2026-03-13 02:42:13,963 INFO [make_runoff_file] stored NEMO runoff forcing for v202108 bathymetry for 2026-03-12: /results/forcing/rivers/R202108Dailies_y2026m03d12.nc
2026-03-13 02:42:13,964 INFO [make_runoff_file] 2026-03-12 runoff file creation completed
2026-03-13 02:42:13,966 INFO [manager] checklist updated with [rivers forcing] items from make_runoff_file worker
2026-03-13 02:42:14,124 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2026m03d14.nc
2026-03-13 02:42:17,352 INFO [get_onc_ctd] 2026-03-12 ONC SCVIP CTD T&S file created
2026-03-13 02:42:17,355 INFO [manager] checklist updated with [ONC CTD data] items from get_onc_ctd worker
2026-03-13 02:42:17,558 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2026m03d15.nc
2026-03-13 02:42:17,565 INFO [grib_to_netcdf] 2026-03-13 NEMO-atmos forcing file for forecast2 created
2026-03-13 02:42:17,726 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['SCVIP-CTD'], host='localhost')
2026-03-13 02:42:17,733 INFO [manager] checklist updated with [weather forcing] items from grib_to_netcdf worker
2026-03-13 02:42:18,036 INFO [get_onc_ctd] 2026-03-12 ONC SEVIP CTD T&S file created
2026-03-13 02:42:18,079 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'forecast2'], host='localhost')
2026-03-13 02:42:18,082 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'forecast2'], host='localhost')
2026-03-13 02:42:18,087 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.nibi', 'forecast2'], host='localhost')
2026-03-13 02:42:18,094 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'forecast2'], host='localhost')
2026-03-13 02:42:18,105 INFO [manager] checklist updated with [ONC CTD data] items from get_onc_ctd worker
2026-03-13 02:42:18,606 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['SEVIP-CTD'], host='localhost')
2026-03-13 02:42:19,552 INFO [get_onc_ferry] requesting ONC TWDP TSG data for 2026-03-12
2026-03-13 02:42:19,632 INFO [ping_erddap] running in process 1319770
2026-03-13 02:42:19,633 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-13 02:42:19,634 INFO [ping_erddap] publishing log messages to tcp://*:5564
2026-03-13 02:42:19,635 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2026-03-13 02:42:19,636 INFO [ping_erddap] SCVIP-CTD ERDDAP dataset flag file(s) created
2026-03-13 02:42:19,638 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2026-03-13 02:42:19,799 INFO [upload_forcing] running in process 1319773
2026-03-13 02:42:19,800 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-13 02:42:19,801 INFO [upload_forcing] publishing log messages to tcp://*:5566
2026-03-13 02:42:19,802 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-03-13 02:42:19,826 INFO [upload_forcing] running in process 1319776
2026-03-13 02:42:19,828 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-13 02:42:19,833 INFO [upload_forcing] publishing log messages to tcp://*:5567
2026-03-13 02:42:19,834 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-03-13 02:42:19,897 INFO [upload_forcing] running in process 1319775
2026-03-13 02:42:19,898 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-13 02:42:19,899 INFO [upload_forcing] publishing log messages to tcp://*:5568
2026-03-13 02:42:19,900 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-03-13 02:42:19,903 INFO [upload_forcing] running in process 1319777
2026-03-13 02:42:19,904 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-13 02:42:19,905 INFO [upload_forcing] publishing log messages to tcp://*:5569
2026-03-13 02:42:19,907 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-03-13 02:42:19,924 INFO [ping_erddap] running in process 1319781
2026-03-13 02:42:19,926 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-13 02:42:19,927 INFO [ping_erddap] publishing log messages to tcp://*:5570
2026-03-13 02:42:19,928 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2026-03-13 02:42:19,929 INFO [ping_erddap] SEVIP-CTD ERDDAP dataset flag file(s) created
2026-03-13 02:42:20,001 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2026-03-13 02:42:21,201 INFO [upload_forcing] LiveOcean boundary conditions file not found; created symlink to /results/forcing/LiveOcean/boundary_conditions/LiveOcean_v201905_y2026m03d12.nc
2026-03-13 02:42:21,252 INFO [upload_forcing] forecast2 2026-03-13 forcing files upload to optimum-hindcast completed
2026-03-13 02:42:21,255 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-03-13 02:42:22,506 INFO [upload_forcing] forecast2 2026-03-13 forcing files upload to arbutus.cloud-nowcast completed
2026-03-13 02:42:22,510 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-03-13 02:42:22,876 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2026-03-13'], host='localhost')
2026-03-13 02:42:24,319 INFO [make_forcing_links] running in process 1319818
2026-03-13 02:42:24,320 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-13 02:42:24,321 INFO [make_forcing_links] publishing log messages to tcp://*:5564
2026-03-13 02:42:24,322 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2026-03-13 02:42:25,445 INFO [make_forcing_links] forecast2 2026-03-13 forcing file links on arbutus.cloud-nowcast created
2026-03-13 02:42:25,610 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2026-03-13 02:42:26,045 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2026-03-13'], host='arbutus.cloud-nowcast')
2026-03-13 02:42:26,617 INFO [upload_forcing] forecast2 2026-03-13 forcing files upload to orcinus-nowcast-agrif completed
2026-03-13 02:42:26,620 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-03-13 02:42:27,484 INFO [upload_forcing] forecast2 2026-03-13 forcing files upload to robot.nibi completed
2026-03-13 02:42:27,488 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-03-13 02:42:28,124 INFO [run_NEMO] running in process 27104
2026-03-13 02:42:28,125 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-13 02:42:28,126 INFO [run_NEMO] publishing log messages to tcp://*:5556
2026-03-13 02:42:28,189 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2026-03-13 02:42:28,989 WARNING [run_NEMO] There are uncommitted changes in /nemoShare/MEOPAR/nowcast-sys/SS-run-sets
2026-03-13 02:42:29,295 INFO [run_NEMO] forecast2: launching /nemoShare/MEOPAR/nowcast-sys/runs/12mar26forecast2_2026-03-13T024228.258064-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2026-03-13 02:42:29,307 INFO [run_NEMO] forecast2 NEMO run for 2026-03-13 on arbutus.cloud-nowcast started
2026-03-13 02:42:29,320 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2026-03-13 02:42:29,977 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'forecast2'], host='arbutus.cloud-nowcast')
2026-03-13 02:42:42,963 INFO [get_onc_ferry] requesting ONC TWDP OXYSENSOR data for 2026-03-12
2026-03-13 02:43:06,585 INFO [get_onc_ferry] requesting ONC TWDP TURBCHLFL data for 2026-03-12
2026-03-13 02:43:29,149 INFO [get_onc_ferry] requesting ONC TWDP CO2SENSOR data for 2026-03-12
2026-03-13 02:43:29,220 WARNING [get_onc_ferry] No ONC TWDP CO2SENSOR partial_pressure,co2 data for 2026-03-12; substituting empty dataset
2026-03-13 02:43:29,228 WARNING [get_onc_ferry] No ONC TWDP CO2SENSOR partial_pressure data for 2026-03-12; substituting empty dataset
2026-03-13 02:43:29,230 WARNING [get_onc_ferry] No ONC TWDP CO2SENSOR co2 data for 2026-03-12; substituting empty dataset
2026-03-13 02:43:29,231 INFO [get_onc_ferry] requesting ONC TWDP TEMPHUMID data for 2026-03-12
2026-03-13 02:43:29,297 WARNING [get_onc_ferry] No ONC TWDP TEMPHUMID air_temperature,rel_humidity data for 2026-03-12; substituting empty dataset
2026-03-13 02:43:29,304 WARNING [get_onc_ferry] No ONC TWDP TEMPHUMID air_temperature data for 2026-03-12; substituting empty dataset
2026-03-13 02:43:29,306 WARNING [get_onc_ferry] No ONC TWDP TEMPHUMID rel_humidity data for 2026-03-12; substituting empty dataset
2026-03-13 02:43:29,307 INFO [get_onc_ferry] requesting ONC TWDP BARPRESS data for 2026-03-12
2026-03-13 02:43:29,403 WARNING [get_onc_ferry] No ONC TWDP BARPRESS barometric_pressure data for 2026-03-12; substituting empty dataset
2026-03-13 02:43:29,407 WARNING [get_onc_ferry] No ONC TWDP BARPRESS barometric_pressure data for 2026-03-12; substituting empty dataset
2026-03-13 02:43:29,408 INFO [get_onc_ferry] requesting ONC TWDP PYRANOMETER data for 2026-03-12
2026-03-13 02:43:29,466 WARNING [get_onc_ferry] No ONC TWDP PYRANOMETER solar_radiation data for 2026-03-12; substituting empty dataset
2026-03-13 02:43:29,469 WARNING [get_onc_ferry] No ONC TWDP PYRANOMETER solar_radiation data for 2026-03-12; substituting empty dataset
2026-03-13 02:43:29,471 INFO [get_onc_ferry] requesting ONC TWDP PYRGEOMETER data for 2026-03-12
2026-03-13 02:43:29,534 WARNING [get_onc_ferry] No ONC TWDP PYRGEOMETER downward_radiation data for 2026-03-12; substituting empty dataset
2026-03-13 02:43:29,538 WARNING [get_onc_ferry] No ONC TWDP PYRGEOMETER downward_radiation data for 2026-03-12; substituting empty dataset
2026-03-13 02:43:29,784 WARNING [get_onc_ferry] ONC TWDP CO2SENSOR partial_pressure data for 2026-03-12 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2026-03-13 02:43:29,801 WARNING [get_onc_ferry] ONC TWDP CO2SENSOR co2 data for 2026-03-12 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2026-03-13 02:43:29,820 WARNING [get_onc_ferry] ONC TWDP TEMPHUMID air_temperature data for 2026-03-12 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2026-03-13 02:43:29,839 WARNING [get_onc_ferry] ONC TWDP TEMPHUMID rel_humidity data for 2026-03-12 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2026-03-13 02:43:29,857 WARNING [get_onc_ferry] ONC TWDP BARPRESS barometric_pressure data for 2026-03-12 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2026-03-13 02:43:29,876 WARNING [get_onc_ferry] ONC TWDP PYRANOMETER solar_radiation data for 2026-03-12 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2026-03-13 02:43:29,894 WARNING [get_onc_ferry] ONC TWDP PYRGEOMETER downward_radiation data for 2026-03-12 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2026-03-13 02:44:08,785 INFO [get_onc_ferry] 2026-03-12 ONC TWDP ferry data file created
2026-03-13 02:44:08,787 INFO [manager] checklist updated with [ONC ferry data] items from get_onc_ferry worker
2026-03-13 02:44:09,301 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['TWDP-ferry'], host='localhost')
2026-03-13 02:44:10,600 INFO [ping_erddap] running in process 1319847
2026-03-13 02:44:10,601 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-13 02:44:10,602 INFO [ping_erddap] publishing log messages to tcp://*:5564
2026-03-13 02:44:10,607 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2026-03-13 02:44:10,609 INFO [ping_erddap] TWDP-ferry ERDDAP dataset flag file(s) created
2026-03-13 02:44:10,610 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2026-03-13 02:47:32,101 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 19097050 = 2026-03-14 05:26:00 UTC, 18.1% complete
2026-03-13 02:52:32,100 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 19097580 = 2026-03-14 11:19:20 UTC, 37.8% complete
2026-03-13 02:57:32,101 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 19098110 = 2026-03-14 17:12:40 UTC, 57.4% complete
2026-03-13 03:02:32,102 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 19098677 = 2026-03-14 23:30:40 UTC, 78.4% complete
2026-03-13 03:07:32,149 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 19099242 = 2026-03-15 05:47:20 UTC, 99.3% complete
2026-03-13 03:12:32,107 INFO [watch_NEMO] forecast2 NEMO run on arbutus.cloud-nowcast completed
2026-03-13 03:12:32,110 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2026-03-13 03:12:32,694 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_wind_file', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2026-03-13'], host='arbutus.cloud-nowcast')
2026-03-13 03:12:32,872 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_current_file', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2026-03-13'], host='arbutus.cloud-nowcast')
2026-03-13 03:12:32,877 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2026-03-12'], host='localhost')
2026-03-13 03:12:34,127 INFO [download_results] running in process 1320109
2026-03-13 03:12:34,128 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-13 03:12:34,129 INFO [download_results] publishing log messages to tcp://*:5564
2026-03-13 03:12:34,130 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2026-03-13 03:12:34,131 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/forecast2/12mar26 to /results/SalishSea/forecast2.202111
2026-03-13 03:12:35,672 INFO [make_ww3_wind_file] running in process 27855
2026-03-13 03:12:35,674 INFO [make_ww3_wind_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-13 03:12:35,675 INFO [make_ww3_wind_file] publishing log messages to tcp://*:5570
2026-03-13 03:12:35,676 INFO [make_ww3_wind_file] connected to skookum.eos.ubc.ca port 5555
2026-03-13 03:12:35,677 INFO [make_ww3_wind_file] Creating wwatch3 wind forcing file for 2026-03-13 forecast2 run
2026-03-13 03:12:35,695 INFO [make_ww3_current_file] running in process 27852
2026-03-13 03:12:35,696 INFO [make_ww3_current_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-13 03:12:35,697 INFO [make_ww3_current_file] publishing log messages to tcp://*:5571
2026-03-13 03:12:35,699 INFO [make_ww3_current_file] connected to skookum.eos.ubc.ca port 5555
2026-03-13 03:12:35,700 INFO [make_ww3_current_file] Creating wwatch3 currents forcing file for 2026-03-13 forecast2 run
2026-03-13 03:12:44,161 INFO [make_ww3_wind_file] wwatch3 wind forcing file created on arbutus.cloud-nowcast for 2026-03-13 forecast2 run
2026-03-13 03:12:44,165 INFO [manager] checklist updated with [WW3 wind forcing] items from make_ww3_wind_file worker
2026-03-13 03:13:55,613 INFO [download_results] forecast2 2026-03-12 results files from arbutus.cloud-nowcast downloaded
2026-03-13 03:13:55,617 INFO [manager] checklist updated with [results files] items from download_results worker
2026-03-13 03:13:55,978 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['forecast2', '--run-date', '2026-03-12'], host='localhost')
2026-03-13 03:13:59,019 INFO [make_CHS_currents_file] running in process 1320121
2026-03-13 03:13:59,020 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-13 03:13:59,021 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5564
2026-03-13 03:13:59,022 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555
2026-03-13 03:14:36,744 INFO [make_CHS_currents_file] Made CHS currents file for 2026-03-12 for forecast2
2026-03-13 03:14:36,747 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker
2026-03-13 03:14:37,104 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['nemo', 'forecast2', '--run-date', '2026-03-12'], host='localhost')
2026-03-13 03:14:38,202 INFO [update_forecast_datasets] running in process 1320154
2026-03-13 03:14:38,203 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-13 03:14:38,204 INFO [update_forecast_datasets] publishing log messages to tcp://*:5564
2026-03-13 03:14:38,205 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2026-03-13 03:14:38,206 INFO [update_forecast_datasets] updating nemo forecast directory for forecast2/12mar26 run
2026-03-13 03:16:14,673 INFO [make_ww3_current_file] wwatch3 currents forcing file created on arbutus.cloud-nowcast for 2026-03-13 forecast2 run
2026-03-13 03:16:14,677 INFO [manager] checklist updated with [WW3 currents forcing] items from make_ww3_current_file worker
2026-03-13 03:16:15,014 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2026-03-13'], host='arbutus.cloud-nowcast')
2026-03-13 03:16:17,045 INFO [run_ww3] running in process 28441
2026-03-13 03:16:17,046 INFO [run_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-13 03:16:17,047 INFO [run_ww3] publishing log messages to tcp://*:5572
2026-03-13 03:16:17,050 INFO [run_ww3] connected to skookum.eos.ubc.ca port 5555
2026-03-13 03:16:17,054 INFO [run_ww3] Created run directory /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast2_2026-03-13T031617.046431-0700
2026-03-13 03:16:17,056 INFO [run_ww3] forecast2: launching /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast2_2026-03-13T031617.046431-0700/SoGWW3.sh on arbutus.cloud-nowcast
2026-03-13 03:16:17,070 INFO [run_ww3] forecast2 WaveWatch3 run for 2026-03-13 on arbutus.cloud-nowcast started
2026-03-13 03:16:17,072 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker
2026-03-13 03:16:17,422 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'forecast2'], host='arbutus.cloud-nowcast')
2026-03-13 03:16:19,422 INFO [watch_ww3] running in process 28528
2026-03-13 03:16:19,423 INFO [watch_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-13 03:16:19,424 INFO [watch_ww3] publishing log messages to tcp://*:5573
2026-03-13 03:16:19,425 INFO [watch_ww3] connected to skookum.eos.ubc.ca port 5555
2026-03-13 03:16:19,458 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2026-03-13 03:19:03,678 INFO [update_forecast_datasets] updated nemo forecast directory for forecast2/12mar26 run: /results/SalishSea/rolling-forecasts/nemo
2026-03-13 03:19:03,679 INFO [update_forecast_datasets] nemo 2026-03-12 forecast2 rolling forecast datasets updated
2026-03-13 03:19:03,682 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2026-03-13 03:19:04,031 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nemo-forecast'], host='localhost')
2026-03-13 03:19:04,038 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'forecast2', 'publish', '--run-date', '2026-03-12'], host='localhost')
2026-03-13 03:19:04,043 INFO [manager] launching NextWorker(module='nowcast.workers.make_surface_current_tiles', args=['forecast2', '--run-date', '2026-03-12'], host='localhost')
2026-03-13 03:19:05,531 INFO [ping_erddap] running in process 1320295
2026-03-13 03:19:05,532 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-13 03:19:05,532 INFO [ping_erddap] publishing log messages to tcp://*:5564
2026-03-13 03:19:05,534 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2026-03-13 03:19:05,545 INFO [ping_erddap] nemo-forecast ERDDAP dataset flag file(s) created
2026-03-13 03:19:05,546 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2026-03-13 03:19:06,914 INFO [make_surface_current_tiles] running in process 1320298
2026-03-13 03:19:06,915 INFO [make_surface_current_tiles] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-13 03:19:06,916 INFO [make_surface_current_tiles] publishing log messages to tcp://*:5564
2026-03-13 03:19:06,918 INFO [make_surface_current_tiles] connected to skookum.eos.ubc.ca port 5555
2026-03-13 03:19:06,981 INFO [make_surface_current_tiles] starting rendering of tiles for 2026-03-12 forecast2 in 6 processes into /results/nowcast-sys/figures/surface_currents/forecast2/12mar26
2026-03-13 03:19:08,612 INFO [make_plots] running in process 1320297
2026-03-13 03:19:08,613 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-13 03:19:08,614 INFO [make_plots] publishing log messages to tcp://*:5565
2026-03-13 03:19:08,615 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2026-03-13 03:19:09,170 INFO [make_plots] preparing render list for 2026-03-12 NEMO forecast2 publish figures
2026-03-13 03:19:09,417 INFO [make_plots] added Website_thumbnail figure to 2026-03-12 NEMO forecast2 publish render list
2026-03-13 03:19:09,418 INFO [make_plots] added Threshold_website figure to 2026-03-12 NEMO forecast2 publish render list
2026-03-13 03:19:09,419 INFO [make_plots] added PA_tidal_predictions figure to 2026-03-12 NEMO forecast2 publish render list
2026-03-13 03:19:09,420 INFO [make_plots] added Boundary Bay figure to 2026-03-12 NEMO forecast2 publish render list
2026-03-13 03:19:09,421 INFO [make_plots] added Campbell River figure to 2026-03-12 NEMO forecast2 publish render list
2026-03-13 03:19:09,422 INFO [make_plots] added Cherry Point figure to 2026-03-12 NEMO forecast2 publish render list
2026-03-13 03:19:09,423 INFO [make_plots] added Friday Harbor figure to 2026-03-12 NEMO forecast2 publish render list
2026-03-13 03:19:09,423 INFO [make_plots] added Halfmoon Bay figure to 2026-03-12 NEMO forecast2 publish render list
2026-03-13 03:19:09,424 INFO [make_plots] added Nanaimo figure to 2026-03-12 NEMO forecast2 publish render list
2026-03-13 03:19:09,425 INFO [make_plots] added Neah Bay figure to 2026-03-12 NEMO forecast2 publish render list
2026-03-13 03:19:09,426 INFO [make_plots] added New Westminster figure to 2026-03-12 NEMO forecast2 publish render list
2026-03-13 03:19:09,426 INFO [make_plots] added Patricia Bay figure to 2026-03-12 NEMO forecast2 publish render list
2026-03-13 03:19:09,427 INFO [make_plots] added Point Atkinson figure to 2026-03-12 NEMO forecast2 publish render list
2026-03-13 03:19:09,428 INFO [make_plots] added Port Renfrew figure to 2026-03-12 NEMO forecast2 publish render list
2026-03-13 03:19:09,429 INFO [make_plots] added Sand Heads figure to 2026-03-12 NEMO forecast2 publish render list
2026-03-13 03:19:09,431 INFO [make_plots] added Sandy Cove figure to 2026-03-12 NEMO forecast2 publish render list
2026-03-13 03:19:09,432 INFO [make_plots] added Squamish figure to 2026-03-12 NEMO forecast2 publish render list
2026-03-13 03:19:09,433 INFO [make_plots] added Victoria figure to 2026-03-12 NEMO forecast2 publish render list
2026-03-13 03:19:09,434 INFO [make_plots] added Woodwards Landing figure to 2026-03-12 NEMO forecast2 publish render list
2026-03-13 03:19:09,435 INFO [make_plots] starting to render nemo forecast2 publish 12mar26 figures
2026-03-13 03:21:19,462 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 90, 16.7% complete
2026-03-13 03:26:19,460 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 189, 35.0% complete
2026-03-13 03:27:49,219 WARNING [make_plots] invalid station number for Boundary Bay station: None
2026-03-13 03:27:49,221 WARNING [make_plots] can't resolve a valid CHS station code for Boundary Bay
2026-03-13 03:28:18,747 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb996/data for station 08074 Campbell River from 2026-03-08 00:00:00Z to 2026-03-15 05:55:00Z
2026-03-13 03:29:31,319 WARNING [make_plots] invalid station number for Halfmoon Bay station: None
2026-03-13 03:29:31,319 WARNING [make_plots] can't resolve a valid CHS station code for Halfmoon Bay
2026-03-13 03:29:52,923 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb96d/data for station 07917 Nanaimo from 2026-03-08 00:00:00Z to 2026-03-15 05:55:00Z
2026-03-13 03:30:33,472 INFO [make_surface_current_tiles] starting PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast2/12mar26
2026-03-13 03:30:33,635 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e13d0f4a073c4bbf93/data for station 07654 New Westminster from 2026-03-08 00:00:00Z to 2026-03-15 05:55:00Z
2026-03-13 03:30:51,012 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd2d/data for station 07277 Patricia Bay from 2026-03-08 00:00:00Z to 2026-03-15 05:55:00Z
2026-03-13 03:31:10,763 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94c/data for station 07795 Point Atkinson from 2026-03-08 00:00:00Z to 2026-03-15 05:55:00Z
2026-03-13 03:31:19,463 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 291, 53.9% complete
2026-03-13 03:31:28,216 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e23d0f4a073c4bc060/data for station 08525 Port Renfrew from 2026-03-08 00:00:00Z to 2026-03-15 05:55:00Z
2026-03-13 03:31:45,237 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb93c/data for station 07594 Sand Heads from 2026-03-08 00:00:00Z to 2026-03-15 05:55:00Z
2026-03-13 03:32:02,326 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc43b/data for station 07786 Sandy Cove from 2026-03-08 00:00:00Z to 2026-03-15 05:55:00Z
2026-03-13 03:32:19,790 WARNING [make_plots] invalid station number for Squamish station: None
2026-03-13 03:32:19,791 WARNING [make_plots] can't resolve a valid CHS station code for Squamish
2026-03-13 03:32:37,015 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd1e/data for station 07120 Victoria from 2026-03-08 00:00:00Z to 2026-03-15 05:55:00Z
2026-03-13 03:32:54,935 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc42d/data for station 07610 Woodwards Landing from 2026-03-08 00:00:00Z to 2026-03-15 05:55:00Z
2026-03-13 03:33:12,683 INFO [make_plots] finished rendering nemo forecast2 publish 12mar26 figures
2026-03-13 03:33:12,685 INFO [make_plots] nemo publish plots for 2026-03-12 forecast2 completed
2026-03-13 03:33:12,694 INFO [manager] checklist updated with [plots] items from make_plots worker
2026-03-13 03:33:12,999 INFO [manager] launching NextWorker(module='nowcast.workers.make_feeds', args=['forecast2', '--run-date', '2026-03-12'], host='localhost')
2026-03-13 03:33:17,185 INFO [make_feeds] running in process 1325330
2026-03-13 03:33:17,187 INFO [make_feeds] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-13 03:33:17,188 INFO [make_feeds] publishing log messages to tcp://*:5565
2026-03-13 03:33:17,189 INFO [make_feeds] connected to skookum.eos.ubc.ca port 5555
2026-03-13 03:33:32,717 INFO [make_feeds] ATOM feeds for 2026-03-12 forecast2 run completed
2026-03-13 03:33:32,720 INFO [manager] checklist updated with [storm surge alert feeds] items from make_feeds worker
2026-03-13 03:33:33,069 INFO [manager] launching NextWorker(module='nemo_nowcast.workers.clear_checklist', args=[], host='localhost')
2026-03-13 03:33:34,316 INFO [clear_checklist] running in process 1325355
2026-03-13 03:33:34,317 INFO [clear_checklist] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-13 03:33:34,318 INFO [clear_checklist] publishing log messages to tcp://*:5565
2026-03-13 03:33:34,319 INFO [clear_checklist] connected to skookum.eos.ubc.ca port 5555
2026-03-13 03:33:34,321 INFO [clear_checklist] requesting that manager clear system state checklist
2026-03-13 03:33:34,323 INFO [manager] writing checklist to log file
2026-03-13 03:33:34,380 INFO [manager] checklist cleared
2026-03-13 03:33:34,381 INFO [clear_checklist] nowcast system checklist cleared
2026-03-13 03:33:34,384 INFO [manager] launching NextWorker(module='nemo_nowcast.workers.rotate_logs', args=[], host='localhost')
2026-03-13 03:33:35,629 INFO [rotate_logs] running in process 1325360
2026-03-13 03:33:35,630 INFO [rotate_logs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-13 03:33:35,631 INFO [rotate_logs] publishing log messages to tcp://*:5565
2026-03-13 03:33:35,632 INFO [rotate_logs] connected to skookum.eos.ubc.ca port 5555
2026-03-13 03:33:35,633 INFO [rotate_logs] rotating log files